Add support for DAX vmemmap optimization for ppc64
Subsystems (15)
v2 cover Add support for DAX vmemmap optimization for ppc64
This patch series implements changes required to support DAX vmemmap optimization for ppc64. The vmemmap optimization is only enabled with radix MMU translation and 1GB PUD mapping with 64K page size. The patch series also split hugetlb vmemmap optimization as a separate Kconfig variable so that architectures can enable DAX vmemmap optimization without enabling hugetlb vmemmap optimization. This should enable architectures like arm64 to enable DAX vmemmap optimization while they can't enable hugetlb vmemmap optimization. More details of the same are in patch "mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization"
Changes from V1:
* Fix make htmldocs warning
* Fix vmemmap allocation bugs with different alignment values.
* Correctly check for section validity to before we free vmemmap area
Aneesh Kumar K.V (16):
powerpc/mm/book3s64: Use pmdp_ptep helper instead of typecasting. powerpc/book3s64/mm: mmu_vmemmap_psize is used by radix powerpc/book3s64/mm: Fix DirectMap stats in /proc/meminfo powerpc/book3s64/mm: Use PAGE_KERNEL instead of opencoding powerpc/mm/dax: Fix the condition when checking if altmap vmemap can
cross-boundary
mm/hugepage pud: Allow arch-specific helper function to check huge
page pud support
mm: Change pudp_huge_get_and_clear_full take vm_area_struct as arg mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to
override
mm/vmemmap: Allow architectures to override how vmemmap optimization
works
mm: Add __HAVE_ARCH_PUD_SAME similar to __HAVE_ARCH_P4D_SAME mm/huge pud: Use transparent huge pud helpers only with
CONFIG_TRANSPARENT_HUGEPAGE
mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization powerpc/book3s64/mm: Enable transparent pud hugepage powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap
handling function
powerpc/book3s64/radix: Add support for vmemmap optimization for radix powerpc/book3s64/radix: Remove mmu_vmemmap_psize
Documentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 +++ arch/loongarch/Kconfig | 2 +- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/pgtable.h | 156 ++++- arch/powerpc/include/asm/book3s/64/radix.h | 47 ++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++ arch/powerpc/mm/book3s64/radix_pgtable.c | 577 ++++++++++++++++-- arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/mm/init_64.c | 39 +- arch/powerpc/platforms/Kconfig.cputype | 1 + arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 3 +- drivers/nvdimm/pfn_devs.c | 2 +- fs/Kconfig | 2 +- include/linux/mm.h | 32 +- include/linux/pgtable.h | 11 +- include/trace/events/thp.h | 17 + mm/Kconfig | 5 +- mm/debug_vm_pgtable.c | 2 +- mm/huge_memory.c | 2 +- mm/mm_init.c | 2 +- mm/mremap.c | 2 +- mm/sparse-vmemmap.c | 3 + 28 files changed, 1032 insertions(+), 77 deletions(-) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst
Signature · 1 line
2.40.1
PATCH v2 1/16 powerpc/mm/book3s64: Use pmdp_ptep helper instead of typecasting.
No functional change in this patch.
--- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/book3s64/radix_pgtable.c +1 -1
@@ -952,7 +952,7 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addassert_spin_locked (pmd_lockptr (mm, pmdp));#endif - old =radix__pte_update (mm, addr, (pte_t *)pmdp, clr, set,1 ); + old =radix__pte_update (mm, addr,pmdp_ptep (pmdp), clr, set,1 );trace_hugepage_update (addr, old, clr, set);return old;
Signature · 1 line
2.40.1
PATCH v2 2/16 powerpc/book3s64/mm: mmu_vmemmap_psize is used by radix
In reply to Aneesh Kumar K.VThis should not be within CONFIG_PPC_64S_HASHS_MMU. We use mmu_vmemmap_psize on radix while mapping the vmemmap area.
--- arch/powerpc/mm/book3s64/radix_pgtable.c | 2 -- 1 file changed, 2 deletions(-)
arch/powerpc/mm/book3s64/radix_pgtable.c +0 -2
@@ -594,7 +594,6 @@ void __init radix__early_init_mmu(void) {unsigned long lpcr; -#ifdef CONFIG_PPC_64S_HASH_MMU #ifdef CONFIG_PPC_64K_PAGES /* PAGE_SIZE mappings */ mmu_virtual_psize =MMU_PAGE_64K ;
@@ -611,7 +610,6 @@ void __init radix__early_init_mmu(void) mmu_vmemmap_psize = MMU_PAGE_2M; }else mmu_vmemmap_psize = mmu_virtual_psize; -#endif #endif /* * initialize page table size
Signature · 1 line
2.40.1
PATCH v2 3/16 powerpc/book3s64/mm: Fix DirectMap stats in /proc/meminfo
In reply to Aneesh Kumar K.VOn memory unplug reduce DirectMap page count correctly.
root@ubuntu-guest:# grep Direct /proc/meminfo DirectMap4k: 0 kB DirectMap64k: 0 kB DirectMap2M: 115343360 kB DirectMap1G: 0 kB
Before fix:
root@ubuntu-guest:# ndctl disable-namespace all disabled 1 namespace root@ubuntu-guest:# grep Direct /proc/meminfo DirectMap4k: 0 kB DirectMap64k: 0 kB DirectMap2M: 115343360 kB DirectMap1G: 0 kB
After fix:
root@ubuntu-guest:# ndctl disable-namespace all disabled 1 namespace root@ubuntu-guest:# grep Direct /proc/meminfo DirectMap4k: 0 kB DirectMap64k: 0 kB DirectMap2M: 104857600 kB DirectMap1G: 0 kB
--- arch/powerpc/mm/book3s64/radix_pgtable.c | 34 +++++++++++++++--------- 1 file changed, 22 insertions(+), 12 deletions(-)
arch/powerpc/mm/book3s64/radix_pgtable.c +22 -12
@@ -743,9 +743,9 @@ static void free_pud_table(pud_t *pud_start, p4d_t *p4d) }static void remove_pte_table (pte_t *pte_start,unsigned long addr, -unsigned long end) +unsigned long end,bool direct) { -unsigned long next; +unsigned long next, pages =0 ;pte_t *pte; pte = pte_start +pte_index (addr);
@@ -767,13 +767,16 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr, }pte_clear (&init_mm, addr, pte); + pages++; } +if (direct) +update_page_count (mmu_virtual_psize, -pages); }static void __meminit remove_pmd_table (pmd_t *pmd_start,unsigned long addr, -unsigned long end) +unsigned long end,bool direct) { -unsigned long next; +unsigned long next, pages =0 ;pte_t *pte_base;pmd_t *pmd;
@@ -791,19 +794,22 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr,continue ; }pte_clear (&init_mm, addr, (pte_t *)pmd); + pages++;continue ; } pte_base = (pte_t *)pmd_page_vaddr (*pmd); -remove_pte_table (pte_base, addr, next); +remove_pte_table (pte_base, addr, next, direct);free_pte_table (pte_base, pmd); } +if (direct) +update_page_count (MMU_PAGE_2M , -pages); }static void __meminit remove_pud_table (pud_t *pud_start,unsigned long addr, -unsigned long end) +unsigned long end,bool direct) { -unsigned long next; +unsigned long next, pages =0 ;pmd_t *pmd_base;pud_t *pud;
@@ -821,16 +827,20 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr,continue ; }pte_clear (&init_mm, addr, (pte_t *)pud); + pages++;continue ; } pmd_base =pud_pgtable (*pud); -remove_pmd_table (pmd_base, addr, next); +remove_pmd_table (pmd_base, addr, next, direct);free_pmd_table (pmd_base, pud); } +if (direct) +update_page_count (MMU_PAGE_1G , -pages); } -static void __meminit remove_pagetable (unsigned long start,unsigned long end) +static void __meminit remove_pagetable (unsigned long start,unsigned long end, +bool direct) {unsigned long addr, next;pud_t *pud_base;
@@ -859,7 +869,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end) } pud_base =p4d_pgtable (*p4d); -remove_pud_table (pud_base, addr, next); +remove_pud_table (pud_base, addr, next, direct);free_pud_table (pud_base, p4d); }
@@ -882,7 +892,7 @@ int __meminit radix__create_section_mapping(unsigned long start,int __meminit radix__remove_section_mapping (unsigned long start,unsigned long end) { -remove_pagetable (start, end); +remove_pagetable (start, end, true);return 0 ; }#endif /* CONFIG_MEMORY_HOTPLUG */
@@ -918,7 +928,7 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start,#ifdef CONFIG_MEMORY_HOTPLUG void __meminit radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size) { -remove_pagetable (start, start + page_size); +remove_pagetable (start, start + page_size, false); }#endif #endif
Signature · 1 line
2.40.1
PATCH v2 4/16 powerpc/book3s64/mm: Use PAGE_KERNEL instead of opencoding
In reply to Aneesh Kumar K.VNo functional change in this patch.
--- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/powerpc/mm/book3s64/radix_pgtable.c +1 -2
@@ -910,7 +910,6 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys) {/* Create a PTE encoding */ -unsigned long flags = _PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_KERNEL_RW;int nid =early_pfn_to_nid (phys >>PAGE_SHIFT );int ret;
@@ -919,7 +918,7 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start,return -1 ; } - ret =__map_kernel_page_nid (start, phys,__pgprot (flags), page_size, nid); + ret =__map_kernel_page_nid (start, phys,PAGE_KERNEL , page_size, nid);BUG_ON (ret);return 0 ;
Signature · 1 line
2.40.1
PATCH v2 5/16 powerpc/mm/dax: Fix the condition when checking if altmap vmemap can cross-boundary
In reply to Aneesh Kumar K.VWithout this fix, the last subsection vmemmap can end up in memory even if the namespace is created with -M mem and has sufficient space in the altmap area.
--- arch/powerpc/mm/init_64.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/powerpc/mm/init_64.c +1 -1
@@ -189,7 +189,7 @@ static bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long starunsigned long nr_pfn = page_size /sizeof (struct page );unsigned long start_pfn =page_to_pfn ((struct page *)start); -if ((start_pfn + nr_pfn) > altmap->end_pfn) +if ((start_pfn + nr_pfn -1 ) > altmap->end_pfn)return true;if (start_pfn < altmap->base_pfn)
Signature · 1 line
2.40.1
PATCH v2 6/16 mm/hugepage pud: Allow arch-specific helper function to check huge page pud support
In reply to Aneesh Kumar K.VArchitectures like powerpc would like to enable transparent huge page pud support only with radix translation. To support that add has_transparent_pud_hugepage() helper that architectures can override.
--- drivers/nvdimm/pfn_devs.c | 2 +- include/linux/pgtable.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/nvdimm/pfn_devs.c +1 -1
@@ -100,7 +100,7 @@ static unsigned long *nd_pfn_supported_alignments(unsigned long *alignments)if (has_transparent_hugepage ()) { alignments[1 ] =HPAGE_PMD_SIZE ; -if (IS_ENABLED (CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD )) +if (has_transparent_pud_hugepage ()) alignments[2 ] =HPAGE_PUD_SIZE ; }
include/linux/pgtable.h +3 -0
@@ -1597,6 +1597,9 @@ typedef unsigned int pgtbl_mod_mask;#define has_transparent_hugepage () IS_BUILTIN(CONFIG_TRANSPARENT_HUGEPAGE)#endif +#ifndef has_transparent_pud_hugepage +#define has_transparent_pud_hugepage () IS_BUILTIN(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +#endif /* * On some architectures it depends on the mm if the p4d/pud or pmd * layer of the page table hierarchy is folded or not.
Signature · 1 line
2.40.1
PATCH v2 7/16 mm: Change pudp_huge_get_and_clear_full take vm_area_struct as arg
In reply to Aneesh Kumar K.VWe will use this in a later patch to do tlb flush when clearing pud entries on powerpc. This is similar to commit 93a98695f2f9 ("mm: change pmdp_huge_get_and_clear_full take vm_area_struct as arg")
--- include/linux/pgtable.h | 4 ++-- mm/debug_vm_pgtable.c | 2 +- mm/huge_memory.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
include/linux/pgtable.h +2 -2
@@ -442,11 +442,11 @@ static inline pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL -static inline pud_t pudp_huge_get_and_clear_full (struct mm_struct *mm, +static inline pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma,unsigned long address,pud_t *pudp,int full) { -return pudp_huge_get_and_clear (mm, address, pudp); +return pudp_huge_get_and_clear (vma->vm_mm, address, pudp); }#endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
mm/debug_vm_pgtable.c +1 -1
@@ -382,7 +382,7 @@ static void __init pud_advanced_tests(struct pgtable_debug_args *args)WARN_ON (!(pud_write (pud) &&pud_dirty (pud)));#ifndef __PAGETABLE_PMD_FOLDED -pudp_huge_get_and_clear_full (args->mm, vaddr, args->pudp,1 ); +pudp_huge_get_and_clear_full (args->vma, vaddr, args->pudp,1 ); pud =READ_ONCE (*args->pudp);WARN_ON (!pud_none (pud));#endif /* __PAGETABLE_PMD_FOLDED */
mm/huge_memory.c +1 -1
@@ -1980,7 +1980,7 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma,if (!ptl)return 0 ; -pudp_huge_get_and_clear_full (tlb->mm, addr, pud, tlb->fullmm); +pudp_huge_get_and_clear_full (vma, addr, pud, tlb->fullmm);tlb_remove_pud_tlb_entry (tlb, pud, addr);if (vma_is_special_huge (vma)) {spin_unlock (ptl);
Signature · 1 line
2.40.1
PATCH v2 8/16 mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override
In reply to Aneesh Kumar K.Vdax vmemmap optimization requires a minimum of 2 PAGE_SIZE area within vmemmap such that tail page mapping can point to the second PAGE_SIZE area. Enforce that in vmemmap_can_optimize() function.
Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override.
--- include/linux/mm.h | 30 ++++++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 27 insertions(+), 5 deletions(-)
include/linux/mm.h +26 -4
@@ -31,6 +31,8 @@#include <linux/memremap.h> #include <linux/slab.h> +#include <asm/page.h> +struct mempolicy ;struct anon_vma ;struct anon_vma_chain ;
@@ -3550,13 +3552,33 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap);#endif +#define VMEMMAP_RESERVE_NR 2#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, -struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) { -return is_power_of_2 (sizeof (struct page )) && - pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +if (pgmap) { +unsigned long nr_pages; +unsigned long nr_vmemmap_pages; + + nr_pages =pgmap_vmemmap_nr (pgmap); + nr_vmemmap_pages = ((nr_pages *sizeof (struct page )) >>PAGE_SHIFT ); +/* +* For vmemmap optimization with DAX we need minimum 2 vmemmap +* pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst +*/ +return is_power_of_2 (sizeof (struct page )) && + (nr_vmemmap_pages >VMEMMAP_RESERVE_NR ) && !altmap; + } +return false; } +/* +* If we don't have an architecture override, use the generic rule +*/ +#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif +#else static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap,struct dev_pagemap *pgmap)
mm/mm_init.c +1 -1
@@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,if (!vmemmap_can_optimize (altmap, pgmap))return pgmap_vmemmap_nr (pgmap); -return 2 * (PAGE_SIZE /sizeof (struct page )); +return VMEMMAP_RESERVE_NR * (PAGE_SIZE /sizeof (struct page )); } static void __ref memmap_init_compound(struct page *head,
Signature · 1 line
2.40.1
Re: [PATCH v2 08/16] mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override
Quoted 8 lines · Aneesh Kumar K.V
dax vmemmap optimization requires a minimum of 2
PAGE_SIZEarea within vmemmap such that tail page mapping can point to the secondPAGE_SIZEarea. Enforce that invmemmap_can_optimize()function.Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override.
This makes sense. The enforcing here is not just for correctness but because you want to use VMEMMAP_RESERVE_NR supposedly?
I would suggest having two patches one for the refactor and another one for the override, but I don't feel particularly strongly about it.
Quoted 11 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mm.h | 30 ++++++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 27 insertions(+), 5 deletions(-)include/linux/mm.h +0
-0
include/linux/mm.h line 35
Quoted 5 lines
include/linux/mm.h +2
-0@@ -31,6 +31,8 @@#include <linux/memremap.h> #include <linux/slab.h> +#include <asm/page.h> +
Why is this include needed?
include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3555
Quoted 8 lines
include/linux/mm.h +1
-0struct mempolicy ;struct anon_vma ;struct anon_vma_chain ;@@ -3550,13 +3552,33 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap);#endif +#define VMEMMAP_RESERVE_NR 2
see below
Quoted 14 lines
include/linux/mm.h +8
-4#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, -struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) { -return is_power_of_2 (sizeof (struct page )) && - pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +if (pgmap) { +unsigned long nr_pages; +unsigned long nr_vmemmap_pages; + + nr_pages =pgmap_vmemmap_nr (pgmap); + nr_vmemmap_pages = ((nr_pages *sizeof (struct page )) >>PAGE_SHIFT );
include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3572
Quoted 10 lines
include/linux/mm.h +7
-0+/* +* For vmemmap optimization with DAX we need minimum 2 vmemmap +* pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst +*/ +return is_power_of_2 (sizeof (struct page )) && + (nr_vmemmap_pages >VMEMMAP_RESERVE_NR ) && !altmap; + }
It would be more readable (i.e. less identation) if you just reverse this:
unsigned long nr_vmemmap_pages; if (!pgmap || !is_power_of_2(sizeof(struct page)) return false; nr_vmemmap_pages = ((pgmap_vmemmap_nr(pgmap) * sizeof(struct page)) >> PAGE_SHIFT); /* * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ return (nr_vmemmap_pages > VMEMMAP_RESERVE_NR) && !altmap;
include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3581
Quoted 9 lines
include/linux/mm.h +8
-0+return false; } +/* +* If we don't have an architecture override, use the generic rule +*/ +#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif +
sparse-vmemmap code is trivial to change to use dedup a single vmemmap page (e.g. to align with hugetlb), hopefully the architecture override to do. this is to say whether VMEMMAP_RESERVE_NR should have similar to above?
Quoted 16 lines
#else
static inline bool vmemmap_can_optimize(structvmem_altmap*altmap,include/linux/mm.h +0
-0struct dev_pagemap *pgmap)mm/mm_init.c +1
-1@@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,if (!vmemmap_can_optimize (altmap, pgmap))return pgmap_vmemmap_nr (pgmap); -return 2 * (PAGE_SIZE /sizeof (struct page )); +return VMEMMAP_RESERVE_NR * (PAGE_SIZE /sizeof (struct page )); } static void __ref memmap_init_compound(struct page *head,
Re: [PATCH v2 08/16] mm/vmemmap: Improve vmemmap_can_optimize and allow architectures to override
Quoted 9 lines · Joao Martins
Quoted 8 lines · Aneesh Kumar K.V
dax vmemmap optimization requires a minimum of 2
PAGE_SIZEarea within vmemmap such that tail page mapping can point to the secondPAGE_SIZEarea. Enforce that invmemmap_can_optimize()function.Architectures like powerpc also want to enable vmemmap optimization conditionally (only with radix MMU translation). Hence allow architecture override.
Quoted 6 lines
This makes sense. The enforcing here is not just for correctness but because you want to use
VMEMMAP_RESERVE_NRsupposedly?I would suggest having two patches one for the refactor and another one for the override, but I don't feel particularly strongly about it.
I will wait for feedback from others. If we have others also suggesting for the split patch I will do that.
Quoted 14 lines
Quoted 14 lines
Signed-off-by Aneesh Kumar K.V <[email protected]>--- include/linux/mm.h | 30 ++++++++++++++++++++++++++---- mm/mm_init.c | 2 +- 2 files changed, 27 insertions(+), 5 deletions(-)include/linux/mm.h +0
-0@@ -31,6 +31,8 @@#include <linux/memremap.h> #include <linux/slab.h>
Quoted 4 lines
include/linux/mm.h line 35
Quoted 2 lines
include/linux/mm.h +2
-0+#include <asm/page.h> +Why is this include needed?
That was for PAGE_SHIFT. But then we do pull that through other include dependencies. I will see if i can drop that.
Quoted 39 lines
include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3555
Quoted 9 lines
include/linux/mm.h +1
-0struct mempolicy ;struct anon_vma ;struct anon_vma_chain ;@@ -3550,13 +3552,33 @@ void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap);#endif +#define VMEMMAP_RESERVE_NR 2see below
Quoted 14 lines
include/linux/mm.h +8
-4#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP -static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, -struct dev_pagemap *pgmap) +static inline bool __vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) { -return is_power_of_2 (sizeof (struct page )) && - pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +if (pgmap) { +unsigned long nr_pages; +unsigned long nr_vmemmap_pages; + + nr_pages =pgmap_vmemmap_nr (pgmap); + nr_vmemmap_pages = ((nr_pages *sizeof (struct page )) >>PAGE_SHIFT );include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3572
Quoted 10 lines
include/linux/mm.h +7
-0+/* +* For vmemmap optimization with DAX we need minimum 2 vmemmap +* pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst +*/ +return is_power_of_2 (sizeof (struct page )) && + (nr_vmemmap_pages >VMEMMAP_RESERVE_NR ) && !altmap; + }It would be more readable (i.e. less identation) if you just reverse this:
Quoted 15 lines
unsigned long nr_vmemmap_pages; if (!pgmap || !is_power_of_2(sizeof(struct page)) return false; nr_vmemmap_pages = ((pgmap_vmemmap_nr(pgmap) * sizeof(struct page)) >> PAGE_SHIFT); /* * For vmemmap optimization with DAX we need minimum 2 vmemmap * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst */ return (nr_vmemmap_pages > VMEMMAP_RESERVE_NR) && !altmap;
Will update
Quoted 5 lines
Quoted 5 lines
include/linux/mm.h +4
-0+return false; } +/* +* If we don't have an architecture override, use the generic rule +*/
Quoted 8 lines
include/linux/mm.h void vmemmap_free(unsigned long start, unsigned long end, · line 3581
Quoted 4 lines
include/linux/mm.h +4
-0+#ifndef vmemmap_can_optimize +#define vmemmap_can_optimize __vmemmap_can_optimize +#endif +sparse-vmemmap code is trivial to change to use dedup a single vmemmap page (e.g. to align with hugetlb), hopefully the architecture override to do. this is to say whether
VMEMMAP_RESERVE_NRshould have similar to above?
VMEMMAP_RESERVE_NR was added to avoid the usage of 2 in the below code. The reason we need the arch override was to add an additional check on ppc64 as shown by patch
https://lore.kernel.org/linux-mm/[email protected]/
bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap)
{
if (radix_enabled())
return __vmemmap_can_optimize(altmap, pgmap);
return false;
}Quoted 17 lines
Quoted 17 lines
#else
static inline bool vmemmap_can_optimize(structvmem_altmap*altmap,include/linux/mm.h +0
-0struct dev_pagemap *pgmap)mm/mm_init.c +1
-1@@ -1020,7 +1020,7 @@ static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,if (!vmemmap_can_optimize (altmap, pgmap))return pgmap_vmemmap_nr (pgmap); -return 2 * (PAGE_SIZE /sizeof (struct page )); +return VMEMMAP_RESERVE_NR * (PAGE_SIZE /sizeof (struct page )); } static void __ref memmap_init_compound(struct page *head,
PATCH v2 9/16 mm/vmemmap: Allow architectures to override how vmemmap optimization works
In reply to Aneesh Kumar K.VArchitectures like powerpc will like to use different page table allocators and mapping mechanisms to implement vmemmap optimization. Similar to vmemmap_populate allow architectures to implement vmemap_populate_compound_pages
--- mm/sparse-vmemmap.c | 3 +++ 1 file changed, 3 insertions(+)
mm/sparse-vmemmap.c +3 -0
@@ -141,6 +141,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } +#ifndef vmemmap_populate_compound_pagespte_t *__meminit vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node,struct vmem_altmap *altmap,struct page *reuse)
@@ -446,6 +447,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return 0; } +#endif +struct page *__meminit __populate_section_memmap (unsigned long pfn,unsigned long nr_pages,int nid,struct vmem_altmap *altmap,struct dev_pagemap *pgmap)
Signature · 1 line
2.40.1
PATCH v2 10/16 mm: Add __HAVE_ARCH_PUD_SAME similar to __HAVE_ARCH_P4D_SAME
In reply to Aneesh Kumar K.VThis helps architectures to override pmd_same and pud_same independently.
--- include/linux/pgtable.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/pgtable.h +2 -0
@@ -681,7 +681,9 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b) {return pmd_val (pmd_a) ==pmd_val (pmd_b); } +#endif +#ifndef __HAVE_ARCH_PUD_SAMEstatic inline int pud_same (pud_t pud_a,pud_t pud_b) {return pud_val (pud_a) ==pud_val (pud_b);
Signature · 1 line
2.40.1
PATCH v2 11/16 mm/huge pud: Use transparent huge pud helpers only with CONFIG_TRANSPARENT_HUGEPAGE
In reply to Aneesh Kumar K.Vpudp_set_wrprotect and move_huge_pud helpers are only used when CONFIG_TRANSPARENT_HUGEPAGE is enabled. Similar to pmdp_set_wrprotect and move_huge_pmd_helpers use architecture override only if CONFIG_TRANSPARENT_HUGEPAGE is set
--- include/linux/pgtable.h | 2 ++ mm/mremap.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/pgtable.h +2 -0
@@ -550,6 +550,7 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,#endif #ifndef __HAVE_ARCH_PUDP_SET_WRPROTECT #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pudp_set_wrprotect(struct mm_struct *mm, unsigned long address, pud_t *pudp) {
@@ -563,6 +564,7 @@ static inline void pudp_set_wrprotect(struct mm_struct *mm, {BUILD_BUG (); } +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ #endif
mm/mremap.c +1 -1
@@ -338,7 +338,7 @@ static inline bool move_normal_pud(struct vm_area_struct *vma, }#endif -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD )static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr,unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) {
Signature · 1 line
2.40.1
PATCH v2 12/16 mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization
In reply to Aneesh Kumar K.VArm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures.
With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config.
loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization
[1] commit 060a2c92d1b6 ("arm64: mm: hugetlb: Disable HUGETLB_PAGE_OPTIMIZE_VMEMMAP")
--- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 6 files changed, 10 insertions(+), 6 deletions(-)
arch/loongarch/Kconfig +1 -1
@@ -55,7 +55,7 @@ config LOONGARCH select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select BUILDTIME_TABLE_SORT select COMMON_CLK
arch/riscv/Kconfig +1 -1
@@ -50,7 +50,7 @@ config RISCV select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL - select ARCH_WANT_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU
arch/x86/Kconfig +2 -1
@@ -127,7 +127,8 @@ config X86 select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_LD_ORPHAN_WARN - select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT
fs/Kconfig +1 -1
@@ -252,7 +252,7 @@ config HUGETLB_PAGE config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON
include/linux/mm.h +1 -1
@@ -3553,7 +3553,7 @@ void vmemmap_free(unsigned long start, unsigned long end,#endif #define VMEMMAP_RESERVE_NR 2 -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize (struct vmem_altmap *altmap,struct dev_pagemap *pgmap) {
mm/Kconfig +4 -1
@@ -461,7 +461,10 @@ config SPARSEMEM_VMEMMAP # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. # -config ARCH_WANT_OPTIMIZE_VMEMMAP +config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP + bool + +config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool config HAVE_MEMBLOCK_PHYS_MAP
Signature · 1 line
2.40.1
Ritesh Harjani (IBM)
reviewerRe: [PATCH v2 12/16] mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization
Quoted 18 lines · Aneesh Kumar K.V
Arm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures.
With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config.
loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization
[1] commit
060a2c92d1b6("arm64: mm: hugetlb: DisableHUGETLB_PAGE_OPTIMIZE_VMEMMAP")Signed-off-by Aneesh Kumar K.V <[email protected]>
Quoted 8 lines
--- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 6 files changed, 10 insertions(+), 6 deletions(-)
what about s390?
git grep "ARCH_WANT_OPTIMIZE_VMEMMAP" .arch/s390/Kconfig: select ARCH_WANT_OPTIMIZE_VMEMMAP
Quoted 18 lines
mm/Kconfig +4
-1@@ -461,7 +461,10 @@ config SPARSEMEM_VMEMMAP # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. # -config ARCH_WANT_OPTIMIZE_VMEMMAP +config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP + bool + +config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool config HAVE_MEMBLOCK_PHYS_MAPSignature · 1 line
2.40.1
-ritesh
Re: [PATCH v2 12/16] mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization
Quoted 29 lines · Ritesh Harjani (IBM)
Quoted 18 lines · Aneesh Kumar K.V
Arm disabled hugetlb vmemmap optimization [1] because hugetlb vmemmap optimization includes an update of both the permissions (writeable to read-only) and the output address (pfn) of the vmemmap ptes. That is not supported without unmapping of pte(marking it invalid) by some architectures.
With DAX vmemmap optimization we don't require such pte updates and architectures can enable DAX vmemmap optimization while having hugetlb vmemmap optimization disabled. Hence split DAX optimization support into a different config.
loongarch and riscv don't have devdax support. So the DAX config is not enabled for them. With this change, arm64 should be able to select DAX optimization
[1] commit
060a2c92d1b6("arm64: mm: hugetlb: DisableHUGETLB_PAGE_OPTIMIZE_VMEMMAP")Signed-off-by Aneesh Kumar K.V <[email protected]>Quoted 8 lines
--- arch/loongarch/Kconfig | 2 +- arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 3 ++- fs/Kconfig | 2 +- include/linux/mm.h | 2 +- mm/Kconfig | 5 ++++- 6 files changed, 10 insertions(+), 6 deletions(-)
Quoted 5 lines
what about s390?
git grep "
ARCH_WANT_OPTIMIZE_VMEMMAP" .arch/s390/Kconfig: selectARCH_WANT_OPTIMIZE_VMEMMAP
Thanks for catching that. Updated
-aneesh
PATCH v2 13/16 powerpc/book3s64/mm: Enable transparent pud hugepage
In reply to Aneesh Kumar K.VThis is enabled only with radix translation and 1G hugepage size. This will be used with devdax device memory with a namespace alignment of 1G.
Anon transparent hugepage is not supported even though we do have helpers checking pud_trans_huge(). We should never find that return true. The only expected pte bit combination is _PAGE_PTE | _PAGE_DEVMAP.
Some of the helpers are never expected to get called on hash translation and hence is marked to call BUG() in such a case.
--- arch/powerpc/include/asm/book3s/64/pgtable.h | 156 ++++++++++++++++-- arch/powerpc/include/asm/book3s/64/radix.h | 37 +++++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++++++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 28 ++++ arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/platforms/Kconfig.cputype | 1 + include/trace/events/thp.h | 17 ++ 9 files changed, 323 insertions(+), 11 deletions(-)
arch/powerpc/include/asm/book3s/64/pgtable.h +145 -11
@@ -921,8 +921,29 @@ static inline pud_t pte_pud(pte_t pte) {return __pud_raw (pte_raw (pte)); } + +static inline pte_t *pudp_ptep (pud_t *pud) +{ +return (pte_t *)pud; +} + +#define pud_pfn (pud) pte_pfn(pud_pte(pud)) +#define pud_dirty (pud) pte_dirty(pud_pte(pud)) +#define pud_young (pud) pte_young(pud_pte(pud)) +#define pud_mkold (pud) pte_pud(pte_mkold(pud_pte(pud))) +#define pud_wrprotect (pud) pte_pud(pte_wrprotect(pud_pte(pud))) +#define pud_mkdirty (pud) pte_pud(pte_mkdirty(pud_pte(pud))) +#define pud_mkclean (pud) pte_pud(pte_mkclean(pud_pte(pud))) +#define pud_mkyoung (pud) pte_pud(pte_mkyoung(pud_pte(pud))) +#define pud_mkwrite (pud) pte_pud(pte_mkwrite(pud_pte(pud)))#define pud_write (pud) pte_write(pud_pte(pud)) +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY +#define pud_soft_dirty (pmd) pte_soft_dirty(pud_pte(pud)) +#define pud_mksoft_dirty (pmd) pte_pud(pte_mksoft_dirty(pud_pte(pud))) +#define pud_clear_soft_dirty (pmd) pte_pud(pte_clear_soft_dirty(pud_pte(pud))) +#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ +static inline int pud_bad (pud_t pud) {if (radix_enabled ())
@@ -1115,15 +1136,24 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write)#ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd (unsigned long pfn,pgprot_t pgprot); +extern pud_t pfn_pud (unsigned long pfn,pgprot_t pgprot);extern pmd_t mk_pmd (struct page *page,pgprot_t pgprot);extern pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot);extern void set_pmd_at (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd); +extern void set_pud_at (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,pud_t pud); +static inline void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd) { } +static inline void update_mmu_cache_pud (struct vm_area_struct *vma, +unsigned long addr,pud_t *pud) +{ +} +extern int hash__has_transparent_hugepage (void );static inline int has_transparent_hugepage (void ) {
@@ -1133,6 +1163,14 @@ static inline int has_transparent_hugepage(void) }#define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_pud_hugepage (void ) +{ +if (radix_enabled ()) +return radix__has_transparent_pud_hugepage (); +return 0 ; +} +#define has_transparent_pud_hugepage has_transparent_pud_hugepage +static inline unsigned long pmd_hugepage_update (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,unsigned long clr,unsigned long set)
@@ -1142,6 +1180,16 @@ pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp,return hash__pmd_hugepage_update (mm, addr, pmdp, clr, set); } +static inline unsigned long +pud_hugepage_update (struct mm_struct *mm,unsigned long addr,pud_t *pudp, +unsigned long clr,unsigned long set) +{ +if (radix_enabled ()) +return radix__pud_hugepage_update (mm, addr, pudp, clr, set); +BUG (); +return pud_val (*pudp); +} +/* * returns true for pmd migration entries, THP, devmap, hugetlb * But compile time dependent on THP config
@@ -1151,6 +1199,11 @@ static inline int pmd_large(pmd_t pmd)return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_PTE)); } +static inline int pud_large (pud_t pud) +{ +return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); +} +/* * For radix we should always find H_PAGE_HASHPTE zero. Hence * the below will work for radix too
@@ -1166,6 +1219,17 @@ static inline int __pmdp_test_and_clear_young(struct mm_struct *mm,return ((old & _PAGE_ACCESSED) !=0 ); } +static inline int __pudp_test_and_clear_young (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +unsigned long old; + +if ((pud_raw (*pudp) &cpu_to_be64 (_PAGE_ACCESSED |H_PAGE_HASHPTE )) ==0 ) +return 0 ; + old =pud_hugepage_update (mm, addr, pudp, _PAGE_ACCESSED,0 ); +return ((old & _PAGE_ACCESSED) !=0 ); +} +#define __HAVE_ARCH_PMDP_SET_WRPROTECTstatic inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)
@@ -1174,6 +1238,14 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr,pmd_hugepage_update (mm, addr, pmdp, _PAGE_WRITE,0 ); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect (struct mm_struct *mm,unsigned long addr, +pud_t *pudp) +{ +if (pud_write (*pudp)) +pud_hugepage_update (mm, addr, pudp, _PAGE_WRITE,0 ); +} +/* * Only returns true for a THP. False for pmd migration entry. * We also need to return true when we come across a pte that
@@ -1195,6 +1267,17 @@ static inline int pmd_trans_huge(pmd_t pmd)return hash__pmd_trans_huge (pmd); } +static inline int pud_trans_huge (pud_t pud) +{ +if (!pud_present (pud)) +return false; + +if (radix_enabled ()) +return radix__pud_trans_huge (pud); +return 0 ; +} + +#define __HAVE_ARCH_PMD_SAMEstatic inline int pmd_same (pmd_t pmd_a,pmd_t pmd_b) {
@@ -1203,6 +1286,16 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b)return hash__pmd_same (pmd_a, pmd_b); } +#define __HAVE_ARCH_PUD_SAME +static inline int pud_same (pud_t pud_a,pud_t pud_b) +{ +if (radix_enabled ()) +return radix__pud_same (pud_a, pud_b); +BUG (); +return 0 ; +} + +static inline pmd_t __pmd_mkhuge (pmd_t pmd) {if (radix_enabled ())
@@ -1210,6 +1303,14 @@ static inline pmd_t __pmd_mkhuge(pmd_t pmd)return hash__pmd_mkhuge (pmd); } +static inline pud_t __pud_mkhuge (pud_t pud) +{ +if (radix_enabled ()) +return radix__pud_mkhuge (pud); +BUG (); +return pud; +} +/* * pfn_pmd return a pmd_t that can be used as pmd pte entry. */
@@ -1225,14 +1326,34 @@ static inline pmd_t pmd_mkhuge(pmd_t pmd)return pmd; } +static inline pud_t pud_mkhuge (pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM +if (radix_enabled ()) +WARN_ON ((pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)) ==0 ); +else +WARN_ON (1 ); +#endif +return pud; +} + +#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGSextern int pmdp_set_access_flags (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp,pmd_t entry,int dirty); +#define __HAVE_ARCH_PUDP_SET_ACCESS_FLAGS +extern int pudp_set_access_flags (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp, +pud_t entry,int dirty);#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNGextern int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_TEST_AND_CLEAR_YOUNG +extern int pudp_test_and_clear_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp); +#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
@@ -1243,6 +1364,16 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,return hash__pmdp_huge_get_and_clear (mm, addr, pmdp); } +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR +static inline pud_t pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +if (radix_enabled ()) +return radix__pudp_huge_get_and_clear (mm, addr, pudp); +BUG (); +return *pudp; +} +static inline pmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) {
@@ -1257,6 +1388,11 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,unsigned long addr, pmd_t *pmdp, int full); +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma, +unsigned long addr, +pud_t *pudp,int full); +#define __HAVE_ARCH_PGTABLE_DEPOSITstatic inline void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pgtable)
@@ -1305,6 +1441,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)return hash__pmd_mkdevmap (pmd); } +static inline pud_t pud_mkdevmap (pud_t pud) +{ +if (radix_enabled ()) +return radix__pud_mkdevmap (pud); +BUG (); +return pud; +} +static inline int pmd_devmap (pmd_t pmd) {return pte_devmap (pmd_pte (pmd));
@@ -1312,7 +1456,7 @@ static inline int pmd_devmap(pmd_t pmd)static inline int pud_devmap (pud_t pud) { -return 0 ; +return pte_devmap (pud_pte (pud)); }static inline int pgd_devmap (pgd_t pgd)
@@ -1321,16 +1465,6 @@ static inline int pgd_devmap(pgd_t pgd) }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int pud_pfn (pud_t pud) -{ -/* -* Currently all calls to pud_pfn() are gated around a pud_devmap() -* check so this should never be used. If it grows another user we -* want to know about it. -*/ -BUILD_BUG (); -return 0 ; -}#define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTIONpte_t ptep_modify_prot_start (struct vm_area_struct *,unsigned long ,pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long,
arch/powerpc/include/asm/book3s/64/radix.h +37 -0
@@ -250,6 +250,10 @@ static inline int radix__pud_bad(pud_t pud)return !!(pud_val (pud) &RADIX_PUD_BAD_BITS ); } +static inline int radix__pud_same (pud_t pud_a,pud_t pud_b) +{ +return ((pud_raw (pud_a) ^pud_raw (pud_b)) ==0 ); +}static inline int radix__p4d_bad (p4d_t p4d) {
@@ -268,9 +272,22 @@ static inline pmd_t radix__pmd_mkhuge(pmd_t pmd)return __pmd (pmd_val (pmd) | _PAGE_PTE); } +static inline int radix__pud_trans_huge (pud_t pud) +{ +return (pud_val (pud) & (_PAGE_PTE | _PAGE_DEVMAP)) == _PAGE_PTE; +} + +static inline pud_t radix__pud_mkhuge (pud_t pud) +{ +return __pud (pud_val (pud) | _PAGE_PTE); +} +extern unsigned long radix__pmd_hugepage_update (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,unsigned long clr,unsigned long set); +extern unsigned long radix__pud_hugepage_update (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,unsigned long clr, +unsigned long set);extern pmd_t radix__pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
@@ -278,6 +295,9 @@ extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,extern pgtable_t radix__pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);extern pmd_t radix__pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp); +pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp); +static inline int radix__has_transparent_hugepage (void ) {/* For radix 2M at PMD level means thp */
@@ -285,6 +305,14 @@ static inline int radix__has_transparent_hugepage(void)return 1 ; return 0; } + +static inline int radix__has_transparent_pud_hugepage (void ) +{ +/* For radix 1G at PUD level means pud hugepage support */ +if (mmu_psize_defs[MMU_PAGE_1G ].shift ==PUD_SHIFT ) +return 1 ; +return 0 ; +}#endif static inline pmd_t radix__pmd_mkdevmap (pmd_t pmd)
@@ -292,9 +320,18 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd)return __pmd (pmd_val (pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); } +static inline pud_t radix__pud_mkdevmap (pud_t pud) +{ +return __pud (pud_val (pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap ; +struct dev_pagemap ;extern int __meminit radix__vmemmap_create_mapping (unsigned long start,unsigned long page_size,unsigned long phys); +int __meminit radix__vmemmap_populate (unsigned long start,unsigned long end, +int node,struct vmem_altmap *altmap);extern void radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size);
arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +2 -0
@@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize);extern void radix__flush_pmd_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end); +extern void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end);extern void radix__flush_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);extern void radix__flush_tlb_kernel_range (unsigned long start,unsigned long end);
arch/powerpc/include/asm/book3s/64/tlbflush.h +8 -0
@@ -50,6 +50,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma,radix__flush_pmd_tlb_range (vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +if (radix_enabled ()) +radix__flush_pud_tlb_range (vma, start, end); +} +#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start,
arch/powerpc/mm/book3s64/pgtable.c +78 -0
112 lines not on this page · raw message
arch/powerpc/mm/book3s64/radix_pgtable.c +28 -0
@@ -965,6 +965,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addreturn old; } +unsigned long radix__pud_hugepage_update (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,unsigned long clr, +unsigned long set) +{ +unsigned long old; + +#ifdef CONFIG_DEBUG_VM +WARN_ON (!pud_devmap (*pudp)); +assert_spin_locked (pud_lockptr (mm, pudp)); +#endif + + old =radix__pte_update (mm, addr,pudp_ptep (pudp), clr, set,1 ); +trace_hugepage_update (addr, old, clr, set); + +return old; +} +pmd_t radix__pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
@@ -1041,6 +1058,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,return old_pmd; } +pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +pud_t old_pud; +unsigned long old; + + old =radix__pud_hugepage_update (mm, addr, pudp, ~0UL ,0 ); + old_pud =__pud (old); +return old_pud; +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,
arch/powerpc/mm/book3s64/radix_tlb.c +7 -0
@@ -1453,6 +1453,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, }EXPORT_SYMBOL (radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +radix__flush_tlb_range_psize (vma->vm_mm, start, end,MMU_PAGE_1G ); +} +EXPORT_SYMBOL (radix__flush_pud_tlb_range); +void radix__flush_tlb_all (void ) {unsigned long rb,prs,r,rs;
arch/powerpc/platforms/Kconfig.cputype +1 -0
@@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
include/trace/events/thp.h +17 -0
@@ -25,6 +25,23 @@ TRACE_EVENT(hugepage_set_pmd,TP_printk ("Set pmd with 0x%lx with 0x%lx" , __entry->addr, __entry->pmd) ); +TRACE_EVENT (hugepage_set_pud, + + TP_PROTO(unsigned long addr, unsigned long pud), +TP_ARGS (addr, pud), +TP_STRUCT__entry ( + __field(unsigned long, addr) + __field(unsigned long, pud) + ), + +TP_fast_assign ( + __entry->addr = addr; + __entry->pud = pud; + ), + +TP_printk ("Set pud with 0x%lx with 0x%lx" , __entry->addr, __entry->pud) + ); + TRACE_EVENT(hugepage_update,
Signature · 1 line
2.40.1
Ritesh Harjani (IBM)
reviewerRe: [PATCH v2 13/16] powerpc/book3s64/mm: Enable transparent pud hugepage
"Aneesh Kumar K.V" <[email protected]> writes:
These are just some minor nits in case you are going to send another revision.
Quoted 378 lines
This is enabled only with radix translation and 1G hugepage size. This will be used with devdax device memory with a namespace alignment of 1G.
Anon transparent hugepage is not supported even though we do have helpers checking
pud_trans_huge(). We should never find that return true. The only expected pte bit combination is_PAGE_PTE|_PAGE_DEVMAP.Some of the helpers are never expected to get called on hash translation and hence is marked to call
BUG()in such a case.Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/include/asm/book3s/64/pgtable.h | 156 ++++++++++++++++-- arch/powerpc/include/asm/book3s/64/radix.h | 37 +++++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++++++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 28 ++++ arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/platforms/Kconfig.cputype | 1 + include/trace/events/thp.h | 17 ++ 9 files changed, 323 insertions(+), 11 deletions(-)arch/powerpc/include/asm/book3s/64/pgtable.h +145
-11@@ -921,8 +921,29 @@ static inline pud_t pte_pud(pte_t pte) {return __pud_raw (pte_raw (pte)); } + +static inline pte_t *pudp_ptep (pud_t *pud) +{ +return (pte_t *)pud; +} + +#define pud_pfn (pud) pte_pfn(pud_pte(pud)) +#define pud_dirty (pud) pte_dirty(pud_pte(pud)) +#define pud_young (pud) pte_young(pud_pte(pud)) +#define pud_mkold (pud) pte_pud(pte_mkold(pud_pte(pud))) +#define pud_wrprotect (pud) pte_pud(pte_wrprotect(pud_pte(pud))) +#define pud_mkdirty (pud) pte_pud(pte_mkdirty(pud_pte(pud))) +#define pud_mkclean (pud) pte_pud(pte_mkclean(pud_pte(pud))) +#define pud_mkyoung (pud) pte_pud(pte_mkyoung(pud_pte(pud))) +#define pud_mkwrite (pud) pte_pud(pte_mkwrite(pud_pte(pud)))#define pud_write (pud) pte_write(pud_pte(pud)) +#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY +#define pud_soft_dirty (pmd) pte_soft_dirty(pud_pte(pud)) +#define pud_mksoft_dirty (pmd) pte_pud(pte_mksoft_dirty(pud_pte(pud))) +#define pud_clear_soft_dirty (pmd) pte_pud(pte_clear_soft_dirty(pud_pte(pud))) +#endif /* CONFIG_HAVE_ARCH_SOFT_DIRTY */ +static inline int pud_bad (pud_t pud) {if (radix_enabled ())@@ -1115,15 +1136,24 @@ static inline bool pmd_access_permitted(pmd_t pmd, bool write)#ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pfn_pmd (unsigned long pfn,pgprot_t pgprot); +extern pud_t pfn_pud (unsigned long pfn,pgprot_t pgprot);extern pmd_t mk_pmd (struct page *page,pgprot_t pgprot);extern pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot);extern void set_pmd_at (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd); +extern void set_pud_at (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,pud_t pud); +static inline void update_mmu_cache_pmd (struct vm_area_struct *vma,unsigned long addr,pmd_t *pmd) { } +static inline void update_mmu_cache_pud (struct vm_area_struct *vma, +unsigned long addr,pud_t *pud) +{ +} +extern int hash__has_transparent_hugepage (void );static inline int has_transparent_hugepage (void ) {@@ -1133,6 +1163,14 @@ static inline int has_transparent_hugepage(void) }#define has_transparent_hugepage has_transparent_hugepage +static inline int has_transparent_pud_hugepage (void ) +{ +if (radix_enabled ()) +return radix__has_transparent_pud_hugepage (); +return 0 ; +} +#define has_transparent_pud_hugepage has_transparent_pud_hugepage +static inline unsigned long pmd_hugepage_update (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,unsigned long clr,unsigned long set)@@ -1142,6 +1180,16 @@ pmd_hugepage_update(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp,return hash__pmd_hugepage_update (mm, addr, pmdp, clr, set); } +static inline unsigned long +pud_hugepage_update (struct mm_struct *mm,unsigned long addr,pud_t *pudp, +unsigned long clr,unsigned long set) +{ +if (radix_enabled ()) +return radix__pud_hugepage_update (mm, addr, pudp, clr, set); +BUG (); +return pud_val (*pudp); +} +/* * returns true for pmd migration entries, THP, devmap, hugetlb * But compile time dependent on THP config @@ -1151,6 +1199,11 @@ static inline int pmd_large(pmd_t pmd)return !!(pmd_raw (pmd) &cpu_to_be64 (_PAGE_PTE)); } +static inline int pud_large (pud_t pud) +{ +return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); +} +/* * For radix we should always find H_PAGE_HASHPTE zero. Hence * the below will work for radix too @@ -1166,6 +1219,17 @@ static inline int __pmdp_test_and_clear_young(struct mm_struct *mm,return ((old & _PAGE_ACCESSED) !=0 ); } +static inline int __pudp_test_and_clear_young (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +unsigned long old; + +if ((pud_raw (*pudp) &cpu_to_be64 (_PAGE_ACCESSED |H_PAGE_HASHPTE )) ==0 ) +return 0 ; + old =pud_hugepage_update (mm, addr, pudp, _PAGE_ACCESSED,0 ); +return ((old & _PAGE_ACCESSED) !=0 ); +} +#define __HAVE_ARCH_PMDP_SET_WRPROTECTstatic inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)@@ -1174,6 +1238,14 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long addr,pmd_hugepage_update (mm, addr, pmdp, _PAGE_WRITE,0 ); } +#define __HAVE_ARCH_PUDP_SET_WRPROTECT +static inline void pudp_set_wrprotect (struct mm_struct *mm,unsigned long addr, +pud_t *pudp) +{ +if (pud_write (*pudp)) +pud_hugepage_update (mm, addr, pudp, _PAGE_WRITE,0 ); +} +/* * Only returns true for a THP. False for pmd migration entry. * We also need to return true when we come across a pte that @@ -1195,6 +1267,17 @@ static inline int pmd_trans_huge(pmd_t pmd)return hash__pmd_trans_huge (pmd); } +static inline int pud_trans_huge (pud_t pud) +{ +if (!pud_present (pud)) +return false; + +if (radix_enabled ()) +return radix__pud_trans_huge (pud); +return 0 ; +} + +#define __HAVE_ARCH_PMD_SAMEstatic inline int pmd_same (pmd_t pmd_a,pmd_t pmd_b) {@@ -1203,6 +1286,16 @@ static inline int pmd_same(pmd_t pmd_a, pmd_t pmd_b)return hash__pmd_same (pmd_a, pmd_b); } +#define __HAVE_ARCH_PUD_SAME +static inline int pud_same (pud_t pud_a,pud_t pud_b) +{ +if (radix_enabled ()) +return radix__pud_same (pud_a, pud_b); +BUG (); +return 0 ; +} + +static inline pmd_t __pmd_mkhuge (pmd_t pmd) {if (radix_enabled ())@@ -1210,6 +1303,14 @@ static inline pmd_t __pmd_mkhuge(pmd_t pmd)return hash__pmd_mkhuge (pmd); } +static inline pud_t __pud_mkhuge (pud_t pud) +{ +if (radix_enabled ()) +return radix__pud_mkhuge (pud); +BUG (); +return pud; +} +/* * pfn_pmd return a pmd_t that can be used as pmd pte entry. */ @@ -1225,14 +1326,34 @@ static inline pmd_t pmd_mkhuge(pmd_t pmd)return pmd; } +static inline pud_t pud_mkhuge (pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM +if (radix_enabled ()) +WARN_ON ((pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)) ==0 ); +else +WARN_ON (1 ); +#endif +return pud; +} + +#define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGSextern int pmdp_set_access_flags (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp,pmd_t entry,int dirty); +#define __HAVE_ARCH_PUDP_SET_ACCESS_FLAGS +extern int pudp_set_access_flags (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp, +pud_t entry,int dirty);#define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNGextern int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_TEST_AND_CLEAR_YOUNG +extern int pudp_test_and_clear_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp); +#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,@@ -1243,6 +1364,16 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,return hash__pmdp_huge_get_and_clear (mm, addr, pmdp); } +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR +static inline pud_t pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +if (radix_enabled ()) +return radix__pudp_huge_get_and_clear (mm, addr, pudp); +BUG (); +return *pudp; +} +static inline pmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) {@@ -1257,6 +1388,11 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,unsigned long addr, pmd_t *pmdp, int full); +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEAR_FULL +pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma, +unsigned long addr, +pud_t *pudp,int full); +#define __HAVE_ARCH_PGTABLE_DEPOSITstatic inline void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pgtable)@@ -1305,6 +1441,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)return hash__pmd_mkdevmap (pmd); } +static inline pud_t pud_mkdevmap (pud_t pud) +{ +if (radix_enabled ()) +return radix__pud_mkdevmap (pud); +BUG (); +return pud; +} +static inline int pmd_devmap (pmd_t pmd) {return pte_devmap (pmd_pte (pmd));@@ -1312,7 +1456,7 @@ static inline int pmd_devmap(pmd_t pmd)static inline int pud_devmap (pud_t pud) { -return 0 ; +return pte_devmap (pud_pte (pud)); }static inline int pgd_devmap (pgd_t pgd)@@ -1321,16 +1465,6 @@ static inline int pgd_devmap(pgd_t pgd) }#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline int pud_pfn (pud_t pud) -{ -/* -* Currently all calls to pud_pfn() are gated around a pud_devmap() -* check so this should never be used. If it grows another user we -* want to know about it. -*/ -BUILD_BUG (); -return 0 ; -}#define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTIONpte_t ptep_modify_prot_start (struct vm_area_struct *,unsigned long ,pte_t *); void ptep_modify_prot_commit(struct vm_area_struct *, unsigned long,arch/powerpc/include/asm/book3s/64/radix.h +28
-0@@ -250,6 +250,10 @@ static inline int radix__pud_bad(pud_t pud)return !!(pud_val (pud) &RADIX_PUD_BAD_BITS ); } +static inline int radix__pud_same (pud_t pud_a,pud_t pud_b) +{ +return ((pud_raw (pud_a) ^pud_raw (pud_b)) ==0 ); +}static inline int radix__p4d_bad (p4d_t p4d) {@@ -268,9 +272,22 @@ static inline pmd_t radix__pmd_mkhuge(pmd_t pmd)return __pmd (pmd_val (pmd) | _PAGE_PTE); } +static inline int radix__pud_trans_huge (pud_t pud) +{ +return (pud_val (pud) & (_PAGE_PTE | _PAGE_DEVMAP)) == _PAGE_PTE; +} + +static inline pud_t radix__pud_mkhuge (pud_t pud) +{ +return __pud (pud_val (pud) | _PAGE_PTE); +} +extern unsigned long radix__pmd_hugepage_update (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,unsigned long clr,unsigned long set); +extern unsigned long radix__pud_hugepage_update (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,unsigned long clr, +unsigned long set);extern pmd_t radix__pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,@@ -278,6 +295,9 @@ extern void radix__pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,extern pgtable_t radix__pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp);extern pmd_t radix__pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp); +pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp); +static inline int radix__has_transparent_hugepage (void ) {/* For radix 2M at PMD level means thp */ @@ -285,6 +305,14 @@ static inline int radix__has_transparent_hugepage(void)return 1 ; return 0; } + +static inline int radix__has_transparent_pud_hugepage (void ) +{ +/* For radix 1G at PUD level means pud hugepage support */ +if (mmu_psize_defs[MMU_PAGE_1G ].shift ==PUD_SHIFT ) +return 1 ; +return 0 ; +}#endif static inline pmd_t radix__pmd_mkdevmap (pmd_t pmd)
arch/powerpc/include/asm/book3s/64/radix.h static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) · line 329
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/radix.h +7
-0@@ -292,9 +320,18 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd)return __pmd (pmd_val (pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); } +static inline pud_t radix__pud_mkdevmap (pud_t pud) +{ +return __pud (pud_val (pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap ; +struct dev_pagemap ;
Minor nit.
I guess this struct dev_pagemap is meant to be added in a later patch.
Quoted 45 lines
arch/powerpc/include/asm/book3s/64/radix.h +2
-0extern int __meminit radix__vmemmap_create_mapping (unsigned long start,unsigned long page_size,unsigned long phys); +int __meminit radix__vmemmap_populate (unsigned long start,unsigned long end, +int node,struct vmem_altmap *altmap);extern void radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size);arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +2
-0@@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize);extern void radix__flush_pmd_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end); +extern void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end);extern void radix__flush_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);extern void radix__flush_tlb_kernel_range (unsigned long start,unsigned long end);arch/powerpc/include/asm/book3s/64/tlbflush.h +8
-0@@ -50,6 +50,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma,radix__flush_pmd_tlb_range (vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +if (radix_enabled ()) +radix__flush_pud_tlb_range (vma, start, end); +} +#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start,arch/powerpc/mm/book3s64/pgtable.c +0
-0
arch/powerpc/mm/book3s64/pgtable.c int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, · line 72
Quoted 9 lines
arch/powerpc/mm/book3s64/pgtable.c +6
-0@@ -64,11 +64,39 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,return changed; } +int pudp_set_access_flags (struct vm_area_struct *vma,unsigned long address, +pud_t *pudp,pud_t entry,int dirty) +{ +int changed; +#ifdef CONFIG_DEBUG_VM +WARN_ON (!pud_devmap (*pudp));
just a query -
for pmdp_set_access_flags() we had WARN_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp));
so why don't we require the same check here?
arch/powerpc/mm/book3s64/pgtable.c int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, · line 78
Quoted 6 lines
arch/powerpc/mm/book3s64/pgtable.c +6
-0+assert_spin_locked (pud_lockptr (vma->vm_mm, pudp)); +#endif + changed = !pud_same (*(pudp), entry); +if (changed) { +/* +* We can use MMU_PAGE_2M here, because only radix
s/MMU_PAGE_2M/MMY_PAGE_1G
Quoted 28 lines
arch/powerpc/mm/book3s64/pgtable.c +16
-0+* path look at the psize. +*/ +__ptep_set_access_flags (vma,pudp_ptep (pudp), +pud_pte (entry), address,MMU_PAGE_1G ); + } +return changed; +} + +int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) {return __pmdp_test_and_clear_young (vma->vm_mm, address, pmdp); } + +int pudp_test_and_clear_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp) +{ +return __pudp_test_and_clear_young (vma->vm_mm, address, pudp); +} +/* * set a new huge pmd. We should not be called for updating * an existing pmd entry. That should go via pmd_hugepage_update. @@ -90,6 +118,23 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr,return set_pte_at (mm, addr,pmdp_ptep (pmdp),pmd_pte (pmd)); }
arch/powerpc/mm/book3s64/pgtable.c void set_pmd_at(struct mm_struct *mm, unsigned long addr, · line 130
Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +10
-0+void set_pud_at (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM +/* +* Make sure hardware valid bit is not set. We don't do +* tlb flush for this update. +*/ + +WARN_ON (pte_hw_valid (pud_pte (*pudp)));
For set_pmd_at() we hadWARN_ON(pte_hw_valid(pmd_pte(*pmdp)) && !pte_protnone(pmd_pte(*pmdp)));
Could you please help me understand why is it different for set_pud_at()
Quoted 73 lines
arch/powerpc/mm/book3s64/pgtable.c +40
-0+assert_spin_locked (pud_lockptr (mm, pudp)); +WARN_ON (!(pud_large (pud))); +#endif +trace_hugepage_set_pud (addr,pud_val (pud)); +return set_pte_at (mm, addr,pudp_ptep (pudp),pud_pte (pud)); +} +static void do_serialize (void *arg) {/* We've taken the IPI, so try to trim the mask while here */ @@ -147,11 +192,35 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,return pmd; } +pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma, +unsigned long addr,pud_t *pudp,int full) +{ +pud_t pud; + +VM_BUG_ON (addr & ~HPAGE_PMD_MASK ); +VM_BUG_ON ((pud_present (*pudp) && !pud_devmap (*pudp)) || + !pud_present (*pudp)); + pud =pudp_huge_get_and_clear (vma->vm_mm, addr, pudp); +/* +* if it not a fullmm flush, then we can possibly end up converting +* this PMD pte entry to a regular level 0 PTE by a parallel page fault. +* Make sure we flush the tlb in this case. +*/ +if (!full) +flush_pud_tlb_range (vma, addr, addr +HPAGE_PUD_SIZE ); +return pud; +} +static pmd_t pmd_set_protbits (pmd_t pmd,pgprot_t pgprot) {return __pmd (pmd_val (pmd) |pgprot_val (pgprot)); } +static pud_t pud_set_protbits (pud_t pud,pgprot_t pgprot) +{ +return __pud (pud_val (pud) |pgprot_val (pgprot)); +} +/* * At some point we should be able to get rid of * pmd_mkhuge() and mk_huge_pmd() when we update all the @@ -166,6 +235,15 @@ pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot)return __pmd_mkhuge (pmd_set_protbits (__pmd (pmdv), pgprot)); } +pud_t pfn_pud (unsigned long pfn,pgprot_t pgprot) +{ +unsigned long pudv; + + pudv = (pfn <<PAGE_SHIFT ) &PTE_RPN_MASK ; + +return __pud_mkhuge (pud_set_protbits (__pud (pudv), pgprot)); +} +pmd_t mk_pmd (struct page *page,pgprot_t pgprot) {return pfn_pmd (page_to_pfn (page), pgprot);arch/powerpc/mm/book3s64/radix_pgtable.c +3
-0@@ -965,6 +965,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addreturn old; } +unsigned long radix__pud_hugepage_update (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,unsigned long clr, +unsigned long set)
arch/powerpc/mm/book3s64/radix_pgtable.c unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add · line 980
Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +10
-0+{ +unsigned long old; + +#ifdef CONFIG_DEBUG_VM +WARN_ON (!pud_devmap (*pudp)); +assert_spin_locked (pud_lockptr (mm, pudp)); +#endif + + old =radix__pte_update (mm, addr,pudp_ptep (pudp), clr, set,1 ); +trace_hugepage_update (addr, old, clr, set);
Here, we are using the same trace event for both pmd and pud update. See the comment in the end.
Quoted 60 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +15
-0+ +return old; +} +pmd_t radix__pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)@@ -1041,6 +1058,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,return old_pmd; } +pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +pud_t old_pud; +unsigned long old; + + old =radix__pud_hugepage_update (mm, addr, pudp, ~0UL ,0 ); + old_pud =__pud (old); +return old_pud; +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,arch/powerpc/mm/book3s64/radix_tlb.c +7
-0@@ -1453,6 +1453,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, }EXPORT_SYMBOL (radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +radix__flush_tlb_range_psize (vma->vm_mm, start, end,MMU_PAGE_1G ); +} +EXPORT_SYMBOL (radix__flush_pud_tlb_range); +void radix__flush_tlb_all (void ) {unsigned long rb,prs,r,rs;arch/powerpc/platforms/Kconfig.cputype +1
-0@@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGEinclude/trace/events/thp.h +0
-0
include/trace/events/thp.h TRACE_EVENT(hugepage_set_pmd, · line 28
Quoted 4 lines
include/trace/events/thp.h +1
-0@@ -25,6 +25,23 @@ TRACE_EVENT(hugepage_set_pmd,TP_printk ("Set pmd with 0x%lx with 0x%lx" , __entry->addr, __entry->pmd) ); +TRACE_EVENT(hugepage_set_pud,
Given we have trace_hugepage_set_pud and trace_hugepage_set_pmd, doing the exact same thing, we can as well have a DECLARE_EVENT_CLASS(hugepage_set) and then DEFINE_EVENT using this event class.
Quoted 8 lines
include/trace/events/thp.h +8
-0+ + TP_PROTO(unsigned long addr, unsigned long pud), +TP_ARGS (addr, pud), +TP_STRUCT__entry ( + __field(unsigned long, addr) +__field (unsigned long ,pud ) + ), +
include/trace/events/thp.h TRACE_EVENT(hugepage_set_pmd, · line 46
Quoted 10 lines
include/trace/events/thp.h +8
-0+TP_fast_assign ( + __entry->addr = addr; + __entry->pud = pud; + ), + +TP_printk ("Set pud with 0x%lx with 0x%lx" , __entry->addr, __entry->pud) + ); + TRACE_EVENT(hugepage_update,
Same here. We can have a DECLARE_EVENT_CLASS(hugepage_update) and then DEFINE_EVENT for hugepage_update_pmd and hugepage_update_pud.
-ritesh
Re: [PATCH v2 13/16] powerpc/book3s64/mm: Enable transparent pud hugepage
Quoted 1 line · Ritesh Harjani (IBM)
"Aneesh Kumar K.V" <[email protected]> writes:
....
Quoted 6 lines
Quoted 6 lines
static inline
pmd_tradix__pmd_mkdevmap(pmd_t pmd)arch/powerpc/include/asm/book3s/64/radix.h +0
-0@@ -292,9 +320,18 @@ static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd)return __pmd (pmd_val (pmd) | (_PAGE_PTE | _PAGE_DEVMAP)); }
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/radix.h static inline pmd_t radix__pmd_mkdevmap(pmd_t pmd) · line 329
Quoted 7 lines
arch/powerpc/include/asm/book3s/64/radix.h +7
-0+static inline pud_t radix__pud_mkdevmap (pud_t pud) +{ +return __pud (pud_val (pud) | (_PAGE_PTE | _PAGE_DEVMAP)); +} + +struct vmem_altmap ; +struct dev_pagemap ;Minor nit.
I guess this structdev_pagemapis meant to be added in a later patch.
Moved that change to a later patch.
Quoted 60 lines
Quoted 46 lines
arch/powerpc/include/asm/book3s/64/radix.h +2
-0extern int __meminit radix__vmemmap_create_mapping (unsigned long start,unsigned long page_size,unsigned long phys); +int __meminit radix__vmemmap_populate (unsigned long start,unsigned long end, +int node,struct vmem_altmap *altmap);extern void radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size);arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +2
-0@@ -68,6 +68,8 @@ void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize);extern void radix__flush_pmd_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end); +extern void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end);extern void radix__flush_tlb_range (struct vm_area_struct *vma,unsigned long start,unsigned long end);extern void radix__flush_tlb_kernel_range (unsigned long start,unsigned long end);arch/powerpc/include/asm/book3s/64/tlbflush.h +8
-0@@ -50,6 +50,14 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma,radix__flush_pmd_tlb_range (vma, start, end); } +#define __HAVE_ARCH_FLUSH_PUD_TLB_RANGE +static inline void flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +if (radix_enabled ()) +radix__flush_pud_tlb_range (vma, start, end); +} +#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start,arch/powerpc/mm/book3s64/pgtable.c +0
-0arch/powerpc/mm/book3s64/pgtable.c int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, · line 72
Quoted 9 lines
arch/powerpc/mm/book3s64/pgtable.c +6
-0@@ -64,11 +64,39 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address,return changed; } +int pudp_set_access_flags (struct vm_area_struct *vma,unsigned long address, +pud_t *pudp,pud_t entry,int dirty) +{ +int changed; +#ifdef CONFIG_DEBUG_VM +WARN_ON (!pud_devmap (*pudp));just a query -
forpmdp_set_access_flags()we hadWARN_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp));
Quoted 2 lines
so why don't we require the same check here?
Because we don't support generic anon PUD level THP. and we want to catch a wrong usage.
Quoted 9 lines
arch/powerpc/mm/book3s64/pgtable.c int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, · line 78
Quoted 6 lines
arch/powerpc/mm/book3s64/pgtable.c +6
-0+assert_spin_locked (pud_lockptr (vma->vm_mm, pudp)); +#endif + changed = !pud_same (*(pudp), entry); +if (changed) { +/* +* We can use MMU_PAGE_2M here, because only radix s/MMU_PAGE_2M/MMY_PAGE_1G
updated
Quoted 42 lines
Quoted 28 lines
arch/powerpc/mm/book3s64/pgtable.c +16
-0+* path look at the psize. +*/ +__ptep_set_access_flags (vma,pudp_ptep (pudp), +pud_pte (entry), address,MMU_PAGE_1G ); + } +return changed; +} + +int pmdp_test_and_clear_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp) {return __pmdp_test_and_clear_young (vma->vm_mm, address, pmdp); } + +int pudp_test_and_clear_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp) +{ +return __pudp_test_and_clear_young (vma->vm_mm, address, pudp); +} +/* * set a new huge pmd. We should not be called for updating * an existing pmd entry. That should go via pmd_hugepage_update. @@ -90,6 +118,23 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr,return set_pte_at (mm, addr,pmdp_ptep (pmdp),pmd_pte (pmd)); }arch/powerpc/mm/book3s64/pgtable.c void set_pmd_at(struct mm_struct *mm, unsigned long addr, · line 130
Quoted 10 lines
arch/powerpc/mm/book3s64/pgtable.c +10
-0+void set_pud_at (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,pud_t pud) +{ +#ifdef CONFIG_DEBUG_VM +/* +* Make sure hardware valid bit is not set. We don't do +* tlb flush for this update. +*/ + +WARN_ON (pte_hw_valid (pud_pte (*pudp)));For
set_pmd_at()we hadWARN_ON(pte_hw_valid(pmd_pte(*pmdp)) && !pte_protnone(pmd_pte(*pmdp)));
Quoted 3 lines
Could you please help me understand why is it different for
set_pud_at()
There is no numa fault support for devdax pages and we want to catch wrong usage.
Quoted 70 lines
Quoted 70 lines
arch/powerpc/mm/book3s64/pgtable.c +40
-0+assert_spin_locked (pud_lockptr (mm, pudp)); +WARN_ON (!(pud_large (pud))); +#endif +trace_hugepage_set_pud (addr,pud_val (pud)); +return set_pte_at (mm, addr,pudp_ptep (pudp),pud_pte (pud)); +} +static void do_serialize (void *arg) {/* We've taken the IPI, so try to trim the mask while here */ @@ -147,11 +192,35 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,return pmd; } +pud_t pudp_huge_get_and_clear_full (struct vm_area_struct *vma, +unsigned long addr,pud_t *pudp,int full) +{ +pud_t pud; + +VM_BUG_ON (addr & ~HPAGE_PMD_MASK ); +VM_BUG_ON ((pud_present (*pudp) && !pud_devmap (*pudp)) || + !pud_present (*pudp)); + pud =pudp_huge_get_and_clear (vma->vm_mm, addr, pudp); +/* +* if it not a fullmm flush, then we can possibly end up converting +* this PMD pte entry to a regular level 0 PTE by a parallel page fault. +* Make sure we flush the tlb in this case. +*/ +if (!full) +flush_pud_tlb_range (vma, addr, addr +HPAGE_PUD_SIZE ); +return pud; +} +static pmd_t pmd_set_protbits (pmd_t pmd,pgprot_t pgprot) {return __pmd (pmd_val (pmd) |pgprot_val (pgprot)); } +static pud_t pud_set_protbits (pud_t pud,pgprot_t pgprot) +{ +return __pud (pud_val (pud) |pgprot_val (pgprot)); +} +/* * At some point we should be able to get rid of * pmd_mkhuge() and mk_huge_pmd() when we update all the @@ -166,6 +235,15 @@ pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot)return __pmd_mkhuge (pmd_set_protbits (__pmd (pmdv), pgprot)); } +pud_t pfn_pud (unsigned long pfn,pgprot_t pgprot) +{ +unsigned long pudv; + + pudv = (pfn <<PAGE_SHIFT ) &PTE_RPN_MASK ; + +return __pud_mkhuge (pud_set_protbits (__pud (pudv), pgprot)); +} +pmd_t mk_pmd (struct page *page,pgprot_t pgprot) {return pfn_pmd (page_to_pfn (page), pgprot);arch/powerpc/mm/book3s64/radix_pgtable.c +0
-0@@ -965,6 +965,23 @@ unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long addreturn old; }
Quoted 17 lines
arch/powerpc/mm/book3s64/radix_pgtable.c unsigned long radix__pmd_hugepage_update(struct mm_struct *mm, unsigned long add · line 980
Quoted 13 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +13
-0+unsigned long radix__pud_hugepage_update (struct mm_struct *mm,unsigned long addr, +pud_t *pudp,unsigned long clr, +unsigned long set) +{ +unsigned long old; + +#ifdef CONFIG_DEBUG_VM +WARN_ON (!pud_devmap (*pudp)); +assert_spin_locked (pud_lockptr (mm, pudp)); +#endif + + old =radix__pte_update (mm, addr,pudp_ptep (pudp), clr, set,1 ); +trace_hugepage_update (addr, old, clr, set);Here, we are using the same trace event for both pmd and pud update. See the comment in the end.
updated
Quoted 86 lines
Quoted 60 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +15
-0+ +return old; +} +pmd_t radix__pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)@@ -1041,6 +1058,17 @@ pmd_t radix__pmdp_huge_get_and_clear(struct mm_struct *mm,return old_pmd; } +pud_t radix__pudp_huge_get_and_clear (struct mm_struct *mm, +unsigned long addr,pud_t *pudp) +{ +pud_t old_pud; +unsigned long old; + + old =radix__pud_hugepage_update (mm, addr, pudp, ~0UL ,0 ); + old_pud =__pud (old); +return old_pud; +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ void radix__ptep_set_access_flags(struct vm_area_struct *vma, pte_t *ptep,arch/powerpc/mm/book3s64/radix_tlb.c +7
-0@@ -1453,6 +1453,13 @@ void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, }EXPORT_SYMBOL (radix__flush_pmd_tlb_range); +void radix__flush_pud_tlb_range (struct vm_area_struct *vma, +unsigned long start,unsigned long end) +{ +radix__flush_tlb_range_psize (vma->vm_mm, start, end,MMU_PAGE_1G ); +} +EXPORT_SYMBOL (radix__flush_pud_tlb_range); +void radix__flush_tlb_all (void ) {unsigned long rb,prs,r,rs;arch/powerpc/platforms/Kconfig.cputype +1
-0@@ -94,6 +94,7 @@ config PPC_BOOK3S_64 select PPC_FPU select PPC_HAVE_PMU_SUPPORT select HAVE_ARCH_TRANSPARENT_HUGEPAGE + select HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGEinclude/trace/events/thp.h +0
-0include/trace/events/thp.h TRACE_EVENT(hugepage_set_pmd, · line 28
Quoted 4 lines
include/trace/events/thp.h +1
-0@@ -25,6 +25,23 @@ TRACE_EVENT(hugepage_set_pmd,TP_printk ("Set pmd with 0x%lx with 0x%lx" , __entry->addr, __entry->pmd) ); +TRACE_EVENT(hugepage_set_pud,Given we have
trace_hugepage_set_pudandtrace_hugepage_set_pmd, doing the exact same thing, we can as well have aDECLARE_EVENT_CLASS(hugepage_set)and thenDEFINE_EVENTusing this event class.Quoted 17 lines
include/trace/events/thp.h +16
-0+ + TP_PROTO(unsigned long addr, unsigned long pud), +TP_ARGS (addr, pud), +TP_STRUCT__entry ( + __field(unsigned long, addr) +__field (unsigned long ,pud ) + ), + +TP_fast_assign ( + __entry->addr = addr; + __entry->pud = pud; + ), + +TP_printk ("Set pud with 0x%lx with 0x%lx" , __entry->addr, __entry->pud) + ); +
Quoted 4 lines
include/trace/events/thp.h TRACE_EVENT(hugepage_set_pmd, · line 46
Quoted 1 line
include/trace/events/thp.h +0
-0TRACE_EVENT(hugepage_update,Same here. We can have a
DECLARE_EVENT_CLASS(hugepage_update)and thenDEFINE_EVENTforhugepage_update_pmdandhugepage_update_pud.
updated. I created a patch that first switch the existing tracepoint to DECLARE_EVENT_CLASS.
-aneesh
PATCH v2 14/16 powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function
In reply to Aneesh Kumar K.VThis is in preparation to update radix to implement vmemmap optimization for devdax. Below are the rules w.r.t radix vmemmap mapping
1. First try to map things using PMD (2M)
2. With altmap if altmap cross-boundary check returns true, fall back to PAGE_SIZE
3. If we can't allocate PMD_SIZE backing memory for vmemmap, fallback to PAGE_SIZE
On removing vmemmap mapping, check if every subsection that is using the vmemmap area is invalid. If found to be invalid, that implies we can safely free the vmemmap area. We don't use the PAGE_UNUSED pattern used by x86 because with 64K page size, we need to do the above check even at the PAGE_SIZE granularity.
--- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/mm/book3s64/radix_pgtable.c | 319 +++++++++++++++++++-- arch/powerpc/mm/init_64.c | 26 +- 4 files changed, 319 insertions(+), 31 deletions(-)
arch/powerpc/include/asm/book3s/64/radix.h +2 -0
@@ -332,6 +332,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys);int __meminit radix__vmemmap_populate (unsigned long start,unsigned long end,int node,struct vmem_altmap *altmap); +void __ref radix__vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap);extern void radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size);
arch/powerpc/include/asm/pgtable.h +3 -0
@@ -168,6 +168,9 @@ static inline bool is_ioremap_addr(const void *x)struct seq_file ;void arch_report_meminfo (struct seq_file *m); +int __meminit vmemmap_populated (unsigned long vmemmap_addr,int vmemmap_map_size); +bool altmap_cross_boundary (struct vmem_altmap *altmap,unsigned long start, +unsigned long page_size);#endif /* CONFIG_PPC64 */ #endif /* __ASSEMBLY__ */
arch/powerpc/mm/book3s64/radix_pgtable.c +293 -26
389 lines not on this page · raw message
arch/powerpc/mm/init_64.c +21 -5
@@ -92,7 +92,7 @@ static struct page * __meminit vmemmap_subsection_start(unsigned long vmemmap_ad* a page table lookup here because with the hash translation we don't keep * vmemmap details in linux page table. */ -static int __meminit vmemmap_populated (unsigned long vmemmap_addr,int vmemmap_map_size) +int __meminit vmemmap_populated (unsigned long vmemmap_addr,int vmemmap_map_size) {struct page *start;unsigned long vmemmap_end = vmemmap_addr + vmemmap_map_size;
@@ -183,8 +183,8 @@ static __meminit int vmemmap_list_populate(unsigned long phys,return 0 ; } -static bool altmap_cross_boundary (struct vmem_altmap *altmap,unsigned long start, -unsigned long page_size) +bool altmap_cross_boundary (struct vmem_altmap *altmap,unsigned long start, +unsigned long page_size) {unsigned long nr_pfn = page_size /sizeof (struct page );unsigned long start_pfn =page_to_pfn ((struct page *)start);
@@ -204,6 +204,11 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,bool altmap_alloc;unsigned long page_size =1 << mmu_psize_defs[mmu_vmemmap_psize].shift; +#ifdef CONFIG_PPC_BOOK3S_64 +if (radix_enabled ()) +return radix__vmemmap_populate (start, end, node, altmap); +#endif +/* Align to the page size of the linear mapping. */ start =ALIGN_DOWN (start, page_size);
@@ -303,8 +308,8 @@ static unsigned long vmemmap_list_free(unsigned long start)return vmem_back->phys; } -void __ref vmemmap_free (unsigned long start,unsigned long end, -struct vmem_altmap *altmap) +void __ref __vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) {unsigned long page_size =1 << mmu_psize_defs[mmu_vmemmap_psize].shift;unsigned long page_order =get_order (page_size);
@@ -362,6 +367,17 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, vmemmap_remove_mapping(start, page_size); } } + +void __ref vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap) +{ +#ifdef CONFIG_PPC_BOOK3S_64 +if (radix_enabled ()) +return radix__vmemmap_free (start, end, altmap); +#endif +return __vmemmap_free (start, end, altmap); +} +#endif void register_page_bootmem_memmap (unsigned long section_nr,struct page *start_page,unsigned long size)
Signature · 1 line
2.40.1
Ritesh Harjani (IBM)
reviewerRe: [PATCH v2 14/16] powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function
Quoted 226 lines · Aneesh Kumar K.V
This is in preparation to update radix to implement vmemmap optimization for devdax. Below are the rules w.r.t radix vmemmap mapping
1. First try to map things using PMD (2M)
2. With altmap if altmap cross-boundary check returns true, fall back toPAGE_SIZE
3. If we can't allocatePMD_SIZEbacking memory for vmemmap, fallback toPAGE_SIZEOn removing vmemmap mapping, check if every subsection that is using the vmemmap area is invalid. If found to be invalid, that implies we can safely free the vmemmap area. We don't use the
PAGE_UNUSEDpattern used by x86 because with 64K page size, we need to do the above check even at thePAGE_SIZEgranularity.Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/mm/book3s64/radix_pgtable.c | 319 +++++++++++++++++++-- arch/powerpc/mm/init_64.c | 26 +- 4 files changed, 319 insertions(+), 31 deletions(-)arch/powerpc/include/asm/book3s/64/radix.h +2
-0@@ -332,6 +332,8 @@ extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long phys);int __meminit radix__vmemmap_populate (unsigned long start,unsigned long end,int node,struct vmem_altmap *altmap); +void __ref radix__vmemmap_free (unsigned long start,unsigned long end, +struct vmem_altmap *altmap);extern void radix__vmemmap_remove_mapping (unsigned long start,unsigned long page_size);arch/powerpc/include/asm/pgtable.h +3
-0@@ -168,6 +168,9 @@ static inline bool is_ioremap_addr(const void *x)struct seq_file ;void arch_report_meminfo (struct seq_file *m); +int __meminit vmemmap_populated (unsigned long vmemmap_addr,int vmemmap_map_size); +bool altmap_cross_boundary (struct vmem_altmap *altmap,unsigned long start, +unsigned long page_size);#endif /* CONFIG_PPC64 */ #endif /* __ASSEMBLY__ */ arch/powerpc/mm/book3s64/radix_pgtable.c +78
-25@@ -742,8 +742,57 @@ static void free_pud_table(pud_t *pud_start, p4d_t *p4d)p4d_clear (p4d); } +static bool __meminit vmemmap_pmd_is_unused (unsigned long addr,unsigned long end) +{ +unsigned long start =ALIGN_DOWN (addr,PMD_SIZE ); + +return !vmemmap_populated (start,PMD_SIZE ); +} + +static bool __meminit vmemmap_page_is_unused (unsigned long addr,unsigned long end) +{ +unsigned long start =ALIGN_DOWN (addr,PAGE_SIZE ); + +return !vmemmap_populated (start,PAGE_SIZE ); + +} + +static void __meminit free_vmemmap_pages (struct page *page, +struct vmem_altmap *altmap, +int order) +{ +unsigned int nr_pages =1 << order; + +if (altmap) { +unsigned long alt_start, alt_end; +unsigned long base_pfn =page_to_pfn (page); + +/* +* with 1G vmemmap mmaping we can have things setup +* such that even though atlmap is specified we never +* used altmap. +*/ + alt_start = altmap->base_pfn; + alt_end = altmap->base_pfn + altmap->reserve + + altmap->free + altmap->alloc + altmap->align; + +if (base_pfn >= alt_start && base_pfn < alt_end) { +vmem_altmap_free (altmap, nr_pages); +return ; + } + } + +if (PageReserved (page)) { +/* allocated from memblock */ +while (nr_pages--) +free_reserved_page (page++); + }else +free_pages ((unsigned long )page_address (page), order); +} +static void remove_pte_table (pte_t *pte_start,unsigned long addr, -unsigned long end,bool direct) +unsigned long end,bool direct, +struct vmem_altmap *altmap) {unsigned long next, pages =0 ;pte_t *pte;@@ -757,24 +806,23 @@ static void remove_pte_table(pte_t *pte_start, unsigned long addr,if (!pte_present (*pte))continue ; -if (!PAGE_ALIGNED (addr) || !PAGE_ALIGNED (next)) { -/* -* The vmemmap_free() and remove_section_mapping() -* codepaths call us with aligned addresses. -*/ -WARN_ONCE (1 ,"%s: unaligned range\n" , __func__); -continue ; +if (PAGE_ALIGNED (addr) &&PAGE_ALIGNED (next)) { +if (!direct) +free_vmemmap_pages (pte_page (*pte), altmap,0 ); +pte_clear (&init_mm, addr, pte); + pages++; + }else if (!direct &&vmemmap_page_is_unused (addr, next)) { +free_vmemmap_pages (pte_page (*pte), altmap,0 ); +pte_clear (&init_mm, addr, pte); } - -pte_clear (&init_mm, addr, pte); - pages++; }if (direct)update_page_count (mmu_virtual_psize, -pages); }static void __meminit remove_pmd_table (pmd_t *pmd_start,unsigned long addr, -unsigned long end,bool direct) +unsigned long end,bool direct, +struct vmem_altmap *altmap) {unsigned long next, pages =0 ;pte_t *pte_base;@@ -788,18 +836,21 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr,continue ;if (pmd_is_leaf (*pmd)) { -if (!IS_ALIGNED (addr,PMD_SIZE ) || - !IS_ALIGNED (next,PMD_SIZE )) { -WARN_ONCE (1 ,"%s: unaligned range\n" , __func__); -continue ; +if (IS_ALIGNED (addr,PMD_SIZE ) && +IS_ALIGNED (next,PMD_SIZE )) { +if (!direct) +free_vmemmap_pages (pmd_page (*pmd), altmap,get_order (PMD_SIZE )); +pte_clear (&init_mm, addr, (pte_t *)pmd); + pages++; + }else if (vmemmap_pmd_is_unused (addr, next)) { +free_vmemmap_pages (pmd_page (*pmd), altmap,get_order (PMD_SIZE )); +pte_clear (&init_mm, addr, (pte_t *)pmd); } -pte_clear (&init_mm, addr, (pte_t *)pmd); - pages++;continue ; } pte_base = (pte_t *)pmd_page_vaddr (*pmd); -remove_pte_table (pte_base, addr, next, direct); +remove_pte_table (pte_base, addr, next, direct, altmap);free_pte_table (pte_base, pmd); }if (direct)@@ -807,7 +858,8 @@ static void __meminit remove_pmd_table(pmd_t *pmd_start, unsigned long addr, }static void __meminit remove_pud_table (pud_t *pud_start,unsigned long addr, -unsigned long end,bool direct) +unsigned long end,bool direct, +struct vmem_altmap *altmap) {unsigned long next, pages =0 ;pmd_t *pmd_base;@@ -832,15 +884,16 @@ static void __meminit remove_pud_table(pud_t *pud_start, unsigned long addr, } pmd_base =pud_pgtable (*pud); -remove_pmd_table (pmd_base, addr, next, direct); +remove_pmd_table (pmd_base, addr, next, direct, altmap);free_pmd_table (pmd_base, pud); }if (direct)update_page_count (MMU_PAGE_1G , -pages); } -static void __meminit remove_pagetable (unsigned long start,unsigned long end, -bool direct) +static void __meminit +remove_pagetable (unsigned long start,unsigned long end,bool direct, +struct vmem_altmap *altmap) {unsigned long addr, next;pud_t *pud_base;@@ -869,7 +922,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end, } pud_base =p4d_pgtable (*p4d); -remove_pud_table (pud_base, addr, next, direct); +remove_pud_table (pud_base, addr, next, direct, altmap);free_pud_table (pud_base, p4d); }@@ -892,7 +945,7 @@ int __meminit radix__create_section_mapping(unsigned long start,int __meminit radix__remove_section_mapping (unsigned long start,unsigned long end) { -remove_pagetable (start, end, true); +remove_pagetable (start, end, true,NULL );return 0 ; }#endif /* CONFIG_MEMORY_HOTPLUG */
arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 985
Quoted 9 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +6
-0@@ -924,10 +977,224 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start,return 0 ; } +int __meminit vmemmap_check_pmd (pmd_t *pmd,int node, +unsigned long addr,unsigned long next) +{ +int large =pmd_large (*pmd); + +if (pmd_large (*pmd))
we already got the value of pmd_large into "large" variable. we can use just if (large) right?
arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 986
Quoted 1 line
arch/powerpc/mm/book3s64/radix_pgtable.c +1
-0+vmemmap_verify ((pte_t *)pmd, node, addr, next);
maybe we can use pmdp_ptep() function here which we used in the 1st patch? also shouldn't this be pmdp in the function argument instead of pmd?
Quoted 33 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +33
-0+ +return large; +} + +void __meminit vmemmap_set_pmd (pmd_t *pmdp,void *p,int node, +unsigned long addr,unsigned long next) +{ +pte_t entry; +pte_t *ptep =pmdp_ptep (pmdp); + +VM_BUG_ON (!IS_ALIGNED (addr,PMD_SIZE )); + entry =pfn_pte (__pa (p) >>PAGE_SHIFT ,PAGE_KERNEL ); +set_pte_at (&init_mm, addr, ptep, entry); + asmvolatile ("ptesync" : : :"memory" ); + +vmemmap_verify (ptep, node, addr, next); +} + +static pte_t *__meminit radix__vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node, +struct vmem_altmap *altmap, +struct page *reuse) +{ +pte_t *pte =pte_offset_kernel (pmd, addr); + +if (pte_none (*pte)) { +pte_t entry; +void *p; + +if (!reuse) { +/* +* make sure we don't create altmap mappings +* covering things outside the device. +*/
arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 1029
Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +10
-0+if (altmap &&altmap_cross_boundary (altmap, addr,PAGE_SIZE )) + altmap =NULL ; + + p =vmemmap_alloc_block_buf (PAGE_SIZE , node, altmap); +if (!p) { +if (altmap) + p =vmemmap_alloc_block_buf (PAGE_SIZE , node,NULL ); +if (!p) +return NULL ; + }
Above if conditions are quite confusing when looking for the 1st time? Can we do this? Did I get it right?
if (!p && altmap)
p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL);
if (!p)
return NULL;-ritesh
Re: [PATCH v2 14/16] powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap handling function
Quoted 1 line · Ritesh Harjani (IBM)
"Aneesh Kumar K.V" <[email protected]> writes:
....
Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 985
Quoted 6 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +6
-0+int __meminit vmemmap_check_pmd (pmd_t *pmd,int node, +unsigned long addr,unsigned long next) +{ +int large =pmd_large (*pmd); + +if (pmd_large (*pmd))we already got the value of
pmd_largeinto "large" variable. we can use just if (large) right?
Quoted 5 lines
arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 986
Quoted 1 line
arch/powerpc/mm/book3s64/radix_pgtable.c +1
-0+vmemmap_verify ((pte_t *)pmd, node, addr, next);maybe we can use
pmdp_ptep()function here which we used in the 1st patch? also shouldn't this be pmdp in the function argument instead of pmd?
updated
Quoted 47 lines
Quoted 33 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +33
-0+ +return large; +} + +void __meminit vmemmap_set_pmd (pmd_t *pmdp,void *p,int node, +unsigned long addr,unsigned long next) +{ +pte_t entry; +pte_t *ptep =pmdp_ptep (pmdp); + +VM_BUG_ON (!IS_ALIGNED (addr,PMD_SIZE )); + entry =pfn_pte (__pa (p) >>PAGE_SHIFT ,PAGE_KERNEL ); +set_pte_at (&init_mm, addr, ptep, entry); + asmvolatile ("ptesync" : : :"memory" ); + +vmemmap_verify (ptep, node, addr, next); +} + +static pte_t *__meminit radix__vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node, +struct vmem_altmap *altmap, +struct page *reuse) +{ +pte_t *pte =pte_offset_kernel (pmd, addr); + +if (pte_none (*pte)) { +pte_t entry; +void *p; + +if (!reuse) { +/* +* make sure we don't create altmap mappings +* covering things outside the device. +*/ arch/powerpc/mm/book3s64/radix_pgtable.c int __meminit radix__vmemmap_create_mapping(unsigned long start, · line 1029
Quoted 10 lines
arch/powerpc/mm/book3s64/radix_pgtable.c +10
-0+if (altmap &&altmap_cross_boundary (altmap, addr,PAGE_SIZE )) + altmap =NULL ; + + p =vmemmap_alloc_block_buf (PAGE_SIZE , node, altmap); +if (!p) { +if (altmap) + p =vmemmap_alloc_block_buf (PAGE_SIZE , node,NULL ); +if (!p) +return NULL ; + }Above if conditions are quite confusing when looking for the 1st time? Can we do this? Did I get it right?
Quoted 6 lines
if (!p && altmap)
p = vmemmap_alloc_block_buf(PAGE_SIZE, node, NULL);
if (!p)
return NULL;updated
-aneesh
PATCH v2 15/16 powerpc/book3s64/radix: Add support for vmemmap optimization for radix
In reply to Aneesh Kumar K.VWith 2M PMD-level mapping, we require 32 struct pages and a single vmemmap page can contain 1024 struct pages (PAGE_SIZE/sizeof(struct page)). Hence with 64K page size, we don't use vmemmap deduplication for PMD-level mapping.
--- Documentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 ++++++++++ arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/radix.h | 8 + arch/powerpc/mm/book3s64/radix_pgtable.c | 203 +++++++++++++++++++++ 6 files changed, 315 insertions(+) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst
Documentation/mm/vmemmap_dedup.rst +1 -0
@@ -210,6 +210,7 @@ the device (altmap). The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). +For powerpc equivalent details see Documentation/powerpc/vmemmap_dedup.rst The differences with HugeTLB are relatively minor.
Documentation/powerpc/index.rst +1 -0
@@ -35,6 +35,7 @@ powerpc ultravisor vas-api vcpudispatch_stats + vmemmap_dedup features
Documentation/powerpc/vmemmap_dedup.rst new file +101 -0
101 lines not on this page · raw message
arch/powerpc/Kconfig +1 -0
@@ -175,6 +175,7 @@ config PPC select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if PPC_RADIX_MMU select ARCH_WANTS_MODULES_DATA_IN_VMALLOC if PPC_BOOK3S_32 || PPC_8xx select ARCH_WEAK_RELEASE_ACQUIRE select BINFMT_ELF
arch/powerpc/include/asm/book3s/64/radix.h +8 -0
@@ -364,5 +364,13 @@ int radix__remove_section_mapping(unsigned long start, unsigned long end);void radix__kernel_map_pages (struct page *page,int numpages,int enable); +#define vmemmap_can_optimize vmemmap_can_optimize +bool vmemmap_can_optimize (struct vmem_altmap *altmap,struct dev_pagemap *pgmap); + +#define vmemmap_populate_compound_pages vmemmap_populate_compound_pages +int __meminit vmemmap_populate_compound_pages (unsigned long start_pfn, +unsigned long start, +unsigned long end,int node, +struct dev_pagemap *pgmap);#endif /* __ASSEMBLY__ */ #endif
arch/powerpc/mm/book3s64/radix_pgtable.c +203 -0
215 lines not on this page · raw message
Signature · 1 line
2.40.1
PATCH v2 16/16 powerpc/book3s64/radix: Remove mmu_vmemmap_psize
In reply to Aneesh Kumar K.VThis is not used by radix anymore.
--- arch/powerpc/mm/book3s64/radix_pgtable.c | 10 ---------- arch/powerpc/mm/init_64.c | 21 ++++++++++++++------- 2 files changed, 14 insertions(+), 17 deletions(-)
arch/powerpc/mm/book3s64/radix_pgtable.c +0 -10
@@ -601,16 +601,6 @@ void __init radix__early_init_mmu(void) mmu_virtual_psize =MMU_PAGE_4K ;#endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP -/* vmemmap mapping */ -if (mmu_psize_defs[MMU_PAGE_2M ].shift) { -/* -* map vmemmap using 2M if available -*/ - mmu_vmemmap_psize =MMU_PAGE_2M ; - }else - mmu_vmemmap_psize = mmu_virtual_psize; -#endif /* * initialize page table size */
arch/powerpc/mm/init_64.c +14 -7
@@ -198,17 +198,12 @@ bool altmap_cross_boundary(struct vmem_altmap *altmap, unsigned long start,return false; } -int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, -struct vmem_altmap *altmap) +int __meminit __vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) {bool altmap_alloc;unsigned long page_size =1 << mmu_psize_defs[mmu_vmemmap_psize].shift; -#ifdef CONFIG_PPC_BOOK3S_64 -if (radix_enabled ()) -return radix__vmemmap_populate (start, end, node, altmap); -#endif -/* Align to the page size of the linear mapping. */ start =ALIGN_DOWN (start, page_size);
@@ -277,6 +272,18 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,return 0 ; } +int __meminit vmemmap_populate (unsigned long start,unsigned long end,int node, +struct vmem_altmap *altmap) +{ + +#ifdef CONFIG_PPC_BOOK3S_64 +if (radix_enabled ()) +return radix__vmemmap_populate (start, end, node, altmap); +#endif + +return __vmemmap_populate (start, end, node, altmap); +} +#ifdef CONFIG_MEMORY_HOTPLUG static unsigned long vmemmap_list_free (unsigned long start) {
Signature · 1 line
2.40.1
Quoted 14 lines · Aneesh Kumar K.V
This patch series implements changes required to support DAX vmemmap optimization for ppc64. The vmemmap optimization is only enabled with radix MMU translation and 1GB PUD mapping with 64K page size. The patch series also split hugetlb vmemmap optimization as a separate
Kconfigvariable so that architectures can enable DAX vmemmap optimization without enabling hugetlb vmemmap optimization. This should enable architectures like arm64 to enable DAX vmemmap optimization while they can't enable hugetlb vmemmap optimization. More details of the same are in patch "mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization"
Quoted 4 lines
Changes from V1:
* Fix make htmldocs warning
* Fix vmemmap allocation bugs with different alignment values.
* Correctly check for section validity to before we free vmemmap area
Thanks for the updated series.
The previously reported WARN_ON is not seen with this series. I also ran few regression tests against this series and did not see any problems.
Based on the test results
-Sachin
Hi Andrew,
Quoted 61 lines · Aneesh Kumar K.V
This patch series implements changes required to support DAX vmemmap optimization for ppc64. The vmemmap optimization is only enabled with radix MMU translation and 1GB PUD mapping with 64K page size. The patch series also split hugetlb vmemmap optimization as a separate
Kconfigvariable so that architectures can enable DAX vmemmap optimization without enabling hugetlb vmemmap optimization. This should enable architectures like arm64 to enable DAX vmemmap optimization while they can't enable hugetlb vmemmap optimization. More details of the same are in patch "mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization"Changes from V1:
* Fix make htmldocs warning
* Fix vmemmap allocation bugs with different alignment values.
* Correctly check for section validity to before we free vmemmap areaAneesh Kumar K.V (16):
powerpc/mm/book3s64: Usepmdp_ptephelper instead of typecasting. powerpc/book3s64/mm:mmu_vmemmap_psizeis used by radix powerpc/book3s64/mm: Fix DirectMap stats in/proc/meminfopowerpc/book3s64/mm: UsePAGE_KERNELinstead of opencoding powerpc/mm/dax: Fix the condition when checking if altmap vmemap can
cross-boundary
mm/hugepage pud: Allow arch-specific helper function to check huge
page pud support
mm: Changepudp_huge_get_and_clear_fulltakevm_area_structas arg mm/vmemmap: Improvevmemmap_can_optimizeand allow architectures to
override
mm/vmemmap: Allow architectures to override how vmemmap optimization
works
mm: Add__HAVE_ARCH_PUD_SAMEsimilar to__HAVE_ARCH_P4D_SAMEmm/huge pud: Use transparent huge pud helpers only with
CONFIG_TRANSPARENT_HUGEPAGE
mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization powerpc/book3s64/mm: Enable transparent pud hugepage powerpc/book3s64/vmemmap: Switch radix to use a different vmemmap
handling function
powerpc/book3s64/radix: Add support for vmemmap optimization for radix powerpc/book3s64/radix: Removemmu_vmemmap_psizeDocumentation/mm/vmemmap_dedup.rst | 1 + Documentation/powerpc/index.rst | 1 + Documentation/powerpc/vmemmap_dedup.rst | 101 +++ arch/loongarch/Kconfig | 2 +- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/book3s/64/pgtable.h | 156 ++++- arch/powerpc/include/asm/book3s/64/radix.h | 47 ++ .../include/asm/book3s/64/tlbflush-radix.h | 2 + arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 + arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/mm/book3s64/pgtable.c | 78 +++ arch/powerpc/mm/book3s64/radix_pgtable.c | 577 ++++++++++++++++-- arch/powerpc/mm/book3s64/radix_tlb.c | 7 + arch/powerpc/mm/init_64.c | 39 +- arch/powerpc/platforms/Kconfig.cputype | 1 + arch/riscv/Kconfig | 2 +- arch/x86/Kconfig | 3 +- drivers/nvdimm/pfn_devs.c | 2 +- fs/Kconfig | 2 +- include/linux/mm.h | 32 +- include/linux/pgtable.h | 11 +- include/trace/events/thp.h | 17 +
Quoted 9 lines
mm/Kconfig | 5 +- mm/debug_vm_pgtable.c | 2 +- mm/huge_memory.c | 2 +- mm/mm_init.c | 2 +- mm/mremap.c | 2 +- mm/sparse-vmemmap.c | 3 + 28 files changed, 1032 insertions(+), 77 deletions(-) create mode 100644 Documentation/powerpc/vmemmap_dedup.rst
Michael Ellerman merged some of the ppc64 patches. How do you like to merge the mm changes? Do you like a repost of patches 6-12?
-aneesh
Andrew Morton
maintainerQuoted 6 lines · On Sat, 24 Jun 2023 20:22:27 +0530 "Aneesh Kumar K.V"…
Quoted 3 lines
28 files changed, 1032 insertions(+), 77 deletions(-) create mode 100644 Documentation/powerpc/vmemmap_dedup.rstMichael Ellerman merged some of the ppc64 patches. How do you like to merge the mm changes? Do you like a repost of patches 6-12?
https://lkml.kernel.org/r/[email protected] leads me to believe there will be a v3 series?
Please send whatever is appropriate, against 6.5-rc1 or thereabouts.
Re: (subset) [PATCH v2 00/16] Add support for DAX vmemmap optimization for ppc64
In reply to Aneesh Kumar K.VQuoted 11 lines · Aneesh Kumar K.V
This patch series implements changes required to support DAX vmemmap optimization for ppc64. The vmemmap optimization is only enabled with radix MMU translation and 1GB PUD mapping with 64K page size. The patch series also split hugetlb vmemmap optimization as a separate
Kconfigvariable so that architectures can enable DAX vmemmap optimization without enabling hugetlb vmemmap optimization. This should enable architectures like arm64 to enable DAX vmemmap optimization while they can't enable hugetlb vmemmap optimization. More details of the same are in patch "mm/vmemmap optimization: Split hugetlb and devdax vmemmap optimization"[...]
Patch 1, 3, 4 and 5 applied to powerpc/next.
[01/16] powerpc/mm/book3s64: Use pmdp_ptep helper instead of typecasting.
https://git.kernel.org/powerpc/c/040ec6202bb89d51e2d8d4c848d8160c0e5e3fe8 [03/16] powerpc/book3s64/mm: Fix DirectMap stats in /proc/meminfo https://git.kernel.org/powerpc/c/0da90af431abc3f497a38ec9ef6e43b0d0dabe80 [04/16] powerpc/book3s64/mm: Use PAGE_KERNEL instead of opencoding https://git.kernel.org/powerpc/c/d933557b858f78d7b702e70614b469a84b72e56c [05/16] powerpc/mm/dax: Fix the condition when checking if altmap vmemap can cross-boundary https://git.kernel.org/powerpc/c/c8eebc4a99f15280654f23e914e746c40a516e50
cheers