Reimplement huge pages without hugepd on powerpc (8xx, e500, book3s/64)
Subsystems (22)
v3 cover Reimplement huge pages without hugepd on powerpc (8xx, e500, book3s/64)
This is the continuation of the RFC v1 series "Reimplement huge pages without hugepd on powerpc 8xx". It now get rid of hugepd completely after handling also e500 and book3s/64
Also see https://github.com/linuxppc/issues/issues/483
Unlike most architectures, powerpc 8xx HW requires a two-level pagetable topology for all page sizes. So a leaf PMD-contig approach is not feasible as such.
Possible sizes on 8xx are 4k, 16k, 512k and 8M.
First level (PGD/PMD) covers 4M per entry. For 8M pages, two PMD entries must point to a single entry level-2 page table. Until now that was done using hugepd. This series changes it to use standard page tables where the entry is replicated 1024 times on each of the two pagetables refered by the two associated PMD entries for that 8M page.
For e500 and book3s/64 there are less constraints because it is not tied to the HW assisted tablewalk like on 8xx, so it is easier to use leaf PMDs (and PUDs).
On e500 the supported page sizes are 4M, 16M, 64M, 256M and 1G. All at PMD level on e500/32 (mpc85xx) and mix of PMD and PUD for e500/64. We encode page size with 4 available bits in PTE entries. On e300/32 PGD entries size is increases to 64 bits in order to allow leaf-PMD entries because PTE are 64 bits on e500.
On book3s/64 only the hash-4k mode is concerned. It supports 16M pages as cont-PMD and 16G pages as cont-PUD. In other modes (radix-4k, radix-6k and hash-64k) the sizes match with PMD and PUD sizes so that's just leaf entries. The hash processing make things a bit more complex. To ease things, __hash_page_huge() is modified to bail out when DIRTY or ACCESSED bits are missing, leaving it to mm core to fix it.
Global changes in v3:
- Removed patches 1 and 2
- Squashed patch 11 into patch 5
- Replaced patches 12 and 13 with a series from Michael
- Reordered patches a bit to have more general patches up front
For more details on changes, see in each patch.
Christophe Leroy (15):
mm: Define __pte_leaf_size() to also take a PMD entry
mm: Provide mm_struct and address to huge_ptep_get() powerpc/mm: Remove _PAGE_PSIZE
powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries powerpc/mm: Allow hugepages without hugepd
powerpc/8xx: Fix size given to set_huge_pte_at() powerpc/8xx: Rework support for 8M pages using contiguous PTE entries powerpc/8xx: Simplify struct mmu_psize_def
powerpc/e500: Remove enc and ind fields from struct mmu_psize_def powerpc/e500: Switch to 64 bits PGD on 85xx (32 bits) powerpc/e500: Encode hugepage size in PTE bits powerpc/e500: Use contiguous PMD instead of hugepd powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD powerpc/mm: Remove hugepd leftovers
mm: Remove CONFIG_ARCH_HAS_HUGEPD
Michael Ellerman (1):
powerpc/64e: Remove unused IBM HTW code [SQUASHED]
arch/arm/include/asm/hugetlb-3level.h | 4 +- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 - arch/powerpc/include/asm/book3s/64/hash-4k.h | 15 - arch/powerpc/include/asm/book3s/64/hash.h | 38 +- arch/powerpc/include/asm/book3s/64/hugetlb.h | 38 -- .../include/asm/book3s/64/pgtable-4k.h | 34 -- .../include/asm/book3s/64/pgtable-64k.h | 20 - arch/powerpc/include/asm/book3s/64/pgtable.h | 18 + arch/powerpc/include/asm/hugetlb.h | 15 +- .../include/asm/nohash/32/hugetlb-8xx.h | 38 +- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 9 +- arch/powerpc/include/asm/nohash/32/pte-40x.h | 3 - arch/powerpc/include/asm/nohash/32/pte-44x.h | 3 - arch/powerpc/include/asm/nohash/32/pte-85xx.h | 3 - arch/powerpc/include/asm/nohash/32/pte-8xx.h | 58 ++- .../powerpc/include/asm/nohash/hugetlb-e500.h | 36 +- arch/powerpc/include/asm/nohash/mmu-e500.h | 6 +- arch/powerpc/include/asm/nohash/pgalloc.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 45 +- arch/powerpc/include/asm/nohash/pte-e500.h | 35 +- arch/powerpc/include/asm/page.h | 32 -- arch/powerpc/include/asm/pgtable-be-types.h | 10 - arch/powerpc/include/asm/pgtable-types.h | 13 +- arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/kernel/exceptions-64e.S | 4 +- arch/powerpc/kernel/head_85xx.S | 33 +- arch/powerpc/kernel/head_8xx.S | 10 +- arch/powerpc/kernel/setup_64.c | 6 +- arch/powerpc/mm/book3s64/hash_utils.c | 11 +- arch/powerpc/mm/book3s64/hugetlbpage.c | 10 + arch/powerpc/mm/book3s64/pgtable.c | 12 - arch/powerpc/mm/hugetlbpage.c | 455 +--------------- arch/powerpc/mm/init-common.c | 8 +- arch/powerpc/mm/kasan/8xx.c | 21 +- arch/powerpc/mm/nohash/8xx.c | 43 +- arch/powerpc/mm/nohash/Makefile | 2 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 4 +- arch/powerpc/mm/nohash/tlb.c | 407 +-------------- arch/powerpc/mm/nohash/tlb_64e.c | 314 ++++++++++++ arch/powerpc/mm/nohash/tlb_low_64e.S | 484 ++---------------- arch/powerpc/mm/pgtable.c | 92 ++-- arch/powerpc/mm/pgtable_32.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +- arch/s390/mm/hugetlbpage.c | 4 +- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 8 +- fs/userfaultfd.c | 2 +- include/asm-generic/hugetlb.h | 2 +- include/linux/hugetlb.h | 6 - include/linux/pgtable.h | 3 + include/linux/swapops.h | 2 +- kernel/events/core.c | 2 +- mm/Kconfig | 10 - mm/damon/vaddr.c | 6 +- mm/gup.c | 105 +--- mm/hmm.c | 2 +- mm/hugetlb.c | 46 +- mm/memory-failure.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 4 +- mm/mincore.c | 2 +- mm/pagewalk.c | 57 +-- mm/userfaultfd.c | 2 +- 68 files changed, 754 insertions(+), 1926 deletions(-) create mode 100644 arch/powerpc/mm/nohash/tlb_64e.c
Signature · 1 line
2.44.0
RFC PATCH v3 1/16 powerpc/64e: Remove unused IBM HTW code [SQUASHED]
From: Michael Ellerman <[email protected]>
This is a squash of series from Michael https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/
The nohash HTW_IBM (Hardware Table Walk) code is unused since support for A2 was removed in commit fb5a515704d7 ("powerpc: Remove platforms/ wsp and associated pieces") (2014).
The remaining supported CPUs use either no HTW (data_tlb_miss_bolted), or the e6500 HTW (data_tlb_miss_e6500).
powerpc/64e: Split out nohash Book3E 64-bit code
A reasonable chunk of nohash/tlb.c is 64-bit only code, split it out into a separate file.
powerpc/64e: Drop E500 ifdefs in 64-bit code
All 64-bit Book3E have E500=y, so drop the unneeded ifdefs.
powerpc/64e: Drop MMU_FTR_TYPE_FSL_E checks in 64-bit code
All 64-bit Book3E have MMU_FTR_TYPE_FSL_E, since A2 was removed, so remove checks for it in 64-bit only code.
powerpc/64e: Consolidate TLB miss handler patching
The 64e TLB miss handler patching is done in setup_mmu_htw(), and then again immediately afterward in early_init_mmu_global(). Consolidate it into a single location.
powerpc/64e: Drop unused TLB miss handlers
There are two possibilities for book3e_htw_mode, PPC_HTW_E6500 or PPC_HTW_NONE.
The TLB miss handlers are patched to use, respectively:
- exc_[data|indstruction]_tlb_miss_e6500_book3e
- exc_[data|indstruction]_tlb_miss_bolted_book3e
Which means the default handlers are never used. Remove those, and use the bolted handlers (PPC_HTW_NONE) by default.
--- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 +- arch/powerpc/kernel/exceptions-64e.S | 4 +- arch/powerpc/kernel/setup_64.c | 6 +- arch/powerpc/mm/nohash/Makefile | 2 +- arch/powerpc/mm/nohash/tlb.c | 398 +------------------ arch/powerpc/mm/nohash/tlb_64e.c | 314 +++++++++++++++ arch/powerpc/mm/nohash/tlb_low_64e.S | 421 --------------------- 7 files changed, 320 insertions(+), 828 deletions(-) create mode 100644 arch/powerpc/mm/nohash/tlb_64e.c
arch/powerpc/include/asm/nohash/mmu-e500.h +1 -2
@@ -303,8 +303,7 @@ extern unsigned long linear_map_top;extern int book3e_htw_mode;#define PPC_HTW_NONE 0 -#define PPC_HTW_IBM 1 -#define PPC_HTW_E6500 2 +#define PPC_HTW_E6500 1/* * 64-bit booke platforms don't load the tlb in the tlb miss handler code.
arch/powerpc/kernel/exceptions-64e.S +2 -2
@@ -485,8 +485,8 @@ interrupt_base_book3e: /* fake trap */ EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */ EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */ EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */ - EXCEPTION_STUB(0x1c0, data_tlb_miss) - EXCEPTION_STUB(0x1e0, instruction_tlb_miss) + EXCEPTION_STUB(0x1c0, data_tlb_miss_bolted) + EXCEPTION_STUB(0x1e0, instruction_tlb_miss_bolted) EXCEPTION_STUB(0x200, altivec_unavailable) EXCEPTION_STUB(0x220, altivec_assist) EXCEPTION_STUB(0x260, perfmon)
arch/powerpc/kernel/setup_64.c +1 -5
@@ -696,11 +696,7 @@ __init u64 ppc64_bolted_size(void) {#ifdef CONFIG_PPC_BOOK3E_64 /* Freescale BookE bolts the entire linear mapping */ -/* XXX: BookE ppc64_rma_limit setup seems to disagree? */ -if (early_mmu_has_feature (MMU_FTR_TYPE_FSL_E )) -return linear_map_top; -/* Other BookE, we assume the first GB is bolted */ -return 1ul <<30 ; +return linear_map_top;#else /* BookS radix, does not take faults on linear mapping */ if (early_radix_enabled ())
arch/powerpc/mm/nohash/Makefile +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-y += mmu_context.o tlb.o tlb_low.o kup.o -obj-$(CONFIG_PPC_BOOK3E_64) += tlb_low_64e.o book3e_pgtable.o +obj-$(CONFIG_PPC_BOOK3E_64) += tlb_64e.o tlb_low_64e.o book3e_pgtable.o obj-$(CONFIG_40x) += 40x.o obj-$(CONFIG_44x) += 44x.o obj-$(CONFIG_PPC_8xx) += 8xx.o
arch/powerpc/mm/nohash/tlb.c +1 -397
410 lines not on this page · raw message
arch/powerpc/mm/nohash/tlb_64e.c new file +314 -0
314 lines not on this page · raw message
arch/powerpc/mm/nohash/tlb_low_64e.S +0 -421
433 lines not on this page · raw message
Signature · 1 line
2.44.0
RFC PATCH v3 2/16 mm: Define __pte_leaf_size() to also take a PMD entry
In reply to Christophe LeroyOn powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide __pte_leaf_size() instead of pte_leaf_size() and provide the PMD entry to that function.
When __pte_leaf_size() is not defined, define it as a pte_leaf_size() so that architectures not interested in the PMD arguments are not impacted.
Only define a default pte_leaf_size() when __pte_leaf_size() is not defined to make sure nobody adds new calls to pte_leaf_size() in the core.
---
v3: Don't change pte_leaf_size() to not impact other architectures
--- include/linux/pgtable.h | 3 +++ kernel/events/core.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/pgtable.h +3 -0
@@ -1801,9 +1801,12 @@ typedef unsigned int pgtbl_mod_mask;#ifndef pmd_leaf_size#define pmd_leaf_size (x) PMD_SIZE#endif +#ifndef __pte_leaf_size#ifndef pte_leaf_size#define pte_leaf_size (x) PAGE_SIZE#endif +#define __pte_leaf_size (x,y) pte_leaf_size(y) +#endif /* * Some architectures have MMUs that are configurable or selectable at boot
kernel/events/core.c +1 -1
@@ -7585,7 +7585,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) pte =ptep_get_lockless (ptep);if (pte_present (pte)) - size =pte_leaf_size (pte); + size =__pte_leaf_size (pmd, pte);pte_unmap (ptep);#endif /* CONFIG_HAVE_FAST_GUP */
Signature · 1 line
2.44.0
Re: [RFC PATCH v3 02/16] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 8 lines · Christophe Leroy
On powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide
__pte_leaf_size()instead ofpte_leaf_size()and provide the PMD entry to that function.When
__pte_leaf_size()is not defined, define it as apte_leaf_size()so that architectures not interested in the PMD arguments are not impacted.
Quoted 5 lines
Only define a default
pte_leaf_size()when__pte_leaf_size()is not defined to make sure nobody adds new calls topte_leaf_size()in the core.Signed-off-by Christophe Leroy <[email protected]>
thanks, this looks much cleaner.
Signature · 2 lines
Oscar Salvador SUSE Labs
RFC PATCH v3 3/16 mm: Provide mm_struct and address to huge_ptep_get()
In reply to Christophe LeroyOn powerpc 8xx huge_ptep_get() will need to know whether the given ptep is a PTE entry or a PMD entry. This cannot be known with the PMD entry itself because there is no easy way to know it from the content of the entry.
So huge_ptep_get() will need to know either the size of the page or get the pmd.
In order to be consistent with huge_ptep_get_and_clear(), give mm and address to huge_ptep_get().
---
v2: Add missing changes in arch implementations
v3: Fixed a comment in ARM and missing changes in S390
--- arch/arm/include/asm/hugetlb-3level.h | 4 +-- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +-- arch/s390/mm/hugetlbpage.c | 4 +-- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 8 ++--- fs/userfaultfd.c | 2 +- include/asm-generic/hugetlb.h | 2 +- include/linux/swapops.h | 2 +- mm/damon/vaddr.c | 6 ++-- mm/gup.c | 2 +- mm/hmm.c | 2 +- mm/hugetlb.c | 46 +++++++++++++-------------- mm/memory-failure.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 4 +-- mm/mincore.c | 2 +- mm/userfaultfd.c | 2 +- 21 files changed, 52 insertions(+), 52 deletions(-)
arch/arm/include/asm/hugetlb-3level.h +2 -2
@@ -13,12 +13,12 @@/* * If our huge pte is non-zero then mark the valid bit. -* This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero +* This allows pte_present(huge_ptep_get(mm,addr,ptep)) to return true for non-zero * ptes. * (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes). */ #define __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (pte_t *ptep) +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {pte_t retval = *ptep;if (pte_val (retval))
arch/arm64/include/asm/hugetlb.h +1 -1
@@ -46,7 +46,7 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,extern void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz);#define __HAVE_ARCH_HUGE_PTEP_GET -extern pte_t huge_ptep_get (pte_t *ptep); +extern pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);void __init arm64_hugetlb_cma_reserve (void );
arch/arm64/mm/hugetlbpage.c +1 -1
@@ -141,7 +141,7 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize)return contig_ptes; } -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {int ncontig, i;size_t pgsize;
arch/riscv/include/asm/hugetlb.h +1 -1
@@ -44,7 +44,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t pte, int dirty);#define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get (pte_t *ptep); +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags);#define arch_make_huge_pte arch_make_huge_pte
arch/riscv/mm/hugetlbpage.c +1 -1
@@ -3,7 +3,7 @@#include <linux/err.h> #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {unsigned long pte_num;int i;
arch/s390/include/asm/hugetlb.h +2 -2
@@ -19,7 +19,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz);void __set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte); -pte_t huge_ptep_get (pte_t *ptep); +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep);
@@ -64,7 +64,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int changed = !pte_same(huge_ptep_get(ptep), pte); + int changed = !pte_same(huge_ptep_get(vma->vm_mm, addr, ptep), pte);if (changed) {huge_ptep_get_and_clear (vma->vm_mm, addr, ptep);__set_huge_pte_at (vma->vm_mm, addr, ptep, pte);
arch/s390/mm/hugetlbpage.c +2 -2
@@ -169,7 +169,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,__set_huge_pte_at (mm, addr, ptep, pte); } -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {return __rste_to_pte (pte_val (*ptep)); }
@@ -177,7 +177,7 @@ pte_t huge_ptep_get(pte_t *ptep)pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { -pte_t pte =huge_ptep_get (ptep); +pte_t pte =huge_ptep_get (mm, addr, ptep);pmd_t *pmdp = (pmd_t *) ptep;pud_t *pudp = (pud_t *) ptep;
fs/hugetlbfs/inode.c +1 -1
@@ -425,7 +425,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma,if (!ptep)return false; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, addr, ptep);if (huge_pte_none (pte) || !pte_present (pte))return false;
fs/proc/task_mmu.c +4 -4
@@ -1572,7 +1572,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,if (vma->vm_flags &VM_SOFTDIRTY ) flags |=PM_SOFT_DIRTY ; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, addr, ptep);if (pte_present (pte)) {struct page *page =pte_page (pte);
@@ -2260,7 +2260,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,if (~p->arg.flags &PM_SCAN_WP_MATCHING ) {/* Go the short route when not write-protecting pages. */ - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, start, ptep); categories = p->cur_vma_category |pagemap_hugetlb_category (pte);if (!pagemap_scan_is_interesting_page (categories, p))
@@ -2272,7 +2272,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,i_mmap_lock_write (vma->vm_file->f_mapping); ptl =huge_pte_lock (hstate_vma (vma), vma->vm_mm, ptep); - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, start, ptep); categories = p->cur_vma_category |pagemap_hugetlb_category (pte);if (!pagemap_scan_is_interesting_page (categories, p))
@@ -2667,7 +2667,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr,static int gather_hugetlb_stats (pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) { -pte_t huge_pte =huge_ptep_get (pte); +pte_t huge_pte =huge_ptep_get (walk->mm, addr, pte);struct numa_maps *md;struct page *page;
fs/userfaultfd.c +1 -1
@@ -256,7 +256,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, goto out; ret = false; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, vmf->address, ptep);/* * Lockless access: we're in a wait_event so it's ok if it
include/asm-generic/hugetlb.h +1 -1
@@ -144,7 +144,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (pte_t *ptep) +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {return ptep_get (ptep); }
include/linux/swapops.h +1 -1
@@ -334,7 +334,7 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry)extern void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address); -extern void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte); +extern void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *pte);#else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry (pgoff_t offset) {
mm/damon/vaddr.c +3 -3
@@ -339,7 +339,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr) { bool referenced = false; -pte_t entry =huge_ptep_get (pte); +pte_t entry =huge_ptep_get (mm, addr, pte);struct folio *folio =pfn_folio (pte_pfn (entry));unsigned long psize =huge_page_size (hstate_vma (vma));
@@ -373,7 +373,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (h, walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) goto out;
@@ -509,7 +509,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (h, walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) goto out;
mm/gup.c +1 -1
@@ -2812,7 +2812,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,if (pte_end < end) end = pte_end; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (NULL , addr, ptep);if (!pte_access_permitted (pte, flags &FOLL_WRITE ))return 0 ;
mm/hmm.c +1 -1
@@ -485,7 +485,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (hstate_vma (vma), walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte); i = (start - range->start) >>PAGE_SHIFT ; pfn_req_flags = range->hmm_pfns[i];
mm/hugetlb.c +23 -23
181 lines not on this page · raw message
mm/memory-failure.c +1 -1
@@ -832,7 +832,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask, struct mm_walk *walk) {struct hwpoison_walk *hwp = walk->private; -pte_t pte =huge_ptep_get (ptep); +pte_t pte =huge_ptep_get (walk->mm, addr, ptep);struct hstate *h =hstate_vma (walk->vma); return check_hwpoisoned_entry(pte, addr, huge_page_shift(h),
mm/mempolicy.c +1 -1
@@ -624,7 +624,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (hstate_vma (walk->vma), walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) {if (unlikely (is_hugetlb_entry_migration (entry))) qp->nr_failed++;
mm/migrate.c +2 -2
@@ -338,14 +338,14 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,* * This function will release the vma lock before returning. */ -void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *ptep) +void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) {spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, ptep);pte_t pte;hugetlb_vma_assert_locked (vma);spin_lock (ptl); - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, addr, ptep);if (unlikely (!is_hugetlb_entry_migration (pte))) {spin_unlock (ptl);
mm/mincore.c +1 -1
@@ -33,7 +33,7 @@ static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,* Hugepages under user process are always in RAM and never * swapped out, but theoretically it needs to be checked. */ - present = pte && !huge_pte_none_mostly (huge_ptep_get (pte)); + present = pte && !huge_pte_none_mostly (huge_ptep_get (walk->mm, addr, pte));for (; addr != end; vec++, addr +=PAGE_SIZE ) *vec = present; walk->private = vec;
mm/userfaultfd.c +1 -1
@@ -555,7 +555,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( }if (!uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && - !huge_pte_none_mostly (huge_ptep_get (dst_pte))) { + !huge_pte_none_mostly (huge_ptep_get (dst_mm, dst_addr, dst_pte))) { err = -EEXIST ;hugetlb_vma_unlock_read (dst_vma);mutex_unlock (&hugetlb_fault_mutex_table[hash]);
Signature · 1 line
2.44.0
kernel test robot
build botRe: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get()
kernel test robot noticed the following build errors:
Hi Christophe,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes arm64/for-next/core v6.9] [cannot apply to akpm-mm/mm-everything s390/features linus/master next-20240523] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Christophe-Leroy/powerpc-64e-Remove-unused-IBM-HTW-code-SQUASHED/20240526-172511 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/fbba60d762faad40ebb959bf9517c5c22301f69e.1716714720.git.christophe.leroy%40csgroup.eu patch subject: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get() config: powerpc-allmodconfig (https://download.01.org/0day-ci/archive/20240526/[email protected]/config) compiler: powerpc64-linux-gcc (GCC) 13.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240526/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
mm/hugetlb.c: In function 'hugetlb_mfill_atomic_pte':
>> mm/hugetlb.c:6692:50: error: 'mm' undeclared (first use in this function); did you mean 'tm'?
6692 | if (!huge_pte_none(huge_ptep_get(mm, dst_addr, dst_pte))) {
| ^~
| tm mm/hugetlb.c:6692:50: note: each undeclared identifier is reported only once for each function it appears in
vim +6692 mm/hugetlb.c
6661
6662 /*
6663 * Used by userfaultfd UFFDIO_* ioctls. Based on userfaultfd's mfill_atomic_pte
6664 * with modifications for hugetlb pages.
6665 */
6666 int hugetlb_mfill_atomic_pte(pte_t *dst_pte,
6667 struct vm_area_struct *dst_vma,
6668 unsigned long dst_addr,
6669 unsigned long src_addr,
6670 uffd_flags_t flags,
6671 struct folio **foliop)
6672 {
6673 struct mm_struct *dst_mm = dst_vma->vm_mm;
6674 bool is_continue = uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE);
6675 bool wp_enabled = (flags & MFILL_ATOMIC_WP);
6676 struct hstate *h = hstate_vma(dst_vma);
6677 struct address_space *mapping = dst_vma->vm_file->f_mapping;
6678 pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr);
6679 unsigned long size;
6680 int vm_shared = dst_vma->vm_flags & VM_SHARED;
6681 pte_t _dst_pte;
6682 spinlock_t *ptl;
6683 int ret = -ENOMEM;
6684 struct folio *folio;
6685 int writable;
6686 bool folio_in_pagecache = false;
6687
6688 if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) {
6689 ptl = huge_pte_lock(h, dst_mm, dst_pte);
6690
6691 /* Don't overwrite any existing PTEs (even markers) */
> 6692 if (!huge_pte_none(huge_ptep_get(mm, dst_addr, dst_pte))) { 6693 spin_unlock(ptl);
6694 return -EEXIST;
6695 }
6696
6697 _dst_pte = make_pte_marker(PTE_MARKER_POISONED);
6698 set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte,
6699 huge_page_size(h));
6700
6701 /* No need to invalidate - it was non-present before */
6702 update_mmu_cache(dst_vma, dst_addr, dst_pte);
6703
6704 spin_unlock(ptl);
6705 return 0;
6706 }
6707
6708 if (is_continue) {
6709 ret = -EFAULT;
6710 folio = filemap_lock_hugetlb_folio(h, mapping, idx);
6711 if (IS_ERR(folio))
6712 goto out;
6713 folio_in_pagecache = true;
6714 } else if (!*foliop) {
6715 /* If a folio already exists, then it's UFFDIO_COPY for
6716 * a non-missing case. Return -EEXIST.
6717 */
6718 if (vm_shared &&
6719 hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) {
6720 ret = -EEXIST;
6721 goto out;
6722 }
6723
6724 folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0);
6725 if (IS_ERR(folio)) {
6726 ret = -ENOMEM;
6727 goto out;
6728 }
6729
6730 ret = copy_folio_from_user(folio, (const void __user *) src_addr,
6731 false);
6732
6733 /* fallback to copy_from_user outside mmap_lock */
6734 if (unlikely(ret)) {
6735 ret = -ENOENT;
6736 /* Free the allocated folio which may have
6737 * consumed a reservation.
6738 */
6739 restore_reserve_on_error(h, dst_vma, dst_addr, folio);
6740 folio_put(folio);
6741
6742 /* Allocate a temporary folio to hold the copied
6743 * contents.
6744 */
6745 folio = alloc_hugetlb_folio_vma(h, dst_vma, dst_addr);
6746 if (!folio) {
6747 ret = -ENOMEM;
6748 goto out;
6749 }
6750 *foliop = folio;
6751 /* Set the outparam foliop and return to the caller to
6752 * copy the contents outside the lock. Don't free the
6753 * folio.
6754 */
6755 goto out;
6756 }
6757 } else {
6758 if (vm_shared &&
6759 hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) {
6760 folio_put(*foliop);
6761 ret = -EEXIST;
6762 *foliop = NULL;
6763 goto out;
6764 }
6765
6766 folio = alloc_hugetlb_folio(dst_vma, dst_addr, 0);
6767 if (IS_ERR(folio)) {
6768 folio_put(*foliop);
6769 ret = -ENOMEM;
6770 *foliop = NULL;
6771 goto out;
6772 }
6773 ret = copy_user_large_folio(folio, *foliop, dst_addr, dst_vma);
6774 folio_put(*foliop);
6775 *foliop = NULL;
6776 if (ret) {
6777 folio_put(folio);
6778 goto out;
6779 }
6780 }
6781
6782 /*
6783 * If we just allocated a new page, we need a memory barrier to ensure
6784 * that preceding stores to the page become visible before the
6785 * set_pte_at() write. The memory barrier inside __folio_mark_uptodate
6786 * is what we need.
6787 *
6788 * In the case where we have not allocated a new page (is_continue),
6789 * the page must already be uptodate. UFFDIO_CONTINUE already includes
6790 * an earlier smp_wmb() to ensure that prior stores will be visible
6791 * before the set_pte_at() write.
6792 */
6793 if (!is_continue)
6794 __folio_mark_uptodate(folio);
6795 else
6796 WARN_ON_ONCE(!folio_test_uptodate(folio));
6797
6798 /* Add shared, newly allocated pages to the page cache. */
6799 if (vm_shared && !is_continue) {
6800 size = i_size_read(mapping->host) >> huge_page_shift(h);
6801 ret = -EFAULT;
6802 if (idx >= size)
6803 goto out_release_nounlock;
6804
6805 /*
6806 * Serialization between remove_inode_hugepages() and
6807 * hugetlb_add_to_page_cache() below happens through the
6808 * hugetlb_fault_mutex_table that here must be hold by
6809 * the caller.
6810 */
6811 ret = hugetlb_add_to_page_cache(folio, mapping, idx);
6812 if (ret)
6813 goto out_release_nounlock;
6814 folio_in_pagecache = true;
6815 }
6816
6817 ptl = huge_pte_lock(h, dst_mm, dst_pte);
6818
6819 ret = -EIO;
6820 if (folio_test_hwpoison(folio))
6821 goto out_release_unlock;
6822
6823 /*
6824 * We allow to overwrite a pte marker: consider when both MISSING|WP
6825 * registered, we firstly wr-protect a none pte which has no page cache
6826 * page backing it, then access the page.
6827 */
6828 ret = -EEXIST;
6829 if (!huge_pte_none_mostly(huge_ptep_get(mm, dst_addr, dst_pte)))
6830 goto out_release_unlock;
6831
6832 if (folio_in_pagecache)
6833 hugetlb_add_file_rmap(folio);
6834 else
6835 hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr);
6836
6837 /*
6838 * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY
6839 * with wp flag set, don't set pte write bit.
6840 */
6841 if (wp_enabled || (is_continue && !vm_shared))
6842 writable = 0;
6843 else
6844 writable = dst_vma->vm_flags & VM_WRITE;
6845
6846 _dst_pte = make_huge_pte(dst_vma, &folio->page, writable);
6847 /*
6848 * Always mark UFFDIO_COPY page dirty; note that this may not be
6849 * extremely important for hugetlbfs for now since swapping is not
6850 * supported, but we should still be clear in that this page cannot be
6851 * thrown away at will, even if write bit not set.
6852 */
6853 _dst_pte = huge_pte_mkdirty(_dst_pte);
6854 _dst_pte = pte_mkyoung(_dst_pte);
6855
6856 if (wp_enabled)
6857 _dst_pte = huge_pte_mkuffd_wp(_dst_pte);
6858
6859 set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte, huge_page_size(h));
6860
6861 hugetlb_count_add(pages_per_huge_page(h), dst_mm);
6862
6863 /* No need to invalidate - it was non-present before */
6864 update_mmu_cache(dst_vma, dst_addr, dst_pte);
6865
6866 spin_unlock(ptl);
6867 if (!is_continue)
6868 folio_set_hugetlb_migratable(folio);
6869 if (vm_shared || is_continue)
6870 folio_unlock(folio);
6871 ret = 0;
6872 out:
6873 return ret;
6874 out_release_unlock:
6875 spin_unlock(ptl);
6876 if (vm_shared || is_continue)
6877 folio_unlock(folio);
6878 out_release_nounlock:
6879 if (!folio_in_pagecache)
6880 restore_reserve_on_error(h, dst_vma, dst_addr, folio);
6881 folio_put(folio);
6882 goto out;
6883 }
6884 #endif /* CONFIG_USERFAULTFD */
6885
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get()
In reply to Christophe Leroykernel test robot noticed the following build errors:
Hi Christophe,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes arm64/for-next/core v6.9] [cannot apply to akpm-mm/mm-everything s390/features linus/master next-20240523] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Christophe-Leroy/powerpc-64e-Remove-unused-IBM-HTW-code-SQUASHED/20240526-172511 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/fbba60d762faad40ebb959bf9517c5c22301f69e.1716714720.git.christophe.leroy%40csgroup.eu patch subject: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get() config: i386-randconfig-003-20240526 (https://download.01.org/0day-ci/archive/20240526/[email protected]/config) compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240526/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/hugetlb.c:6532:42: error: too many arguments to function call, expected 2, have 3
6532 | migration_entry_wait_huge(vma, haddr, ptep);
| ~~~~~~~~~~~~~~~~~~~~~~~~~ ^~~~
include/linux/swapops.h:361:20: note: 'migration_entry_wait_huge' declared here
361 | static inline void migration_entry_wait_huge(struct vm_area_struct *vma,
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
362 | pte_t *pte) { }
| ~~~~~~~~~~
mm/hugetlb.c:6692:36: error: use of undeclared identifier 'mm'
6692 | if (!huge_pte_none(huge_ptep_get(mm, dst_addr, dst_pte))) {
| ^
mm/hugetlb.c:6829:42: error: use of undeclared identifier 'mm'
6829 | if (!huge_pte_none_mostly(huge_ptep_get(mm, dst_addr, dst_pte)))
| ^3 errors generated.
vim +6532 mm/hugetlb.c
6441
6442 vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
6443 unsigned long address, unsigned int flags)
6444 {
6445 pte_t *ptep, entry;
6446 spinlock_t *ptl;
6447 vm_fault_t ret;
6448 u32 hash;
6449 struct folio *folio = NULL;
6450 struct folio *pagecache_folio = NULL;
6451 struct hstate *h = hstate_vma(vma);
6452 struct address_space *mapping;
6453 int need_wait_lock = 0;
6454 unsigned long haddr = address & huge_page_mask(h);
6455 struct vm_fault vmf = {
6456 .vma = vma,
6457 .address = haddr,
6458 .real_address = address,
6459 .flags = flags,
6460 .pgoff = vma_hugecache_offset(h, vma, haddr),
6461 /* TODO: Track hugetlb faults using vm_fault */
6462
6463 /*
6464 * Some fields may not be initialized, be careful as it may
6465 * be hard to debug if called functions make assumptions
6466 */
6467 };
6468
6469 /*
6470 * Serialize hugepage allocation and instantiation, so that we don't
6471 * get spurious allocation failures if two CPUs race to instantiate
6472 * the same page in the page cache.
6473 */
6474 mapping = vma->vm_file->f_mapping;
6475 hash = hugetlb_fault_mutex_hash(mapping, vmf.pgoff);
6476 mutex_lock(&hugetlb_fault_mutex_table[hash]);
6477
6478 /*
6479 * Acquire vma lock before calling huge_pte_alloc and hold
6480 * until finished with ptep. This prevents huge_pmd_unshare from
6481 * being called elsewhere and making the ptep no longer valid.
6482 */
6483 hugetlb_vma_lock_read(vma);
6484 ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h));
6485 if (!ptep) {
6486 hugetlb_vma_unlock_read(vma);
6487 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6488 return VM_FAULT_OOM;
6489 }
6490
6491 entry = huge_ptep_get(mm, address, ptep);
6492 if (huge_pte_none_mostly(entry)) {
6493 if (is_pte_marker(entry)) {
6494 pte_marker marker =
6495 pte_marker_get(pte_to_swp_entry(entry));
6496
6497 if (marker & PTE_MARKER_POISONED) {
6498 ret = VM_FAULT_HWPOISON_LARGE;
6499 goto out_mutex;
6500 }
6501 }
6502
6503 /*
6504 * Other PTE markers should be handled the same way as none PTE.
6505 *
6506 * hugetlb_no_page will drop vma lock and hugetlb fault
6507 * mutex internally, which make us return immediately.
6508 */
6509 return hugetlb_no_page(mm, vma, mapping, vmf.pgoff, address,
6510 ptep, entry, flags, &vmf);
6511 }
6512
6513 ret = 0;
6514
6515 /*
6516 * entry could be a migration/hwpoison entry at this point, so this
6517 * check prevents the kernel from going below assuming that we have
6518 * an active hugepage in pagecache. This goto expects the 2nd page
6519 * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will
6520 * properly handle it.
6521 */
6522 if (!pte_present(entry)) {
6523 if (unlikely(is_hugetlb_entry_migration(entry))) {
6524 /*
6525 * Release the hugetlb fault lock now, but retain
6526 * the vma lock, because it is needed to guard the
6527 * huge_pte_lockptr() later in
6528 * migration_entry_wait_huge(). The vma lock will
6529 * be released there.
6530 */
6531 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
> 6532 migration_entry_wait_huge(vma, haddr, ptep);
6533 return 0;
6534 } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
6535 ret = VM_FAULT_HWPOISON_LARGE |
6536 VM_FAULT_SET_HINDEX(hstate_index(h));
6537 goto out_mutex;
6538 }
6539
6540 /*
6541 * If we are going to COW/unshare the mapping later, we examine the
6542 * pending reservations for this page now. This will ensure that any
6543 * allocations necessary to record that reservation occur outside the
6544 * spinlock. Also lookup the pagecache page now as it is used to
6545 * determine if a reservation has been consumed.
6546 */
6547 if ((flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) &&
6548 !(vma->vm_flags & VM_MAYSHARE) && !huge_pte_write(entry)) {
6549 if (vma_needs_reservation(h, vma, haddr) < 0) {
6550 ret = VM_FAULT_OOM;
6551 goto out_mutex;
6552 }
6553 /* Just decrements count, does not deallocate */
6554 vma_end_reservation(h, vma, haddr);
6555
6556 pagecache_folio = filemap_lock_hugetlb_folio(h, mapping,
6557 vmf.pgoff);
6558 if (IS_ERR(pagecache_folio))
6559 pagecache_folio = NULL;
6560 }
6561
6562 ptl = huge_pte_lock(h, mm, ptep);
6563
6564 /* Check for a racing update before calling hugetlb_wp() */
6565 if (unlikely(!pte_same(entry, huge_ptep_get(mm, address, ptep))))
6566 goto out_ptl;
6567
6568 /* Handle userfault-wp first, before trying to lock more pages */
6569 if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(mm, address, ptep)) &&
6570 (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) {
6571 if (!userfaultfd_wp_async(vma)) {
6572 spin_unlock(ptl);
6573 if (pagecache_folio) {
6574 folio_unlock(pagecache_folio);
6575 folio_put(pagecache_folio);
6576 }
6577 hugetlb_vma_unlock_read(vma);
6578 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6579 return handle_userfault(&vmf, VM_UFFD_WP);
6580 }
6581
6582 entry = huge_pte_clear_uffd_wp(entry);
6583 set_huge_pte_at(mm, haddr, ptep, entry,
6584 huge_page_size(hstate_vma(vma)));
6585 /* Fallthrough to CoW */
6586 }
6587
6588 /*
6589 * hugetlb_wp() requires page locks of pte_page(entry) and
6590 * pagecache_folio, so here we need take the former one
6591 * when folio != pagecache_folio or !pagecache_folio.
6592 */
6593 folio = page_folio(pte_page(entry));
6594 if (folio != pagecache_folio)
6595 if (!folio_trylock(folio)) {
6596 need_wait_lock = 1;
6597 goto out_ptl;
6598 }
6599
6600 folio_get(folio);
6601
6602 if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) {
6603 if (!huge_pte_write(entry)) {
6604 ret = hugetlb_wp(mm, vma, address, ptep, flags,
6605 pagecache_folio, ptl, &vmf);
6606 goto out_put_page;
6607 } else if (likely(flags & FAULT_FLAG_WRITE)) {
6608 entry = huge_pte_mkdirty(entry);
6609 }
6610 }
6611 entry = pte_mkyoung(entry);
6612 if (huge_ptep_set_access_flags(vma, haddr, ptep, entry,
6613 flags & FAULT_FLAG_WRITE))
6614 update_mmu_cache(vma, haddr, ptep);
6615 out_put_page:
6616 if (folio != pagecache_folio)
6617 folio_unlock(folio);
6618 folio_put(folio);
6619 out_ptl:
6620 spin_unlock(ptl);
6621
6622 if (pagecache_folio) {
6623 folio_unlock(pagecache_folio);
6624 folio_put(pagecache_folio);
6625 }
6626 out_mutex:
6627 hugetlb_vma_unlock_read(vma);
6628 mutex_unlock(&hugetlb_fault_mutex_table[hash]);
6629 /*
6630 * Generally it's safe to hold refcount during waiting page lock. But
6631 * here we just wait to defer the next page fault to avoid busy loop and
6632 * the page is not used after unlocked before returning from the current
6633 * page fault. So we are safe from accessing freed page, even if we wait
6634 * here without taking refcount.
6635 */
6636 if (need_wait_lock)
6637 folio_wait_locked(folio);
6638 return ret;
6639 }
6640
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Oscar Salvador
maintainerRe: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get()
In reply to Christophe LeroyQuoted 15 lines · Christophe Leroy
On powerpc 8xx
huge_ptep_get()will need to know whether the given ptep is a PTE entry or a PMD entry. This cannot be known with the PMD entry itself because there is no easy way to know it from the content of the entry.So
huge_ptep_get()will need to know either the size of the page or get the pmd.In order to be consistent with
huge_ptep_get_and_clear(), give mm and address tohuge_ptep_get().Signed-off-by Christophe Leroy <[email protected]>---
v2: Add missing changes in arch implementations
v3: Fixed a comment in ARM and missing changes in S390
Quoted 8 lines
--- arch/arm/include/asm/hugetlb-3level.h | 4 +-- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +-- arch/s390/mm/hugetlbpage.c | 4 +--
I was wondering whether we could do something similar for what we did in patch#1, so we do not touch architectures code.
mm/gup.c static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, · line 2815
Quoted 10 lines
mm/gup.c +1
-1@@ -2812,7 +2812,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,if (pte_end < end) end = pte_end; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (NULL , addr, ptep);
I know that after this series all this code is gone, but I was not sure about the behaviour between this patch and the last one.
It made me nervous, until I realized that this code is only used on CONFIG_ARCH_HAS_HUGEPD, which should not be the case anymore for 8xx after patch#8, and since 8xx is the only one that will use the mm parameter from huge_ptep_get, we are all good.
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get()
Le 27/05/2024 à 13:19, Oscar Salvador a écrit :
Quoted 25 lines · Christophe Leroy
Quoted 15 lines
On powerpc 8xx
huge_ptep_get()will need to know whether the given ptep is a PTE entry or a PMD entry. This cannot be known with the PMD entry itself because there is no easy way to know it from the content of the entry.So
huge_ptep_get()will need to know either the size of the page or get the pmd.In order to be consistent with
huge_ptep_get_and_clear(), give mm and address tohuge_ptep_get().Signed-off-by Christophe Leroy <[email protected]>---
v2: Add missing changes in arch implementations
v3: Fixed a comment in ARM and missing changes in S390Quoted 8 lines
--- arch/arm/include/asm/hugetlb-3level.h | 4 +-- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +-- arch/s390/mm/hugetlbpage.c | 4 +--
Quoted 2 lines
I was wondering whether we could do something similar for what we did in patch#1, so we do not touch architectures code.
We could be is that worth the churn ?
With patch 1 there was only one callsite.
Here we have many callsites, and we also have huge_ptep_get_and_clear() which already takes three arguments. So for me it make more sense to adapt huge_ptep_get() here.
Today several of the huge-related functions already have parameters that are used only by a few architectures and everytime one architecture needs a new parameter it is added for all of them, and there are exemples in the past of new functions added to get new parameters for only a few architectures that ended up with a mess and a need to re-factor at the end.
See for instance the story around arch_make_huge_pte() and pte_mkhuge(), both do the same but arch_make_huge_pte() was added to take additional parameters by commit d9ed9faac283 ("mm: add new arch_make_huge_pte() method for tile support") then they were merged by commit 16785bd77431 ("mm: merge pte_mkhuge() call into arch_make_huge_pte()")
So I'm open to any suggestion but we need to try not make it a bigger mess at the end.
By the way, I think most if not all huge related helpers should all take the same parameters even if not all of them are used, then it would make things easier. And maybe the cleanest would be to give the page size to all those functions instead of having them guess it.
So let's have your ideas here on the most straight forward way to handle that.
Quoted 16 lines
mm/gup.c static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, · line 2815
Quoted 12 lines
mm/gup.c +1
-1@@ -2812,7 +2812,7 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,if (pte_end < end) end = pte_end; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (NULL , addr, ptep);I know that after this series all this code is gone, but I was not sure about the behaviour between this patch and the last one.
Quoted 5 lines
It made me nervous, until I realized that this code is only used on
CONFIG_ARCH_HAS_HUGEPD, which should not be the case anymore for 8xx after patch#8, and since 8xx is the only one that will use the mm parameter fromhuge_ptep_get, we are all good.
By the way, after commit 01d89b93e176 ("mm/gup: fix hugepd handling in hugetlb rework") we now have the vma in gup_hugepte() so we now pass vma->vm_mm
Thanks for the review
Christophe
Oscar Salvador
maintainerRe: [RFC PATCH v3 03/16] mm: Provide mm_struct and address to huge_ptep_get()
Quoted 1 line · Christophe Leroy
We could be is that worth the churn ?
Probably not.
Quoted 1 line
With patch 1 there was only one callsite.
Yes, you are right here.
Quoted 20 lines
Here we have many callsites, and we also have
huge_ptep_get_and_clear()which already takes three arguments. So for me it make more sense to adapthuge_ptep_get()here.Today several of the huge-related functions already have parameters that are used only by a few architectures and everytime one architecture needs a new parameter it is added for all of them, and there are exemples in the past of new functions added to get new parameters for only a few architectures that ended up with a mess and a need to re-factor at the end.
See for instance the story around
arch_make_huge_pte()andpte_mkhuge(), both do the same butarch_make_huge_pte()was added to take additional parameters by commitd9ed9faac283("mm: add newarch_make_huge_pte()method for tile support") then they were merged by commit16785bd77431("mm: mergepte_mkhuge()call intoarch_make_huge_pte()")So I'm open to any suggestion but we need to try not make it a bigger mess at the end.
Quoted 7 lines
By the way, I think most if not all huge related helpers should all take the same parameters even if not all of them are used, then it would make things easier. And maybe the cleanest would be to give the page size to all those functions instead of having them guess it.
So let's have your ideas here on the most straight forward way to handle that.
It is probably not worth pursuing this then.
As you said, there are many callers and we would have to create some kind of hook for only those interested places, which I guess would end up looking just too ugly in order to save little code in arch code.
So please disregard my comment here, and stick with what we have.
Quoted 3 lines
By the way, after commit
01d89b93e176("mm/gup: fix hugepd handling in hugetlb rework") we now have the vma ingup_hugepte()so we now passvma->vm_mm
I did not notice, thanks.
Signature · 2 lines
Oscar Salvador SUSE Labs
RFC PATCH v3 4/16 powerpc/mm: Remove _PAGE_PSIZE
In reply to Christophe Leroy_PAGE_PSIZE macro is never used outside the place it is defined and is used only on 8xx and e500.
Remove indirection, remove it and use its content directly.
--- arch/powerpc/include/asm/nohash/32/pte-40x.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-44x.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-85xx.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 5 ++--- arch/powerpc/include/asm/nohash/pte-e500.h | 4 +--- 5 files changed, 3 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/nohash/32/pte-40x.h +0 -3
@@ -49,9 +49,6 @@#define _PAGE_EXEC 0x200/* hardware: EX permission */ #define _PAGE_ACCESSED 0x400/* software: R: page referenced */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -/* cache related flags non existing on 40x */ #define _PAGE_COHERENT 0
arch/powerpc/include/asm/nohash/32/pte-44x.h +0 -3
@@ -75,9 +75,6 @@#define _PAGE_NO_CACHE 0x00000400/* H: I bit */ #define _PAGE_WRITETHRU 0x00000800/* H: W bit */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -/* TODO: Add large page lowmem mapping support */ #define _PMD_PRESENT 0#define _PMD_PRESENT_MASK (PAGE_MASK)
arch/powerpc/include/asm/nohash/32/pte-85xx.h +0 -3
@@ -31,9 +31,6 @@#define _PAGE_WRITETHRU 0x00400/* H: W bit */ #define _PAGE_SPECIAL 0x00800/* S: Special page */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -#define _PMD_PRESENT 0#define _PMD_PRESENT_MASK (PAGE_MASK)#define _PMD_BAD (~PAGE_MASK)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +2 -3
@@ -74,12 +74,11 @@#define _PTE_NONE_MASK 0#ifdef CONFIG_PPC_16K_PAGES -#define _PAGE_PSIZE _PAGE_SPS +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_SPS)#else -#define _PAGE_PSIZE 0 +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED)#endif -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE)#define _PAGE_BASE (_PAGE_BASE_NC)#include <asm/pgtable-masks.h>
arch/powerpc/include/asm/nohash/pte-e500.h +1 -3
@@ -65,8 +65,6 @@#define _PAGE_SPECIAL _PAGE_SW0 -/* Base page size */ -#define _PAGE_PSIZE _PAGE_PSIZE_4K#define PTE_RPN_SHIFT (24)#define PTE_WIMGE_SHIFT (19)
@@ -89,7 +87,7 @@* pages. We always set _PAGE_COHERENT when SMP is enabled or * the processor might need it for DMA coherency. */ -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE_4K)#if defined(CONFIG_SMP )#define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT)#else
Signature · 1 line
2.44.0
RFC PATCH v3 5/16 powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
In reply to Christophe LeroyBuilding on 32 bits with pmd_leaf() not returning always false leads to the following error:
CC arch/powerpc/mm/pgtable.oarch/powerpc/mm/pgtable.c: In function '__find_linux_pte':arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr]
506 | }
| ^
arch/powerpc/mm/pgtable.c:394:15: note: declared here
394 | pud_t pud, *pudp;
| ^~~arch/powerpc/mm/pgtable.c:394:15: note: declared here
This is due to pmd_offset() being a no-op in that case.
So rework it for powerpc/32 so that pXd_offset() are used on real pointers and not on on-stack copies.
---
v3: Removed p4dp and pudp locals for PPC32 and add a comment.
--- arch/powerpc/mm/pgtable.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/powerpc/mm/pgtable.c +9 -0
@@ -382,8 +382,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, bool *is_thp, unsigned *hpage_shift) { pgd_t *pgdp; +#ifdef CONFIG_PPC64 p4d_t p4d, *p4dp;pud_t pud, *pudp; +#endif pmd_t pmd, *pmdp;pte_t *ret_pte;hugepd_t *hpdp =NULL ;
@@ -401,8 +403,12 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,* page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. * Top level is an exception because it is folded into p4d. +* +* On PPC32, P4D/PUD/PMD are folded into PGD so go straight to +* PMD level. */ pgdp = pgdir +pgd_index (ea); +#ifdef CONFIG_PPC64 p4dp =p4d_offset (pgdp, ea); p4d =READ_ONCE (*p4dp); pdshift =P4D_SHIFT ;
@@ -444,6 +450,9 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, pdshift =PMD_SHIFT ; pmdp =pmd_offset (&pud, ea); +#else + pmdp =pmd_offset (pud_offset (p4d_offset (pgdp, ea), ea), ea); +#endif pmd =READ_ONCE (*pmdp);/*
Signature · 1 line
2.44.0
kernel test robot
build botRe: [RFC PATCH v3 05/16] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
kernel test robot noticed the following build warnings:
Hi Christophe,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on powerpc/next]
[also build test WARNING on powerpc/fixes arm64/for-next/core v6.9] [cannot apply to akpm-mm/mm-everything s390/features linus/master next-20240523] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Christophe-Leroy/powerpc-64e-Remove-unused-IBM-HTW-code-SQUASHED/20240526-172511 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/aa54834e9ee5e40179def32ff5834a8a2a02c413.1716714720.git.christophe.leroy%40csgroup.eu patch subject: [RFC PATCH v3 05/16] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries config: powerpc-randconfig-001-20240526 (https://download.01.org/0day-ci/archive/20240526/[email protected]/config) compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240526/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> arch/powerpc/mm/pgtable.c:503:18: warning: variable 'pdshift' is uninitialized when used here [-Wuninitialized]
503 | *hpage_shift = pdshift;
| ^~~~~~~
arch/powerpc/mm/pgtable.c:392:18: note: initialize the variable 'pdshift' to silence this warning
392 | unsigned pdshift;
| ^
| = 01 warning generated.
vim +/pdshift +503 arch/powerpc/mm/pgtable.c
a00196a27216133 Nicholas Piggin 2019-06-07 475
0caed4de502c769 Christophe Leroy 2019-04-26 476 if (pmd_trans_huge(pmd) || pmd_devmap(pmd)) {
0caed4de502c769 Christophe Leroy 2019-04-26 477 if (is_thp)
0caed4de502c769 Christophe Leroy 2019-04-26 478 *is_thp = true;
0caed4de502c769 Christophe Leroy 2019-04-26 479 ret_pte = (pte_t *)pmdp;
0caed4de502c769 Christophe Leroy 2019-04-26 480 goto out;
0caed4de502c769 Christophe Leroy 2019-04-26 481 }
d6eacedd1f0ebf0 Aneesh Kumar K.V 2019-05-14 482
bd18b688220c722 Peter Xu 2024-03-05 483 if (pmd_leaf(pmd)) {
0caed4de502c769 Christophe Leroy 2019-04-26 484 ret_pte = (pte_t *)pmdp;
0caed4de502c769 Christophe Leroy 2019-04-26 485 goto out;
fab9a1165bcda99 Christophe Leroy 2019-04-26 486 }
d6eacedd1f0ebf0 Aneesh Kumar K.V 2019-05-14 487
fab9a1165bcda99 Christophe Leroy 2019-04-26 488 if (is_hugepd(__hugepd(pmd_val(pmd)))) {
0caed4de502c769 Christophe Leroy 2019-04-26 489 hpdp = (hugepd_t *)&pmd;
fab9a1165bcda99 Christophe Leroy 2019-04-26 490 goto out_huge;
fab9a1165bcda99 Christophe Leroy 2019-04-26 491 }
fab9a1165bcda99 Christophe Leroy 2019-04-26 492
0caed4de502c769 Christophe Leroy 2019-04-26 493 return pte_offset_kernel(&pmd, ea);
26e66b08c3376b6 Christophe Leroy 2019-04-26 494
fab9a1165bcda99 Christophe Leroy 2019-04-26 495 out_huge:
0caed4de502c769 Christophe Leroy 2019-04-26 496 if (!hpdp)
0caed4de502c769 Christophe Leroy 2019-04-26 497 return NULL;
0caed4de502c769 Christophe Leroy 2019-04-26 498
0caed4de502c769 Christophe Leroy 2019-04-26 499 ret_pte = hugepte_offset(*hpdp, ea, pdshift);
0caed4de502c769 Christophe Leroy 2019-04-26 500 pdshift = hugepd_shift(*hpdp);
0caed4de502c769 Christophe Leroy 2019-04-26 501 out:
0caed4de502c769 Christophe Leroy 2019-04-26 502 if (hpage_shift)
0caed4de502c769 Christophe Leroy 2019-04-26 @503 *hpage_shift = pdshift;
0caed4de502c769 Christophe Leroy 2019-04-26 504 return ret_pte;
0caed4de502c769 Christophe Leroy 2019-04-26 505 }
0caed4de502c769 Christophe Leroy 2019-04-26 506 EXPORT_SYMBOL_GPL(__find_linux_pte);6eac1eaf2105dd8 Anshuman Khandual 2022-07-11 507
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH v3 05/16] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
In reply to Christophe LeroyQuoted 13 lines · Christophe Leroy
Building on 32 bits with
pmd_leaf()not returning always false leads to the following error:CC
arch/powerpc/mm/pgtable.oarch/powerpc/mm/pgtable.c: In function '__find_linux_pte':arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr]
506 | }| ^
arch/powerpc/mm/pgtable.c:394:15: note: declared here
394 |pud_tpud, *pudp;
| ^~~arch/powerpc/mm/pgtable.c:394:15: note: declared here
Quoted 6 lines
This is due to
pmd_offset()being a no-op in that case.So rework it for powerpc/32 so that
pXd_offset()are used on real pointers and not on on-stack copies.Signed-off-by Christophe Leroy <[email protected]>
Maybe this could be folded into the patch that makes pmd_leaf() not returning always false, but no strong feelings:
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [RFC PATCH v3 05/16] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
Le 27/05/2024 à 06:55, Oscar Salvador a écrit :
Quoted 16 lines · Christophe Leroy
Quoted 15 lines
Building on 32 bits with
pmd_leaf()not returning always false leads to the following error:CC
arch/powerpc/mm/pgtable.oarch/powerpc/mm/pgtable.c: In function '__find_linux_pte':arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr]
506 | }| ^
arch/powerpc/mm/pgtable.c:394:15: note: declared here
394 |pud_tpud, *pudp;
| ^~~arch/powerpc/mm/pgtable.c:394:15: note: declared hereThis is due to
pmd_offset()being a no-op in that case.
Quoted 7 lines
Quoted 4 lines
So rework it for powerpc/32 so that
pXd_offset()are used on real pointers and not on on-stack copies.Signed-off-by Christophe Leroy <[email protected]>Maybe this could be folded into the patch that makes
pmd_leaf()not returning always false, but no strong feelings:
I prefer to keep it separate, the patch introducing pmd_leaf() is already big enough.
Quoted 4 lines
Reviewed-by Oscar Salvador <[email protected]>
Re: [RFC PATCH v3 05/16] powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
In reply to Christophe LeroyQuoted 13 lines · Christophe Leroy
Building on 32 bits with
pmd_leaf()not returning always false leads to the following error:CC
arch/powerpc/mm/pgtable.oarch/powerpc/mm/pgtable.c: In function '__find_linux_pte':arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr]
506 | }| ^
arch/powerpc/mm/pgtable.c:394:15: note: declared here
394 |pud_tpud, *pudp;
| ^~~arch/powerpc/mm/pgtable.c:394:15: note: declared here
Quoted 6 lines
This is due to
pmd_offset()being a no-op in that case.So rework it for powerpc/32 so that
pXd_offset()are used on real pointers and not on on-stack copies.Signed-off-by Christophe Leroy <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
RFC PATCH v3 6/16 powerpc/mm: Allow hugepages without hugepd
In reply to Christophe LeroyIn preparation of implementing huge pages on powerpc 8xx without hugepd, enclose hugepd related code inside an ifdef CONFIG_ARCH_HAS_HUGEPD
This also allows removing some stubs.
---
v3:
- Prepare huge_pte_alloc() for full standard topology, not only for 2-level
- Reordered last part of huge_pte_alloc()
--- arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 -- arch/powerpc/include/asm/hugetlb.h | 10 ++---- arch/powerpc/include/asm/nohash/pgtable.h | 8 +++-- arch/powerpc/mm/hugetlbpage.c | 33 ++++++++++++++++++++ arch/powerpc/mm/pgtable.c | 2 ++ 5 files changed, 42 insertions(+), 13 deletions(-)
arch/powerpc/include/asm/book3s/32/pgalloc.h +0 -2
@@ -47,8 +47,6 @@ static inline void pgtable_free(void *table, unsigned index_size) } } -#define get_hugepd_cache_index (x) (x) -static inline void pgtable_free_tlb (struct mmu_gather *tlb,void *table,int shift) {
arch/powerpc/include/asm/hugetlb.h +2 -8
@@ -30,10 +30,12 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, }#define is_hugepage_only_range is_hugepage_only_range +#ifdef CONFIG_ARCH_HAS_HUGEPD #define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGEvoid hugetlb_free_pgd_range (struct mmu_gather *tlb,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling); +#endif #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
@@ -67,14 +69,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, { } -#define hugepd_shift (x) 0 -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned pdshift) -{ -return NULL ; -} - -static inline void __init gigantic_hugetlb_cma_reserve (void ) { }
arch/powerpc/include/asm/nohash/pgtable.h +5 -3
@@ -340,7 +340,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,#define pgprot_writecombine pgprot_noncached_wc -#ifdef CONFIG_HUGETLB_PAGE +#ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok (hugepd_t hpd) {#ifdef CONFIG_PPC_8xx
@@ -351,6 +351,10 @@ static inline int hugepd_ok(hugepd_t hpd)#endif } +#define is_hugepd (hpd) (hugepd_ok(hpd)) +#endif + +#ifdef CONFIG_HUGETLB_PAGE static inline int pmd_huge (pmd_t pmd) {return 0 ;
@@ -360,8 +364,6 @@ static inline int pud_huge(pud_t pud) {return 0 ; } - -#define is_hugepd (hpd) (hugepd_ok(hpd))#endif int map_kernel_page (unsigned long va,phys_addr_t pa,pgprot_t prot);
arch/powerpc/mm/hugetlbpage.c +33 -0
@@ -42,6 +42,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sreturn __find_linux_pte (mm->pgd, addr,NULL ,NULL ); } +#ifdef CONFIG_ARCH_HAS_HUGEPD static int __hugepte_alloc (struct mm_struct *mm,hugepd_t *hpdp,unsigned long address,unsigned int pdshift,unsigned int pshift,spinlock_t *ptl)
@@ -193,6 +194,36 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,return hugepte_offset (*hpdp, addr, pdshift); } +#else +pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr,unsigned long sz) +{ +p4d_t *p4d; +pud_t *pud; +pmd_t *pmd; + + addr &= ~(sz -1 ); + + p4d =p4d_offset (pgd_offset (mm, addr), addr); +if (!mm_pud_folded (mm) && sz >=P4D_SIZE ) +return (pte_t *)p4d; + + pud =pud_alloc (mm, p4d, addr); +if (!pud) +return NULL ; +if (!mm_pmd_folded (mm) && sz >=PUD_SIZE ) +return (pte_t *)pud; + + pmd =pmd_alloc (mm, pud, addr); +if (!pmd) +return NULL ; + +if (sz >=PMD_SIZE ) +return (pte_t *)pmd; + +return pte_alloc_huge (mm, pmd, addr); +} +#endif #ifdef CONFIG_PPC_BOOK3S_64 /*
@@ -248,6 +279,7 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid)return __alloc_bootmem_huge_page (h, nid); } +#ifdef CONFIG_ARCH_HAS_HUGEPD #ifndef CONFIG_PPC_BOOK3S_64 #define HUGEPD_FREELIST_SIZE \ ((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
@@ -505,6 +537,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, } }while (addr = next, addr != end); } +#endif bool __init arch_hugetlb_valid_size (unsigned long size) {
arch/powerpc/mm/pgtable.c +2 -0
@@ -496,8 +496,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,if (!hpdp)return NULL ; +#ifdef CONFIG_ARCH_HAS_HUGEPD ret_pte =hugepte_offset (*hpdp, ea, pdshift); pdshift =hugepd_shift (*hpdp); +#endif out:if (hpage_shift) *hpage_shift = pdshift;
Signature · 1 line
2.44.0
Re: [RFC PATCH v3 06/16] powerpc/mm: Allow hugepages without hugepd
Quoted 7 lines · Christophe Leroy
In preparation of implementing huge pages on powerpc 8xx without hugepd, enclose hugepd related code inside an ifdef
CONFIG_ARCH_HAS_HUGEPDThis also allows removing some stubs.
Signed-off-by Christophe Leroy <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
RFC PATCH v3 7/16 powerpc/8xx: Fix size given to set_huge_pte_at()
In reply to Christophe Leroyset_huge_pte_at() expects the size of the hugepage as an int, not the psize which is the index of the page definition in table mmu_psize_defs[]
--- arch/powerpc/mm/nohash/8xx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/powerpc/mm/nohash/8xx.c +2 -1
@@ -94,7 +94,8 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,return -EINVAL ;set_huge_pte_at (&init_mm, va, ptep, -pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), psize); +pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), +1UL <<mmu_psize_to_shift (psize));return 0 ; }
Signature · 1 line
2.44.0
Re: [RFC PATCH v3 07/16] powerpc/8xx: Fix size given to set_huge_pte_at()
Quoted 5 lines · Christophe Leroy
set_huge_pte_at()expects the size of the hugepage as an int, not the psize which is the index of the page definition in table mmu_psize_defs[]Fixes 935d4f0c6dc8 ("mm: hugetlb: add huge page size param to set_huge_pte_at()")Signed-off-by Christophe Leroy <[email protected]>
Quoted 21 lines
--- arch/powerpc/mm/nohash/8xx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/powerpc/mm/nohash/8xx.c +2
-1@@ -94,7 +94,8 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,return -EINVAL ;set_huge_pte_at (&init_mm, va, ptep, -pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), psize); +pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), +1UL <<mmu_psize_to_shift (psize));return 0 ; }Signature · 1 line
2.44.0
Signature · 2 lines
Oscar Salvador SUSE Labs
RFC PATCH v3 8/16 powerpc/8xx: Rework support for 8M pages using contiguous PTE entries
In reply to Christophe LeroyIn order to fit better with standard Linux page tables layout, add support for 8M pages using contiguous PTE entries in a standard page table. Page tables will then be populated with 1024 similar entries and two PMD entries will point to that page table.
The PMD entries also get a flag to tell it is addressing an 8M page, this is required for the HW tablewalk assistance.
---
v3:
- Move huge_ptep_get() for a more readable commit diff
- Flag PMD as 8Mbytes in set_huge_pte_at()
- Define __pte_leaf_size()
- Change pte_update() instead of all huge callers of pte_update()
- Added ptep_is_8m_pmdp() helper
- Fixed kasan early memory 8M allocation
--- arch/powerpc/Kconfig | 1 - .../include/asm/nohash/32/hugetlb-8xx.h | 38 +++---------- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 53 ++++++++++++------- arch/powerpc/include/asm/nohash/pgtable.h | 4 -- arch/powerpc/include/asm/page.h | 5 -- arch/powerpc/include/asm/pgtable.h | 3 ++ arch/powerpc/kernel/head_8xx.S | 10 +--- arch/powerpc/mm/hugetlbpage.c | 18 ++++--- arch/powerpc/mm/kasan/8xx.c | 21 +++++--- arch/powerpc/mm/nohash/8xx.c | 40 +++++++------- arch/powerpc/mm/pgtable.c | 27 +++++++--- arch/powerpc/mm/pgtable_32.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 2 + 13 files changed, 112 insertions(+), 112 deletions(-)
arch/powerpc/Kconfig +0 -1
@@ -135,7 +135,6 @@ config PPC select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_HAS_KCOV select ARCH_HAS_MEMBARRIER_CALLBACKS select ARCH_HAS_MEMBARRIER_SYNC_CORE
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h +8 -30
@@ -4,42 +4,12 @@#define PAGE_SHIFT_8M 23 -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -BUG_ON (!hugepd_ok (hpd)); - -return (pte_t *)__va (hpd_val (hpd) & ~HUGEPD_SHIFT_MASK ); -} - -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return PAGE_SHIFT_8M ; -} - -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -unsigned long idx = (addr & (SZ_4M -1 )) >>PAGE_SHIFT ; - -return hugepd_page (hpd) + idx; -} -static inline void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr) {flush_tlb_page (vma, vmaddr); } -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) | _PMD_USER | _PMD_PRESENT | _PMD_PAGE_8M); -} - -static inline void hugepd_populate_kernel (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) | _PMD_PRESENT | _PMD_PAGE_8M); -} -static inline int check_and_get_huge_psize (int shift) {return shift_to_mmu_psize (shift);
@@ -49,6 +19,14 @@ static inline int check_and_get_huge_psize(int shift)void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ +if (ptep_is_8m_pmdp (mm, addr, ptep)) + ptep =pte_offset_kernel ((pmd_t *)ptep,0 ); +return ptep_get (ptep); +} +#define __HAVE_ARCH_HUGE_PTE_CLEARstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +34 -19
104 lines not on this page · raw message
arch/powerpc/include/asm/nohash/pgtable.h +0 -4
@@ -343,12 +343,8 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,#ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok (hugepd_t hpd) { -#ifdef CONFIG_PPC_8xx -return ((hpd_val (hpd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M); -#else /* We clear the top bit to indicate hugepd */ return (hpd_val (hpd) && (hpd_val (hpd) &PD_HUGE ) ==0 ); -#endif }#define is_hugepd (hpd) (hugepd_ok(hpd))
arch/powerpc/include/asm/page.h +0 -5
@@ -293,13 +293,8 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)/* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits. -* On 8xx, HW assistance requires 4k alignment for the hugepte. */ -#ifdef CONFIG_PPC_8xx -#define HUGEPD_SHIFT_MASK 0xfff -#else #define HUGEPD_SHIFT_MASK 0x3f -#endif #ifndef __ASSEMBLY__
arch/powerpc/include/asm/pgtable.h +3 -0
@@ -106,6 +106,9 @@ unsigned long vmalloc_to_phys(void *vmalloc_addr);void pgtable_cache_add (unsigned int shift); +#ifdef CONFIG_PPC32 +void __init *early_alloc_pgtable (unsigned long size); +#endif pte_t *early_pte_alloc_kernel (pmd_t *pmdp,unsigned long va);#if defined(CONFIG_STRICT_KERNEL_RWX ) || defined(CONFIG_PPC32 )
arch/powerpc/kernel/head_8xx.S +2 -8
@@ -415,14 +415,13 @@ FixupDAR:/* Entry point for dcbx workaround. */ oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha 3: lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + rlwinm r11, r11, 0, ~_PMD_PAGE_8M mtspr SPRN_MD_TWC, r11 - mtcrf 0x01, r11 mfspr r11, SPRN_MD_TWC lwz r11, 0(r11) /* Get the pte */ - bt 28,200f /* bit 28 = Large page (8M) */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 -201: lwz r11,0(r11) + lwz r11,0(r11) /* Check if it really is a dcbx instruction. */ /* dcbt and dcbtst does not generate DTLB Misses/Errors, * no need to include them here */
@@ -441,11 +440,6 @@ FixupDAR:/* Entry point for dcbx workaround. */ 141: mfspr r10,SPRN_M_TW b DARFixed /* Nope, go back to normal TLB processing */ -200: - /* concat physical page address(r11) and page offset(r10) */ - rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 - b 201b - 144: mfspr r10, SPRN_DSISR rlwinm r10, r10,0,7,5 /* Clear store bit for buggy dcbst insn */ mtspr SPRN_DSISR, r10
arch/powerpc/mm/hugetlbpage.c +12 -6
@@ -183,9 +183,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,if (!hpdp)return NULL ; -if (IS_ENABLED (CONFIG_PPC_8xx) && pshift <PMD_SHIFT ) -return pte_alloc_huge (mm, (pmd_t *)hpdp, addr); -BUG_ON (!hugepd_none (*hpdp) && !hugepd_ok (*hpdp)); if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr,
@@ -218,8 +215,18 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,if (!pmd)return NULL ; -if (sz >=PMD_SIZE ) +if (sz >=PMD_SIZE ) { +/* On 8xx, all hugepages are handled as contiguous PTEs */ +if (IS_ENABLED (CONFIG_PPC_8xx)) { +int i; + +for (i =0 ; i < sz /PMD_SIZE ; i++) { +if (!pte_alloc_huge (mm, pmd + i, addr)) +return NULL ; + } + }return (pte_t *)pmd; + }return pte_alloc_huge (mm, pmd, addr); }
@@ -619,8 +626,7 @@ static int __init hugetlbpage_init(void)if (pdshift > shift) {if (!IS_ENABLED (CONFIG_PPC_8xx))pgtable_cache_add (pdshift - shift); - }else if (IS_ENABLED (CONFIG_PPC_E500 ) || -IS_ENABLED (CONFIG_PPC_8xx)) { + }else if (IS_ENABLED (CONFIG_PPC_E500 )) {pgtable_cache_add (PTE_T_ORDER ); }
arch/powerpc/mm/kasan/8xx.c +13 -8
@@ -6,28 +6,33 @@#include <linux/memblock.h> #include <linux/hugetlb.h> +#include <asm/pgalloc.h> +static int __init kasan_init_shadow_8M (unsigned long k_start,unsigned long k_end,void *block) {pmd_t *pmd =pmd_off_k (k_start);unsigned long k_cur, k_next; -for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd +=2 , block +=SZ_8M ) { -pte_basic_t *new; +for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++, block +=SZ_4M ) { +pte_t *ptep; +int i; k_next =pgd_addr_end (k_cur, k_end); - k_next =pgd_addr_end (k_next, k_end);if ((void *)pmd_page_vaddr (*pmd) != kasan_early_shadow_pte)continue ; - new =memblock_alloc (sizeof (pte_basic_t),SZ_4K ); -if (!new) + ptep =memblock_alloc (PTE_FRAG_SIZE ,PTE_FRAG_SIZE ); +if (!ptep)return -ENOMEM ; - *new =pte_val (pte_mkhuge (pfn_pte (PHYS_PFN (__pa (block)),PAGE_KERNEL ))); +for (i =0 ; i <PTRS_PER_PTE ; i++) { +pte_t pte =pte_mkhuge (pfn_pte (PHYS_PFN (__pa (block + i *PAGE_SIZE )),PAGE_KERNEL )); -hugepd_populate_kernel ((hugepd_t *)pmd, (pte_t *)new,PAGE_SHIFT_8M ); -hugepd_populate_kernel ((hugepd_t *)pmd +1 , (pte_t *)new,PAGE_SHIFT_8M ); +__set_pte_at (&init_mm, k_cur, ptep + i, pte,1 ); + } +pmd_populate_kernel (&init_mm, pmd, ptep); + *pmd =__pmd (pmd_val (*pmd) | _PMD_PAGE_8M); }return 0 ; }
arch/powerpc/mm/nohash/8xx.c +18 -22
@@ -11,6 +11,7 @@#include <linux/hugetlb.h> #include <asm/fixmap.h> +#include <asm/pgalloc.h> #include <mm/mmu_decl.h>
@@ -48,20 +49,6 @@ unsigned long p_block_mapped(phys_addr_t pa)return 0 ; } -static pte_t __init *early_hugepd_alloc_kernel (hugepd_t *pmdp,unsigned long va) -{ -if (hpd_val (*pmdp) ==0 ) { -pte_t *ptep =memblock_alloc (sizeof (pte_basic_t),SZ_4K ); - -if (!ptep) -return NULL ; - -hugepd_populate_kernel ((hugepd_t *)pmdp, ptep,PAGE_SHIFT_8M ); -hugepd_populate_kernel ((hugepd_t *)pmdp +1 , ptep,PAGE_SHIFT_8M ); - } -return hugepte_offset (*(hugepd_t *)pmdp, va,PGDIR_SHIFT ); -} -static int __ref __early_map_kernel_hugepage (unsigned long va,phys_addr_t pa,pgprot_t prot,int psize,bool new) {
@@ -75,24 +62,33 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,if (WARN_ON (slab_is_available ()))return -EINVAL ; -if (psize ==MMU_PAGE_512K ) +if (psize ==MMU_PAGE_512K ) { ptep =early_pte_alloc_kernel (pmdp, va); -else - ptep =early_hugepd_alloc_kernel ((hugepd_t *)pmdp, va); +/* The PTE should never be already present */ +if (WARN_ON (pte_present (*ptep) &&pgprot_val (prot))) +return -EINVAL ; + }else { +if (WARN_ON (!pmd_none (*pmdp) || !pmd_none (*(pmdp +1 )))) +return -EINVAL ; + + ptep =early_alloc_pgtable (PTE_FRAG_SIZE ); +pmd_populate_kernel (&init_mm, pmdp, ptep); + + ptep =early_alloc_pgtable (PTE_FRAG_SIZE ); +pmd_populate_kernel (&init_mm, pmdp +1 , ptep); + + ptep = (pte_t *)pmdp; + } }else {if (psize ==MMU_PAGE_512K ) ptep =pte_offset_kernel (pmdp, va);else - ptep =hugepte_offset (*(hugepd_t *)pmdp, va,PGDIR_SHIFT ); + ptep = (pte_t *)pmdp; }if (WARN_ON (!ptep))return -ENOMEM ; -/* The PTE should never be already present */ -if (new &&WARN_ON (pte_present (*ptep) &&pgprot_val (prot))) -return -EINVAL ; -set_huge_pte_at (&init_mm, va, ptep,pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)),1UL <<mmu_psize_to_shift (psize));
arch/powerpc/mm/pgtable.c +19 -8
@@ -297,11 +297,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, }#if defined(CONFIG_PPC_8xx) -void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte,unsigned long sz) +static void __set_huge_pte_at (pmd_t *pmd,pte_t *ptep,pte_basic_t val) { -pmd_t *pmd =pmd_off (mm, addr); -pte_basic_t val;pte_basic_t *entry = (pte_basic_t *)ptep;int num, i;
@@ -311,15 +308,29 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,*/ VM_WARN_ON (pte_hw_valid (*ptep) && !pte_protnone (*ptep)); - pte =set_pte_filter (pte, addr); - - val =pte_val (pte); - num =number_of_cells_per_pte (pmd, val,1 );for (i =0 ; i < num; i++, entry++, val +=SZ_4K ) *entry = val; } + +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz) +{ +pmd_t *pmdp =pmd_off (mm, addr); + + pte =set_pte_filter (pte, addr); + +if (sz ==SZ_8M ) {/* Flag both PMD entries as 8M and fill both page tables */ + *pmdp =__pmd (pmd_val (*pmdp) | _PMD_PAGE_8M); + *(pmdp +1 ) =__pmd (pmd_val (*(pmdp +1 )) | _PMD_PAGE_8M); + +__set_huge_pte_at (pmdp,pte_offset_kernel (pmdp,0 ),pte_val (pte)); +__set_huge_pte_at (pmdp,pte_offset_kernel (pmdp +1 ,0 ),pte_val (pte) +SZ_4M ); + }else { +__set_huge_pte_at (pmdp, ptep,pte_val (pte)); + } +}#endif #endif /* CONFIG_HUGETLB_PAGE */
arch/powerpc/mm/pgtable_32.c +1 -1
@@ -48,7 +48,7 @@ notrace void __init early_ioremap_init(void)early_ioremap_setup (); } -static void __init *early_alloc_pgtable (unsigned long size) +void __init *early_alloc_pgtable (unsigned long size) {void *ptr =memblock_alloc (size, size);
arch/powerpc/platforms/Kconfig.cputype +2 -0
@@ -98,6 +98,7 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD
@@ -290,6 +291,7 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL
Signature · 1 line
2.44.0
Re: [RFC PATCH v3 08/16] powerpc/8xx: Rework support for 8M pages using contiguous PTE entries
Quoted 9 lines · Christophe Leroy
In order to fit better with standard Linux page tables layout, add support for 8M pages using contiguous PTE entries in a standard page table. Page tables will then be populated with 1024 similar entries and two PMD entries will point to that page table.
The PMD entries also get a flag to tell it is addressing an 8M page, this is required for the HW tablewalk assistance.
Signed-off-by Christophe Leroy <[email protected]>
I did not look close into KSAN bits, and I trust you with the assembly part, but other than that looks good to me, so FWIW:
Just a nit below:
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h static inline int check_and_get_huge_psize(int shift) · line 26
Quoted 5 lines
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h +5
-0+#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ +if (ptep_is_8m_pmdp (mm, addr, ptep)) + ptep =pte_offset_kernel ((pmd_t *)ptep,0 );
Would it not be more clear to use pmd_page_vaddr directly there?
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [RFC PATCH v3 08/16] powerpc/8xx: Rework support for 8M pages using contiguous PTE entries
Le 27/05/2024 à 14:10, Oscar Salvador a écrit :
Quoted 18 lines · Christophe Leroy
Quoted 9 lines
In order to fit better with standard Linux page tables layout, add support for 8M pages using contiguous PTE entries in a standard page table. Page tables will then be populated with 1024 similar entries and two PMD entries will point to that page table.
The PMD entries also get a flag to tell it is addressing an 8M page, this is required for the HW tablewalk assistance.
Signed-off-by Christophe Leroy <[email protected]>I did not look close into KSAN bits, and I trust you with the assembly part, but other than that looks good to me, so FWIW:
Reviewed-by Oscar Salvador <[email protected]>Just a nit below:
Quoted 9 lines
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h static inline int check_and_get_huge_psize(int shift) · line 26
Quoted 5 lines
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h +5
-0+#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ +if (ptep_is_8m_pmdp (mm, addr, ptep)) + ptep =pte_offset_kernel ((pmd_t *)ptep,0 );Would it not be more clear to use
pmd_page_vaddrdirectly there?
Well, the correct way should have been:
ptep = pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M));
Now, is it more clear with:
ptep = (pte_t *)pmd_page_vaddr(*(pmd_t *)ptep);
I don't know.
RFC PATCH v3 9/16 powerpc/8xx: Simplify struct mmu_psize_def
In reply to Christophe LeroyOn 8xx, only the shift field is used in struct mmu_psize_def
Remove other fields and related macros.
--- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
arch/powerpc/include/asm/nohash/32/mmu-8xx.h +2 -7
@@ -189,19 +189,14 @@ typedef struct {#define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000) -/* Page size definitions, common between 32 and 64-bit +/* +* Page size definitions for 8xx * * shift : is the "PAGE_SHIFT" value for that page size -* penc : is the pte encoding mask * */ struct mmu_psize_def {unsigned int shift;/* number of bits */ -unsigned int enc;/* PTE encoding */ -unsigned int ind;/* Corresponding indirect page size shift */ -unsigned int flags; -#define MMU_PAGE_SIZE_DIRECT 0x1/* Supported as a direct size */ -#define MMU_PAGE_SIZE_INDIRECT 0x2/* Supported as an indirect size */ };extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ];
Signature · 1 line
2.44.0
RFC PATCH v3 10/16 powerpc/e500: Remove enc and ind fields from struct mmu_psize_def
In reply to Christophe Leroyenc field is hidden behind BOOK3E_PAGESZ_XX macros, and when you look closer you realise that this field is nothing else than the value of shift minus ten.
So remove enc field and calculate tsize from shift field.
Also remove inc field which is unused.
--- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 --- arch/powerpc/mm/nohash/book3e_pgtable.c | 4 ++-- arch/powerpc/mm/nohash/tlb.c | 9 +-------- arch/powerpc/mm/nohash/tlb_64e.c | 2 +- 4 files changed, 4 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/nohash/mmu-e500.h +0 -3
@@ -244,14 +244,11 @@ typedef struct {/* Page size definitions, common between 32 and 64-bit * * shift : is the "PAGE_SHIFT" value for that page size -* penc : is the pte encoding mask * */ struct mmu_psize_def {unsigned int shift;/* number of bits */ -unsigned int enc;/* PTE encoding */ -unsigned int ind;/* Corresponding indirect page size shift */ unsigned int flags;#define MMU_PAGE_SIZE_DIRECT 0x1/* Supported as a direct size */ #define MMU_PAGE_SIZE_INDIRECT 0x2/* Supported as an indirect size */
arch/powerpc/mm/nohash/book3e_pgtable.c +2 -2
@@ -29,10 +29,10 @@ int __meminit vmemmap_create_mapping(unsigned long start, _PAGE_KERNEL_RW;/* PTEs only contain page size encodings up to 32M */ -BUG_ON (mmu_psize_defs[mmu_vmemmap_psize].enc >0xf ); +BUG_ON (mmu_psize_defs[mmu_vmemmap_psize].shift -10 >0xf );/* Encode the size in the PTE */ - flags |= mmu_psize_defs[mmu_vmemmap_psize].enc <<8 ; + flags |= (mmu_psize_defs[mmu_vmemmap_psize].shift -10 ) <<8 ;/* For each PTE for that area, map things. Note that we don't * increment phys because all PTEs are of the large size and
arch/powerpc/mm/nohash/tlb.c +1 -8
@@ -53,37 +53,30 @@struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ] = { [MMU_PAGE_4K ] = { .shift =12 , - .enc =BOOK3E_PAGESZ_4K , }, [MMU_PAGE_2M ] = { .shift =21 , - .enc =BOOK3E_PAGESZ_2M , }, [MMU_PAGE_4M ] = { .shift =22 , - .enc =BOOK3E_PAGESZ_4M , }, [MMU_PAGE_16M ] = { .shift =24 , - .enc =BOOK3E_PAGESZ_16M , }, [MMU_PAGE_64M ] = { .shift =26 , - .enc =BOOK3E_PAGESZ_64M , }, [MMU_PAGE_256M ] = { .shift =28 , - .enc =BOOK3E_PAGESZ_256M , }, [MMU_PAGE_1G ] = { .shift =30 , - .enc =BOOK3E_PAGESZ_1GB , }, };static inline int mmu_get_tsize (int psize) { -return mmu_psize_defs[psize].enc; +return mmu_psize_defs[psize].shift -10 ; }#else static inline int mmu_get_tsize (int psize)
arch/powerpc/mm/nohash/tlb_64e.c +1 -1
@@ -53,7 +53,7 @@ int extlb_level_exc;*/ void tlb_flush_pgtable (struct mmu_gather *tlb,unsigned long address) { -int tsize = mmu_psize_defs[mmu_pte_psize].enc; +int tsize = mmu_psize_defs[mmu_pte_psize].shift -10 ;if (book3e_htw_mode !=PPC_HTW_NONE ) {unsigned long start = address &PMD_MASK ;
Signature · 1 line
2.44.0
RFC PATCH v3 11/16 powerpc/e500: Switch to 64 bits PGD on 85xx (32 bits)
In reply to Christophe LeroyAt the time being when CONFIG_PTE_64BIT is selected, PTE entries are 64 bits but PGD entries are still 32 bits.
In order to allow leaf PMD entries, switch the PGD to 64 bits entries.
--- arch/powerpc/include/asm/pgtable-types.h | 4 ++++ arch/powerpc/kernel/head_85xx.S | 10 ++++++---- 2 files changed, 10 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/pgtable-types.h +4 -0
@@ -49,7 +49,11 @@ static inline unsigned long pud_val(pud_t x)#endif /* CONFIG_PPC64 */ /* PGD level */ +#if defined(CONFIG_PPC_E500 ) && defined(CONFIG_PTE_64BIT ) +typedef struct {unsigned long long pgd; }pgd_t ; +#else typedef struct {unsigned long pgd; }pgd_t ; +#endif #define __pgd (x) ((pgd_t) { (x) })static inline unsigned long pgd_val (pgd_t x) {
arch/powerpc/kernel/head_85xx.S +6 -4
@@ -307,8 +307,9 @@ set_ivor: #ifdef CONFIG_PTE_64BIT #ifdef CONFIG_HUGETLB_PAGE #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ blt 1000f; /* Normal non-huge page */ \ beq 2f; /* Bail if no table */ \
@@ -321,8 +322,9 @@ set_ivor: 1001: lwz r11, 4(r12); /* Get pte entry */ #else #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ beq 2f; /* Bail if no table */ \ rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \
Signature · 1 line
2.44.0
RFC PATCH v3 12/16 powerpc/e500: Encode hugepage size in PTE bits
In reply to Christophe LeroyUse U0-U3 bits to encode hugepage size, more exactly page shift.
As we start using hugepages at shift 21 (2Mbytes), substract 20 so that it fits into 4 bits. That may change in the future if we want to use smaller hugepages.
--- arch/powerpc/include/asm/nohash/hugetlb-e500.h | 6 ++++++ arch/powerpc/include/asm/nohash/pte-e500.h | 3 +++ 2 files changed, 9 insertions(+)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +6 -0
@@ -42,4 +42,10 @@ static inline int check_and_get_huge_psize(int shift) return shift_to_mmu_psize(shift); } +static inline pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags) +{ +return __pte (pte_val (entry) | (_PAGE_U3 * (shift -20 ))); +} +#define arch_make_huge_pte arch_make_huge_pte +#endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */
arch/powerpc/include/asm/nohash/pte-e500.h +3 -0
@@ -46,6 +46,9 @@#define _PAGE_NO_CACHE 0x400000/* I: cache inhibit */ #define _PAGE_WRITETHRU 0x800000/* W: cache write-through */ +#define _PAGE_HSIZE_MSK (_PAGE_U0 | _PAGE_U1 | _PAGE_U2 | _PAGE_U3) +#define _PAGE_HSIZE_SHIFT 14 +/* "Higher level" linux bit combinations */ #define _PAGE_EXEC (_PAGE_BAP_SX | _PAGE_BAP_UX)/* .. and was cache cleaned */ #define _PAGE_READ (_PAGE_BAP_SR | _PAGE_BAP_UR)/* User read permission */
Signature · 1 line
2.44.0
RFC PATCH v3 13/16 powerpc/e500: Use contiguous PMD instead of hugepd
In reply to Christophe Leroye500 supports many page sizes among which the following size are implemented in the kernel at the time being: 4M, 16M, 64M, 256M, 1G.
On e500, TLB miss for hugepages is exclusively handled by SW even on e6500 which has HW assistance for 4k pages, so there are no constraints like on the 8xx.
On e500/32, all are at PGD/PMD level and can be handled as cont-PMD.
On e500/64, smaller ones are on PMD while bigger ones are on PUD. Again, they can easily be handled as cont-PMD and cont-PUD instead of hugepd.
---
v3: Add missing pmd_leaf_size() and pud_leaf_size()
--- .../powerpc/include/asm/nohash/hugetlb-e500.h | 32 +--------- arch/powerpc/include/asm/nohash/pgalloc.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 43 +++++++++---- arch/powerpc/include/asm/nohash/pte-e500.h | 28 +++++++++ arch/powerpc/include/asm/page.h | 15 +---- arch/powerpc/kernel/head_85xx.S | 23 +++---- arch/powerpc/mm/hugetlbpage.c | 2 - arch/powerpc/mm/nohash/tlb_low_64e.S | 63 +++++++++++-------- arch/powerpc/mm/pgtable.c | 31 +++++++++ arch/powerpc/platforms/Kconfig.cputype | 1 - 10 files changed, 144 insertions(+), 96 deletions(-)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +3 -29
@@ -2,38 +2,12 @@#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -if (WARN_ON (!hugepd_ok (hpd))) -return NULL ; - -return (pte_t *)((hpd_val (hpd) & ~HUGEPD_SHIFT_MASK ) |PD_HUGE ); -} - -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return hpd_val (hpd) &HUGEPD_SHIFT_MASK ; -} - -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -/* -* On FSL BookE, we have multiple higher-level table entries that -* point to the same hugepte. Just use the first one since they're all -* identical. So for that case, idx=0. -*/ -return hugepd_page (hpd); -} +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz);void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr); -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ -/* We use the old format for PPC_E500 */ - *hpdp =__hugepd (((unsigned long )new & ~PD_HUGE ) | pshift); -} -static inline int check_and_get_huge_psize (int shift) {if (shift &1 )/* Not a power of 4 */
arch/powerpc/include/asm/nohash/pgalloc.h +0 -2
@@ -44,8 +44,6 @@ static inline void pgtable_free(void *table, int shift) } } -#define get_hugepd_cache_index (x) (x) -static inline void pgtable_free_tlb (struct mmu_gather *tlb,void *table,int shift) {unsigned long pgf = (unsigned long )table;
arch/powerpc/include/asm/nohash/pgtable.h +31 -12
67 lines not on this page · raw message
arch/powerpc/include/asm/nohash/pte-e500.h +28 -0
@@ -67,6 +67,7 @@#define _PAGE_RWX (_PAGE_READ | _PAGE_WRITE | _PAGE_BAP_UX)#define _PAGE_SPECIAL _PAGE_SW0 +#define _PAGE_PTE _PAGE_PSIZE_4K#define PTE_RPN_SHIFT (24)
@@ -106,6 +107,33 @@ static inline pte_t pte_mkexec(pte_t pte) }#define pte_mkexec pte_mkexec +static inline int pmd_leaf (pmd_t pmd) +{ +return pmd_val (pmd) & _PAGE_PTE; +} +#define pmd_leaf pmd_leaf + +static inline unsigned long pmd_leaf_size (pmd_t pmd) +{ +return 1UL << (((pmd_val (pmd) & _PAGE_HSIZE_MSK) >> _PAGE_HSIZE_SHIFT) +20 ); +} +#define pmd_leaf_size pmd_leaf_size + +#ifdef CONFIG_PPC64 +static inline int pud_leaf (pud_t pud) +{ +return pud_val (pud) & _PAGE_PTE; +} +#define pud_leaf pud_leaf + +static inline unsigned long pud_leaf_size (pud_t pud) +{ +return 1UL << (((pud_val (pud) & _PAGE_HSIZE_MSK) >> _PAGE_HSIZE_SHIFT) +20 ); +} +#define pud_leaf_size pud_leaf_size + +#endif +#endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */
arch/powerpc/include/asm/page.h +1 -14
@@ -269,20 +269,7 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -#ifndef CONFIG_PPC_BOOK3S_64 -/* -* Use the top bit of the higher-level page table entries to indicate whether -* the entries we point to contain hugepages. This works because we know that -* the page tables live in kernel space. If we ever decide to support having -* page tables at arbitrary addresses, this breaks and will have to change. -*/ -#ifdef CONFIG_PPC64 -#define PD_HUGE 0x8000000000000000UL -#else -#define PD_HUGE 0x80000000 -#endif - -#else /* CONFIG_PPC_BOOK3S_64 */ +#ifdef CONFIG_PPC_BOOK3S_64 /* * Book3S 64 stores real addresses in the hugepd entries to * avoid overlaps with _PAGE_PRESENT and _PAGE_PTE.
arch/powerpc/kernel/head_85xx.S +12 -11
@@ -310,16 +310,17 @@ set_ivor: rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ - rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ - blt 1000f; /* Normal non-huge page */ \ - beq 2f; /* Bail if no table */ \ - oris r11, r11, PD_HUGE@h; /* Put back address bit */ \ - andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ - xor r12, r10, r11; /* drop size bits from pointer */ \ + rotlwi. r11, r11, 22; /* Leaf entry (_PAGE_PTE set) */\ + bge 1000f; /* Normal non-huge page */ \ + rlwinm r10, r11, 64 - _PAGE_HSIZE_SHIFT - 22, 0xf; \ + rotrwi r11, r11, 22; /* Restore entry */ \ b 1001f; \ -1000: rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ +1000: rlwinm. r12, r11, 32 - 22, 0, 20; /* Extract pt base address */ \ + beq 2f; /* Bail if no table */ \ + rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ li r10, 0; /* clear r10 */ \ -1001: lwz r11, 4(r12); /* Get pte entry */ + lwz r11, 4(r12); /* Get pte entry */ \ +1001: #else #define FIND_PTE \ rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \
@@ -749,16 +750,16 @@ finish_tlb_load: 100: stw r15, 0(r17) /* - * Calc MAS1_TSIZE from r10 (which has pshift encoded) + * Calc MAS1_TSIZE from r10 (which has pshift - 20 encoded) * tlb_enc = (pshift - 10). */ - subi r15, r10, 10 + addi r15, r10, 10 mfspr r16, SPRN_MAS1 rlwimi r16, r15, 7, 20, 24 mtspr SPRN_MAS1, r16 /* copy the pshift for use later */ - mr r14, r10 + addi r14, r10, 20 /* fall through */
arch/powerpc/mm/hugetlbpage.c +0 -2
@@ -626,8 +626,6 @@ static int __init hugetlbpage_init(void)if (pdshift > shift) {if (!IS_ENABLED (CONFIG_PPC_8xx))pgtable_cache_add (pdshift - shift); - }else if (IS_ENABLED (CONFIG_PPC_E500 )) { -pgtable_cache_add (PTE_T_ORDER ); } configured = true;
arch/powerpc/mm/nohash/tlb_low_64e.S +38 -25
118 lines not on this page · raw message
arch/powerpc/mm/pgtable.c +31 -0
@@ -331,6 +331,37 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,__set_huge_pte_at (pmdp, ptep,pte_val (pte)); } } +#elif defined(CONFIG_PPC_E500 ) +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz) +{ +unsigned long pdsize; +int i; + + pte =set_pte_filter (pte, addr); + +/* +* Make sure hardware valid bit is not set. We don't do +* tlb flush for this update. +*/ +VM_WARN_ON (pte_hw_valid (*ptep) && !pte_protnone (*ptep)); + +if (sz <PMD_SIZE ) + pdsize =PAGE_SIZE ; +else if (sz <PUD_SIZE ) + pdsize =PMD_SIZE ; +else if (sz <P4D_SIZE ) + pdsize =PUD_SIZE ; +else if (sz <PGDIR_SIZE ) + pdsize =P4D_SIZE ; +else + pdsize =PGDIR_SIZE ; + +for (i =0 ; i < sz / pdsize; i++, ptep++, addr += pdsize) { +__set_pte_at (mm, addr, ptep, pte,0 ); + pte =__pte (pte_val (pte) + ((unsigned long long )pdsize /PAGE_SIZE <<PFN_PTE_SHIFT )); + } +}#endif #endif /* CONFIG_HUGETLB_PAGE */
arch/powerpc/platforms/Kconfig.cputype +0 -1
@@ -291,7 +291,6 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL
Signature · 1 line
2.44.0
RFC PATCH v3 14/16 powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD
In reply to Christophe LeroyOn book3s/64, the only user of hugepd is hash in 4k mode.
All other setups (hash-64, radix-4, radix-64) use leaf PMD/PUD.
Rework hash-4k to use contiguous PMD and PUD instead.
In that setup there are only two huge page sizes: 16M and 16G.
16M sits at PMD level and 16G at PUD level.
pte_update doesn't know page size, lets use the same trick as hpte_need_flush() to get page size from segment properties. That's not the most efficient way but let's do that until callers of pte_update() provide page size instead of just a huge flag.
---
v3:
- Add missing pmd_leaf_size() and pud_leaf_size()
- More cleanup in hugetlbpage_init()
- Take a page fault when DIRTY or ACCESSED is missing on hash-4 hugepage
--- arch/powerpc/include/asm/book3s/64/hash-4k.h | 15 -------- arch/powerpc/include/asm/book3s/64/hash.h | 38 +++++++++++++++---- arch/powerpc/include/asm/book3s/64/hugetlb.h | 38 ------------------- .../include/asm/book3s/64/pgtable-4k.h | 34 ----------------- .../include/asm/book3s/64/pgtable-64k.h | 20 ---------- arch/powerpc/include/asm/book3s/64/pgtable.h | 18 +++++++++ arch/powerpc/include/asm/hugetlb.h | 4 ++ .../powerpc/include/asm/nohash/hugetlb-e500.h | 4 -- arch/powerpc/include/asm/page.h | 8 ---- arch/powerpc/mm/book3s64/hash_utils.c | 11 ++++-- arch/powerpc/mm/book3s64/hugetlbpage.c | 10 +++++ arch/powerpc/mm/book3s64/pgtable.c | 12 ------ arch/powerpc/mm/hugetlbpage.c | 27 ------------- arch/powerpc/mm/pgtable.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 1 - 15 files changed, 71 insertions(+), 171 deletions(-)
arch/powerpc/include/asm/book3s/64/hash-4k.h +0 -15
@@ -74,21 +74,6 @@#define remap_4k_pfn (vma, addr, pfn, prot) \ remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot)) -#ifdef CONFIG_HUGETLB_PAGE -static inline int hash__hugepd_ok (hugepd_t hpd) -{ -unsigned long hpdval =hpd_val (hpd); -/* -* if it is not a pte and have hugepd shift mask -* set, then it is a hugepd directory pointer -*/ -if (!(hpdval & _PAGE_PTE) && (hpdval & _PAGE_PRESENT) && - ((hpdval &HUGEPD_SHIFT_MASK ) !=0 )) -return true; -return false; -} -#endif -/* * 4K PTE format is different from 64K PTE format. Saving the hash_slot is just * a matter of returning the PTE bits that need to be modified. On 64K PTE,
arch/powerpc/include/asm/book3s/64/hash.h +31 -7
@@ -4,6 +4,7 @@#ifdef __KERNEL__#include <asm/asm-const.h> +#include <asm/book3s/64/slice.h> /* * Common bits between 4K and 64K pages in a linux-style PTE.
@@ -161,14 +162,10 @@ extern void hpte_need_flush(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long pte, int huge);unsigned long htab_convert_pte_flags (unsigned long pteflags,unsigned long flags);/* Atomic PTE updates */ -static inline unsigned long hash__pte_update (struct mm_struct *mm, -unsigned long addr, -pte_t *ptep,unsigned long clr, -unsigned long set, -int huge) +static inline unsigned long hash__pte_update_one (pte_t *ptep,unsigned long clr, +unsigned long set) {__be64 old_be, tmp_be; -unsigned long old; __asm__ __volatile__( "1: ldarx %0,0,%3 # pte_update\n\
@@ -182,11 +179,38 @@ static inline unsigned long hash__pte_update(struct mm_struct *mm, : "r" (ptep), "r" (cpu_to_be64(clr)), "m" (*ptep), "r" (cpu_to_be64(H_PAGE_BUSY)), "r" (cpu_to_be64(set)) : "cc" ); + +return be64_to_cpu (old_be); +} + +static inline unsigned long hash__pte_update (struct mm_struct *mm, +unsigned long addr, +pte_t *ptep,unsigned long clr, +unsigned long set, +int huge) +{ +unsigned long old; + + old =hash__pte_update_one (ptep, clr, set); + +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && huge) { +unsigned int psize =get_slice_psize (mm, addr); +int nb, i; + +if (psize ==MMU_PAGE_16M ) + nb =SZ_16M /PMD_SIZE ; +else if (psize ==MMU_PAGE_16G ) + nb =SZ_16G /PUD_SIZE ; +else + nb =1 ; + +for (i =1 ; i < nb; i++) +hash__pte_update_one (ptep + i, clr, set); + }/* huge pages use the old page table lock */ if (!huge)assert_pte_locked (mm, addr); - old =be64_to_cpu (old_be);if (old &H_PAGE_HASHPTE )hpte_need_flush (mm, addr, ptep, old, huge);
arch/powerpc/include/asm/book3s/64/hugetlb.h +0 -38
@@ -49,9 +49,6 @@ static inline bool gigantic_page_runtime_supported(void)return true; } -/* hugepd entry valid bit */ -#define HUGEPD_VAL_BITS (0x8000000000000000UL) -#define huge_ptep_modify_prot_start huge_ptep_modify_prot_startextern pte_t huge_ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep);
@@ -60,29 +57,7 @@ extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,extern void huge_ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t new_pte); -/* -* This should work for other subarchs too. But right now we use the -* new format only for 64bit book3s -*/ -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -BUG_ON (!hugepd_ok (hpd)); -/* -* We have only four bits to encode, MMU page size -*/ -BUILD_BUG_ON ((MMU_PAGE_COUNT -1 ) >0xf ); -return __va (hpd_val (hpd) &HUGEPD_ADDR_MASK ); -} - -static inline unsigned int hugepd_mmu_psize (hugepd_t hpd) -{ -return (hpd_val (hpd) &HUGEPD_SHIFT_MASK ) >>2 ; -} -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return mmu_psize_to_shift (hugepd_mmu_psize (hpd)); -}static inline void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr) {
@@ -90,19 +65,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma,return radix__flush_hugetlb_page (vma, vmaddr); } -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -unsigned long idx = (addr & ((1UL << pdshift) -1 )) >>hugepd_shift (hpd); - -return hugepd_page (hpd) + idx; -} - -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) |HUGEPD_VAL_BITS | (shift_to_mmu_psize (pshift) <<2 )); -} -void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr);static inline int check_and_get_huge_psize (int shift)
arch/powerpc/include/asm/book3s/64/pgtable-4k.h +0 -34
@@ -26,40 +26,6 @@ static inline int pud_huge(pud_t pud)return 0 ; } -/* -* With radix , we have hugepage ptes in the pud and pmd entries. We don't -* need to setup hugepage directory for them. Our pte and page directory format -* enable us to have this enabled. -*/ -static inline int hugepd_ok (hugepd_t hpd) -{ -if (radix_enabled ()) -return 0 ; -return hash__hugepd_ok (hpd); -} -#define is_hugepd (hpd) (hugepd_ok(hpd)) - -/* -* 16M and 16G huge page directory tables are allocated from slab cache -* -*/ -#define H_16M_CACHE_INDEX (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE - 24) -#define H_16G_CACHE_INDEX \ - (PAGE_SHIFT + H_PTE_INDEX_SIZE + H_PMD_INDEX_SIZE + H_PUD_INDEX_SIZE - 34) - -static inline int get_hugepd_cache_index (int index) -{ -switch (index) { -case H_16M_CACHE_INDEX : -return HTLB_16M_INDEX ; -case H_16G_CACHE_INDEX : -return HTLB_16G_INDEX ; -default : -BUG (); - } -/* should not reach */ -} -#endif /* CONFIG_HUGETLB_PAGE */ #endif /* __ASSEMBLY__ */
arch/powerpc/include/asm/book3s/64/pgtable-64k.h +0 -20
@@ -30,26 +30,6 @@ static inline int pud_huge(pud_t pud)return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); } -/* -* With 64k page size, we have hugepage ptes in the pgd and pmd entries. We don't -* need to setup hugepage directory for them. Our pte and page directory format -* enable us to have this enabled. -*/ -static inline int hugepd_ok (hugepd_t hpd) -{ -return 0 ; -} - -#define is_hugepd (pdep) 0 - -/* -* This should never get called -*/ -static __always_inline int get_hugepd_cache_index(int index) -{ -BUILD_BUG (); -} -#endif /* CONFIG_HUGETLB_PAGE */ static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr,
arch/powerpc/include/asm/book3s/64/pgtable.h +18 -0
@@ -1441,5 +1441,23 @@ static inline bool pud_leaf(pud_t pud)return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); } +#define pmd_leaf_size pmd_leaf_size +static inline unsigned long pmd_leaf_size (pmd_t pmd) +{ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) +return SZ_16M ; +else +return PMD_SIZE ; +} + +#define pud_leaf_size pud_leaf_size +static inline unsigned long pud_leaf_size (pud_t pud) +{ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) +return SZ_16G ; +else +return PUD_SIZE ; +} +#endif /* __ASSEMBLY__ */ #endif /* _ASM_POWERPC_BOOK3S_64_PGTABLE_H_ */
arch/powerpc/include/asm/hugetlb.h +4 -0
@@ -37,6 +37,10 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long ceiling);#endif +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEARstatic inline pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +0 -4
@@ -2,10 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte,unsigned long sz); -void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr);static inline int check_and_get_huge_psize (int shift)
arch/powerpc/include/asm/page.h +0 -8
@@ -269,14 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -#ifdef CONFIG_PPC_BOOK3S_64 -/* -* Book3S 64 stores real addresses in the hugepd entries to -* avoid overlaps with _PAGE_PRESENT and _PAGE_PTE. -*/ -#define HUGEPD_ADDR_MASK (0x0ffffffffffffffful & ~HUGEPD_SHIFT_MASK) -#endif /* CONFIG_PPC_BOOK3S_64 */ -/* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits.
arch/powerpc/mm/book3s64/hash_utils.c +7 -4
@@ -1233,10 +1233,6 @@ void __init hash__early_init_mmu(void) __pmd_table_size =H_PMD_TABLE_SIZE ; __pud_table_size =H_PUD_TABLE_SIZE ; __pgd_table_size =H_PGD_TABLE_SIZE ; -/* -* 4k use hugepd format, so for hash set then to -* zero -*/ __pmd_val_bits =HASH_PMD_VAL_BITS ; __pud_val_bits =HASH_PUD_VAL_BITS ; __pgd_val_bits =HASH_PGD_VAL_BITS ;
@@ -1546,6 +1542,13 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, goto bail; } +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) { +if (hugeshift ==PMD_SHIFT && psize ==MMU_PAGE_16M ) + hugeshift = mmu_psize_defs[MMU_PAGE_16M ].shift; +if (hugeshift ==PUD_SHIFT && psize ==MMU_PAGE_16G ) + hugeshift = mmu_psize_defs[MMU_PAGE_16G ].shift; + } +/* * Add _PAGE_PRESENT to the required access perm. If there are parallel * updates to the pte that can possibly clear _PAGE_PTE, catch that too.
arch/powerpc/mm/book3s64/hugetlbpage.c +10 -0
@@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,/* If PTE permissions don't match, take page fault */ if (unlikely (!check_pte_access (access, old_pte)))return 1 ; +/* +* If hash-4k, hugepages use seeral contiguous PxD entries +* so bail out and let mm make the page young or dirty +*/ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES )) { +if (!(old_pte & _PAGE_ACCESSED)) +return 1 ; +if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY)) +return 1 ; + }/* * Try to lock the PTE, add ACCESSED and DIRTY if it was
arch/powerpc/mm/book3s64/pgtable.c +0 -12
@@ -460,18 +460,6 @@ static inline void pgtable_free(void *table, int index)case PUD_INDEX :__pud_free (table);break ; -#if defined(CONFIG_PPC_4K_PAGES ) && defined(CONFIG_HUGETLB_PAGE ) -/* 16M hugepd directory at pud level */ -case HTLB_16M_INDEX : -BUILD_BUG_ON (H_16M_CACHE_INDEX <=0 ); -kmem_cache_free (PGT_CACHE (H_16M_CACHE_INDEX ), table); -break ; -/* 16G hugepd directory at the pgd level */ -case HTLB_16G_INDEX : -BUILD_BUG_ON (H_16G_CACHE_INDEX <=0 ); -kmem_cache_free (PGT_CACHE (H_16G_CACHE_INDEX ), table); -break ; -#endif /* We don't free pgd table via RCU callback */ default :BUG ();
arch/powerpc/mm/hugetlbpage.c +0 -27
@@ -592,41 +592,14 @@ static int __init hugetlbpage_init(void)for (psize =0 ; psize <MMU_PAGE_COUNT ; ++psize) {unsigned shift; -unsigned pdshift;if (!mmu_psize_defs[psize].shift)continue ; shift =mmu_psize_to_shift (psize); -#ifdef CONFIG_PPC_BOOK3S_64 -if (shift >PGDIR_SHIFT ) -continue ; -else if (shift >PUD_SHIFT ) - pdshift =PGDIR_SHIFT ; -else if (shift >PMD_SHIFT ) - pdshift =PUD_SHIFT ; -else - pdshift =PMD_SHIFT ; -#else -if (shift <PUD_SHIFT ) - pdshift =PMD_SHIFT ; -else if (shift <PGDIR_SHIFT ) - pdshift =PUD_SHIFT ; -else - pdshift =PGDIR_SHIFT ; -#endif -if (add_huge_page_size (1ULL << shift) <0 )continue ; -/* -* if we have pdshift and shift value same, we don't -* use pgt cache for hugepd. -*/ -if (pdshift > shift) { -if (!IS_ENABLED (CONFIG_PPC_8xx)) -pgtable_cache_add (pdshift - shift); - } configured = true; }
arch/powerpc/mm/pgtable.c +1 -1
@@ -331,7 +331,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, __set_huge_pte_at(pmdp, ptep, pte_val(pte)); } } -#elif defined(CONFIG_PPC_E500) +#else void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz) {
arch/powerpc/platforms/Kconfig.cputype +0 -1
@@ -98,7 +98,6 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD
Signature · 1 line
2.44.0
RFC PATCH v3 15/16 powerpc/mm: Remove hugepd leftovers
In reply to Christophe LeroyAll targets have now opted out of CONFIG_ARCH_HAS_HUGEPD so remove left over code.
--- arch/powerpc/include/asm/hugetlb.h | 7 - arch/powerpc/include/asm/page.h | 6 - arch/powerpc/include/asm/pgtable-be-types.h | 10 - arch/powerpc/include/asm/pgtable-types.h | 9 - arch/powerpc/mm/hugetlbpage.c | 412 -------------------- arch/powerpc/mm/init-common.c | 8 +- arch/powerpc/mm/pgtable.c | 27 +- 7 files changed, 3 insertions(+), 476 deletions(-)
arch/powerpc/include/asm/hugetlb.h +0 -7
@@ -30,13 +30,6 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, }#define is_hugepage_only_range is_hugepage_only_range -#ifdef CONFIG_ARCH_HAS_HUGEPD -#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE -void hugetlb_free_pgd_range (struct mmu_gather *tlb,unsigned long addr, -unsigned long end,unsigned long floor, -unsigned long ceiling); -#endif -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_ATvoid set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz);
arch/powerpc/include/asm/page.h +0 -6
@@ -269,12 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -/* -* Some number of bits at the level of the page table that points to -* a hugepte are used to encode the size. This masks those bits. -*/ -#define HUGEPD_SHIFT_MASK 0x3f -#ifndef __ASSEMBLY__#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/include/asm/pgtable-be-types.h +0 -10
@@ -101,14 +101,4 @@ static inline bool pmd_xchg(pmd_t *pmdp, pmd_t old, pmd_t new) return pmd_raw(old) == prev; } -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct {__be64 pdbe; }hugepd_t ; -#define __hugepd (x) ((hugepd_t) { cpu_to_be64(x) }) - -static inline unsigned long hpd_val (hugepd_t x) -{ -return be64_to_cpu (x.pdbe); -} -#endif -#endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */
arch/powerpc/include/asm/pgtable-types.h +0 -9
@@ -87,13 +87,4 @@ static inline bool pte_xchg(pte_t *ptep, pte_t old, pte_t new) }#endif -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct {unsigned long pd; }hugepd_t ; -#define __hugepd (x) ((hugepd_t) { (x) }) -static inline unsigned long hpd_val (hugepd_t x) -{ -return x.pd; -} -#endif -#endif /* _ASM_POWERPC_PGTABLE_TYPES_H */
arch/powerpc/mm/hugetlbpage.c +0 -412
430 lines not on this page · raw message
arch/powerpc/mm/init-common.c +2 -6
@@ -120,12 +120,8 @@ void pgtable_cache_add(unsigned int shift)/* When batching pgtable pointers for RCU freeing, we store * the index size in the low bits. Table alignment must be * big enough to fit it. -* -* Likewise, hugeapge pagetable pointers contain a (different) -* shift value in the low bits. All tables must be aligned so -* as to leave enough 0 bits in the address to contain it. */ -unsigned long minalign =max (MAX_PGTABLE_INDEX_SIZE +1 , -HUGEPD_SHIFT_MASK +1 ); +*/ +unsigned long minalign =MAX_PGTABLE_INDEX_SIZE +1 ;struct kmem_cache *new =NULL ;/* It would be nice if this was a BUILD_BUG_ON(), but at the
arch/powerpc/mm/pgtable.c +1 -26
@@ -409,11 +409,10 @@ unsigned long vmalloc_to_phys(void *va)EXPORT_SYMBOL_GPL (vmalloc_to_phys);/* -* We have 4 cases for pgds and pmds: +* We have 3 cases for pgds and pmds: * (1) invalid (all zeroes) * (2) pointer to next table, as normal; bottom 6 bits == 0 * (3) leaf pte for huge page _PAGE_PTE set -* (4) hugepd pointer, _PAGE_PTE = 0 and bits [2..6] indicate size of table * * So long as we atomically load page table pointers we are safe against teardown, * we can follow the address down to the page and take a ref on it.
@@ -430,7 +429,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,#endif pmd_t pmd, *pmdp;pte_t *ret_pte; -hugepd_t *hpdp =NULL ;unsigned pdshift;if (hpage_shift)
@@ -463,11 +461,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (p4d_val (p4d)))) { - hpdp = (hugepd_t *)&p4d; - goto out_huge; - } -/* * Even if we end up with an unmap, the pgtable will not * be freed, because we do an rcu free and here we are
@@ -485,11 +478,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (pud_val (pud)))) { - hpdp = (hugepd_t *)&pud; - goto out_huge; - } - pdshift =PMD_SHIFT ; pmdp =pmd_offset (&pud, ea);#else
@@ -527,21 +515,8 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (pmd_val (pmd)))) { - hpdp = (hugepd_t *)&pmd; - goto out_huge; - } -return pte_offset_kernel (&pmd, ea); -out_huge: -if (!hpdp) -return NULL ; - -#ifdef CONFIG_ARCH_HAS_HUGEPD - ret_pte =hugepte_offset (*hpdp, ea, pdshift); - pdshift =hugepd_shift (*hpdp); -#endif out:if (hpage_shift) *hpage_shift = pdshift;
Signature · 1 line
2.44.0
RFC PATCH v3 16/16 mm: Remove CONFIG_ARCH_HAS_HUGEPD
In reply to Christophe Leroypowerpc was the only user of CONFIG_ARCH_HAS_HUGEPD and doesn't use it anymore, so remove all related code.
--- arch/powerpc/mm/hugetlbpage.c | 1 - include/linux/hugetlb.h | 6 -- mm/Kconfig | 10 ---- mm/gup.c | 105 +--------------------------------- mm/pagewalk.c | 57 ++---------------- 5 files changed, 5 insertions(+), 174 deletions(-)
arch/powerpc/mm/hugetlbpage.c +0 -1
@@ -78,7 +78,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,return pte_alloc_huge (mm, pmd, addr); } -#endif #ifdef CONFIG_PPC_BOOK3S_64 /*
include/linux/hugetlb.h +0 -6
@@ -20,12 +20,6 @@ struct user_struct;struct mmu_gather ;struct node ; -#ifndef CONFIG_ARCH_HAS_HUGEPD -typedef struct {unsigned long pd; }hugepd_t ; -#define is_hugepd (hugepd) (0) -#define __hugepd (x) ((hugepd_t) { (x) }) -#endif -void free_huge_folio (struct folio *folio);#ifdef CONFIG_HUGETLB_PAGE
mm/Kconfig +0 -10
@@ -1114,16 +1114,6 @@ config DMAPOOL_TEST config ARCH_HAS_PTE_SPECIAL bool -# -# Some architectures require a special hugepage directory format that is -# required to support multiple hugepage sizes. For example a4fe3ce76 -# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" -# introduced it on powerpc. This allows for a more flexible hugepage -# pagetable layouts. -# -config ARCH_HAS_HUGEPD - bool - config MAPPING_DIRTY_HELPERS bool
mm/gup.c +1 -104
135 lines not on this page · raw message
mm/pagewalk.c +4 -53
87 lines not on this page · raw message
Signature · 1 line
2.44.0
Oscar Salvador
maintainerQuoted 43 lines · Christophe Leroy
This is the continuation of the RFC v1 series "Reimplement huge pages without hugepd on powerpc 8xx". It now get rid of hugepd completely after handling also e500 and book3s/64
Also see https://github.com/linuxppc/issues/issues/483
Unlike most architectures, powerpc 8xx HW requires a two-level pagetable topology for all page sizes. So a leaf PMD-contig approach is not feasible as such.
Possible sizes on 8xx are 4k, 16k, 512k and 8M.
First level (PGD/PMD) covers 4M per entry. For 8M pages, two PMD entries must point to a single entry level-2 page table. Until now that was done using hugepd. This series changes it to use standard page tables where the entry is replicated 1024 times on each of the two pagetables refered by the two associated PMD entries for that 8M page.
For e500 and book3s/64 there are less constraints because it is not tied to the HW assisted tablewalk like on 8xx, so it is easier to use leaf PMDs (and PUDs).
On e500 the supported page sizes are 4M, 16M, 64M, 256M and 1G. All at PMD level on e500/32 (mpc85xx) and mix of PMD and PUD for e500/64. We encode page size with 4 available bits in PTE entries. On e300/32 PGD entries size is increases to 64 bits in order to allow leaf-PMD entries because PTE are 64 bits on e500.
On book3s/64 only the hash-4k mode is concerned. It supports 16M pages as cont-PMD and 16G pages as cont-PUD. In other modes (radix-4k, radix-6k and hash-64k) the sizes match with PMD and PUD sizes so that's just leaf entries. The hash processing make things a bit more complex. To ease things,
__hash_page_huge()is modified to bail out when DIRTY or ACCESSED bits are missing, leaving it to mm core to fix it.Global changes in v3:
- Removed patches 1 and 2
- Squashed patch 11 into patch 5
- Replaced patches 12 and 13 with a series from Michael
- Reordered patches a bit to have more general patches up frontFor more details on changes, see in each patch.
Quoted 16 lines
Christophe Leroy (15):
mm: Define__pte_leaf_size()to also take a PMD entry
mm: Providemm_structand address tohuge_ptep_get()powerpc/mm: Remove_PAGE_PSIZE
powerpc/mm: Fix__find_linux_pte()on 32 bits with PMD leaf entries powerpc/mm: Allow hugepages without hugepd
powerpc/8xx: Fix size given toset_huge_pte_at()powerpc/8xx: Rework support for 8M pages using contiguous PTE entries powerpc/8xx: Simplify structmmu_psize_def
powerpc/e500: Remove enc and ind fields from structmmu_psize_defpowerpc/e500: Switch to 64 bits PGD on 85xx (32 bits) powerpc/e500: Encode hugepage size in PTE bits powerpc/e500: Use contiguous PMD instead of hugepd powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD powerpc/mm: Remove hugepd leftovers
mm: RemoveCONFIG_ARCH_HAS_HUGEPD
I glanced over it and it looks much better, not having to fiddle with other arch code and generic declarations is a big plus.
I plan to do a proper review tomorrow.
Thanks for working on this Christophe!
Signature · 2 lines
Oscar Salvador SUSE Labs