Reimplement huge pages without hugepd on powerpc (8xx, e500, book3s/64)
Subsystems (22)
v5 cover Reimplement huge pages without hugepd on powerpc (8xx, e500, book3s/64)
This series has reached maturity to not send it as an RFC anymore. Only the book3s/64 part maybe needs more attention. Alternatively we could simply disable HUGE pages on book3s/64 in hash-4k mode if we want to be on the safe side.
Also see https://github.com/linuxppc/issues/issues/483
Unlike most architectures, powerpc 8xx HW requires a two-level pagetable topology for all page sizes. So a leaf PMD-contig approach is not feasible as such.
Possible sizes on 8xx are 4k, 16k, 512k and 8M.
First level (PGD/PMD) covers 4M per entry. For 8M pages, two PMD entries must point to a single entry level-2 page table. Until now that was done using hugepd. This series changes it to use standard page tables where the entry is replicated 1024 times on each of the two pagetables refered by the two associated PMD entries for that 8M page.
For e500 and book3s/64 there are less constraints because it is not tied to the HW assisted tablewalk like on 8xx, so it is easier to use leaf PMDs (and PUDs).
On e500 the supported page sizes are 4M, 16M, 64M, 256M and 1G. All at PMD level on e500/32 (mpc85xx) and mix of PMD and PUD for e500/64. We encode page size with 4 available bits in PTE entries. On e300/32 PGD entries size is increases to 64 bits in order to allow leaf-PMD entries because PTE are 64 bits on e500.
On book3s/64 only the hash-4k mode is concerned. It supports 16M pages as cont-PMD and 16G pages as cont-PUD. In other modes (radix-4k, radix-6k and hash-64k) the sizes match with PMD and PUD sizes so that's just leaf entries. The hash processing make things a bit more complex. To ease things, __hash_page_huge() is modified to bail out when DIRTY or ACCESSED bits are missing, leaving it to mm core to fix it.
Global changes in v5:
- Now use PAGE SIZE field in e500's PTE to store TSIZE instead of using U0-U3
- On e500/64, use highest bit to discriminate leaf entries because PUD entries are not garantied to be 4k aligned so PAGE SIZE field is not garantied to be 0 on a non-leaf entry.
Global changes in v4:
- Fixed a few issues reported privately by robots
- Rebased on top of v6.10-rc1
Global changes in v3:
- Removed patches 1 and 2
- Squashed patch 11 into patch 5
- Replaced patches 12 and 13 with a series from Michael
- Reordered patches a bit to have more general patches up front
For more details on changes, see in each patch.
Christophe Leroy (17):
mm: Define __pte_leaf_size() to also take a PMD entry
mm: Provide mm_struct and address to huge_ptep_get() powerpc/mm: Remove _PAGE_PSIZE
powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries powerpc/mm: Allow hugepages without hugepd
powerpc/8xx: Fix size given to set_huge_pte_at() powerpc/8xx: Rework support for 8M pages using contiguous PTE entries powerpc/8xx: Simplify struct mmu_psize_def
powerpc/e500: Remove enc and ind fields from struct mmu_psize_def powerpc/e500: Switch to 64 bits PGD on 85xx (32 bits) powerpc/e500: Encode hugepage size in PTE bits powerpc/e500: Don't pre-check write access on data TLB error powerpc/e500: Free r10 for FIND_PTE
powerpc/e500: Use contiguous PMD instead of hugepd powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD powerpc/mm: Remove hugepd leftovers
mm: Remove CONFIG_ARCH_HAS_HUGEPD
Michael Ellerman (1):
powerpc/64e: Remove unused IBM HTW code [SQUASHED]
arch/arm/include/asm/hugetlb-3level.h | 4 +- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 - arch/powerpc/include/asm/book3s/64/hash-4k.h | 15 - arch/powerpc/include/asm/book3s/64/hash.h | 38 +- arch/powerpc/include/asm/book3s/64/hugetlb.h | 38 -- .../include/asm/book3s/64/pgtable-4k.h | 47 -- .../include/asm/book3s/64/pgtable-64k.h | 20 - arch/powerpc/include/asm/book3s/64/pgtable.h | 22 +- arch/powerpc/include/asm/hugetlb.h | 15 +- .../include/asm/nohash/32/hugetlb-8xx.h | 38 +- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 9 +- arch/powerpc/include/asm/nohash/32/pte-40x.h | 3 - arch/powerpc/include/asm/nohash/32/pte-44x.h | 3 - arch/powerpc/include/asm/nohash/32/pte-85xx.h | 3 - arch/powerpc/include/asm/nohash/32/pte-8xx.h | 58 ++- .../powerpc/include/asm/nohash/hugetlb-e500.h | 39 +- arch/powerpc/include/asm/nohash/mmu-e500.h | 6 +- arch/powerpc/include/asm/nohash/pgalloc.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 46 +- arch/powerpc/include/asm/nohash/pte-e500.h | 63 ++- arch/powerpc/include/asm/page.h | 32 -- arch/powerpc/include/asm/pgtable-be-types.h | 10 - arch/powerpc/include/asm/pgtable-types.h | 13 +- arch/powerpc/include/asm/pgtable.h | 3 + arch/powerpc/kernel/exceptions-64e.S | 4 +- arch/powerpc/kernel/head_85xx.S | 70 +-- arch/powerpc/kernel/head_8xx.S | 10 +- arch/powerpc/kernel/setup_64.c | 6 +- arch/powerpc/mm/book3s64/hash_utils.c | 11 +- arch/powerpc/mm/book3s64/hugetlbpage.c | 10 + arch/powerpc/mm/book3s64/pgtable.c | 12 - arch/powerpc/mm/hugetlbpage.c | 455 +----------------- arch/powerpc/mm/init-common.c | 8 +- arch/powerpc/mm/kasan/8xx.c | 21 +- arch/powerpc/mm/nohash/8xx.c | 43 +- arch/powerpc/mm/nohash/Makefile | 2 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 4 +- arch/powerpc/mm/nohash/tlb.c | 407 +--------------- arch/powerpc/mm/nohash/tlb_64e.c | 314 ++++++++++++ arch/powerpc/mm/nohash/tlb_low_64e.S | 428 +--------------- arch/powerpc/mm/pgtable.c | 94 ++-- arch/powerpc/mm/pgtable_32.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +- arch/s390/mm/hugetlbpage.c | 4 +- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 10 +- fs/userfaultfd.c | 2 +- include/asm-generic/hugetlb.h | 2 +- include/linux/hugetlb.h | 6 - include/linux/pgtable.h | 3 + include/linux/swapops.h | 4 +- kernel/events/core.c | 2 +- mm/Kconfig | 10 - mm/damon/vaddr.c | 6 +- mm/gup.c | 183 +------ mm/hmm.c | 2 +- mm/hugetlb.c | 44 +- mm/memory-failure.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 4 +- mm/mincore.c | 2 +- mm/pagewalk.c | 57 +-- mm/userfaultfd.c | 2 +- 68 files changed, 749 insertions(+), 2043 deletions(-) delete mode 100644 arch/powerpc/include/asm/book3s/64/pgtable-4k.h create mode 100644 arch/powerpc/mm/nohash/tlb_64e.c
Signature · 1 line
2.44.0
PATCH v5 1/18 powerpc/64e: Remove unused IBM HTW code [SQUASHED]
From: Michael Ellerman <[email protected]>
This is a squash of series from Michael https://patchwork.ozlabs.org/project/linuxppc-dev/patch/[email protected]/
The nohash HTW_IBM (Hardware Table Walk) code is unused since support for A2 was removed in commit fb5a515704d7 ("powerpc: Remove platforms/ wsp and associated pieces") (2014).
The remaining supported CPUs use either no HTW (data_tlb_miss_bolted), or the e6500 HTW (data_tlb_miss_e6500).
powerpc/64e: Split out nohash Book3E 64-bit code
A reasonable chunk of nohash/tlb.c is 64-bit only code, split it out into a separate file.
powerpc/64e: Drop E500 ifdefs in 64-bit code
All 64-bit Book3E have E500=y, so drop the unneeded ifdefs.
powerpc/64e: Drop MMU_FTR_TYPE_FSL_E checks in 64-bit code
All 64-bit Book3E have MMU_FTR_TYPE_FSL_E, since A2 was removed, so remove checks for it in 64-bit only code.
powerpc/64e: Consolidate TLB miss handler patching
The 64e TLB miss handler patching is done in setup_mmu_htw(), and then again immediately afterward in early_init_mmu_global(). Consolidate it into a single location.
powerpc/64e: Drop unused TLB miss handlers
There are two possibilities for book3e_htw_mode, PPC_HTW_E6500 or PPC_HTW_NONE.
The TLB miss handlers are patched to use, respectively:
- exc_[data|indstruction]_tlb_miss_e6500_book3e
- exc_[data|indstruction]_tlb_miss_bolted_book3e
Which means the default handlers are never used. Remove those, and use the bolted handlers (PPC_HTW_NONE) by default.
--- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 +- arch/powerpc/kernel/exceptions-64e.S | 4 +- arch/powerpc/kernel/setup_64.c | 6 +- arch/powerpc/mm/nohash/Makefile | 2 +- arch/powerpc/mm/nohash/tlb.c | 398 +------------------ arch/powerpc/mm/nohash/tlb_64e.c | 314 +++++++++++++++ arch/powerpc/mm/nohash/tlb_low_64e.S | 421 --------------------- 7 files changed, 320 insertions(+), 828 deletions(-) create mode 100644 arch/powerpc/mm/nohash/tlb_64e.c
arch/powerpc/include/asm/nohash/mmu-e500.h +1 -2
@@ -303,8 +303,7 @@ extern unsigned long linear_map_top;extern int book3e_htw_mode;#define PPC_HTW_NONE 0 -#define PPC_HTW_IBM 1 -#define PPC_HTW_E6500 2 +#define PPC_HTW_E6500 1/* * 64-bit booke platforms don't load the tlb in the tlb miss handler code.
arch/powerpc/kernel/exceptions-64e.S +2 -2
@@ -485,8 +485,8 @@ interrupt_base_book3e: /* fake trap */ EXCEPTION_STUB(0x160, decrementer) /* 0x0900 */ EXCEPTION_STUB(0x180, fixed_interval) /* 0x0980 */ EXCEPTION_STUB(0x1a0, watchdog) /* 0x09f0 */ - EXCEPTION_STUB(0x1c0, data_tlb_miss) - EXCEPTION_STUB(0x1e0, instruction_tlb_miss) + EXCEPTION_STUB(0x1c0, data_tlb_miss_bolted) + EXCEPTION_STUB(0x1e0, instruction_tlb_miss_bolted) EXCEPTION_STUB(0x200, altivec_unavailable) EXCEPTION_STUB(0x220, altivec_assist) EXCEPTION_STUB(0x260, perfmon)
arch/powerpc/kernel/setup_64.c +1 -5
@@ -696,11 +696,7 @@ __init u64 ppc64_bolted_size(void) {#ifdef CONFIG_PPC_BOOK3E_64 /* Freescale BookE bolts the entire linear mapping */ -/* XXX: BookE ppc64_rma_limit setup seems to disagree? */ -if (early_mmu_has_feature (MMU_FTR_TYPE_FSL_E )) -return linear_map_top; -/* Other BookE, we assume the first GB is bolted */ -return 1ul <<30 ; +return linear_map_top;#else /* BookS radix, does not take faults on linear mapping */ if (early_radix_enabled ())
arch/powerpc/mm/nohash/Makefile +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 obj-y += mmu_context.o tlb.o tlb_low.o kup.o -obj-$(CONFIG_PPC_BOOK3E_64) += tlb_low_64e.o book3e_pgtable.o +obj-$(CONFIG_PPC_BOOK3E_64) += tlb_64e.o tlb_low_64e.o book3e_pgtable.o obj-$(CONFIG_40x) += 40x.o obj-$(CONFIG_44x) += 44x.o obj-$(CONFIG_PPC_8xx) += 8xx.o
arch/powerpc/mm/nohash/tlb.c +1 -397
410 lines not on this page · raw message
arch/powerpc/mm/nohash/tlb_64e.c new file +314 -0
314 lines not on this page · raw message
arch/powerpc/mm/nohash/tlb_low_64e.S +0 -421
433 lines not on this page · raw message
Signature · 1 line
2.44.0
PATCH v5 2/18 mm: Define __pte_leaf_size() to also take a PMD entry
In reply to Christophe LeroyOn powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide __pte_leaf_size() instead of pte_leaf_size() and provide the PMD entry to that function.
When __pte_leaf_size() is not defined, define it as a pte_leaf_size() so that architectures not interested in the PMD arguments are not impacted.
Only define a default pte_leaf_size() when __pte_leaf_size() is not defined to make sure nobody adds new calls to pte_leaf_size() in the core.
---
v3: Don't change pte_leaf_size() to not impact other architectures
--- include/linux/pgtable.h | 3 +++ kernel/events/core.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/pgtable.h +3 -0
@@ -1888,9 +1888,12 @@ typedef unsigned int pgtbl_mod_mask;#ifndef pmd_leaf_size#define pmd_leaf_size (x) PMD_SIZE#endif +#ifndef __pte_leaf_size#ifndef pte_leaf_size#define pte_leaf_size (x) PAGE_SIZE#endif +#define __pte_leaf_size (x,y) pte_leaf_size(y) +#endif /* * We always define pmd_pfn for all archs as it's used in lots of generic
kernel/events/core.c +1 -1
@@ -7596,7 +7596,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) pte =ptep_get_lockless (ptep);if (pte_present (pte)) - size =pte_leaf_size (pte); + size =__pte_leaf_size (pmd, pte);pte_unmap (ptep);#endif /* CONFIG_HAVE_GUP_FAST */
Signature · 1 line
2.44.0
Oscar Salvador
maintainerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 11 lines · Christophe Leroy
On powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide
__pte_leaf_size()instead ofpte_leaf_size()and provide the PMD entry to that function.When
__pte_leaf_size()is not defined, define it as apte_leaf_size()so that architectures not interested in the PMD arguments are not impacted.Only define a default
pte_leaf_size()when__pte_leaf_size()is not defined to make sure nobody adds new calls topte_leaf_size()in the core.
Hi Christophe,
Now I am going to give you a hard time, so sorry in advance. I should have raised this before, but I was not fully aware of it.
There is an ongoing effort of unifying pagewalkers [1], so hugetlb does not have to be special-cased anymore, and the operations we do for THP on page-table basis work for hugetlb as well.
The most special bit about this is huge_ptep_get. huge_ptep_get() gets special handled on arm/arm64/riscv and s390.
arm64 and riscv is about cont-pmd/pte and propagate the dirty/young bits bits, so that is fine as walkers can already understand that.
s390 is a funny one because it converts pud/pmd to pte and viceversa, because hugetlb *works* with ptes, so before returning the pte it has to transfer all bits from PUD/PMD level into a something that PTE level can understand. As you can imagine, this can be gone as we already have all the information in PUD/PMD and that is all pagewalkers need.
But we are left with the one you will introduce in patch#8.
8MB pages get mapped as cont-pte, but all the information is stored in the PMD entries (size, dirtiness, present etc).huge_ptep_get() will return the PMD for 8MB, and so all operations hugetlb code performs with what huge_ptep_get returns will be performed on those PMDs.
Which brings me to this point:
I do not think __pte_leaf_size is needed. AFAICS, it should be enough to define pmd_leaf on 8xx, and return 8MB if it is a 8MB hugepage.
#define pmd_leaf pmd_leaf
static inline bool pmd_leaf(pmd_t pmd)
{
return pmd_val(pmd) & _PMD_PAGE_8M);
} and then pmd_leaf_size to return _PMD_PAGE_8M.
This will help because on the ongoing effort of unifying hugetlb and getting rid of huge_ptep_get() [1], pagewalkers will stumble upon the 8mb-PMD as they do for regular PMDs.
Which means that they would be caught in the following code:
ptl = pmd_huge_lock(pmd, vma);
if (ptl) {
- 8MB hugepages will be handled here
smaps_pmd_entry(pmd, addr, walk);
spin_unlock(ptl);
}
/* pte stuff */
...where pmd_huge_lock is:
static inline spinlock_t *pmd_huge_lock(pmd_t *pmd, struct vm_area_struct *vma)
{
spinlock_t *ptl = pmd_lock(vma->vm_mm, pmd);
if (pmd_leaf(*pmd) || pmd_devmap(*pmd))
return ptl;
spin_unlock(ptl);
return NULL;}
So, since pmd_leaf() will return true for 8MB hugepages, we are fine, because anyway we want to perform pagetable operations on *that* PMD and not the ptes that are cont-mapped, which is different for e.g: 512K hugepages, where we perform it on pte level.
So I would suggest that instead of this patch, we have one implementing pmd_leaf and pmd_leaf_size for 8Mb hugepages on power8xx, as that takes us closer to our goal of unifying hugetlb.
[1] https://github.com/leberus/linux/tree/hugetlb-pagewalk-v2
Signature · 2 lines
Oscar Salvador SUSE Labs
Peter Xu
reviewerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Oscar,
Quoted 10 lines · Oscar Salvador
Which means that they would be caught in the following code:
ptl = pmd_huge_lock(pmd, vma); if (ptl) { - 8MB hugepages will be handled here smaps_pmd_entry(pmd, addr, walk); spin_unlock(ptl); } /* pte stuff */ ...
Just one quick comment: I think there's one challenge though as this is also not a generic "pmd leaf", but a pgtable page underneath. I think it means smaps_pmd_entry() won't trivially work here, e.g., it will start to do this:
if (pmd_present(*pmd)) {
page = vm_normal_page_pmd(vma, addr, *pmd);Here vm_normal_page_pmd() will only work if pmd_leaf() satisfies its definition as:
* - It should contain a huge PFN, which points to a huge page larger than
* PAGE_SIZE of the platform. The PFN format isn't important here.
But now it's a pgtable page, containing cont-ptes. Similarly, I think most pmd_*() helpers will stop working there if we report it as a leaf.
Thanks,
Signature · 1 line
Peter Xu
Oscar Salvador
maintainerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 22 lines · Peter Xu
Oscar,
Quoted 10 lines · Oscar Salvador
Which means that they would be caught in the following code:
ptl = pmd_huge_lock(pmd, vma); if (ptl) { - 8MB hugepages will be handled here smaps_pmd_entry(pmd, addr, walk); spin_unlock(ptl); } /* pte stuff */ ...Just one quick comment: I think there's one challenge though as this is also not a generic "pmd leaf", but a pgtable page underneath. I think it means
smaps_pmd_entry()won't trivially work here, e.g., it will start to do this:if (pmd_present(*pmd)) { page = vm_normal_page_pmd(vma, addr, *pmd);
Quoted 8 lines
Here
vm_normal_page_pmd()will only work ifpmd_leaf()satisfies its definition as:* - It should contain a huge PFN, which points to a huge page larger than
*PAGE_SIZEof the platform. The PFN format isn't important here.But now it's a pgtable page, containing cont-ptes. Similarly, I think most pmd_*() helpers will stop working there if we report it as a leaf.
Heh, I think I managed to confuse myself.
I do not why but I thought that
static inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
if (ptep_is_8m_pmdp(mm, addr, ptep))
ptep = pte_offset_kernel((pmd_t *)ptep, 0);
return ptep_get(ptep);}
would return the address of the pmd for 8MB hugepages, but it will return the address of the first pte?
Then yeah, this will not work as I thought.
The problem is that we do not have spare bits for 8xx to mark these ptes as cont-ptes or mark them pte as 8MB, so I do not see a clear path on how we could remove huge_ptep_get for 8xx.
I am really curious though how we handle that for THP? Or THP on 8xx does not support that size?
Signature · 2 lines
Oscar Salvador SUSE Labs
Peter Xu
reviewerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 3 lines · Oscar Salvador
The problem is that we do not have spare bits for 8xx to mark these ptes as cont-ptes or mark them pte as 8MB, so I do not see a clear path on how we could remove
huge_ptep_getfor 8xx.
Right, I remember I thought about this too when I initially looked at one previous version of the series, I didn't come up yet with a good solution, but I guess we probably need to get rid of hugepd first anyway. We may somehow still need to identify this is a 8M large leaf, and I guess this is again the only special case where contpte can go over >1 pmds.
Quoted 3 lines
I am really curious though how we handle that for THP? Or THP on 8xx does not support that size?
I'll leave this to Christophe, but IIUC thp is only PMD_ORDER sized, so shouldn't apply to the 8MB pages.
Thanks,
Signature · 1 line
Peter Xu
Oscar Salvador
maintainerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 10 lines · Peter Xu
Quoted 3 lines · Oscar Salvador
The problem is that we do not have spare bits for 8xx to mark these ptes as cont-ptes or mark them pte as 8MB, so I do not see a clear path on how we could remove
huge_ptep_getfor 8xx.Right, I remember I thought about this too when I initially looked at one previous version of the series, I didn't come up yet with a good solution, but I guess we probably need to get rid of hugepd first anyway. We may somehow still need to identify this is a 8M large leaf, and I guess this is again the only special case where contpte can go over >1 pmds.
Yes, we definitely need first to get rid of hugepd, which is a huge step, and one step closer to our goal, but at some point we will have to see what can we do about 8MB cont-ptes for 8xx and how to mark them, so ptep_get can work the same way as e.g: arm64 (ptep_get->contpte_ptep_get).
@Christophe: Can you think of a way to flag those ptes? (e.g: a combination of bits etc)
Quoted 2 lines
I'll leave this to Christophe, but IIUC thp is only
PMD_ORDERsized, so shouldn't apply to the 8MB pages.
That might be it, yes.
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Le 11/06/2024 à 18:10, Oscar Salvador a écrit :
Quoted 14 lines
[Vous ne recevez pas souvent de courriers de [email protected]. D?couvrez pourquoi ceci est important ? https://aka.ms/LearnAboutSenderIdentification ]
Quoted 10 lines · Peter Xu
Quoted 3 lines · Oscar Salvador
The problem is that we do not have spare bits for 8xx to mark these ptes as cont-ptes or mark them pte as 8MB, so I do not see a clear path on how we could remove
huge_ptep_getfor 8xx.Right, I remember I thought about this too when I initially looked at one previous version of the series, I didn't come up yet with a good solution, but I guess we probably need to get rid of hugepd first anyway. We may somehow still need to identify this is a 8M large leaf, and I guess this is again the only special case where contpte can go over >1 pmds.
Quoted 8 lines
Yes, we definitely need first to get rid of hugepd, which is a huge step, and one step closer to our goal, but at some point we will have to see what can we do about 8MB cont-ptes for 8xx and how to mark them, so
ptep_getcan work the same way as e.g: arm64 (ptep_get->contpte_ptep_get).@Christophe: Can you think of a way to flag those ptes? (e.g: a combination of bits etc)
We have space available in PMD if we need more flags, but in PTE I can't see anything possible without additional churn that would require additional instructions in TLB miss handlers, which is what I want to avoid most.
Bits mapped to HW PTE:
#define _PAGE_PRESENT 0x0001 /* V: Page is valid */
#define _PAGE_NO_CACHE 0x0002 /* CI: cache inhibit */
#define _PAGE_SH 0x0004 /* SH: No ASID (context) compare */
#define _PAGE_SPS 0x0008 /* SPS: Small Page Size (1 if 16k, 512k or 8M)*/
#define _PAGE_DIRTY 0x0100 /* C: page changed */
#define _PAGE_NA 0x0200 /* Supervisor NA, User no access */
#define _PAGE_RO 0x0600 /* Supervisor RO, User no access */
SW bits masked out in TLB miss handler:
#define _PAGE_GUARDED 0x0010 /* Copied to L1 G entry in DTLB */
#define _PAGE_ACCESSED 0x0020 /* Copied to L1 APG 1 entry in I/DTLB */
#define _PAGE_EXEC 0x0040 /* Copied to PP (bit 21) in ITLB */
#define _PAGE_SPECIAL 0x0080 /* SW entry */
#define _PAGE_HUGE 0x0800 /* Copied to L1 PS bit 29 */
All bits are used. The only thing would could do but that would have a performance cost is to retrieve _PAGE_SH from the PMD and use that bit for something else.
But I was maybe thinking another way. Lets take the exemple of pmd_write() helper:
#define pmd_write(pmd) pte_write(pmd_pte(pmd))
At the time being we have
static inline pte_t pmd_pte(pmd_t pmd)
{
return __pte(pmd_val(pmd));
}But what about something like
static inline pte_t pmd_pte(pmd_t pmd)
{
return *(pte_t *)pmd_page_vaddr(pmd);
}Would it do the trick ?
Of course it would require to carefully make sure all accesses are done through pmd_pte().
Would that work ?
Christophe
Peter Xu
reviewerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 7 lines · LEROY Christophe
But what about something like
static inline
pte_tpmd_pte(pmd_t pmd){ return *(pte_t *)pmd_page_vaddr(pmd); }
Quoted 6 lines
Would it do the trick ?
Of course it would require to carefully make sure all accesses are done through
pmd_pte().Would that work ?
Looks possible to me. It's just that we may miss some spots, and it can hide in the details.
I'm looking at Power's pmd_access_permitted() right below pmd_write(), which indeed uses pmd_pte() already. However before that there's also the other call to pmd_is_serializing(), which doesn't look alright to work on pgtable pages.. In this case maybe it's easy, as I assume pgtable page is stable. Didn't further look, though.
Said that, this doesn't look like a blocker. So maybe worth trying if we're careful and with some good testing coverages.
Thanks,
Signature · 1 line
Peter Xu
Oscar Salvador
maintainerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
In reply to LEROY ChristopheQuoted 23 lines · LEROY Christophe
We have space available in PMD if we need more flags, but in PTE I can't see anything possible without additional churn that would require additional instructions in TLB miss handlers, which is what I want to avoid most.
Bits mapped to HW PTE:
#define
_PAGE_PRESENT0x0001/* V: Page is valid */
#define_PAGE_NO_CACHE0x0002/* CI: cache inhibit */
#define_PAGE_SH0x0004/* SH: No ASID (context) compare */
#define_PAGE_SPS0x0008/* SPS: Small Page Size (1 if 16k, 512k or 8M)*/
#define_PAGE_DIRTY0x0100/* C: page changed */
#define_PAGE_NA0x0200/* Supervisor NA, User no access */
#define_PAGE_RO0x0600/* Supervisor RO, User no access */SW bits masked out in TLB miss handler:
#define
_PAGE_GUARDED0x0010/* Copied to L1 G entry in DTLB */
#define_PAGE_ACCESSED0x0020/* Copied to L1 APG 1 entry in I/DTLB */
#define_PAGE_EXEC0x0040/* Copied to PP (bit 21) in ITLB */
#define_PAGE_SPECIAL0x0080/* SW entry */
#define_PAGE_HUGE0x0800/* Copied to L1 PS bit 29 */
Quoted 3 lines
All bits are used. The only thing would could do but that would have a performance cost is to retrieve
_PAGE_SHfrom the PMD and use that bit for something else.
I guess that this would be the last resort if we run out of options. But at least it is good to know that there is a plan B (or Z if you will :-))
Quoted 12 lines
But I was maybe thinking another way. Lets take the exemple of
pmd_write()helper:#define
pmd_write(pmd)pte_write(pmd_pte(pmd))At the time being we have
static inline
pte_tpmd_pte(pmd_t pmd){ return __pte(pmd_val(pmd)); }
Quoted 6 lines
But what about something like
static inline
pte_tpmd_pte(pmd_t pmd){ return *(pte_t *)pmd_page_vaddr(pmd); }
I think this could work, yes.
So, we should define all pmd_*(pmd) operations for 8xx the way they are defined in include/asm/book3s/64/pgtable.h.
Other page size would not interfere because they already can perform operations on pte level.
Ok, I think we might have a shot here.
I would help testing, but I do not have 8xx hardware, and Qemu does not support 8xx emulation, but I think that if we are careful enough, this can work.
Actually, as a smoketest would be enough to have a task with a 8MB huge mapped, and then do:
static const struct mm_walk_ops test_walk_ops = {
.pmd_entry = test_8mbp_hugepage, .pte_entry = test_16k_and_512k_hugepage, .hugetlb_entry = check_hugetlb_entry, .walk_lock = PGWALK_RDLOCK,
};
static int test(void)
{
pr_info("%s: %s [0 - %lx]\n", __func__, current->comm, TASK_SIZE);
mmap_read_lock(current->mm);
ret = walk_page_range(current->mm, 0, TASK_SIZE, &test_walk_ops, NULL);
mmap_read_unlock(current->mm);
pr_info("%s: %s ret: %d\n", __func__, current->comm, ret);
return 0;}
This is an extract of a debugging mechanism I have to check that I am not going off rails when unifying hugetlb and normal walkers.
test_8mbp_hugepage() could so some checks with pmd_ operations, print the results, and then compare them with those that check_hugetlb_entry() would give us.
If everything is alright, both results should be the same.
I can write the tests up, so we run some sort of smoketests.
So yes, I do think that this is a good initiative.
Thanks a lot Christoph
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Le 13/06/2024 à 09:19, Oscar Salvador a écrit :
Quoted 90 lines · LEROY Christophe
Quoted 23 lines
We have space available in PMD if we need more flags, but in PTE I can't see anything possible without additional churn that would require additional instructions in TLB miss handlers, which is what I want to avoid most.
Bits mapped to HW PTE:
#define
_PAGE_PRESENT0x0001/* V: Page is valid */
#define_PAGE_NO_CACHE0x0002/* CI: cache inhibit */
#define_PAGE_SH0x0004/* SH: No ASID (context) compare */
#define_PAGE_SPS0x0008/* SPS: Small Page Size (1 if 16k, 512k or 8M)*/
#define_PAGE_DIRTY0x0100/* C: page changed */
#define_PAGE_NA0x0200/* Supervisor NA, User no access */
#define_PAGE_RO0x0600/* Supervisor RO, User no access */SW bits masked out in TLB miss handler:
#define
_PAGE_GUARDED0x0010/* Copied to L1 G entry in DTLB */
#define_PAGE_ACCESSED0x0020/* Copied to L1 APG 1 entry in I/DTLB */
#define_PAGE_EXEC0x0040/* Copied to PP (bit 21) in ITLB */
#define_PAGE_SPECIAL0x0080/* SW entry */
#define_PAGE_HUGE0x0800/* Copied to L1 PS bit 29 */Quoted 3 lines
All bits are used. The only thing would could do but that would have a performance cost is to retrieve
_PAGE_SHfrom the PMD and use that bit for something else.I guess that this would be the last resort if we run out of options. But at least it is good to know that there is a plan B (or Z if you will :-))
Quoted 12 lines
But I was maybe thinking another way. Lets take the exemple of
pmd_write()helper:#define
pmd_write(pmd)pte_write(pmd_pte(pmd))At the time being we have
static inline
pte_tpmd_pte(pmd_t pmd){ return __pte(pmd_val(pmd)); }Quoted 6 lines
But what about something like
static inline
pte_tpmd_pte(pmd_t pmd){ return *(pte_t *)pmd_page_vaddr(pmd); }I think this could work, yes.
So, we should define all pmd_*(pmd) operations for 8xx the way they are defined in
include/asm/book3s/64/pgtable.h.Other page size would not interfere because they already can perform operations on pte level.
Ok, I think we might have a shot here.
I would help testing, but I do not have 8xx hardware, and Qemu does not support 8xx emulation, but I think that if we are careful enough, this can work.
Actually, as a smoketest would be enough to have a task with a 8MB huge mapped, and then do:
static const struct mm_walk_ops test_walk_ops = { .pmd_entry = test_8mbp_hugepage, .pte_entry = test_16k_and_512k_hugepage, .hugetlb_entry = check_hugetlb_entry, .walk_lock = PGWALK_RDLOCK, }; static int test(void) {pr_info("%s: %s [0 - %lx]\n", __func__, current->comm, TASK_SIZE); mmap_read_lock(current->mm); ret = walk_page_range(current->mm, 0, TASK_SIZE, &test_walk_ops, NULL); mmap_read_unlock(current->mm); pr_info("%s: %s ret: %d\n", __func__, current->comm, ret); return 0;}
This is an extract of a debugging mechanism I have to check that I am not going off rails when unifying hugetlb and normal walkers.
Quoted 9 lines
test_8mbp_hugepage()could so some checks withpmd_operations, print the results, and then compare them with those thatcheck_hugetlb_entry()would give us.
If everything is alright, both results should be the same.I can write the tests up, so we run some sort of smoketests.
So yes, I do think that this is a good initiative.
I can test whatever you want on my 8xx boards.
I have two types of board:
- One with MPC866 microcontroller and 32Mbytes memory
- One with MPC885 microcontroller and 128Mbytes memory
Christophe
Oscar Salvador
maintainerRe: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
Quoted 5 lines · LEROY Christophe
I can test whatever you want on my 8xx boards.
I have two types of board:
- One with MPC866 microcontroller and 32Mbytes memory
- One with MPC885 microcontroller and 128Mbytes memory
That is great.
I will code up some tests, and once you have the pmd_* stuff on 8xx we can give it a shot.
Thanks!
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
In reply to Oscar SalvadorLe 11/06/2024 à 17:08, Oscar Salvador a écrit :
Quoted 47 lines
[Vous ne recevez pas souvent de courriers de [email protected]. D?couvrez pourquoi ceci est important ? https://aka.ms/LearnAboutSenderIdentification ]
Quoted 22 lines · Peter Xu
Oscar,
Quoted 10 lines · Oscar Salvador
Which means that they would be caught in the following code:
ptl = pmd_huge_lock(pmd, vma); if (ptl) { - 8MB hugepages will be handled here smaps_pmd_entry(pmd, addr, walk); spin_unlock(ptl); } /* pte stuff */ ...Just one quick comment: I think there's one challenge though as this is also not a generic "pmd leaf", but a pgtable page underneath. I think it means
smaps_pmd_entry()won't trivially work here, e.g., it will start to do this:if (pmd_present(*pmd)) { page = vm_normal_page_pmd(vma, addr, *pmd);Quoted 8 lines
Here
vm_normal_page_pmd()will only work ifpmd_leaf()satisfies its definition as:* - It should contain a huge PFN, which points to a huge page larger than
*PAGE_SIZEof the platform. The PFN format isn't important here.But now it's a pgtable page, containing cont-ptes. Similarly, I think most pmd_*() helpers will stop working there if we report it as a leaf.
Heh, I think I managed to confuse myself.
I do not why but I thought thatstatic inline pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { if (ptep_is_8m_pmdp(mm, addr, ptep)) ptep = pte_offset_kernel((pmd_t *)ptep, 0); return ptep_get(ptep); }would return the address of the pmd for 8MB hugepages, but it will return the address of the first pte?
Quoted 8 lines
Then yeah, this will not work as I thought.
The problem is that we do not have spare bits for 8xx to mark these ptes as cont-ptes or mark them pte as 8MB, so I do not see a clear path on how we could remove
huge_ptep_getfor 8xx.I am really curious though how we handle that for THP? Or THP on 8xx does not support that size?
8xx doesn't support THP, as far as I know THP is only supported on single leaf PMD/PUD, not on cont-PUD/PMD allthough there is some work in progress on arm64 to add that.
But honestly I'm not too much interested in 8M transparent pages, what I'd like to have first is 512k transparent pages.
Christophe
Re: [PATCH v5 02/18] mm: Define __pte_leaf_size() to also take a PMD entry
In reply to Oscar SalvadorLe 11/06/2024 à 11:34, Oscar Salvador a écrit :
Quoted 35 lines
[Vous ne recevez pas souvent de courriers de [email protected]. D?couvrez pourquoi ceci est important ? https://aka.ms/LearnAboutSenderIdentification ]
Quoted 11 lines · Christophe Leroy
On powerpc 8xx, when a page is 8M size, the information is in the PMD entry. So allow architectures to provide
__pte_leaf_size()instead ofpte_leaf_size()and provide the PMD entry to that function.When
__pte_leaf_size()is not defined, define it as apte_leaf_size()so that architectures not interested in the PMD arguments are not impacted.Only define a default
pte_leaf_size()when__pte_leaf_size()is not defined to make sure nobody adds new calls topte_leaf_size()in the core.Hi Christophe,
Now I am going to give you a hard time, so sorry in advance. I should have raised this before, but I was not fully aware of it.
There is an ongoing effort of unifying pagewalkers [1], so hugetlb does not have to be special-cased anymore, and the operations we do for THP on page-table basis work for hugetlb as well.
The most special bit about this is
huge_ptep_get.huge_ptep_get()gets special handled on arm/arm64/riscv and s390.arm64 and riscv is about cont-pmd/pte and propagate the dirty/young bits bits, so that is fine as walkers can already understand that.
s390 is a funny one because it converts pud/pmd to pte and viceversa, because hugetlb *works* with ptes, so before returning the pte it has to transfer all bits from PUD/PMD level into a something that PTE level can understand. As you can imagine, this can be gone as we already have all the information in PUD/PMD and that is all pagewalkers need.
Quoted 4 lines
But we are left with the one you will introduce in patch#8.
8MB pages get mapped as cont-pte, but all the information is stored in the PMD entries (size, dirtiness, present etc).
I'm not sure I understand what you mean.
In my case, the PMD entry is almost standard, the only thing it contains is a bit telling that the pointed PTEs are to be mapped 8M.
Quoted 2 lines
huge_ptep_get()will return the PMD for 8MB, and so all operations hugetlb code performs with whathuge_ptep_getreturns will be performed on those PMDs.
Indeed no, my huge_ptep_get() doesn't return the PMD but the PTE.
Quoted 5 lines
Which brings me to this point:
I do not think
__pte_leaf_sizeis needed. AFAICS, it should be enough to definepmd_leafon 8xx, and return 8MB if it is a 8MB hugepage.
If I declare it as a PMD leaf, then many places will expect the PTE entry to be the PMD entry, which is not the case here. As far as I understand, in order that the walker walks down to the page table, we need it flaged as non-leaf by PMD-leaf.
Quoted 9 lines
#define pmd_leaf pmd_leaf
static inline bool pmd_leaf(pmd_t pmd)
{
return pmd_val(pmd) & _PMD_PAGE_8M);
}
and then pmd_leaf_size to return _PMD_PAGE_8M.Quoted 3 lines
This will help because on the ongoing effort of unifying hugetlb and getting rid of
huge_ptep_get()[1], pagewalkers will stumble upon the 8mb-PMD as they do for regular PMDs.
But AFAIU, it won't work that simple, because *pmd is definitely not a PTE but still a pointer to a page table which contains the PTE.
Quoted 22 lines
Which means that they would be caught in the following code:
ptl = pmd_huge_lock(pmd, vma); if (ptl) { - 8MB hugepages will be handled here smaps_pmd_entry(pmd, addr, walk); spin_unlock(ptl); } /* pte stuff */ ...where
pmd_huge_lockis:static inline spinlock_t *pmd_huge_lock(pmd_t *pmd, struct vm_area_struct *vma) { spinlock_t *ptl = pmd_lock(vma->vm_mm, pmd);if (pmd_leaf(*pmd) || pmd_devmap(*pmd)) return ptl; spin_unlock(ptl); return NULL;
Quoted 6 lines
}
So, since
pmd_leaf()will return true for 8MB hugepages, we are fine, because anyway we want to perform pagetable operations on *that* PMD and not the ptes that are cont-mapped, which is different for e.g: 512K hugepages, where we perform it on pte level.
We still want to do the operation on the cont-PTE, in fact in both 4M page tables so that we cover the 8M. There is no operation to do on the PMD entry itself except that telling it is a 8M page table underneath.
Quoted 4 lines
So I would suggest that instead of this patch, we have one implementing
pmd_leafandpmd_leaf_sizefor 8Mb hugepages on power8xx, as that takes us closer to our goal of unifying hugetlb.
But then, how will it work to go down the PTE road ?
Christophe
PATCH v5 3/18 mm: Provide mm_struct and address to huge_ptep_get()
In reply to Christophe LeroyOn powerpc 8xx huge_ptep_get() will need to know whether the given ptep is a PTE entry or a PMD entry. This cannot be known with the PMD entry itself because there is no easy way to know it from the content of the entry.
So huge_ptep_get() will need to know either the size of the page or get the pmd.
In order to be consistent with huge_ptep_get_and_clear(), give mm and address to huge_ptep_get().
---
v2: Add missing changes in arch implementations
v3: Fixed a comment in ARM and missing changes in S390
v4: Fix missing or bad changes in mm/hugetlb.c and rebased on v6.10-rc1
v5: Fix vma->mm to vma->vm_mm
--- arch/arm/include/asm/hugetlb-3level.h | 4 +-- arch/arm64/include/asm/hugetlb.h | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/mm/hugetlbpage.c | 2 +- arch/s390/include/asm/hugetlb.h | 4 +-- arch/s390/mm/hugetlbpage.c | 4 +-- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 10 +++--- fs/userfaultfd.c | 2 +- include/asm-generic/hugetlb.h | 2 +- include/linux/swapops.h | 4 +-- mm/damon/vaddr.c | 6 ++-- mm/gup.c | 2 +- mm/hmm.c | 2 +- mm/hugetlb.c | 44 +++++++++++++-------------- mm/memory-failure.c | 2 +- mm/mempolicy.c | 2 +- mm/migrate.c | 4 +-- mm/mincore.c | 2 +- mm/userfaultfd.c | 2 +- 21 files changed, 53 insertions(+), 53 deletions(-)
arch/arm/include/asm/hugetlb-3level.h +2 -2
@@ -13,12 +13,12 @@/* * If our huge pte is non-zero then mark the valid bit. -* This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero +* This allows pte_present(huge_ptep_get(mm,addr,ptep)) to return true for non-zero * ptes. * (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes). */ #define __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (pte_t *ptep) +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {pte_t retval = *ptep;if (pte_val (retval))
arch/arm64/include/asm/hugetlb.h +1 -1
@@ -46,7 +46,7 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,extern void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz);#define __HAVE_ARCH_HUGE_PTEP_GET -extern pte_t huge_ptep_get (pte_t *ptep); +extern pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);void __init arm64_hugetlb_cma_reserve (void );
arch/arm64/mm/hugetlbpage.c +1 -1
@@ -127,7 +127,7 @@ static inline int num_contig_ptes(unsigned long size, size_t *pgsize)return contig_ptes; } -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {int ncontig, i;size_t pgsize;
arch/riscv/include/asm/hugetlb.h +1 -1
@@ -44,7 +44,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t pte, int dirty);#define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get (pte_t *ptep); +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags);#define arch_make_huge_pte arch_make_huge_pte
arch/riscv/mm/hugetlbpage.c +1 -1
@@ -3,7 +3,7 @@#include <linux/err.h> #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {unsigned long pte_num;int i;
arch/s390/include/asm/hugetlb.h +2 -2
@@ -19,7 +19,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz);void __set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte); -pte_t huge_ptep_get (pte_t *ptep); +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep);pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep);
@@ -64,7 +64,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int changed = !pte_same(huge_ptep_get(ptep), pte); + int changed = !pte_same(huge_ptep_get(vma->vm_mm, addr, ptep), pte);if (changed) {huge_ptep_get_and_clear (vma->vm_mm, addr, ptep);__set_huge_pte_at (vma->vm_mm, addr, ptep, pte);
arch/s390/mm/hugetlbpage.c +2 -2
@@ -169,7 +169,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,__set_huge_pte_at (mm, addr, ptep, pte); } -pte_t huge_ptep_get (pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {return __rste_to_pte (pte_val (*ptep)); }
@@ -177,7 +177,7 @@ pte_t huge_ptep_get(pte_t *ptep)pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { -pte_t pte =huge_ptep_get (ptep); +pte_t pte =huge_ptep_get (mm, addr, ptep);pmd_t *pmdp = (pmd_t *) ptep;pud_t *pudp = (pud_t *) ptep;
fs/hugetlbfs/inode.c +1 -1
@@ -422,7 +422,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma,if (!ptep)return false; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, addr, ptep);if (huge_pte_none (pte) || !pte_present (pte))return false;
fs/proc/task_mmu.c +5 -5
@@ -730,7 +730,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, {struct mem_size_stats *mss = walk->private;struct vm_area_struct *vma = walk->vma; -pte_t ptent =huge_ptep_get (pte); +pte_t ptent =huge_ptep_get (walk->mm, addr, pte);struct folio *folio =NULL ;if (pte_present (ptent)) {
@@ -1582,7 +1582,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,if (vma->vm_flags &VM_SOFTDIRTY ) flags |=PM_SOFT_DIRTY ; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, addr, ptep);if (pte_present (pte)) {struct folio *folio =page_folio (pte_page (pte));
@@ -2271,7 +2271,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,if (~p->arg.flags &PM_SCAN_WP_MATCHING ) {/* Go the short route when not write-protecting pages. */ - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, start, ptep); categories = p->cur_vma_category |pagemap_hugetlb_category (pte);if (!pagemap_scan_is_interesting_page (categories, p))
@@ -2283,7 +2283,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,i_mmap_lock_write (vma->vm_file->f_mapping); ptl =huge_pte_lock (hstate_vma (vma), vma->vm_mm, ptep); - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (walk->mm, start, ptep); categories = p->cur_vma_category |pagemap_hugetlb_category (pte);if (!pagemap_scan_is_interesting_page (categories, p))
@@ -2679,7 +2679,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr,static int gather_hugetlb_stats (pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) { -pte_t huge_pte =huge_ptep_get (pte); +pte_t huge_pte =huge_ptep_get (walk->mm, addr, pte);struct numa_maps *md;struct page *page;
fs/userfaultfd.c +1 -1
@@ -257,7 +257,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, goto out; ret = false; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, vmf->address, ptep);/* * Lockless access: we're in a wait_event so it's ok if it
include/asm-generic/hugetlb.h +1 -1
@@ -144,7 +144,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (pte_t *ptep) +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) {return ptep_get (ptep); }
include/linux/swapops.h +2 -2
@@ -334,7 +334,7 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry)extern void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address); -extern void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *pte); +extern void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *pte);#else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry (pgoff_t offset) {
@@ -359,7 +359,7 @@ static inline int is_migration_entry(swp_entry_t swp)static inline void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address) { }static inline void migration_entry_wait_huge (struct vm_area_struct *vma, -pte_t *pte) { } +unsigned long addr,pte_t *pte) { }static inline int is_writable_migration_entry (swp_entry_t entry) {return 0 ;
mm/damon/vaddr.c +3 -3
@@ -339,7 +339,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr) { bool referenced = false; -pte_t entry =huge_ptep_get (pte); +pte_t entry =huge_ptep_get (mm, addr, pte);struct folio *folio =pfn_folio (pte_pfn (entry));unsigned long psize =huge_page_size (hstate_vma (vma));
@@ -373,7 +373,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (h, walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) goto out;
@@ -509,7 +509,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (h, walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) goto out;
mm/gup.c +1 -1
@@ -547,7 +547,7 @@ static int gup_hugepte(struct vm_area_struct *vma, pte_t *ptep, unsigned long szif (pte_end < end) end = pte_end; - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, addr, ptep);if (!pte_access_permitted (pte, flags &FOLL_WRITE ))return 0 ;
mm/hmm.c +1 -1
@@ -480,7 +480,7 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (hstate_vma (vma), walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte); i = (start - range->start) >>PAGE_SHIFT ; pfn_req_flags = range->hmm_pfns[i];
mm/hugetlb.c +22 -22
173 lines not on this page · raw message
mm/memory-failure.c +1 -1
@@ -834,7 +834,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask, struct mm_walk *walk) {struct hwpoison_walk *hwp = walk->private; -pte_t pte =huge_ptep_get (ptep); +pte_t pte =huge_ptep_get (walk->mm, addr, ptep);struct hstate *h =hstate_vma (walk->vma); return check_hwpoisoned_entry(pte, addr, huge_page_shift(h),
mm/mempolicy.c +1 -1
@@ -624,7 +624,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,pte_t entry; ptl =huge_pte_lock (hstate_vma (walk->vma), walk->mm, pte); - entry =huge_ptep_get (pte); + entry =huge_ptep_get (walk->mm, addr, pte);if (!pte_present (entry)) {if (unlikely (is_hugetlb_entry_migration (entry))) qp->nr_failed++;
mm/migrate.c +2 -2
@@ -338,14 +338,14 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,* * This function will release the vma lock before returning. */ -void migration_entry_wait_huge (struct vm_area_struct *vma,pte_t *ptep) +void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) {spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, ptep);pte_t pte;hugetlb_vma_assert_locked (vma);spin_lock (ptl); - pte =huge_ptep_get (ptep); + pte =huge_ptep_get (vma->vm_mm, addr, ptep);if (unlikely (!is_hugetlb_entry_migration (pte))) {spin_unlock (ptl);
mm/mincore.c +1 -1
@@ -33,7 +33,7 @@ static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr,* Hugepages under user process are always in RAM and never * swapped out, but theoretically it needs to be checked. */ - present = pte && !huge_pte_none_mostly (huge_ptep_get (pte)); + present = pte && !huge_pte_none_mostly (huge_ptep_get (walk->mm, addr, pte));for (; addr != end; vec++, addr +=PAGE_SIZE ) *vec = present; walk->private = vec;
mm/userfaultfd.c +1 -1
@@ -587,7 +587,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( }if (!uffd_flags_mode_is (flags,MFILL_ATOMIC_CONTINUE ) && - !huge_pte_none_mostly (huge_ptep_get (dst_pte))) { + !huge_pte_none_mostly (huge_ptep_get (dst_mm, dst_addr, dst_pte))) { err = -EEXIST ;hugetlb_vma_unlock_read (dst_vma);mutex_unlock (&hugetlb_fault_mutex_table[hash]);
Signature · 1 line
2.44.0
PATCH v5 4/18 powerpc/mm: Remove _PAGE_PSIZE
In reply to Christophe Leroy_PAGE_PSIZE macro is never used outside the place it is defined and is used only on 8xx and e500.
Remove indirection, remove it and use its content directly.
--- arch/powerpc/include/asm/nohash/32/pte-40x.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-44x.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-85xx.h | 3 --- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 5 ++--- arch/powerpc/include/asm/nohash/pte-e500.h | 4 +--- 5 files changed, 3 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/nohash/32/pte-40x.h +0 -3
@@ -49,9 +49,6 @@#define _PAGE_EXEC 0x200/* hardware: EX permission */ #define _PAGE_ACCESSED 0x400/* software: R: page referenced */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -/* cache related flags non existing on 40x */ #define _PAGE_COHERENT 0
arch/powerpc/include/asm/nohash/32/pte-44x.h +0 -3
@@ -75,9 +75,6 @@#define _PAGE_NO_CACHE 0x00000400/* H: I bit */ #define _PAGE_WRITETHRU 0x00000800/* H: W bit */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -/* TODO: Add large page lowmem mapping support */ #define _PMD_PRESENT 0#define _PMD_PRESENT_MASK (PAGE_MASK)
arch/powerpc/include/asm/nohash/32/pte-85xx.h +0 -3
@@ -31,9 +31,6 @@#define _PAGE_WRITETHRU 0x00400/* H: W bit */ #define _PAGE_SPECIAL 0x00800/* S: Special page */ -/* No page size encoding in the linux PTE */ -#define _PAGE_PSIZE 0 -#define _PMD_PRESENT 0#define _PMD_PRESENT_MASK (PAGE_MASK)#define _PMD_BAD (~PAGE_MASK)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +2 -3
@@ -74,12 +74,11 @@#define _PTE_NONE_MASK 0#ifdef CONFIG_PPC_16K_PAGES -#define _PAGE_PSIZE _PAGE_SPS +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_SPS)#else -#define _PAGE_PSIZE 0 +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED)#endif -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE)#define _PAGE_BASE (_PAGE_BASE_NC)#include <asm/pgtable-masks.h>
arch/powerpc/include/asm/nohash/pte-e500.h +1 -3
@@ -65,8 +65,6 @@#define _PAGE_SPECIAL _PAGE_SW0 -/* Base page size */ -#define _PAGE_PSIZE _PAGE_PSIZE_4K#define PTE_RPN_SHIFT (24)#define PTE_WIMGE_SHIFT (19)
@@ -89,7 +87,7 @@* pages. We always set _PAGE_COHERENT when SMP is enabled or * the processor might need it for DMA coherency. */ -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE) +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE_4K)#if defined(CONFIG_SMP )#define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT)#else
Signature · 1 line
2.44.0
PATCH v5 5/18 powerpc/mm: Fix __find_linux_pte() on 32 bits with PMD leaf entries
In reply to Christophe LeroyBuilding on 32 bits with pmd_leaf() not returning always false leads to the following error:
CC arch/powerpc/mm/pgtable.oarch/powerpc/mm/pgtable.c: In function '__find_linux_pte':arch/powerpc/mm/pgtable.c:506:1: error: function may return address of local variable [-Werror=return-local-addr]
506 | }
| ^
arch/powerpc/mm/pgtable.c:394:15: note: declared here
394 | pud_t pud, *pudp;
| ^~~arch/powerpc/mm/pgtable.c:394:15: note: declared here
This is due to pmd_offset() being a no-op in that case.
So rework it for powerpc/32 so that pXd_offset() are used on real pointers and not on on-stack copies.
Behind fixing the problem, it also has the advantage of simplifying __find_linux_pte() including the removal of stack frame:
After this patch:
00000018 <__find_linux_pte>: 18: 2c 06 00 00 cmpwi r6,0 1c: 41 82 00 0c beq 28 <__find_linux_pte+0x10> 20: 39 20 00 00 li r9,0 24: 91 26 00 00 stw r9,0(r6) 28: 2f 85 00 00 cmpwi cr7,r5,0 2c: 41 9e 00 0c beq cr7,38 <__find_linux_pte+0x20> 30: 39 20 00 00 li r9,0 34: 99 25 00 00 stb r9,0(r5) 38: 54 89 65 3a rlwinm r9,r4,12,20,29 3c: 7c 63 48 2e lwzx r3,r3,r9 40: 2f 83 00 00 cmpwi cr7,r3,0 44: 41 9e 00 30 beq cr7,74 <__find_linux_pte+0x5c> 48: 54 69 07 3a rlwinm r9,r3,0,28,29 4c: 2f 89 00 0c cmpwi cr7,r9,12 50: 54 63 00 26 clrrwi r3,r3,12 54: 54 84 b5 36 rlwinm r4,r4,22,20,27 58: 3c 63 c0 00 addis r3,r3,-16384 5c: 7c 63 22 14 add r3,r3,r4 60: 4c be 00 20 bnelr+ cr7 64: 4d 82 00 20 beqlr 68: 39 20 00 17 li r9,23 6c: 91 26 00 00 stw r9,0(r6) 70: 4e 80 00 20 blr 74: 38 60 00 00 li r3,0 78: 4e 80 00 20 blr
Before this patch:
00000018 <__find_linux_pte>: 18: 2c 06 00 00 cmpwi r6,0 1c: 94 21 ff e0 stwu r1,-32(r1) 20: 41 82 00 0c beq 2c <__find_linux_pte+0x14> 24: 39 20 00 00 li r9,0 28: 91 26 00 00 stw r9,0(r6) 2c: 2f 85 00 00 cmpwi cr7,r5,0 30: 41 9e 00 0c beq cr7,3c <__find_linux_pte+0x24> 34: 39 20 00 00 li r9,0 38: 99 25 00 00 stb r9,0(r5) 3c: 54 89 65 3a rlwinm r9,r4,12,20,29 40: 7c 63 48 2e lwzx r3,r3,r9 44: 54 69 07 3a rlwinm r9,r3,0,28,29 48: 2f 89 00 0c cmpwi cr7,r9,12 4c: 90 61 00 0c stw r3,12(r1) 50: 41 9e 00 4c beq cr7,9c <__find_linux_pte+0x84> 54: 80 61 00 0c lwz r3,12(r1) 58: 54 69 07 3a rlwinm r9,r3,0,28,29 5c: 2f 89 00 0c cmpwi cr7,r9,12 60: 90 61 00 08 stw r3,8(r1) 64: 41 9e 00 38 beq cr7,9c <__find_linux_pte+0x84> 68: 80 61 00 08 lwz r3,8(r1) 6c: 2f 83 00 00 cmpwi cr7,r3,0 70: 41 9e 00 54 beq cr7,c4 <__find_linux_pte+0xac> 74: 54 69 07 3a rlwinm r9,r3,0,28,29 78: 2f 89 00 0c cmpwi cr7,r9,12 7c: 54 69 00 26 clrrwi r9,r3,12 80: 54 8a b5 36 rlwinm r10,r4,22,20,27 84: 3c 69 c0 00 addis r3,r9,-16384 88: 7c 63 52 14 add r3,r3,r10 8c: 54 84 93 be srwi r4,r4,14 90: 41 9e 00 14 beq cr7,a4 <__find_linux_pte+0x8c> 94: 38 21 00 20 addi r1,r1,32 98: 4e 80 00 20 blr 9c: 54 69 00 26 clrrwi r9,r3,12 a0: 54 84 93 be srwi r4,r4,14 a4: 3c 69 c0 00 addis r3,r9,-16384 a8: 54 84 25 36 rlwinm r4,r4,4,20,27 ac: 7c 63 22 14 add r3,r3,r4 b0: 41 a2 ff e4 beq 94 <__find_linux_pte+0x7c> b4: 39 20 00 17 li r9,23 b8: 91 26 00 00 stw r9,0(r6) bc: 38 21 00 20 addi r1,r1,32 c0: 4e 80 00 20 blr c4: 38 60 00 00 li r3,0 c8: 38 21 00 20 addi r1,r1,32 cc: 4e 80 00 20 blr
---
v3: Removed p4dp and pudp locals for PPC32 and add a comment.
v4: Properly set pdshift on PPC32 case
v5: Enhanced commit message
--- arch/powerpc/mm/pgtable.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/powerpc/mm/pgtable.c +10 -1
@@ -382,8 +382,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, bool *is_thp, unsigned *hpage_shift) { pgd_t *pgdp; +#ifdef CONFIG_PPC64 p4d_t p4d, *p4dp;pud_t pud, *pudp; +#endif pmd_t pmd, *pmdp;pte_t *ret_pte;hugepd_t *hpdp =NULL ;
@@ -401,8 +403,12 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,* page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. * Top level is an exception because it is folded into p4d. +* +* On PPC32, P4D/PUD/PMD are folded into PGD so go straight to +* PMD level. */ pgdp = pgdir +pgd_index (ea); +#ifdef CONFIG_PPC64 p4dp =p4d_offset (pgdp, ea); p4d =READ_ONCE (*p4dp); pdshift =P4D_SHIFT ;
@@ -442,8 +448,11 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out_huge; } - pdshift =PMD_SHIFT ; pmdp =pmd_offset (&pud, ea); +#else + pmdp =pmd_offset (pud_offset (p4d_offset (pgdp, ea), ea), ea); +#endif + pdshift =PMD_SHIFT ; pmd =READ_ONCE (*pmdp);/*
Signature · 1 line
2.44.0
PATCH v5 6/18 powerpc/mm: Allow hugepages without hugepd
In reply to Christophe LeroyIn preparation of implementing huge pages on powerpc 8xx without hugepd, enclose hugepd related code inside an ifdef CONFIG_ARCH_HAS_HUGEPD
This also allows removing some stubs.
---
v3:
- Prepare huge_pte_alloc() for full standard topology, not only for 2-level
- Reordered last part of huge_pte_alloc()
v4:
- Rebased of v6.10-rc1
v5:
- Moved the Kconfig split in this patch.
--- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 -- arch/powerpc/include/asm/hugetlb.h | 10 ++---- arch/powerpc/include/asm/nohash/pgtable.h | 2 +- arch/powerpc/mm/hugetlbpage.c | 33 ++++++++++++++++++++ arch/powerpc/mm/pgtable.c | 2 ++ arch/powerpc/platforms/Kconfig.cputype | 3 ++ 7 files changed, 41 insertions(+), 12 deletions(-)
arch/powerpc/Kconfig +0 -1
@@ -135,7 +135,6 @@ config PPC select ARCH_HAS_DMA_MAP_DIRECT if PPC_PSERIES select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_HAS_KCOV select ARCH_HAS_KERNEL_FPU_SUPPORT if PPC64 && PPC_FPU select ARCH_HAS_MEMBARRIER_CALLBACKS
arch/powerpc/include/asm/book3s/32/pgalloc.h +0 -2
@@ -47,8 +47,6 @@ static inline void pgtable_free(void *table, unsigned index_size) } } -#define get_hugepd_cache_index (x) (x) -static inline void pgtable_free_tlb (struct mmu_gather *tlb,void *table,int shift) {
arch/powerpc/include/asm/hugetlb.h +2 -8
@@ -30,10 +30,12 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, }#define is_hugepage_only_range is_hugepage_only_range +#ifdef CONFIG_ARCH_HAS_HUGEPD #define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGEvoid hugetlb_free_pgd_range (struct mmu_gather *tlb,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling); +#endif #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
@@ -67,14 +69,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma, { } -#define hugepd_shift (x) 0 -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned pdshift) -{ -return NULL ; -} - -static inline void __init gigantic_hugetlb_cma_reserve (void ) { }
arch/powerpc/include/asm/nohash/pgtable.h +1 -1
@@ -340,7 +340,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,#define pgprot_writecombine pgprot_noncached_wc -#ifdef CONFIG_HUGETLB_PAGE +#ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok (hugepd_t hpd) {#ifdef CONFIG_PPC_8xx
arch/powerpc/mm/hugetlbpage.c +33 -0
@@ -42,6 +42,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sreturn __find_linux_pte (mm->pgd, addr,NULL ,NULL ); } +#ifdef CONFIG_ARCH_HAS_HUGEPD static int __hugepte_alloc (struct mm_struct *mm,hugepd_t *hpdp,unsigned long address,unsigned int pdshift,unsigned int pshift,spinlock_t *ptl)
@@ -193,6 +194,36 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,return hugepte_offset (*hpdp, addr, pdshift); } +#else +pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma, +unsigned long addr,unsigned long sz) +{ +p4d_t *p4d; +pud_t *pud; +pmd_t *pmd; + + addr &= ~(sz -1 ); + + p4d =p4d_offset (pgd_offset (mm, addr), addr); +if (!mm_pud_folded (mm) && sz >=P4D_SIZE ) +return (pte_t *)p4d; + + pud =pud_alloc (mm, p4d, addr); +if (!pud) +return NULL ; +if (!mm_pmd_folded (mm) && sz >=PUD_SIZE ) +return (pte_t *)pud; + + pmd =pmd_alloc (mm, pud, addr); +if (!pmd) +return NULL ; + +if (sz >=PMD_SIZE ) +return (pte_t *)pmd; + +return pte_alloc_huge (mm, pmd, addr); +} +#endif #ifdef CONFIG_PPC_BOOK3S_64 /*
@@ -248,6 +279,7 @@ int __init alloc_bootmem_huge_page(struct hstate *h, int nid)return __alloc_bootmem_huge_page (h, nid); } +#ifdef CONFIG_ARCH_HAS_HUGEPD #ifndef CONFIG_PPC_BOOK3S_64 #define HUGEPD_FREELIST_SIZE \ ((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
@@ -505,6 +537,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, } }while (addr = next, addr != end); } +#endif bool __init arch_hugetlb_valid_size (unsigned long size) {
arch/powerpc/mm/pgtable.c +2 -0
@@ -496,8 +496,10 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,if (!hpdp)return NULL ; +#ifdef CONFIG_ARCH_HAS_HUGEPD ret_pte =hugepte_offset (*hpdp, ea, pdshift); pdshift =hugepd_shift (*hpdp); +#endif out:if (hpage_shift) *hpage_shift = pdshift;
arch/powerpc/platforms/Kconfig.cputype +3 -0
@@ -37,6 +37,7 @@ config PPC_85xx config PPC_8xx bool "Freescale 8xx" + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select FSL_SOC select PPC_KUEP
@@ -98,6 +99,7 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD
@@ -290,6 +292,7 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool + select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL
Signature · 1 line
2.44.0
PATCH v5 7/18 powerpc/8xx: Fix size given to set_huge_pte_at()
In reply to Christophe Leroyset_huge_pte_at() expects the size of the hugepage as an int, not the psize which is the index of the page definition in table mmu_psize_defs[]
--- arch/powerpc/mm/nohash/8xx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/powerpc/mm/nohash/8xx.c +2 -1
@@ -94,7 +94,8 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,return -EINVAL ;set_huge_pte_at (&init_mm, va, ptep, -pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), psize); +pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)), +1UL <<mmu_psize_to_shift (psize));return 0 ; }
Signature · 1 line
2.44.0
PATCH v5 8/18 powerpc/8xx: Rework support for 8M pages using contiguous PTE entries
In reply to Christophe LeroyIn order to fit better with standard Linux page tables layout, add support for 8M pages using contiguous PTE entries in a standard page table. Page tables will then be populated with 1024 similar entries and two PMD entries will point to that page table.
The PMD entries also get a flag to tell it is addressing an 8M page, this is required for the HW tablewalk assistance.
---
v3:
- Move huge_ptep_get() for a more readable commit diff
- Flag PMD as 8Mbytes in set_huge_pte_at()
- Define __pte_leaf_size()
- Change pte_update() instead of all huge callers of pte_update()
- Added ptep_is_8m_pmdp() helper
- Fixed kasan early memory 8M allocation
v5:
- In huge_ptep_get(), change pte_offset_kernel((pmd_t *)ptep, 0) to pte_offset_kernel((pmd_t *)ptep, ALIGN_DOWN(addr, SZ_8M)) which is more correct allthough not different.
--- .../include/asm/nohash/32/hugetlb-8xx.h | 38 +++---------- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 53 ++++++++++++------- arch/powerpc/include/asm/nohash/pgtable.h | 4 -- arch/powerpc/include/asm/page.h | 5 -- arch/powerpc/include/asm/pgtable.h | 3 ++ arch/powerpc/kernel/head_8xx.S | 10 +--- arch/powerpc/mm/hugetlbpage.c | 21 +++++--- arch/powerpc/mm/kasan/8xx.c | 21 +++++--- arch/powerpc/mm/nohash/8xx.c | 40 +++++++------- arch/powerpc/mm/pgtable.c | 27 +++++++--- arch/powerpc/mm/pgtable_32.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 1 - 12 files changed, 111 insertions(+), 114 deletions(-)
arch/powerpc/include/asm/nohash/32/hugetlb-8xx.h +8 -30
@@ -4,42 +4,12 @@#define PAGE_SHIFT_8M 23 -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -BUG_ON (!hugepd_ok (hpd)); - -return (pte_t *)__va (hpd_val (hpd) & ~HUGEPD_SHIFT_MASK ); -} - -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return PAGE_SHIFT_8M ; -} - -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -unsigned long idx = (addr & (SZ_4M -1 )) >>PAGE_SHIFT ; - -return hugepd_page (hpd) + idx; -} -static inline void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr) {flush_tlb_page (vma, vmaddr); } -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) | _PMD_USER | _PMD_PRESENT | _PMD_PAGE_8M); -} - -static inline void hugepd_populate_kernel (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) | _PMD_PRESENT | _PMD_PAGE_8M); -} -static inline int check_and_get_huge_psize (int shift) {return shift_to_mmu_psize (shift);
@@ -49,6 +19,14 @@ static inline int check_and_get_huge_psize(int shift)void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +{ +if (ptep_is_8m_pmdp (mm, addr, ptep)) + ptep =pte_offset_kernel ((pmd_t *)ptep,ALIGN_DOWN (addr,SZ_8M )); +return ptep_get (ptep); +} +#define __HAVE_ARCH_HUGE_PTE_CLEARstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep,unsigned long sz)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +34 -19
104 lines not on this page · raw message
arch/powerpc/include/asm/nohash/pgtable.h +0 -4
@@ -343,12 +343,8 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,#ifdef CONFIG_ARCH_HAS_HUGEPD static inline int hugepd_ok (hugepd_t hpd) { -#ifdef CONFIG_PPC_8xx -return ((hpd_val (hpd) & _PMD_PAGE_MASK) == _PMD_PAGE_8M); -#else /* We clear the top bit to indicate hugepd */ return (hpd_val (hpd) && (hpd_val (hpd) &PD_HUGE ) ==0 ); -#endif }#define is_hugepd (hpd) (hugepd_ok(hpd))
arch/powerpc/include/asm/page.h +0 -5
@@ -293,13 +293,8 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)/* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits. -* On 8xx, HW assistance requires 4k alignment for the hugepte. */ -#ifdef CONFIG_PPC_8xx -#define HUGEPD_SHIFT_MASK 0xfff -#else #define HUGEPD_SHIFT_MASK 0x3f -#endif #ifndef __ASSEMBLY__
arch/powerpc/include/asm/pgtable.h +3 -0
@@ -106,6 +106,9 @@ unsigned long vmalloc_to_phys(void *vmalloc_addr);void pgtable_cache_add (unsigned int shift); +#ifdef CONFIG_PPC32 +void __init *early_alloc_pgtable (unsigned long size); +#endif pte_t *early_pte_alloc_kernel (pmd_t *pmdp,unsigned long va);#if defined(CONFIG_STRICT_KERNEL_RWX ) || defined(CONFIG_PPC32 )
arch/powerpc/kernel/head_8xx.S +2 -8
@@ -415,14 +415,13 @@ FixupDAR:/* Entry point for dcbx workaround. */ oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha 3: lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + rlwinm r11, r11, 0, ~_PMD_PAGE_8M mtspr SPRN_MD_TWC, r11 - mtcrf 0x01, r11 mfspr r11, SPRN_MD_TWC lwz r11, 0(r11) /* Get the pte */ - bt 28,200f /* bit 28 = Large page (8M) */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 -201: lwz r11,0(r11) + lwz r11,0(r11) /* Check if it really is a dcbx instruction. */ /* dcbt and dcbtst does not generate DTLB Misses/Errors, * no need to include them here */
@@ -441,11 +440,6 @@ FixupDAR:/* Entry point for dcbx workaround. */ 141: mfspr r10,SPRN_M_TW b DARFixed /* Nope, go back to normal TLB processing */ -200: - /* concat physical page address(r11) and page offset(r10) */ - rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 - b 201b - 144: mfspr r10, SPRN_DSISR rlwinm r10, r10,0,7,5 /* Clear store bit for buggy dcbst insn */ mtspr SPRN_DSISR, r10
arch/powerpc/mm/hugetlbpage.c +13 -8
@@ -183,9 +183,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,if (!hpdp)return NULL ; -if (IS_ENABLED (CONFIG_PPC_8xx) && pshift <PMD_SHIFT ) -return pte_alloc_huge (mm, (pmd_t *)hpdp, addr); -BUG_ON (!hugepd_none (*hpdp) && !hugepd_ok (*hpdp)); if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr,
@@ -218,8 +215,18 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,if (!pmd)return NULL ; -if (sz >=PMD_SIZE ) +if (sz >=PMD_SIZE ) { +/* On 8xx, all hugepages are handled as contiguous PTEs */ +if (IS_ENABLED (CONFIG_PPC_8xx)) { +int i; + +for (i =0 ; i < sz /PMD_SIZE ; i++) { +if (!pte_alloc_huge (mm, pmd + i, addr)) +return NULL ; + } + }return (pte_t *)pmd; + }return pte_alloc_huge (mm, pmd, addr); }
@@ -617,10 +624,8 @@ static int __init hugetlbpage_init(void)* use pgt cache for hugepd. */ if (pdshift > shift) { -if (!IS_ENABLED (CONFIG_PPC_8xx)) -pgtable_cache_add (pdshift - shift); - }else if (IS_ENABLED (CONFIG_PPC_E500 ) || -IS_ENABLED (CONFIG_PPC_8xx)) { +pgtable_cache_add (pdshift - shift); + }else if (IS_ENABLED (CONFIG_PPC_E500 )) {pgtable_cache_add (PTE_T_ORDER ); }
arch/powerpc/mm/kasan/8xx.c +13 -8
@@ -6,28 +6,33 @@#include <linux/memblock.h> #include <linux/hugetlb.h> +#include <asm/pgalloc.h> +static int __init kasan_init_shadow_8M (unsigned long k_start,unsigned long k_end,void *block) {pmd_t *pmd =pmd_off_k (k_start);unsigned long k_cur, k_next; -for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd +=2 , block +=SZ_8M ) { -pte_basic_t *new; +for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++, block +=SZ_4M ) { +pte_t *ptep; +int i; k_next =pgd_addr_end (k_cur, k_end); - k_next =pgd_addr_end (k_next, k_end);if ((void *)pmd_page_vaddr (*pmd) != kasan_early_shadow_pte)continue ; - new =memblock_alloc (sizeof (pte_basic_t),SZ_4K ); -if (!new) + ptep =memblock_alloc (PTE_FRAG_SIZE ,PTE_FRAG_SIZE ); +if (!ptep)return -ENOMEM ; - *new =pte_val (pte_mkhuge (pfn_pte (PHYS_PFN (__pa (block)),PAGE_KERNEL ))); +for (i =0 ; i <PTRS_PER_PTE ; i++) { +pte_t pte =pte_mkhuge (pfn_pte (PHYS_PFN (__pa (block + i *PAGE_SIZE )),PAGE_KERNEL )); -hugepd_populate_kernel ((hugepd_t *)pmd, (pte_t *)new,PAGE_SHIFT_8M ); -hugepd_populate_kernel ((hugepd_t *)pmd +1 , (pte_t *)new,PAGE_SHIFT_8M ); +__set_pte_at (&init_mm, k_cur, ptep + i, pte,1 ); + } +pmd_populate_kernel (&init_mm, pmd, ptep); + *pmd =__pmd (pmd_val (*pmd) | _PMD_PAGE_8M); }return 0 ; }
arch/powerpc/mm/nohash/8xx.c +18 -22
@@ -11,6 +11,7 @@#include <linux/hugetlb.h> #include <asm/fixmap.h> +#include <asm/pgalloc.h> #include <mm/mmu_decl.h>
@@ -48,20 +49,6 @@ unsigned long p_block_mapped(phys_addr_t pa)return 0 ; } -static pte_t __init *early_hugepd_alloc_kernel (hugepd_t *pmdp,unsigned long va) -{ -if (hpd_val (*pmdp) ==0 ) { -pte_t *ptep =memblock_alloc (sizeof (pte_basic_t),SZ_4K ); - -if (!ptep) -return NULL ; - -hugepd_populate_kernel ((hugepd_t *)pmdp, ptep,PAGE_SHIFT_8M ); -hugepd_populate_kernel ((hugepd_t *)pmdp +1 , ptep,PAGE_SHIFT_8M ); - } -return hugepte_offset (*(hugepd_t *)pmdp, va,PGDIR_SHIFT ); -} -static int __ref __early_map_kernel_hugepage (unsigned long va,phys_addr_t pa,pgprot_t prot,int psize,bool new) {
@@ -75,24 +62,33 @@ static int __ref __early_map_kernel_hugepage(unsigned long va, phys_addr_t pa,if (WARN_ON (slab_is_available ()))return -EINVAL ; -if (psize ==MMU_PAGE_512K ) +if (psize ==MMU_PAGE_512K ) { ptep =early_pte_alloc_kernel (pmdp, va); -else - ptep =early_hugepd_alloc_kernel ((hugepd_t *)pmdp, va); +/* The PTE should never be already present */ +if (WARN_ON (pte_present (*ptep) &&pgprot_val (prot))) +return -EINVAL ; + }else { +if (WARN_ON (!pmd_none (*pmdp) || !pmd_none (*(pmdp +1 )))) +return -EINVAL ; + + ptep =early_alloc_pgtable (PTE_FRAG_SIZE ); +pmd_populate_kernel (&init_mm, pmdp, ptep); + + ptep =early_alloc_pgtable (PTE_FRAG_SIZE ); +pmd_populate_kernel (&init_mm, pmdp +1 , ptep); + + ptep = (pte_t *)pmdp; + } }else {if (psize ==MMU_PAGE_512K ) ptep =pte_offset_kernel (pmdp, va);else - ptep =hugepte_offset (*(hugepd_t *)pmdp, va,PGDIR_SHIFT ); + ptep = (pte_t *)pmdp; }if (WARN_ON (!ptep))return -ENOMEM ; -/* The PTE should never be already present */ -if (new &&WARN_ON (pte_present (*ptep) &&pgprot_val (prot))) -return -EINVAL ; -set_huge_pte_at (&init_mm, va, ptep,pte_mkhuge (pfn_pte (pa >>PAGE_SHIFT , prot)),1UL <<mmu_psize_to_shift (psize));
arch/powerpc/mm/pgtable.c +19 -8
@@ -297,11 +297,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, }#if defined(CONFIG_PPC_8xx) -void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte,unsigned long sz) +static void __set_huge_pte_at (pmd_t *pmd,pte_t *ptep,pte_basic_t val) { -pmd_t *pmd =pmd_off (mm, addr); -pte_basic_t val;pte_basic_t *entry = (pte_basic_t *)ptep;int num, i;
@@ -311,15 +308,29 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,*/ VM_WARN_ON (pte_hw_valid (*ptep) && !pte_protnone (*ptep)); - pte =set_pte_filter (pte, addr); - - val =pte_val (pte); - num =number_of_cells_per_pte (pmd, val,1 );for (i =0 ; i < num; i++, entry++, val +=SZ_4K ) *entry = val; } + +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz) +{ +pmd_t *pmdp =pmd_off (mm, addr); + + pte =set_pte_filter (pte, addr); + +if (sz ==SZ_8M ) {/* Flag both PMD entries as 8M and fill both page tables */ + *pmdp =__pmd (pmd_val (*pmdp) | _PMD_PAGE_8M); + *(pmdp +1 ) =__pmd (pmd_val (*(pmdp +1 )) | _PMD_PAGE_8M); + +__set_huge_pte_at (pmdp,pte_offset_kernel (pmdp,0 ),pte_val (pte)); +__set_huge_pte_at (pmdp,pte_offset_kernel (pmdp +1 ,0 ),pte_val (pte) +SZ_4M ); + }else { +__set_huge_pte_at (pmdp, ptep,pte_val (pte)); + } +}#endif #endif /* CONFIG_HUGETLB_PAGE */
arch/powerpc/mm/pgtable_32.c +1 -1
@@ -48,7 +48,7 @@ notrace void __init early_ioremap_init(void)early_ioremap_setup (); } -static void __init *early_alloc_pgtable (unsigned long size) +void __init *early_alloc_pgtable (unsigned long size) {void *ptr =memblock_alloc (size, size);
arch/powerpc/platforms/Kconfig.cputype +0 -1
@@ -37,7 +37,6 @@ config PPC_85xx config PPC_8xx bool "Freescale 8xx" - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select FSL_SOC select PPC_KUEP
Signature · 1 line
2.44.0
PATCH v5 9/18 powerpc/8xx: Simplify struct mmu_psize_def
In reply to Christophe LeroyOn 8xx, only the shift field is used in struct mmu_psize_def
Remove other fields and related macros.
--- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
arch/powerpc/include/asm/nohash/32/mmu-8xx.h +2 -7
@@ -189,19 +189,14 @@ typedef struct {#define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000) -/* Page size definitions, common between 32 and 64-bit +/* +* Page size definitions for 8xx * * shift : is the "PAGE_SHIFT" value for that page size -* penc : is the pte encoding mask * */ struct mmu_psize_def {unsigned int shift;/* number of bits */ -unsigned int enc;/* PTE encoding */ -unsigned int ind;/* Corresponding indirect page size shift */ -unsigned int flags; -#define MMU_PAGE_SIZE_DIRECT 0x1/* Supported as a direct size */ -#define MMU_PAGE_SIZE_INDIRECT 0x2/* Supported as an indirect size */ };extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ];
Signature · 1 line
2.44.0
PATCH v5 10/18 powerpc/e500: Remove enc and ind fields from struct mmu_psize_def
In reply to Christophe Leroyenc field is hidden behind BOOK3E_PAGESZ_XX macros, and when you look closer you realise that this field is nothing else than the value of shift minus ten.
So remove enc field and calculate tsize from shift field.
Also remove inc field which is unused.
--- arch/powerpc/include/asm/nohash/mmu-e500.h | 3 --- arch/powerpc/mm/nohash/book3e_pgtable.c | 4 ++-- arch/powerpc/mm/nohash/tlb.c | 9 +-------- arch/powerpc/mm/nohash/tlb_64e.c | 2 +- 4 files changed, 4 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/nohash/mmu-e500.h +0 -3
@@ -244,14 +244,11 @@ typedef struct {/* Page size definitions, common between 32 and 64-bit * * shift : is the "PAGE_SHIFT" value for that page size -* penc : is the pte encoding mask * */ struct mmu_psize_def {unsigned int shift;/* number of bits */ -unsigned int enc;/* PTE encoding */ -unsigned int ind;/* Corresponding indirect page size shift */ unsigned int flags;#define MMU_PAGE_SIZE_DIRECT 0x1/* Supported as a direct size */ #define MMU_PAGE_SIZE_INDIRECT 0x2/* Supported as an indirect size */
arch/powerpc/mm/nohash/book3e_pgtable.c +2 -2
@@ -29,10 +29,10 @@ int __meminit vmemmap_create_mapping(unsigned long start, _PAGE_KERNEL_RW;/* PTEs only contain page size encodings up to 32M */ -BUG_ON (mmu_psize_defs[mmu_vmemmap_psize].enc >0xf ); +BUG_ON (mmu_psize_defs[mmu_vmemmap_psize].shift -10 >0xf );/* Encode the size in the PTE */ - flags |= mmu_psize_defs[mmu_vmemmap_psize].enc <<8 ; + flags |= (mmu_psize_defs[mmu_vmemmap_psize].shift -10 ) <<8 ;/* For each PTE for that area, map things. Note that we don't * increment phys because all PTEs are of the large size and
arch/powerpc/mm/nohash/tlb.c +1 -8
@@ -53,37 +53,30 @@struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ] = { [MMU_PAGE_4K ] = { .shift =12 , - .enc =BOOK3E_PAGESZ_4K , }, [MMU_PAGE_2M ] = { .shift =21 , - .enc =BOOK3E_PAGESZ_2M , }, [MMU_PAGE_4M ] = { .shift =22 , - .enc =BOOK3E_PAGESZ_4M , }, [MMU_PAGE_16M ] = { .shift =24 , - .enc =BOOK3E_PAGESZ_16M , }, [MMU_PAGE_64M ] = { .shift =26 , - .enc =BOOK3E_PAGESZ_64M , }, [MMU_PAGE_256M ] = { .shift =28 , - .enc =BOOK3E_PAGESZ_256M , }, [MMU_PAGE_1G ] = { .shift =30 , - .enc =BOOK3E_PAGESZ_1GB , }, };static inline int mmu_get_tsize (int psize) { -return mmu_psize_defs[psize].enc; +return mmu_psize_defs[psize].shift -10 ; }#else static inline int mmu_get_tsize (int psize)
arch/powerpc/mm/nohash/tlb_64e.c +1 -1
@@ -53,7 +53,7 @@ int extlb_level_exc;*/ void tlb_flush_pgtable (struct mmu_gather *tlb,unsigned long address) { -int tsize = mmu_psize_defs[mmu_pte_psize].enc; +int tsize = mmu_psize_defs[mmu_pte_psize].shift -10 ;if (book3e_htw_mode !=PPC_HTW_NONE ) {unsigned long start = address &PMD_MASK ;
Signature · 1 line
2.44.0
PATCH v5 11/18 powerpc/e500: Switch to 64 bits PGD on 85xx (32 bits)
In reply to Christophe LeroyAt the time being when CONFIG_PTE_64BIT is selected, PTE entries are 64 bits but PGD entries are still 32 bits.
In order to allow leaf PMD entries, switch the PGD to 64 bits entries.
--- arch/powerpc/include/asm/pgtable-types.h | 4 ++++ arch/powerpc/kernel/head_85xx.S | 10 ++++++---- 2 files changed, 10 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/pgtable-types.h +4 -0
@@ -49,7 +49,11 @@ static inline unsigned long pud_val(pud_t x)#endif /* CONFIG_PPC64 */ /* PGD level */ +#if defined(CONFIG_PPC_E500 ) && defined(CONFIG_PTE_64BIT ) +typedef struct {unsigned long long pgd; }pgd_t ; +#else typedef struct {unsigned long pgd; }pgd_t ; +#endif #define __pgd (x) ((pgd_t) { (x) })static inline unsigned long pgd_val (pgd_t x) {
arch/powerpc/kernel/head_85xx.S +6 -4
@@ -307,8 +307,9 @@ set_ivor: #ifdef CONFIG_PTE_64BIT #ifdef CONFIG_HUGETLB_PAGE #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ blt 1000f; /* Normal non-huge page */ \ beq 2f; /* Bail if no table */ \
@@ -321,8 +322,9 @@ set_ivor: 1001: lwz r11, 4(r12); /* Get pte entry */ #else #define FIND_PTE \ - rlwinm r12, r10, 13, 19, 29; /* Compute pgdir/pmd offset */ \ - lwzx r11, r12, r11; /* Get pgd/pmd entry */ \ + rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + add r12, r11, r12; \ + lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ beq 2f; /* Bail if no table */ \ rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \
Signature · 1 line
2.44.0
PATCH v5 12/18 powerpc/e500: Encode hugepage size in PTE bits
In reply to Christophe LeroyUse PTE page size bits to encode hugepage size with the following format corresponding to the values expected in bits 52-55 in MAS1 register. Those bits are called TSIZE:
0001 4 Kbyte 0010 16 Kbyte 0011 64 Kbyte 0100 256 Kbyte 0101 1 Mbyte 0110 4 Mbyte 0111 16 Mbyte 1000 64 Mbyte 1001 256 Mbyte 1010 1 Gbyte 1011 4 Gbyte 1100 16 Gbyte 1101 64 Gbyte 1110 256 Gbyte 1111 1 Tbyte
It corresponds to shift value minus 10 with lowest bit removed.
It is not the value expected in the PTE in that field, but only e6500 performs HW based TLB loading and the e6500 reference manual explicitely says that this field is ignored.
Also add pte_huge_size() which will be used later.
---
v5: Use PAGE SIZE field instead of U0-U3 because on some HW U2-U3 are used for something else.
--- .../powerpc/include/asm/nohash/hugetlb-e500.h | 9 ++++++ arch/powerpc/include/asm/nohash/pte-e500.h | 28 +++++++++---------- 2 files changed, 22 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +9 -0
@@ -42,4 +42,13 @@ static inline int check_and_get_huge_psize(int shift) return shift_to_mmu_psize(shift); } +static inline pte_t arch_make_huge_pte (pte_t entry,unsigned int shift,vm_flags_t flags) +{ +unsigned int tsize = shift - _PAGE_PSIZE_SHIFT_OFFSET; +pte_basic_t val = (tsize << _PAGE_PSIZE_SHIFT) & _PAGE_PSIZE_MSK; + +return __pte ((pte_val (entry) & ~(pte_basic_t )_PAGE_PSIZE_MSK) | val); +} +#define arch_make_huge_pte arch_make_huge_pte +#endif /* _ASM_POWERPC_NOHASH_HUGETLB_E500_H */
arch/powerpc/include/asm/nohash/pte-e500.h +13 -15
@@ -19,20 +19,7 @@#define _PAGE_BAP_SX 0x000040#define _PAGE_BAP_UX 0x000080#define _PAGE_PSIZE_MSK 0x000f00 -#define _PAGE_PSIZE_4K 0x000200 -#define _PAGE_PSIZE_8K 0x000300 -#define _PAGE_PSIZE_16K 0x000400 -#define _PAGE_PSIZE_32K 0x000500 -#define _PAGE_PSIZE_64K 0x000600 -#define _PAGE_PSIZE_128K 0x000700 -#define _PAGE_PSIZE_256K 0x000800 -#define _PAGE_PSIZE_512K 0x000900 -#define _PAGE_PSIZE_1M 0x000a00 -#define _PAGE_PSIZE_2M 0x000b00 -#define _PAGE_PSIZE_4M 0x000c00 -#define _PAGE_PSIZE_8M 0x000d00 -#define _PAGE_PSIZE_16M 0x000e00 -#define _PAGE_PSIZE_32M 0x000f00 +#define _PAGE_TSIZE_4K 0x000100#define _PAGE_DIRTY 0x001000/* C: page changed */ #define _PAGE_SW0 0x002000#define _PAGE_U3 0x004000
@@ -46,6 +33,9 @@#define _PAGE_NO_CACHE 0x400000/* I: cache inhibit */ #define _PAGE_WRITETHRU 0x800000/* W: cache write-through */ +#define _PAGE_PSIZE_SHIFT 7 +#define _PAGE_PSIZE_SHIFT_OFFSET 10 +/* "Higher level" linux bit combinations */ #define _PAGE_EXEC (_PAGE_BAP_SX | _PAGE_BAP_UX)/* .. and was cache cleaned */ #define _PAGE_READ (_PAGE_BAP_SR | _PAGE_BAP_UR)/* User read permission */
@@ -87,7 +77,7 @@* pages. We always set _PAGE_COHERENT when SMP is enabled or * the processor might need it for DMA coherency. */ -#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_PSIZE_4K) +#define _PAGE_BASE_NC (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_TSIZE_4K)#if defined(CONFIG_SMP )#define _PAGE_BASE (_PAGE_BASE_NC | _PAGE_COHERENT)#else
@@ -103,6 +93,14 @@ static inline pte_t pte_mkexec(pte_t pte) }#define pte_mkexec pte_mkexec +static inline unsigned long pte_huge_size (pte_t pte) +{ +pte_basic_t val =pte_val (pte); + +return 1UL << (((val & _PAGE_PSIZE_MSK) >> _PAGE_PSIZE_SHIFT) + _PAGE_PSIZE_SHIFT_OFFSET); +} +#define pte_huge_size pte_huge_size +#endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */
Signature · 1 line
2.44.0
PATCH v5 13/18 powerpc/e500: Don't pre-check write access on data TLB error
In reply to Christophe LeroyDon't pre-check write access on read-only pages on data TLB error.
Load the TLB anyway and take a DSI exception when it happens. This avoids reading SPRN_ESR at every data TLB error exception.
---
v5: New
--- arch/powerpc/kernel/head_85xx.S | 15 --------------- 1 file changed, 15 deletions(-)
arch/powerpc/kernel/head_85xx.S +0 -15
@@ -472,27 +472,12 @@ END_BTB_FLUSH_SECTION #endif 4: - /* Mask of required permission bits. Note that while we - * do copy ESR:ST to _PAGE_WRITE position as trying to write - * to an RO page is pretty common, we don't do it with - * _PAGE_DIRTY. We could do it, but it's a fairly rare - * event so I'd rather take the overhead when it happens - * rather than adding an instruction here. We should measure - * whether the whole thing is worth it in the first place - * as we could avoid loading SPRN_ESR completely in the first - * place... - * - * TODO: Is it worth doing that mfspr & rlwimi in the first - * place or can we save a couple of instructions here ? - */ - mfspr r12,SPRN_ESR #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT|_PAGE_BAP_SR oris r13,r13,_PAGE_ACCESSED@h #else li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED #endif - rlwimi r13,r12,11,29,29 FIND_PTE andc. r13,r13,r11 /* Check permission */
Signature · 1 line
2.44.0
PATCH v5 14/18 powerpc/e500: Free r10 for FIND_PTE
In reply to Christophe LeroyMove r13 load after the call to FIND_PTE, and use r13 instead of r10 for storing fault address. This will allow using r10 freely in FIND_PTE in following patch to handle hugepage size.
---
v5: New
--- arch/powerpc/kernel/head_85xx.S | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-)
arch/powerpc/kernel/head_85xx.S +16 -14
@@ -294,9 +294,10 @@ set_ivor: /* Macros to hide the PTE size differences * * FIND_PTE -- walks the page tables given EA & pgdir pointer - * r10 -- EA of fault + * r10 -- free * r11 -- PGDIR pointer * r12 -- free + * r13 -- EA of fault * label 2: is the bailout case * * if we find the pte (fall through):
@@ -307,7 +308,7 @@ set_ivor: #ifdef CONFIG_PTE_64BIT #ifdef CONFIG_HUGETLB_PAGE #define FIND_PTE \ - rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \
@@ -317,26 +318,26 @@ set_ivor: andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ xor r12, r10, r11; /* drop size bits from pointer */ \ b 1001f; \ -1000: rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ +1000: rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ li r10, 0; /* clear r10 */ \ 1001: lwz r11, 4(r12); /* Get pte entry */ #else #define FIND_PTE \ - rlwinm r12, r10, 14, 18, 28; /* Compute pgdir/pmd offset */ \ + rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ beq 2f; /* Bail if no table */ \ - rlwimi r12, r10, 23, 20, 28; /* Compute pte address */ \ + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ lwz r11, 4(r12); /* Get pte entry */ #endif /* HUGEPAGE */ #else /* !PTE_64BIT */ #define FIND_PTE \ - rlwimi r11, r10, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ + rlwimi r11, r13, 12, 20, 29; /* Create L1 (pgdir/pmd) address */ \ lwz r11, 0(r11); /* Get L1 entry */ \ rlwinm. r12, r11, 0, 0, 19; /* Extract L2 (pte) base address */ \ beq 2f; /* Bail if no table */ \ - rlwimi r12, r10, 22, 20, 29; /* Compute PTE address */ \ + rlwimi r12, r13, 22, 20, 29; /* Compute PTE address */ \ lwz r11, 0(r12); /* Get Linux PTE */ #endif
@@ -443,13 +444,13 @@ START_BTB_FLUSH_SECTION BTB_FLUSH(r10) 1: END_BTB_FLUSH_SECTION - mfspr r10, SPRN_DEAR /* Get faulting address */ + mfspr r13, SPRN_DEAR /* Get faulting address */ /* If we are faulting a kernel address, we have to use the * kernel page tables. */ lis r11, PAGE_OFFSET@h - cmplw 5, r10, r11 + cmplw 5, r13, r11 blt 5, 3f lis r11, swapper_pg_dir@h ori r11, r11, swapper_pg_dir@l
@@ -472,14 +473,14 @@ END_BTB_FLUSH_SECTION #endif 4: + FIND_PTE + #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT|_PAGE_BAP_SR oris r13,r13,_PAGE_ACCESSED@h #else li r13,_PAGE_PRESENT|_PAGE_READ|_PAGE_ACCESSED #endif - - FIND_PTE andc. r13,r13,r11 /* Check permission */ #ifdef CONFIG_PTE_64BIT
@@ -536,13 +537,13 @@ START_BTB_FLUSH_SECTION 1: END_BTB_FLUSH_SECTION - mfspr r10, SPRN_SRR0 /* Get faulting address */ + mfspr r13, SPRN_SRR0 /* Get faulting address */ /* If we are faulting a kernel address, we have to use the * kernel page tables. */ lis r11, PAGE_OFFSET@h - cmplw 5, r10, r11 + cmplw 5, r13, r11 blt 5, 3f lis r11, swapper_pg_dir@h ori r11, r11, swapper_pg_dir@l
@@ -551,6 +552,7 @@ END_BTB_FLUSH_SECTION rlwinm r12,r12,0,16,1 mtspr SPRN_MAS1,r12 + FIND_PTE /* Make up the required permissions for kernel code */ #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT | _PAGE_BAP_SX
@@ -571,6 +573,7 @@ END_BTB_FLUSH_SECTION beq 2f /* KUAP fault */ #endif + FIND_PTE /* Make up the required permissions for user code */ #ifdef CONFIG_PTE_64BIT li r13,_PAGE_PRESENT | _PAGE_BAP_UX
@@ -580,7 +583,6 @@ END_BTB_FLUSH_SECTION #endif 4: - FIND_PTE andc. r13,r13,r11 /* Check permission */ #ifdef CONFIG_PTE_64BIT
Signature · 1 line
2.44.0
PATCH v5 15/18 powerpc/e500: Use contiguous PMD instead of hugepd
In reply to Christophe Leroye500 supports many page sizes among which the following size are implemented in the kernel at the time being: 4M, 16M, 64M, 256M, 1G.
On e500, TLB miss for hugepages is exclusively handled by SW even on e6500 which has HW assistance for 4k pages, so there are no constraints like on the 8xx.
On e500/32, all are at PGD/PMD level and can be handled as cont-PMD.
On e500/64, smaller ones are on PMD while bigger ones are on PUD. Again, they can easily be handled as cont-PMD and cont-PUD instead of hugepd.
On e500/32, use the pagesize bits in PTE to know if it is a PMD or a leaf entry. This works because the pagesize bits are in the last 12 bits and page tables are 4k aligned.
On e500/64, use highest bit which is always 1 on PxD (Because PxD contains virtual address of a kernel memory) and always 0 on PTEs because not all bits of RPN are used/possible.
---
v3: Add missing pmd_leaf_size() and pud_leaf_size()
v4: Rebased of v6.10-rc1 : pmd_huge() and pud_huge() are gone
v5:
- Define pte_huge_size() to reduce redundant and handle the case of mpc85xx with 32 bits PTE.
- Replace hard-coded page shift offset (10) with new macro _PAGE_PSIZE_SHIFT_OFFSET
- Use TSIZE encoding in SIZE field in PTE
- On 64 bits, use highest bit to know if it is a PUD/PMD or leaf entry. Avoids problem with page table alignment.
--- .../powerpc/include/asm/nohash/hugetlb-e500.h | 32 ++------------ arch/powerpc/include/asm/nohash/pgalloc.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 42 ++++++++++++++----- arch/powerpc/include/asm/nohash/pte-e500.h | 33 +++++++++++++++ arch/powerpc/include/asm/page.h | 15 +------ arch/powerpc/kernel/head_85xx.S | 21 ++++------ arch/powerpc/mm/hugetlbpage.c | 2 - arch/powerpc/mm/nohash/tlb_low_64e.S | 7 +--- arch/powerpc/mm/pgtable.c | 31 ++++++++++++++ arch/powerpc/platforms/Kconfig.cputype | 1 - 10 files changed, 107 insertions(+), 79 deletions(-)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +3 -29
@@ -2,38 +2,12 @@#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -if (WARN_ON (!hugepd_ok (hpd))) -return NULL ; - -return (pte_t *)((hpd_val (hpd) & ~HUGEPD_SHIFT_MASK ) |PD_HUGE ); -} - -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return hpd_val (hpd) &HUGEPD_SHIFT_MASK ; -} - -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -/* -* On FSL BookE, we have multiple higher-level table entries that -* point to the same hugepte. Just use the first one since they're all -* identical. So for that case, idx=0. -*/ -return hugepd_page (hpd); -} +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz);void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr); -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ -/* We use the old format for PPC_E500 */ - *hpdp =__hugepd (((unsigned long )new & ~PD_HUGE ) | pshift); -} -static inline int check_and_get_huge_psize (int shift) {if (shift &1 )/* Not a power of 4 */
arch/powerpc/include/asm/nohash/pgalloc.h +0 -2
@@ -44,8 +44,6 @@ static inline void pgtable_free(void *table, int shift) } } -#define get_hugepd_cache_index (x) (x) -static inline void pgtable_free_tlb (struct mmu_gather *tlb,void *table,int shift) {unsigned long pgf = (unsigned long )table;
arch/powerpc/include/asm/nohash/pgtable.h +31 -11
64 lines not on this page · raw message
arch/powerpc/include/asm/nohash/pte-e500.h +33 -0
@@ -101,6 +101,39 @@ static inline unsigned long pte_huge_size(pte_t pte) }#define pte_huge_size pte_huge_size +static inline int pmd_leaf (pmd_t pmd) +{ +if (IS_ENABLED (CONFIG_PPC64 )) +return (long )pmd_val (pmd) >0 ; +else +return pmd_val (pmd) & _PAGE_PSIZE_MSK; +} +#define pmd_leaf pmd_leaf + +static inline unsigned long pmd_leaf_size (pmd_t pmd) +{ +return pte_huge_size (__pte (pmd_val (pmd))); +} +#define pmd_leaf_size pmd_leaf_size + +#ifdef CONFIG_PPC64 +static inline int pud_leaf (pud_t pud) +{ +if (IS_ENABLED (CONFIG_PPC64 )) +return (long )pud_val (pud) >0 ; +else +return pud_val (pud) & _PAGE_PSIZE_MSK; +} +#define pud_leaf pud_leaf + +static inline unsigned long pud_leaf_size (pud_t pud) +{ +return pte_huge_size (__pte (pud_val (pud))); +} +#define pud_leaf_size pud_leaf_size + +#endif +#endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */
arch/powerpc/include/asm/page.h +1 -14
@@ -269,20 +269,7 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -#ifndef CONFIG_PPC_BOOK3S_64 -/* -* Use the top bit of the higher-level page table entries to indicate whether -* the entries we point to contain hugepages. This works because we know that -* the page tables live in kernel space. If we ever decide to support having -* page tables at arbitrary addresses, this breaks and will have to change. -*/ -#ifdef CONFIG_PPC64 -#define PD_HUGE 0x8000000000000000UL -#else -#define PD_HUGE 0x80000000 -#endif - -#else /* CONFIG_PPC_BOOK3S_64 */ +#ifdef CONFIG_PPC_BOOK3S_64 /* * Book3S 64 stores real addresses in the hugepd entries to * avoid overlaps with _PAGE_PRESENT and _PAGE_PTE.
arch/powerpc/kernel/head_85xx.S +7 -14
@@ -311,16 +311,14 @@ set_ivor: rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \ add r12, r11, r12; \ lwz r11, 4(r12); /* Get pgd/pmd entry */ \ + rlwinm. r10, r11, 32 - _PAGE_PSIZE_SHIFT, 0x1e; /* get tsize*/ \ + bne 1000f; /* Huge page (leaf entry) */ \ rlwinm. r12, r11, 0, 0, 20; /* Extract pt base address */ \ - blt 1000f; /* Normal non-huge page */ \ beq 2f; /* Bail if no table */ \ - oris r11, r11, PD_HUGE@h; /* Put back address bit */ \ - andi. r10, r11, HUGEPD_SHIFT_MASK@l; /* extract size field */ \ - xor r12, r10, r11; /* drop size bits from pointer */ \ - b 1001f; \ -1000: rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ + rlwimi r12, r13, 23, 20, 28; /* Compute pte address */ \ li r10, 0; /* clear r10 */ \ -1001: lwz r11, 4(r12); /* Get pte entry */ + lwz r11, 4(r12); /* Get pte entry */ \ +1000: #else #define FIND_PTE \ rlwinm r12, r13, 14, 18, 28; /* Compute pgdir/pmd offset */ \
@@ -735,17 +733,12 @@ finish_tlb_load: lwz r15, 0(r14) 100: stw r15, 0(r17) - /* - * Calc MAS1_TSIZE from r10 (which has pshift encoded) - * tlb_enc = (pshift - 10). - */ - subi r15, r10, 10 mfspr r16, SPRN_MAS1 - rlwimi r16, r15, 7, 20, 24 + rlwimi r16, r10, MAS1_TSIZE_SHIFT, MAS1_TSIZE_MASK mtspr SPRN_MAS1, r16 /* copy the pshift for use later */ - mr r14, r10 + addi r14, r10, _PAGE_PSIZE_SHIFT_OFFSET /* fall through */
arch/powerpc/mm/hugetlbpage.c +0 -2
@@ -625,8 +625,6 @@ static int __init hugetlbpage_init(void)*/ if (pdshift > shift) {pgtable_cache_add (pdshift - shift); - }else if (IS_ENABLED (CONFIG_PPC_E500 )) { -pgtable_cache_add (PTE_T_ORDER ); } configured = true;
arch/powerpc/mm/nohash/tlb_low_64e.S +1 -6
@@ -450,11 +450,7 @@ END_FTR_SECTION_IFSET(CPU_FTR_SMT) tlb_miss_huge_e6500: beq tlb_miss_fault_e6500 - li r10,1 - andi. r15,r14,HUGEPD_SHIFT_MASK@l /* r15 = psize */ - rldimi r14,r10,63,0 /* Set PD_HUGE */ - xor r14,r14,r15 /* Clear size bits */ - ldx r14,0,r14 + rlwinm r15,r14,32-_PAGE_PSIZE_SHIFT,0x1e /* * Now we build the MAS for a huge page.
@@ -465,7 +461,6 @@ tlb_miss_huge_e6500: * MAS 2,3+7: Needs to be redone similar to non-tablewalk handler */ - subi r15,r15,10 /* Convert psize to tsize */ mfspr r10,SPRN_MAS1 rlwinm r10,r10,0,~MAS1_IND rlwimi r10,r15,MAS1_TSIZE_SHIFT,MAS1_TSIZE_MASK
arch/powerpc/mm/pgtable.c +31 -0
@@ -331,6 +331,37 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep,__set_huge_pte_at (pmdp, ptep,pte_val (pte)); } } +#elif defined(CONFIG_PPC_E500 ) +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz) +{ +unsigned long pdsize; +int i; + + pte =set_pte_filter (pte, addr); + +/* +* Make sure hardware valid bit is not set. We don't do +* tlb flush for this update. +*/ +VM_WARN_ON (pte_hw_valid (*ptep) && !pte_protnone (*ptep)); + +if (sz <PMD_SIZE ) + pdsize =PAGE_SIZE ; +else if (sz <PUD_SIZE ) + pdsize =PMD_SIZE ; +else if (sz <P4D_SIZE ) + pdsize =PUD_SIZE ; +else if (sz <PGDIR_SIZE ) + pdsize =P4D_SIZE ; +else + pdsize =PGDIR_SIZE ; + +for (i =0 ; i < sz / pdsize; i++, ptep++, addr += pdsize) { +__set_pte_at (mm, addr, ptep, pte,0 ); + pte =__pte (pte_val (pte) + ((unsigned long long )pdsize /PAGE_SIZE <<PFN_PTE_SHIFT )); + } +}#endif #endif /* CONFIG_HUGETLB_PAGE */
arch/powerpc/platforms/Kconfig.cputype +0 -1
@@ -291,7 +291,6 @@ config PPC_BOOK3S config PPC_E500 select FSL_EMB_PERFMON bool - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS if PHYS_64BIT || PPC64 select PPC_SMP_MUXED_IPI select PPC_DOORBELL
Signature · 1 line
2.44.0
PATCH v5 16/18 powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD
In reply to Christophe LeroyOn book3s/64, the only user of hugepd is hash in 4k mode.
All other setups (hash-64, radix-4, radix-64) use leaf PMD/PUD.
Rework hash-4k to use contiguous PMD and PUD instead.
In that setup there are only two huge page sizes: 16M and 16G.
16M sits at PMD level and 16G at PUD level.
pte_update doesn't know page size, lets use the same trick as hpte_need_flush() to get page size from segment properties. That's not the most efficient way but let's do that until callers of pte_update() provide page size instead of just a huge flag.
---
v3:
- Add missing pmd_leaf_size() and pud_leaf_size()
- More cleanup in hugetlbpage_init()
- Take a page fault when DIRTY or ACCESSED is missing on hash-4 hugepage
v4: Rebased on v6.10-rc1
--- arch/powerpc/include/asm/book3s/64/hash-4k.h | 15 ------ arch/powerpc/include/asm/book3s/64/hash.h | 38 ++++++++++++--- arch/powerpc/include/asm/book3s/64/hugetlb.h | 38 --------------- .../include/asm/book3s/64/pgtable-4k.h | 47 ------------------- .../include/asm/book3s/64/pgtable-64k.h | 20 -------- arch/powerpc/include/asm/book3s/64/pgtable.h | 22 +++++++-- arch/powerpc/include/asm/hugetlb.h | 4 ++ .../powerpc/include/asm/nohash/hugetlb-e500.h | 4 -- arch/powerpc/include/asm/page.h | 8 ---- arch/powerpc/mm/book3s64/hash_utils.c | 11 +++-- arch/powerpc/mm/book3s64/hugetlbpage.c | 10 ++++ arch/powerpc/mm/book3s64/pgtable.c | 12 ----- arch/powerpc/mm/hugetlbpage.c | 26 ---------- arch/powerpc/mm/pgtable.c | 2 +- arch/powerpc/platforms/Kconfig.cputype | 1 - 15 files changed, 72 insertions(+), 186 deletions(-) delete mode 100644 arch/powerpc/include/asm/book3s/64/pgtable-4k.h
arch/powerpc/include/asm/book3s/64/hash-4k.h +0 -15
@@ -74,21 +74,6 @@#define remap_4k_pfn (vma, addr, pfn, prot) \ remap_pfn_range((vma), (addr), (pfn), PAGE_SIZE, (prot)) -#ifdef CONFIG_HUGETLB_PAGE -static inline int hash__hugepd_ok (hugepd_t hpd) -{ -unsigned long hpdval =hpd_val (hpd); -/* -* if it is not a pte and have hugepd shift mask -* set, then it is a hugepd directory pointer -*/ -if (!(hpdval & _PAGE_PTE) && (hpdval & _PAGE_PRESENT) && - ((hpdval &HUGEPD_SHIFT_MASK ) !=0 )) -return true; -return false; -} -#endif -/* * 4K PTE format is different from 64K PTE format. Saving the hash_slot is just * a matter of returning the PTE bits that need to be modified. On 64K PTE,
arch/powerpc/include/asm/book3s/64/hash.h +31 -7
@@ -4,6 +4,7 @@#ifdef __KERNEL__#include <asm/asm-const.h> +#include <asm/book3s/64/slice.h> /* * Common bits between 4K and 64K pages in a linux-style PTE.
@@ -161,14 +162,10 @@ extern void hpte_need_flush(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long pte, int huge);unsigned long htab_convert_pte_flags (unsigned long pteflags,unsigned long flags);/* Atomic PTE updates */ -static inline unsigned long hash__pte_update (struct mm_struct *mm, -unsigned long addr, -pte_t *ptep,unsigned long clr, -unsigned long set, -int huge) +static inline unsigned long hash__pte_update_one (pte_t *ptep,unsigned long clr, +unsigned long set) {__be64 old_be, tmp_be; -unsigned long old; __asm__ __volatile__( "1: ldarx %0,0,%3 # pte_update\n\
@@ -182,11 +179,38 @@ static inline unsigned long hash__pte_update(struct mm_struct *mm, : "r" (ptep), "r" (cpu_to_be64(clr)), "m" (*ptep), "r" (cpu_to_be64(H_PAGE_BUSY)), "r" (cpu_to_be64(set)) : "cc" ); + +return be64_to_cpu (old_be); +} + +static inline unsigned long hash__pte_update (struct mm_struct *mm, +unsigned long addr, +pte_t *ptep,unsigned long clr, +unsigned long set, +int huge) +{ +unsigned long old; + + old =hash__pte_update_one (ptep, clr, set); + +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && huge) { +unsigned int psize =get_slice_psize (mm, addr); +int nb, i; + +if (psize ==MMU_PAGE_16M ) + nb =SZ_16M /PMD_SIZE ; +else if (psize ==MMU_PAGE_16G ) + nb =SZ_16G /PUD_SIZE ; +else + nb =1 ; + +for (i =1 ; i < nb; i++) +hash__pte_update_one (ptep + i, clr, set); + }/* huge pages use the old page table lock */ if (!huge)assert_pte_locked (mm, addr); - old =be64_to_cpu (old_be);if (old &H_PAGE_HASHPTE )hpte_need_flush (mm, addr, ptep, old, huge);
arch/powerpc/include/asm/book3s/64/hugetlb.h +0 -38
@@ -49,9 +49,6 @@ static inline bool gigantic_page_runtime_supported(void)return true; } -/* hugepd entry valid bit */ -#define HUGEPD_VAL_BITS (0x8000000000000000UL) -#define huge_ptep_modify_prot_start huge_ptep_modify_prot_startextern pte_t huge_ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep);
@@ -60,29 +57,7 @@ extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,extern void huge_ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep,pte_t old_pte,pte_t new_pte); -/* -* This should work for other subarchs too. But right now we use the -* new format only for 64bit book3s -*/ -static inline pte_t *hugepd_page (hugepd_t hpd) -{ -BUG_ON (!hugepd_ok (hpd)); -/* -* We have only four bits to encode, MMU page size -*/ -BUILD_BUG_ON ((MMU_PAGE_COUNT -1 ) >0xf ); -return __va (hpd_val (hpd) &HUGEPD_ADDR_MASK ); -} - -static inline unsigned int hugepd_mmu_psize (hugepd_t hpd) -{ -return (hpd_val (hpd) &HUGEPD_SHIFT_MASK ) >>2 ; -} -static inline unsigned int hugepd_shift (hugepd_t hpd) -{ -return mmu_psize_to_shift (hugepd_mmu_psize (hpd)); -}static inline void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr) {
@@ -90,19 +65,6 @@ static inline void flush_hugetlb_page(struct vm_area_struct *vma,return radix__flush_hugetlb_page (vma, vmaddr); } -static inline pte_t *hugepte_offset (hugepd_t hpd,unsigned long addr, -unsigned int pdshift) -{ -unsigned long idx = (addr & ((1UL << pdshift) -1 )) >>hugepd_shift (hpd); - -return hugepd_page (hpd) + idx; -} - -static inline void hugepd_populate (hugepd_t *hpdp,pte_t *new,unsigned int pshift) -{ - *hpdp =__hugepd (__pa (new) |HUGEPD_VAL_BITS | (shift_to_mmu_psize (pshift) <<2 )); -} -void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr);static inline int check_and_get_huge_psize (int shift)
arch/powerpc/include/asm/book3s/64/pgtable-4k.h deleted +0 -47
47 lines not on this page · raw message
arch/powerpc/include/asm/book3s/64/pgtable-64k.h +0 -20
@@ -5,26 +5,6 @@#ifndef __ASSEMBLY__#ifdef CONFIG_HUGETLB_PAGE -/* -* With 64k page size, we have hugepage ptes in the pgd and pmd entries. We don't -* need to setup hugepage directory for them. Our pte and page directory format -* enable us to have this enabled. -*/ -static inline int hugepd_ok (hugepd_t hpd) -{ -return 0 ; -} - -#define is_hugepd (pdep) 0 - -/* -* This should never get called -*/ -static __always_inline int get_hugepd_cache_index(int index) -{ -BUILD_BUG (); -} -#endif /* CONFIG_HUGETLB_PAGE */ static inline int remap_4k_pfn(struct vm_area_struct *vma, unsigned long addr,
arch/powerpc/include/asm/book3s/64/pgtable.h +19 -3
@@ -274,6 +274,24 @@ static inline bool pud_leaf(pud_t pud) {return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); } + +#define pmd_leaf_size pmd_leaf_size +static inline unsigned long pmd_leaf_size (pmd_t pmd) +{ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) +return SZ_16M ; +else +return PMD_SIZE ; +} + +#define pud_leaf_size pud_leaf_size +static inline unsigned long pud_leaf_size (pud_t pud) +{ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) +return SZ_16G ; +else +return PUD_SIZE ; +}#endif /* __ASSEMBLY__ */ #include <asm/book3s/64/hash.h>
@@ -285,11 +303,9 @@ static inline bool pud_leaf(pud_t pud)#define MAX_PHYSMEM_BITS R_MAX_PHYSMEM_BITS#endif - +/* hash 4k can't share hugetlb and also doesn't support THP */ #ifdef CONFIG_PPC_64K_PAGES #include <asm/book3s/64/pgtable-64k.h> -#else -#include <asm/book3s/64/pgtable-4k.h> #endif #include <asm/barrier.h>
arch/powerpc/include/asm/hugetlb.h +4 -0
@@ -37,6 +37,10 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long ceiling);#endif +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +pte_t pte,unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEARstatic inline pte_t huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr,pte_t *ptep)
arch/powerpc/include/asm/nohash/hugetlb-e500.h +0 -4
@@ -2,10 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_HUGETLB_E500_H#define _ASM_POWERPC_NOHASH_HUGETLB_E500_H -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep, -pte_t pte,unsigned long sz); -void flush_hugetlb_page (struct vm_area_struct *vma,unsigned long vmaddr);static inline int check_and_get_huge_psize (int shift)
arch/powerpc/include/asm/page.h +0 -8
@@ -269,14 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -#ifdef CONFIG_PPC_BOOK3S_64 -/* -* Book3S 64 stores real addresses in the hugepd entries to -* avoid overlaps with _PAGE_PRESENT and _PAGE_PTE. -*/ -#define HUGEPD_ADDR_MASK (0x0ffffffffffffffful & ~HUGEPD_SHIFT_MASK) -#endif /* CONFIG_PPC_BOOK3S_64 */ -/* * Some number of bits at the level of the page table that points to * a hugepte are used to encode the size. This masks those bits.
arch/powerpc/mm/book3s64/hash_utils.c +7 -4
@@ -1233,10 +1233,6 @@ void __init hash__early_init_mmu(void) __pmd_table_size =H_PMD_TABLE_SIZE ; __pud_table_size =H_PUD_TABLE_SIZE ; __pgd_table_size =H_PGD_TABLE_SIZE ; -/* -* 4k use hugepd format, so for hash set then to -* zero -*/ __pmd_val_bits =HASH_PMD_VAL_BITS ; __pud_val_bits =HASH_PUD_VAL_BITS ; __pgd_val_bits =HASH_PGD_VAL_BITS ;
@@ -1546,6 +1542,13 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, goto bail; } +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && !radix_enabled ()) { +if (hugeshift ==PMD_SHIFT && psize ==MMU_PAGE_16M ) + hugeshift = mmu_psize_defs[MMU_PAGE_16M ].shift; +if (hugeshift ==PUD_SHIFT && psize ==MMU_PAGE_16G ) + hugeshift = mmu_psize_defs[MMU_PAGE_16G ].shift; + } +/* * Add _PAGE_PRESENT to the required access perm. If there are parallel * updates to the pte that can possibly clear _PAGE_PTE, catch that too.
arch/powerpc/mm/book3s64/hugetlbpage.c +10 -0
@@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,/* If PTE permissions don't match, take page fault */ if (unlikely (!check_pte_access (access, old_pte)))return 1 ; +/* +* If hash-4k, hugepages use seeral contiguous PxD entries +* so bail out and let mm make the page young or dirty +*/ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES )) { +if (!(old_pte & _PAGE_ACCESSED)) +return 1 ; +if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY)) +return 1 ; + }/* * Try to lock the PTE, add ACCESSED and DIRTY if it was
arch/powerpc/mm/book3s64/pgtable.c +0 -12
@@ -461,18 +461,6 @@ static inline void pgtable_free(void *table, int index)case PUD_INDEX :__pud_free (table);break ; -#if defined(CONFIG_PPC_4K_PAGES ) && defined(CONFIG_HUGETLB_PAGE ) -/* 16M hugepd directory at pud level */ -case HTLB_16M_INDEX : -BUILD_BUG_ON (H_16M_CACHE_INDEX <=0 ); -kmem_cache_free (PGT_CACHE (H_16M_CACHE_INDEX ), table); -break ; -/* 16G hugepd directory at the pgd level */ -case HTLB_16G_INDEX : -BUILD_BUG_ON (H_16G_CACHE_INDEX <=0 ); -kmem_cache_free (PGT_CACHE (H_16G_CACHE_INDEX ), table); -break ; -#endif /* We don't free pgd table via RCU callback */ default :BUG ();
arch/powerpc/mm/hugetlbpage.c +0 -26
@@ -592,40 +592,14 @@ static int __init hugetlbpage_init(void)for (psize =0 ; psize <MMU_PAGE_COUNT ; ++psize) {unsigned shift; -unsigned pdshift;if (!mmu_psize_defs[psize].shift)continue ; shift =mmu_psize_to_shift (psize); -#ifdef CONFIG_PPC_BOOK3S_64 -if (shift >PGDIR_SHIFT ) -continue ; -else if (shift >PUD_SHIFT ) - pdshift =PGDIR_SHIFT ; -else if (shift >PMD_SHIFT ) - pdshift =PUD_SHIFT ; -else - pdshift =PMD_SHIFT ; -#else -if (shift <PUD_SHIFT ) - pdshift =PMD_SHIFT ; -else if (shift <PGDIR_SHIFT ) - pdshift =PUD_SHIFT ; -else - pdshift =PGDIR_SHIFT ; -#endif -if (add_huge_page_size (1ULL << shift) <0 )continue ; -/* -* if we have pdshift and shift value same, we don't -* use pgt cache for hugepd. -*/ -if (pdshift > shift) { -pgtable_cache_add (pdshift - shift); - } configured = true; }
arch/powerpc/mm/pgtable.c +1 -1
@@ -331,7 +331,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, __set_huge_pte_at(pmdp, ptep, pte_val(pte)); } } -#elif defined(CONFIG_PPC_E500) +#else void set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz) {
arch/powerpc/platforms/Kconfig.cputype +0 -1
@@ -98,7 +98,6 @@ config PPC_BOOK3S_64 select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION select ARCH_ENABLE_SPLIT_PMD_PTLOCK select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE - select ARCH_HAS_HUGEPD if HUGETLB_PAGE select ARCH_SUPPORTS_HUGETLBFS select ARCH_SUPPORTS_NUMA_BALANCING select HAVE_MOVE_PMD
Signature · 1 line
2.44.0
Oscar Salvador
maintainerRe: [PATCH v5 16/18] powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD
Quoted 17 lines · Christophe Leroy
On book3s/64, the only user of hugepd is hash in 4k mode.
All other setups (hash-64, radix-4, radix-64) use leaf PMD/PUD.
Rework hash-4k to use contiguous PMD and PUD instead.
In that setup there are only two huge page sizes: 16M and 16G.
16M sits at PMD level and 16G at PUD level.
pte_updatedoesn't know page size, lets use the same trick ashpte_need_flush()to get page size from segment properties. That's not the most efficient way but let's do that until callers ofpte_update()provide page size instead of just a huge flag.Signed-off-by Christophe Leroy <[email protected]>---
...
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/hash.h +10
-0+static inline unsigned long hash__pte_update (struct mm_struct *mm, +unsigned long addr, +pte_t *ptep,unsigned long clr, +unsigned long set, +int huge) +{ +unsigned long old; + + old =hash__pte_update_one (ptep, clr, set); +
arch/powerpc/include/asm/book3s/64/hash.h static inline unsigned long hash__pte_update(struct mm_struct *mm, · line 205
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/hash.h +10
-0+if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && huge) { +unsigned int psize =get_slice_psize (mm, addr); +int nb, i; + +if (psize ==MMU_PAGE_16M ) + nb =SZ_16M /PMD_SIZE ; +else if (psize ==MMU_PAGE_16G ) + nb =SZ_16G /PUD_SIZE ; +else + nb =1 ;
Although that might be a safe default, it might carry consequences down the road? It might not, but if we reach that, something went wrong, so I would put a WARN_ON_ONCE at least.
Quoted 6 lines
arch/powerpc/mm/book3s64/hugetlbpage.c +0
-0@@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,/* If PTE permissions don't match, take page fault */ if (unlikely (!check_pte_access (access, old_pte)))return 1 ;
arch/powerpc/mm/book3s64/hugetlbpage.c int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · line 65
Quoted 10 lines
arch/powerpc/mm/book3s64/hugetlbpage.c +10
-0+/* +* If hash-4k, hugepages use seeral contiguous PxD entries +* so bail out and let mm make the page young or dirty +*/ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES )) { +if (!(old_pte & _PAGE_ACCESSED)) +return 1 ; +if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY)) +return 1 ; + }
You mentioned that we need to bail out otherwise only the first PxD would be updated.
In the comment you say that mm will take care of making the page young or dirty.
Does this mean that the PxDs underneath will not have its bits updated?
Signature · 2 lines
Oscar Salvador SUSE Labs
Re: [PATCH v5 16/18] powerpc/64s: Use contiguous PMD/PUD instead of HUGEPD
Le 13/06/2024 à 09:39, Oscar Salvador a écrit :
Quoted 35 lines · Christophe Leroy
Quoted 17 lines
On book3s/64, the only user of hugepd is hash in 4k mode.
All other setups (hash-64, radix-4, radix-64) use leaf PMD/PUD.
Rework hash-4k to use contiguous PMD and PUD instead.
In that setup there are only two huge page sizes: 16M and 16G.
16M sits at PMD level and 16G at PUD level.
pte_updatedoesn't know page size, lets use the same trick ashpte_need_flush()to get page size from segment properties. That's not the most efficient way but let's do that until callers ofpte_update()provide page size instead of just a huge flag.Signed-off-by Christophe Leroy <[email protected]>---
...
Quoted 16 lines
arch/powerpc/include/asm/book3s/64/hash.h +16
-0+static inline unsigned long hash__pte_update (struct mm_struct *mm, +unsigned long addr, +pte_t *ptep,unsigned long clr, +unsigned long set, +int huge) +{ +unsigned long old; + + old =hash__pte_update_one (ptep, clr, set); + +if (IS_ENABLED (CONFIG_PPC_4K_PAGES ) && huge) { +unsigned int psize =get_slice_psize (mm, addr); +int nb, i; + +if (psize ==MMU_PAGE_16M ) + nb =SZ_16M /PMD_SIZE ;
Quoted 8 lines
arch/powerpc/include/asm/book3s/64/hash.h static inline unsigned long hash__pte_update(struct mm_struct *mm, · line 205
Quoted 4 lines
arch/powerpc/include/asm/book3s/64/hash.h +4
-0+else if (psize ==MMU_PAGE_16G ) + nb =SZ_16G /PUD_SIZE ; +else + nb =1 ;Although that might be a safe default, it might carry consequences down the road? It might not, but if we reach that, something went wrong, so I would put a
WARN_ON_ONCEat least.
Ah right you said it already and I forgot it.
It is now in v6.
Quoted 15 lines
Quoted 15 lines
arch/powerpc/mm/book3s64/hugetlbpage.c +8
-0@@ -53,6 +53,16 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,/* If PTE permissions don't match, take page fault */ if (unlikely (!check_pte_access (access, old_pte)))return 1 ; +/* +* If hash-4k, hugepages use seeral contiguous PxD entries +* so bail out and let mm make the page young or dirty +*/ +if (IS_ENABLED (CONFIG_PPC_4K_PAGES )) { +if (!(old_pte & _PAGE_ACCESSED)) +return 1 ; +if ((access & _PAGE_WRITE) && !(old_pte & _PAGE_DIRTY))
Quoted 10 lines
arch/powerpc/mm/book3s64/hugetlbpage.c int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid, · line 65
Quoted 2 lines
arch/powerpc/mm/book3s64/hugetlbpage.c +2
-0+return 1 ; + }You mentioned that we need to bail out otherwise only the first PxD would be updated.
In the comment you say that mm will take care of making the page young or dirty.
Does this mean that the PxDs underneath will not have its bits updated?
No it just means that mm will do it properly by using huge_ptep_set_access_flags()
Christophe
PATCH v5 17/18 powerpc/mm: Remove hugepd leftovers
In reply to Christophe LeroyAll targets have now opted out of CONFIG_ARCH_HAS_HUGEPD so remove left over code.
---
v5: Fix a forgotten #endif which ended up in following patch
--- arch/powerpc/include/asm/hugetlb.h | 7 - arch/powerpc/include/asm/page.h | 6 - arch/powerpc/include/asm/pgtable-be-types.h | 10 - arch/powerpc/include/asm/pgtable-types.h | 9 - arch/powerpc/mm/hugetlbpage.c | 413 -------------------- arch/powerpc/mm/init-common.c | 8 +- arch/powerpc/mm/pgtable.c | 27 +- 7 files changed, 3 insertions(+), 477 deletions(-)
arch/powerpc/include/asm/hugetlb.h +0 -7
@@ -30,13 +30,6 @@ static inline int is_hugepage_only_range(struct mm_struct *mm, }#define is_hugepage_only_range is_hugepage_only_range -#ifdef CONFIG_ARCH_HAS_HUGEPD -#define __HAVE_ARCH_HUGETLB_FREE_PGD_RANGE -void hugetlb_free_pgd_range (struct mmu_gather *tlb,unsigned long addr, -unsigned long end,unsigned long floor, -unsigned long ceiling); -#endif -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_ATvoid set_huge_pte_at (struct mm_struct *mm,unsigned long addr,pte_t *ptep,pte_t pte,unsigned long sz);
arch/powerpc/include/asm/page.h +0 -6
@@ -269,12 +269,6 @@ static inline const void *pfn_to_kaddr(unsigned long pfn)#define is_kernel_addr (x) ((x) >= TASK_SIZE)#endif -/* -* Some number of bits at the level of the page table that points to -* a hugepte are used to encode the size. This masks those bits. -*/ -#define HUGEPD_SHIFT_MASK 0x3f -#ifndef __ASSEMBLY__#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/include/asm/pgtable-be-types.h +0 -10
@@ -101,14 +101,4 @@ static inline bool pmd_xchg(pmd_t *pmdp, pmd_t old, pmd_t new) return pmd_raw(old) == prev; } -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct {__be64 pdbe; }hugepd_t ; -#define __hugepd (x) ((hugepd_t) { cpu_to_be64(x) }) - -static inline unsigned long hpd_val (hugepd_t x) -{ -return be64_to_cpu (x.pdbe); -} -#endif -#endif /* _ASM_POWERPC_PGTABLE_BE_TYPES_H */
arch/powerpc/include/asm/pgtable-types.h +0 -9
@@ -87,13 +87,4 @@ static inline bool pte_xchg(pte_t *ptep, pte_t old, pte_t new) }#endif -#ifdef CONFIG_ARCH_HAS_HUGEPD -typedef struct {unsigned long pd; }hugepd_t ; -#define __hugepd (x) ((hugepd_t) { (x) }) -static inline unsigned long hpd_val (hugepd_t x) -{ -return x.pd; -} -#endif -#endif /* _ASM_POWERPC_PGTABLE_TYPES_H */
arch/powerpc/mm/hugetlbpage.c +0 -413
437 lines not on this page · raw message
arch/powerpc/mm/init-common.c +2 -6
@@ -120,12 +120,8 @@ void pgtable_cache_add(unsigned int shift)/* When batching pgtable pointers for RCU freeing, we store * the index size in the low bits. Table alignment must be * big enough to fit it. -* -* Likewise, hugeapge pagetable pointers contain a (different) -* shift value in the low bits. All tables must be aligned so -* as to leave enough 0 bits in the address to contain it. */ -unsigned long minalign =max (MAX_PGTABLE_INDEX_SIZE +1 , -HUGEPD_SHIFT_MASK +1 ); +*/ +unsigned long minalign =MAX_PGTABLE_INDEX_SIZE +1 ;struct kmem_cache *new =NULL ;/* It would be nice if this was a BUILD_BUG_ON(), but at the
arch/powerpc/mm/pgtable.c +1 -26
@@ -409,11 +409,10 @@ unsigned long vmalloc_to_phys(void *va)EXPORT_SYMBOL_GPL (vmalloc_to_phys);/* -* We have 4 cases for pgds and pmds: +* We have 3 cases for pgds and pmds: * (1) invalid (all zeroes) * (2) pointer to next table, as normal; bottom 6 bits == 0 * (3) leaf pte for huge page _PAGE_PTE set -* (4) hugepd pointer, _PAGE_PTE = 0 and bits [2..6] indicate size of table * * So long as we atomically load page table pointers we are safe against teardown, * we can follow the address down to the page and take a ref on it.
@@ -430,7 +429,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,#endif pmd_t pmd, *pmdp;pte_t *ret_pte; -hugepd_t *hpdp =NULL ;unsigned pdshift;if (hpage_shift)
@@ -463,11 +461,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (p4d_val (p4d)))) { - hpdp = (hugepd_t *)&p4d; - goto out_huge; - } -/* * Even if we end up with an unmap, the pgtable will not * be freed, because we do an rcu free and here we are
@@ -485,11 +478,6 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (pud_val (pud)))) { - hpdp = (hugepd_t *)&pud; - goto out_huge; - } - pmdp =pmd_offset (&pud, ea);#else pmdp =pmd_offset (pud_offset (p4d_offset (pgdp, ea), ea), ea);
@@ -527,21 +515,8 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out; } -if (is_hugepd (__hugepd (pmd_val (pmd)))) { - hpdp = (hugepd_t *)&pmd; - goto out_huge; - } -return pte_offset_kernel (&pmd, ea); -out_huge: -if (!hpdp) -return NULL ; - -#ifdef CONFIG_ARCH_HAS_HUGEPD - ret_pte =hugepte_offset (*hpdp, ea, pdshift); - pdshift =hugepd_shift (*hpdp); -#endif out:if (hpage_shift) *hpage_shift = pdshift;
Signature · 1 line
2.44.0
PATCH v5 18/18 mm: Remove CONFIG_ARCH_HAS_HUGEPD
In reply to Christophe Leroypowerpc was the only user of CONFIG_ARCH_HAS_HUGEPD and doesn't use it anymore, so remove all related code.
---
v4: Rebased on v6.10-rc1
--- include/linux/hugetlb.h | 6 -- mm/Kconfig | 10 --- mm/gup.c | 183 ++-------------------------------------- mm/pagewalk.c | 57 +------------ 4 files changed, 9 insertions(+), 247 deletions(-)
include/linux/hugetlb.h +0 -6
@@ -20,12 +20,6 @@ struct user_struct;struct mmu_gather ;struct node ; -#ifndef CONFIG_ARCH_HAS_HUGEPD -typedef struct {unsigned long pd; }hugepd_t ; -#define is_hugepd (hugepd) (0) -#define __hugepd (x) ((hugepd_t) { (x) }) -#endif -void free_huge_folio (struct folio *folio);#ifdef CONFIG_HUGETLB_PAGE
mm/Kconfig +0 -10
@@ -1119,16 +1119,6 @@ config DMAPOOL_TEST config ARCH_HAS_PTE_SPECIAL bool -# -# Some architectures require a special hugepage directory format that is -# required to support multiple hugepage sizes. For example a4fe3ce76 -# "powerpc/mm: Allow more flexible layouts for hugepage pagetables" -# introduced it on powerpc. This allows for a more flexible hugepage -# pagetable layouts. -# -config ARCH_HAS_HUGEPD - bool - config MAPPING_DIRTY_HELPERS bool
mm/gup.c +5 -178
243 lines not on this page · raw message
mm/pagewalk.c +4 -53
87 lines not on this page · raw message
Signature · 1 line
2.44.0