mm: remove __ARCH_HAS_5LEVEL_HACK
Subsystems (14)
Mike Rapoport
reviewerv2 cover mm: remove __ARCH_HAS_5LEVEL_HACK
From: Mike Rapoport <[email protected]>
Hi,
These patches convert several architectures to use page table folding and remove __ARCH_HAS_5LEVEL_HACK along with include/asm-generic/5level-fixup.h.
The changes are mostly about mechanical replacement of pgd accessors with p4d ones and the addition of higher levels to page table traversals.
All the patches were sent separately to the respective arch lists and maintainers hence the "v2" prefix.
Geert Uytterhoeven (1):
sh: fault: Modernize printing of kernel messages
Mike Rapoport (12):
arm/arm64: add support for folded p4d page tables
h8300: remove usage of __ARCH_USE_5LEVEL_HACK
hexagon: remove __ARCH_USE_5LEVEL_HACK
ia64: add support for folded p4d page tables
nios2: add support for folded p4d page tables
openrisc: add support for folded p4d page tables
powerpc: add support for folded p4d page tables
sh: drop __pXd_offset() macros that duplicate pXd_index() ones
sh: add support for folded p4d page tables
unicore32: remove __ARCH_USE_5LEVEL_HACK
asm-generic: remove pgtable-nop4d-hack.h
mm: remove __ARCH_HAS_5LEVEL_HACK and include/asm-generic/5level-fixup.h
arch/arm/include/asm/kvm_mmu.h | 5 +- arch/arm/include/asm/pgtable.h | 1 - arch/arm/include/asm/stage2_pgtable.h | 15 +- arch/arm/lib/uaccess_with_memcpy.c | 9 +- arch/arm/mach-sa1100/assabet.c | 2 +- arch/arm/mm/dump.c | 29 ++- arch/arm/mm/fault-armv.c | 7 +- arch/arm/mm/fault.c | 28 ++- arch/arm/mm/idmap.c | 3 +- arch/arm/mm/init.c | 2 +- arch/arm/mm/ioremap.c | 12 +- arch/arm/mm/mm.h | 2 +- arch/arm/mm/mmu.c | 35 ++- arch/arm/mm/pgd.c | 40 +++- arch/arm64/include/asm/kvm_mmu.h | 10 +- arch/arm64/include/asm/pgalloc.h | 10 +- arch/arm64/include/asm/pgtable-types.h | 5 +- arch/arm64/include/asm/pgtable.h | 37 ++-- arch/arm64/include/asm/stage2_pgtable.h | 48 +++- arch/arm64/kernel/hibernate.c | 44 +++- arch/arm64/mm/fault.c | 9 +- arch/arm64/mm/hugetlbpage.c | 15 +- arch/arm64/mm/kasan_init.c | 26 ++- arch/arm64/mm/mmu.c | 52 +++-- arch/arm64/mm/pageattr.c | 7 +- arch/h8300/include/asm/pgtable.h | 1 - arch/hexagon/include/asm/fixmap.h | 4 +- arch/hexagon/include/asm/pgtable.h | 1 - arch/ia64/include/asm/pgalloc.h | 4 +- arch/ia64/include/asm/pgtable.h | 17 +- arch/ia64/mm/fault.c | 7 +- arch/ia64/mm/hugetlbpage.c | 18 +- arch/ia64/mm/init.c | 28 ++- arch/nios2/include/asm/pgtable.h | 3 +- arch/nios2/mm/fault.c | 9 +- arch/nios2/mm/ioremap.c | 6 +- arch/openrisc/include/asm/pgtable.h | 1 - arch/openrisc/mm/fault.c | 10 +- arch/openrisc/mm/init.c | 4 +- arch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 58 +++-- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +-- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 8 + arch/powerpc/kvm/book3s_64_mmu_radix.c | 59 ++++- arch/powerpc/lib/code-patching.c | 7 +- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 19 +- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 ++- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 +- arch/powerpc/mm/pgtable.c | 25 ++- arch/powerpc/mm/pgtable_32.c | 28 ++- arch/powerpc/mm/pgtable_64.c | 10 +- arch/powerpc/mm/ptdump/hashpagetable.c | 20 +- arch/powerpc/mm/ptdump/ptdump.c | 22 +- arch/powerpc/xmon/xmon.c | 17 +- arch/sh/include/asm/pgtable-2level.h | 1 - arch/sh/include/asm/pgtable-3level.h | 1 - arch/sh/include/asm/pgtable_32.h | 5 +- arch/sh/include/asm/pgtable_64.h | 5 +- arch/sh/kernel/io_trapped.c | 7 +- arch/sh/mm/cache-sh4.c | 4 +- arch/sh/mm/cache-sh5.c | 7 +- arch/sh/mm/fault.c | 65 ++++-- arch/sh/mm/hugetlbpage.c | 28 ++- arch/sh/mm/init.c | 15 +- arch/sh/mm/kmap.c | 2 +- arch/sh/mm/tlbex_32.c | 6 +- arch/sh/mm/tlbex_64.c | 7 +- arch/unicore32/include/asm/pgtable.h | 1 - arch/unicore32/kernel/hibernate.c | 4 +- include/asm-generic/5level-fixup.h | 58 ----- include/asm-generic/pgtable-nop4d-hack.h | 64 ------ include/asm-generic/pgtable-nopud.h | 4 - include/linux/mm.h | 6 - mm/kasan/init.c | 11 - mm/memory.c | 8 - virt/kvm/arm/mmu.c | 209 +++++++++++++++--- 89 files changed, 988 insertions(+), 500 deletions(-) delete mode 100644 include/asm-generic/5level-fixup.h delete mode 100644 include/asm-generic/pgtable-nop4d-hack.h
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 1/13 arm/arm64: add support for folded p4d page tables
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate, replace 5level-fixup.h with pgtable-nop4d.h and remove __ARCH_USE_5LEVEL_HACK.
Since arm and arm64 share kvm memory management bits, make the conversion for both variants at once to avoid breaking the builds in the middle.
--- arch/arm/include/asm/kvm_mmu.h | 5 +- arch/arm/include/asm/pgtable.h | 1 - arch/arm/include/asm/stage2_pgtable.h | 15 +- arch/arm/lib/uaccess_with_memcpy.c | 9 +- arch/arm/mach-sa1100/assabet.c | 2 +- arch/arm/mm/dump.c | 29 +++- arch/arm/mm/fault-armv.c | 7 +- arch/arm/mm/fault.c | 28 +++- arch/arm/mm/idmap.c | 3 +- arch/arm/mm/init.c | 2 +- arch/arm/mm/ioremap.c | 12 +- arch/arm/mm/mm.h | 2 +- arch/arm/mm/mmu.c | 35 +++- arch/arm/mm/pgd.c | 40 ++++- arch/arm64/include/asm/kvm_mmu.h | 10 +- arch/arm64/include/asm/pgalloc.h | 10 +- arch/arm64/include/asm/pgtable-types.h | 5 +- arch/arm64/include/asm/pgtable.h | 37 +++-- arch/arm64/include/asm/stage2_pgtable.h | 48 ++++-- arch/arm64/kernel/hibernate.c | 44 ++++- arch/arm64/mm/fault.c | 9 +- arch/arm64/mm/hugetlbpage.c | 15 +- arch/arm64/mm/kasan_init.c | 26 ++- arch/arm64/mm/mmu.c | 52 ++++-- arch/arm64/mm/pageattr.c | 7 +- virt/kvm/arm/mmu.c | 209 ++++++++++++++++++++---- 26 files changed, 522 insertions(+), 140 deletions(-)
arch/arm/include/asm/kvm_mmu.h +4 -1
@@ -68,7 +68,8 @@ void kvm_clear_hyp_idmap(void);#define kvm_mk_pmd (ptep) __pmd(__pa(ptep) | PMD_TYPE_TABLE)#define kvm_mk_pud (pmdp) __pud(__pa(pmdp) | PMD_TYPE_TABLE) -#define kvm_mk_pgd (pudp) ({ BUILD_BUG(); 0; }) +#define kvm_mk_p4d (pudp) ({ BUILD_BUG(); __p4d(0); }) +#define kvm_mk_pgd (p4dp) ({ BUILD_BUG(); 0; })#define kvm_pfn_pte (pfn, prot) pfn_pte(pfn, prot)#define kvm_pfn_pmd (pfn, prot) pfn_pmd(pfn, prot)
@@ -194,10 +195,12 @@ static inline bool kvm_page_empty(void *ptr)#define kvm_pte_table_empty (kvm, ptep) kvm_page_empty(ptep)#define kvm_pmd_table_empty (kvm, pmdp) kvm_page_empty(pmdp)#define kvm_pud_table_empty (kvm, pudp) false +#define kvm_p4d_table_empty (kvm, p4dp) false#define hyp_pte_table_empty (ptep) kvm_page_empty(ptep)#define hyp_pmd_table_empty (pmdp) kvm_page_empty(pmdp)#define hyp_pud_table_empty (pudp) false +#define hyp_p4d_table_empty (p4dp) falsestruct kvm ;
arch/arm/include/asm/pgtable.h +0 -1
@@ -17,7 +17,6 @@#else -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopud.h> #include <asm/memory.h> #include <asm/pgtable-hwdef.h>
arch/arm/include/asm/stage2_pgtable.h +13 -2
@@ -19,8 +19,17 @@#define stage2_pgd_none (kvm, pgd) pgd_none(pgd)#define stage2_pgd_clear (kvm, pgd) pgd_clear(pgd)#define stage2_pgd_present (kvm, pgd) pgd_present(pgd) -#define stage2_pgd_populate (kvm, pgd, pud) pgd_populate(NULL, pgd, pud) -#define stage2_pud_offset (kvm, pgd, address) pud_offset(pgd, address) +#define stage2_pgd_populate (kvm, pgd, p4d) pgd_populate(NULL, pgd, p4d) + +#define stage2_p4d_offset (kvm, pgd, address) p4d_offset(pgd, address) +#define stage2_p4d_free (kvm, p4d) do { } while (0) + +#define stage2_p4d_none (kvm, p4d) p4d_none(p4d) +#define stage2_p4d_clear (kvm, p4d) p4d_clear(p4d) +#define stage2_p4d_present (kvm, p4d) p4d_present(p4d) +#define stage2_p4d_populate (kvm, p4d, pud) p4d_populate(NULL, p4d, pud) + +#define stage2_pud_offset (kvm, p4d, address) pud_offset(p4d, address)#define stage2_pud_free (kvm, pud) do { } while (0)#define stage2_pud_none (kvm, pud) pud_none(pud)
@@ -41,6 +50,7 @@ stage2_pgd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)return (boundary -1 < end -1 ) ? boundary : end; } +#define stage2_p4d_addr_end (kvm, addr, end) (end)#define stage2_pud_addr_end (kvm, addr, end) (end) static inline phys_addr_t
@@ -56,6 +66,7 @@ stage2_pmd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end)#define stage2_pte_table_empty (kvm, ptep) kvm_page_empty(ptep)#define stage2_pmd_table_empty (kvm, pmdp) kvm_page_empty(pmdp)#define stage2_pud_table_empty (kvm, pudp) false +#define stage2_p4d_table_empty (kvm, p4dp) falsestatic inline bool kvm_stage2_has_pud (struct kvm *kvm) {
arch/arm/lib/uaccess_with_memcpy.c +7 -2
@@ -24,6 +24,7 @@ pin_page_for_write(const void __user *_addr, pte_t **ptep, spinlock_t **ptlp) {unsigned long addr = (unsigned long )_addr;pgd_t *pgd; +p4d_t *p4d;pmd_t *pmd;pte_t *pte;pud_t *pud;
@@ -33,7 +34,11 @@ pin_page_for_write(const void __user *_addr, pte_t **ptep, spinlock_t **ptlp)if (unlikely (pgd_none (*pgd) ||pgd_bad (*pgd)))return 0 ; - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +if (unlikely (p4d_none (*p4d) ||p4d_bad (*p4d))) +return 0 ; + + pud =pud_offset (p4d, addr);if (unlikely (pud_none (*pud) ||pud_bad (*pud)))return 0 ;
@@ -154,7 +159,7 @@ arm_copy_to_user(void __user *to, const void *from, unsigned long n) }return n; } - +static unsigned long noinline __clear_user_memset (void __user *addr,unsigned long n) {
arch/arm/mach-sa1100/assabet.c +1 -1
@@ -633,7 +633,7 @@ static void __init map_sa1100_gpio_regs( void )int prot =PMD_TYPE_SECT |PMD_SECT_AP_WRITE |PMD_DOMAIN (DOMAIN_IO );pmd_t *pmd; - pmd =pmd_offset (pud_offset (pgd_offset_k (virt), virt), virt); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (virt), virt), virt), virt); *pmd =__pmd (phys | prot);flush_pmd_entry (pmd); }
arch/arm/mm/dump.c +23 -6
@@ -207,6 +207,7 @@ struct pg_level {static struct pg_level pg_level[] = { { }, {/* pgd */ + }, {/* p4d */ }, {/* pud */ }, {/* pmd */ .bits = section_bits,
@@ -308,7 +309,7 @@ static void walk_pte(struct pg_state *st, pmd_t *pmd, unsigned long start,for (i =0 ; i <PTRS_PER_PTE ; i++, pte++) { addr = start + i *PAGE_SIZE ; -note_page (st, addr,4 ,pte_val (*pte), domain); +note_page (st, addr,5 ,pte_val (*pte), domain); } }
@@ -350,14 +351,14 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) addr +=SECTION_SIZE ; pmd++; domain =get_domain_name (pmd); -note_page (st, addr,3 ,pmd_val (*pmd), domain); +note_page (st, addr,4 ,pmd_val (*pmd), domain); } } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned i;
@@ -366,7 +367,23 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start)if (!pud_none (*pud)) {walk_pmd (st, pud, addr); }else { -note_page (st, addr,2 ,pud_val (*pud),NULL ); +note_page (st, addr,3 ,pud_val (*pud),NULL ); + } + } +} + +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d)) { +walk_pud (st, p4d, addr); + }else { +note_page (st, addr,2 ,p4d_val (*p4d),NULL ); } } }
@@ -381,7 +398,7 @@ static void walk_pgd(struct pg_state *st, struct mm_struct *mm,for (i =0 ; i <PTRS_PER_PGD ; i++, pgd++) { addr = start + i *PGDIR_SIZE ;if (!pgd_none (*pgd)) { -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr); }else {note_page (st, addr,1 ,pgd_val (*pgd),NULL ); }
arch/arm/mm/fault-armv.c +6 -1
@@ -91,6 +91,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, {spinlock_t *ptl;pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -100,7 +101,11 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address,if (pgd_none_or_clear_bad (pgd))return 0 ; - pud =pud_offset (pgd, address); + p4d =p4d_offset (pgd, address); +if (p4d_none_or_clear_bad (p4d)) +return 0 ; + + pud =pud_offset (p4d, address);if (pud_none_or_clear_bad (pud))return 0 ;
arch/arm/mm/fault.c +25 -3
@@ -43,6 +43,7 @@ void show_pte(const char *lvl, struct mm_struct *mm, unsigned long addr)printk ("%s[%08lx] *pgd=%08llx" , lvl, addr, (long long )pgd_val (*pgd)); do { + p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte;
@@ -55,7 +56,19 @@ void show_pte(const char *lvl, struct mm_struct *mm, unsigned long addr)break ; } - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +if (PTRS_PER_P4D !=1 ) +pr_cont (", *p4d=%08llx" , (long long )p4d_val (*p4d)); + +if (p4d_none (*p4d)) +break ; + +if (p4d_bad (*p4d)) { +pr_cont ("(bad)" ); +break ; + } + + pud =pud_offset (p4d, addr);if (PTRS_PER_PUD !=1 )pr_cont (", *pud=%08llx" , (long long )pud_val (*pud));
@@ -408,6 +421,7 @@ do_translation_fault(unsigned long addr, unsigned int fsr, {unsigned int index;pgd_t *pgd, *pgd_k; +p4d_t *p4d, *p4d_k;pud_t *pud, *pud_k;pmd_t *pmd, *pmd_k;
@@ -427,8 +441,16 @@ do_translation_fault(unsigned long addr, unsigned int fsr,if (!pgd_present (*pgd))set_pgd (pgd, *pgd_k); - pud =pud_offset (pgd, addr); - pud_k =pud_offset (pgd_k, addr); + p4d =p4d_offset (pgd, addr); + p4d_k =p4d_offset (pgd_k, addr); + +if (p4d_none (*p4d_k)) + goto bad_area; +if (!p4d_present (*p4d)) +set_p4d (p4d, *p4d_k); + + pud =pud_offset (p4d, addr); + pud_k =pud_offset (p4d_k, addr);if (pud_none (*pud_k)) goto bad_area;
arch/arm/mm/idmap.c +2 -1
@@ -68,7 +68,8 @@ static void idmap_add_pmd(pud_t *pud, unsigned long addr, unsigned long end,static void idmap_add_pud (pgd_t *pgd,unsigned long addr,unsigned long end,unsigned long prot) { -pud_t *pud =pud_offset (pgd, addr); +p4d_t *p4d =p4d_offset (pgd, addr); +pud_t *pud =pud_offset (p4d, addr);unsigned long next; do {
arch/arm/mm/init.c +1 -1
@@ -571,7 +571,7 @@ static inline void section_update(unsigned long addr, pmdval_t mask, {pmd_t *pmd; - pmd =pmd_offset (pud_offset (pgd_offset (mm, addr), addr), addr); + pmd =pmd_off_k (addr);#ifdef CONFIG_ARM_LPAE pmd[0 ] =__pmd ((pmd_val (pmd[0 ]) & mask) | prot);
arch/arm/mm/ioremap.c +9 -3
@@ -142,12 +142,14 @@ static void unmap_area_sections(unsigned long virt, unsigned long size) {unsigned long addr = virt, end = virt + (size & ~(SZ_1M -1 ));pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmdp;flush_cache_vunmap (addr, end); pgd =pgd_offset_k (addr); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); + pud =pud_offset (p4d, addr); pmdp =pmd_offset (pud, addr); do { pmd_t pmd = *pmdp;
@@ -190,6 +192,7 @@ remap_area_sections(unsigned long virt, unsigned long pfn, {unsigned long addr = virt, end = virt + size;pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;
@@ -200,7 +203,8 @@ remap_area_sections(unsigned long virt, unsigned long pfn,unmap_area_sections (virt, size); pgd =pgd_offset_k (addr); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); + pud =pud_offset (p4d, addr); pmd =pmd_offset (pud, addr); do { pmd[0] = __pmd(__pfn_to_phys(pfn) | type->prot_sect);
@@ -222,6 +226,7 @@ remap_area_supersections(unsigned long virt, unsigned long pfn, {unsigned long addr = virt, end = virt + size;pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;
@@ -232,7 +237,8 @@ remap_area_supersections(unsigned long virt, unsigned long pfn,unmap_area_sections (virt, size); pgd =pgd_offset_k (virt); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); + pud =pud_offset (p4d, addr); pmd =pmd_offset (pud, addr); do { unsigned long super_pmd_val, i;
arch/arm/mm/mm.h +1 -1
@@ -38,7 +38,7 @@ static inline pte_t get_top_pte(unsigned long va)static inline pmd_t *pmd_off_k (unsigned long virt) { -return pmd_offset (pud_offset (pgd_offset_k (virt), virt), virt); +return pmd_offset (pud_offset (p4d_offset (pgd_offset_k (virt), virt), virt), virt); }struct mem_type {
arch/arm/mm/mmu.c +29 -6
@@ -375,7 +375,8 @@ static pte_t *pte_offset_late_fixmap(pmd_t *dir, unsigned long addr)static inline pmd_t *__init fixmap_pmd (unsigned long addr) {pgd_t *pgd =pgd_offset_k (addr); -pud_t *pud =pud_offset (pgd, addr); +p4d_t *p4d =p4d_offset (pgd, addr); +pud_t *pud =pud_offset (p4d, addr);pmd_t *pmd =pmd_offset (pud, addr);return pmd;
@@ -827,12 +828,12 @@ static void __init alloc_init_pmd(pud_t *pud, unsigned long addr, }while (pmd++, addr = next, addr != end); } -static void __init alloc_init_pud (pgd_t *pgd,unsigned long addr, +static void __init alloc_init_pud (p4d_t *p4d,unsigned long addr,unsigned long end,phys_addr_t phys,const struct mem_type *type,void *(*alloc)(unsigned long sz),bool ng) { -pud_t *pud =pud_offset (pgd, addr); +pud_t *pud =pud_offset (p4d, addr);unsigned long next; do {
@@ -842,6 +843,21 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr, }while (pud++, addr = next, addr != end); } +static void __init alloc_init_p4d (pgd_t *pgd,unsigned long addr, +unsigned long end,phys_addr_t phys, +const struct mem_type *type, +void *(*alloc)(unsigned long sz),bool ng) +{ +p4d_t *p4d =p4d_offset (pgd, addr); +unsigned long next; + +do { + next =p4d_addr_end (addr, end); +alloc_init_pud (p4d, addr, next, phys, type, alloc, ng); + phys += next - addr; + }while (p4d++, addr = next, addr != end); +} +#ifndef CONFIG_ARM_LPAE static void __init create_36bit_mapping(struct mm_struct *mm, struct map_desc *md,
@@ -889,7 +905,8 @@ static void __init create_36bit_mapping(struct mm_struct *mm, pgd =pgd_offset (mm, addr); end = addr + length; do { - pud_t *pud = pud_offset(pgd, addr); + p4d_t *p4d = p4d_offset(pgd, addr); + pud_t *pud = pud_offset(p4d, addr); pmd_t *pmd = pmd_offset(pud, addr); int i;
@@ -940,7 +957,7 @@ static void __init __create_mapping(struct mm_struct *mm, struct map_desc *md, do { unsigned long next = pgd_addr_end(addr, end); - alloc_init_pud(pgd, addr, next, phys, type, alloc, ng); + alloc_init_p4d(pgd, addr, next, phys, type, alloc, ng); phys += next - addr; addr = next;
@@ -976,7 +993,13 @@ void __init create_mapping_late(struct mm_struct *mm, struct map_desc *md, bool ng) {#ifdef CONFIG_ARM_LPAE -pud_t *pud =pud_alloc (mm,pgd_offset (mm, md->virtual), md->virtual); +p4d_t *p4d; +pud_t *pud; + + p4d =p4d_alloc (mm,pgd_offset (mm, md->virtual), md->virtual); +if (!WARN_ON (!p4d)) +return ; + pud =pud_alloc (mm, p4d, md->virtual);if (WARN_ON (!pud))return ;pmd_alloc (mm, pud,0 );
arch/arm/mm/pgd.c +33 -7
@@ -30,6 +30,7 @@pgd_t *pgd_alloc (struct mm_struct *mm) {pgd_t *new_pgd, *init_pgd; +p4d_t *new_p4d, *init_p4d;pud_t *new_pud, *init_pud;pmd_t *new_pmd, *init_pmd;pte_t *new_pte, *init_pte;
@@ -53,8 +54,12 @@ pgd_t *pgd_alloc(struct mm_struct *mm)/* * Allocate PMD table for modules and pkmap mappings. */ - new_pud =pud_alloc (mm, new_pgd +pgd_index (MODULES_VADDR ), + new_p4d =p4d_alloc (mm, new_pgd +pgd_index (MODULES_VADDR ),MODULES_VADDR ); +if (!new_p4d) + goto no_p4d; + + new_pud =pud_alloc (mm, new_p4d,MODULES_VADDR );if (!new_pud) goto no_pud;
@@ -69,7 +74,11 @@ pgd_t *pgd_alloc(struct mm_struct *mm)* contains the machine vectors. The vectors are always high * with LPAE. */ - new_pud =pud_alloc (mm, new_pgd,0 ); + new_p4d =p4d_alloc (mm, new_pgd,0 ); +if (!new_p4d) + goto no_p4d; + + new_pud =pud_alloc (mm, new_p4d,0 );if (!new_pud) goto no_pud;
@@ -91,7 +100,8 @@ pgd_t *pgd_alloc(struct mm_struct *mm)pmd_val (*new_pmd) |=PMD_DOMAIN (DOMAIN_VECTORS );#endif - init_pud =pud_offset (init_pgd,0 ); + init_p4d =p4d_offset (init_pgd,0 ); + init_pud =pud_offset (init_p4d,0 ); init_pmd =pmd_offset (init_pud,0 ); init_pte =pte_offset_map (init_pmd,0 );set_pte_ext (new_pte +0 , init_pte[0 ],0 );
@@ -108,6 +118,8 @@ pgd_t *pgd_alloc(struct mm_struct *mm) no_pmd:pud_free (mm, new_pud); no_pud: +p4d_free (mm, new_p4d); +no_p4d:__pgd_free (new_pgd); no_pgd:return NULL ;
@@ -116,6 +128,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm)void pgd_free (struct mm_struct *mm,pgd_t *pgd_base) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pgtable_t pte;
@@ -127,7 +140,11 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd_base)if (pgd_none_or_clear_bad (pgd)) goto no_pgd; - pud =pud_offset (pgd,0 ); + p4d =p4d_offset (pgd,0 ); +if (p4d_none_or_clear_bad (p4d)) + goto no_p4d; + + pud =pud_offset (p4d,0 );if (pud_none_or_clear_bad (pud)) goto no_pud;
@@ -144,8 +161,11 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd_base)pmd_free (mm, pmd);mm_dec_nr_pmds (mm); no_pud: -pgd_clear (pgd); +p4d_clear (p4d);pud_free (mm, pud); +no_p4d: +pgd_clear (pgd); +p4d_free (mm, p4d); no_pgd: #ifdef CONFIG_ARM_LPAE/*
@@ -156,15 +176,21 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd_base)continue ;if (pgd_val (*pgd) &L_PGD_SWAPPER )continue ; - pud =pud_offset (pgd,0 ); + p4d =p4d_offset (pgd,0 ); +if (p4d_none_or_clear_bad (p4d)) +continue ; + pud =pud_offset (p4d,0 );if (pud_none_or_clear_bad (pud))continue ; pmd =pmd_offset (pud,0 );pud_clear (pud);pmd_free (mm, pmd);mm_dec_nr_pmds (mm); -pgd_clear (pgd); +p4d_clear (p4d);pud_free (mm, pud); +mm_dec_nr_puds (mm); +pgd_clear (pgd); + p4d_free(mm, p4d); }#endif __pgd_free (pgd_base);
arch/arm64/include/asm/kvm_mmu.h +8 -2
@@ -172,8 +172,8 @@ void kvm_clear_hyp_idmap(void); __pmd(__phys_to_pmd_val(__pa(ptep)) | PMD_TYPE_TABLE)#define kvm_mk_pud (pmdp) \ __pud(__phys_to_pud_val(__pa(pmdp)) | PMD_TYPE_TABLE) -#define kvm_mk_pgd (pudp) \ - __pgd(__phys_to_pgd_val(__pa(pudp)) | PUD_TYPE_TABLE) +#define kvm_mk_p4d (pmdp) \ + __p4d(__phys_to_p4d_val(__pa(pmdp)) | PUD_TYPE_TABLE)#define kvm_set_pud (pudp, pud) set_pud(pudp, pud)
@@ -299,6 +299,12 @@ static inline bool kvm_s2pud_young(pud_t pud)#define hyp_pud_table_empty (pudp) kvm_page_empty(pudp)#endif +#ifdef __PAGETABLE_P4D_FOLDED +#define hyp_p4d_table_empty (p4dp) (0) +#else +#define hyp_p4d_table_empty (p4dp) kvm_page_empty(p4dp) +#endif +struct kvm ;#define kvm_flush_dcache_to_poc (a,l) __flush_dcache_area((a), (l))
arch/arm64/include/asm/pgalloc.h +5 -5
@@ -73,17 +73,17 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pudp) free_page((unsigned long)pudp); } -static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t pudp,pgdval_t prot) +static inline void __p4d_populate (p4d_t *p4dp,phys_addr_t pudp,p4dval_t prot) { -set_pgd (pgdp,__pgd (__phys_to_pgd_val (pudp) | prot)); +set_p4d (p4dp,__p4d (__phys_to_p4d_val (pudp) | prot)); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgdp,pud_t *pudp) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *p4dp,pud_t *pudp) { -__pgd_populate (pgdp,__pa (pudp),PUD_TYPE_TABLE ); +__p4d_populate (p4dp,__pa (pudp),PUD_TYPE_TABLE ); }#else -static inline void __pgd_populate (pgd_t *pgdp,phys_addr_t pudp,pgdval_t prot) +static inline void __p4d_populate (p4d_t *p4dp,phys_addr_t pudp,p4dval_t prot) {BUILD_BUG (); }
arch/arm64/include/asm/pgtable-types.h +2 -3
@@ -14,6 +14,7 @@typedef u64 pteval_t ;typedef u64 pmdval_t ;typedef u64 pudval_t ; +typedef u64 p4dval_t ;typedef u64 pgdval_t ;/*
@@ -44,13 +45,11 @@ typedef struct { pteval_t pgprot; } pgprot_t;#define __pgprot (x) ((pgprot_t) { (x) } ) #if CONFIG_PGTABLE_LEVELS = 2 -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #elif CONFIG_PGTABLE_LEVELS = 3 -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopud.h> #elif CONFIG_PGTABLE_LEVELS = 4 -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #endif #endif /* __ASM_PGTABLE_TYPES_H */
arch/arm64/include/asm/pgtable.h +23 -14
@@ -298,6 +298,11 @@ static inline pte_t pgd_pte(pgd_t pgd)return __pte (pgd_val (pgd)); } +static inline pte_t p4d_pte (p4d_t p4d) +{ +return __pte (p4d_val (p4d)); +} +static inline pte_t pud_pte (pud_t pud) {return __pte (pud_val (pud));
@@ -401,6 +406,9 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)#define set_pmd_at (mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd)) +#define __p4d_to_phys (p4d) __pte_to_phys(p4d_pte(p4d)) +#define __phys_to_p4d_val (phys) __phys_to_pte_val(phys) +#define __pgd_to_phys (pgd) __pte_to_phys(pgd_pte(pgd))#define __phys_to_pgd_val (phys) __phys_to_pte_val(phys)
@@ -588,49 +596,50 @@ static inline phys_addr_t pud_page_paddr(pud_t pud)#define pud_ERROR (pud) __pud_error(__FILE__, __LINE__, pud_val(pud)) -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (!(pgd_val(pgd) & 2)) -#define pgd_present (pgd) (pgd_val(pgd)) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (!(p4d_val(p4d) & 2)) +#define p4d_present (p4d) (p4d_val(p4d)) -static inline void set_pgd (pgd_t *pgdp,pgd_t pgd) +static inline void set_p4d (p4d_t *p4dp,p4d_t p4d) { -if (in_swapper_pgdir (pgdp)) { -set_swapper_pgd (pgdp, pgd); +if (in_swapper_pgdir (p4dp)) { +set_swapper_pgd ((pgd_t *)p4dp,__pgd (p4d_val (p4d)));return ; } -WRITE_ONCE (*pgdp, pgd); +WRITE_ONCE (*p4dp, p4d);dsb (ishst);isb (); } -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { -set_pgd (pgdp,__pgd (0 )); +set_p4d (p4dp,__p4d (0 )); } -static inline phys_addr_t pgd_page_paddr (pgd_t pgd) +static inline phys_addr_t p4d_page_paddr (p4d_t p4d) { -return __pgd_to_phys (pgd); +return __p4d_to_phys (p4d); }/* Find an entry in the frst-level page table. */ #define pud_index (addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)) -#define pud_offset_phys (dir, addr) (pgd_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t)) +#define pud_offset_phys (dir, addr) (p4d_page_paddr(READ_ONCE(*(dir))) + pud_index(addr) * sizeof(pud_t))#define pud_offset (dir, addr) ((pud_t *)__va(pud_offset_phys((dir), (addr))))#define pud_set_fixmap (addr) ((pud_t *)set_fixmap_offset(FIX_PUD, addr)) -#define pud_set_fixmap_offset (pgd, addr) pud_set_fixmap(pud_offset_phys(pgd, addr)) +#define pud_set_fixmap_offset (p4d, addr) pud_set_fixmap(pud_offset_phys(p4d, addr))#define pud_clear_fixmap () clear_fixmap(FIX_PUD) -#define pgd_page (pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) +#define p4d_page (p4d) pfn_to_page(__phys_to_pfn(__p4d_to_phys(p4d)))/* use ONLY for statically allocated translation tables */ #define pud_offset_kimg (dir,addr) ((pud_t *)__phys_to_kimg(pud_offset_phys((dir), (addr))))#else +#define p4d_page_paddr (p4d) ({ BUILD_BUG(); 0;})#define pgd_page_paddr (pgd) ({ BUILD_BUG(); 0;})/* Match pud_offset folding in <asm/generic/pgtable-nopud.h> */
arch/arm64/include/asm/stage2_pgtable.h +37 -11
78 lines not on this page · raw message
arch/arm64/kernel/hibernate.c +37 -7
85 lines not on this page · raw message
arch/arm64/mm/fault.c +8 -1
@@ -145,6 +145,7 @@ static void show_pte(unsigned long addr)pr_alert ("[%016lx] pgd=%016llx" , addr,pgd_val (pgd)); do { + p4d_t *p4dp, p4d; pud_t *pudp, pud; pmd_t *pmdp, pmd; pte_t *ptep, pte;
@@ -152,7 +153,13 @@ static void show_pte(unsigned long addr)if (pgd_none (pgd) ||pgd_bad (pgd))break ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); + p4d =READ_ONCE (*p4dp); +pr_cont (", p4d=%016llx" ,p4d_val (p4d)); +if (p4d_none (p4d) ||p4d_bad (p4d)) +break ; + + pudp =pud_offset (p4dp, addr); pud =READ_ONCE (*pudp);pr_cont (", pud=%016llx" ,pud_val (pud));if (pud_none (pud) ||pud_bad (pud))
arch/arm64/mm/hugetlbpage.c +12 -3
@@ -67,11 +67,13 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr, pte_t *ptep, size_t *pgsize) {pgd_t *pgdp =pgd_offset (mm, addr); +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp; *pgsize =PAGE_SIZE ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); + pudp =pud_offset (p4dp, addr); pmdp =pmd_offset (pudp, addr);if ((pte_t *)pmdp = ptep) { *pgsize =PMD_SIZE ;
@@ -217,12 +219,14 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep =NULL ; pgdp =pgd_offset (mm, addr); - pudp =pud_alloc (mm, pgdp, addr); + p4dp =p4d_offset (pgdp, addr); + pudp =pud_alloc (mm, p4dp, addr);if (!pudp)return NULL ;
@@ -259,6 +263,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp, pud;pmd_t *pmdp, pmd;
@@ -266,7 +271,11 @@ pte_t *huge_pte_offset(struct mm_struct *mm,if (!pgd_present (READ_ONCE (*pgdp)))return NULL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (!p4d_present (READ_ONCE (*p4dp))) +return NULL ; + + pudp =pud_offset (p4dp, addr); pud =READ_ONCE (*pudp);if (sz !=PUD_SIZE &&pud_none (pud))return NULL ;
arch/arm64/mm/kasan_init.c +19 -7
@@ -84,17 +84,17 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,return early ?pmd_offset_kimg (pudp, addr) :pmd_offset (pudp, addr); } -static pud_t *__init kasan_pud_offset (pgd_t *pgdp,unsigned long addr,int node, +static pud_t *__init kasan_pud_offset (p4d_t *p4dp,unsigned long addr,int node,bool early) { -if (pgd_none (READ_ONCE (*pgdp))) { +if (p4d_none (READ_ONCE (*p4dp))) {phys_addr_t pud_phys = early ?__pa_symbol (kasan_early_shadow_pud) :kasan_alloc_zeroed_page (node); -__pgd_populate (pgdp, pud_phys,PMD_TYPE_TABLE ); +__p4d_populate (p4dp, pud_phys,PMD_TYPE_TABLE ); } -return early ?pud_offset_kimg (pgdp, addr) :pud_offset (pgdp, addr); +return early ?pud_offset_kimg (p4dp, addr) :pud_offset (p4dp, addr); } static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr,
@@ -126,11 +126,11 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, }while (pmdp++, addr = next, addr != end &&pmd_none (READ_ONCE (*pmdp))); } -static void __init kasan_pud_populate (pgd_t *pgdp,unsigned long addr, +static void __init kasan_pud_populate (p4d_t *p4dp,unsigned long addr,unsigned long end,int node,bool early) {unsigned long next; -pud_t *pudp =kasan_pud_offset (pgdp, addr, node, early); +pud_t *pudp =kasan_pud_offset (p4dp, addr, node, early); do { next = pud_addr_end(addr, end);
@@ -138,6 +138,18 @@ static void __init kasan_pud_populate(pgd_t *pgdp, unsigned long addr, }while (pudp++, addr = next, addr != end &&pud_none (READ_ONCE (*pudp))); } +static void __init kasan_p4d_populate (pgd_t *pgdp,unsigned long addr, +unsigned long end,int node,bool early) +{ +unsigned long next; +p4d_t *p4dp =p4d_offset (pgdp, addr); + +do { + next =p4d_addr_end (addr, end); +kasan_pud_populate (p4dp, addr, next, node, early); + }while (p4dp++, addr = next, addr != end); +} +static void __init kasan_pgd_populate (unsigned long addr,unsigned long end,int node,bool early) {
@@ -147,7 +159,7 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end, pgdp =pgd_offset_k (addr);do { next =pgd_addr_end (addr, end); -kasan_pud_populate (pgdp, addr, next, node, early); +kasan_p4d_populate (pgdp, addr, next, node, early); }while (pgdp++, addr = next, addr != end); }
arch/arm64/mm/mmu.c +34 -18
127 lines not on this page · raw message
arch/arm64/mm/pageattr.c +6 -1
@@ -198,6 +198,7 @@ void __kernel_map_pages(struct page *page, int numpages, int enable)bool kernel_page_present (struct page *page) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp, pud;pmd_t *pmdp, pmd;pte_t *ptep;
@@ -210,7 +211,11 @@ bool kernel_page_present(struct page *page)if (pgd_none (READ_ONCE (*pgdp)))return false; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (p4d_none (READ_ONCE (*p4dp))) +return false; + + pudp =pud_offset (p4dp, addr); pud =READ_ONCE (*pudp);if (pud_none (pud))return false;
virt/kvm/arm/mmu.c +177 -32
388 lines not on this page · raw message
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 2/13 h8300: remove usage of __ARCH_USE_5LEVEL_HACK
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
h8300 is a nommu architecture and does not require fixup for upper layers of the page tables because it is already handled by the generic nommu implementation.
Remove definition of __ARCH_USE_5LEVEL_HACK inarch/h8300/include/asm/pgtable.h
--- arch/h8300/include/asm/pgtable.h | 1 - 1 file changed, 1 deletion(-)
arch/h8300/include/asm/pgtable.h +0 -1
@@ -1,7 +1,6 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _H8300_PGTABLE_H#define _H8300_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopud.h> #include <asm-generic/pgtable.h> extern void paging_init (void );
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 3/13 hexagon: remove __ARCH_USE_5LEVEL_HACK
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
The hexagon architecture has 2 level page tables and as such most of the page table folding is already implemented in asm-generic/pgtable-nopmd.h.
Fixup the only place in arch/hexagon to unfold the p4d level and remove __ARCH_USE_5LEVEL_HACK.
--- arch/hexagon/include/asm/fixmap.h | 4 ++-- arch/hexagon/include/asm/pgtable.h | 1 - 2 files changed, 2 insertions(+), 3 deletions(-)
arch/hexagon/include/asm/fixmap.h +2 -2
@@ -16,7 +16,7 @@#include <asm-generic/fixmap.h> #define kmap_get_fixmap_pte (vaddr) \ - pte_offset_kernel(pmd_offset(pud_offset(pgd_offset_k(vaddr), \ - (vaddr)), (vaddr)), (vaddr)) + pte_offset_kernel(pmd_offset(pud_offset(p4d_offset(pgd_offset_k(vaddr), \ + (vaddr)), (vaddr)), (vaddr)), (vaddr))#endif
arch/hexagon/include/asm/pgtable.h +0 -1
@@ -12,7 +12,6 @@* Page table definitions for Qualcomm Hexagon processor. */ #include <asm/page.h> -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> /* A handy thing to have if one has the RAM. Declared in head.S */
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 4/13 ia64: add support for folded p4d page tables
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate, remove usage of __ARCH_USE_5LEVEL_HACK and replace 5level-fixup.h with pgtable-nop4d.h
--- arch/ia64/include/asm/pgalloc.h | 4 ++-- arch/ia64/include/asm/pgtable.h | 17 ++++++++--------- arch/ia64/mm/fault.c | 7 ++++++- arch/ia64/mm/hugetlbpage.c | 18 ++++++++++++------ arch/ia64/mm/init.c | 28 ++++++++++++++++++++++++---- 5 files changed, 52 insertions(+), 22 deletions(-)
arch/ia64/include/asm/pgalloc.h +2 -2
@@ -36,9 +36,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) #if CONFIG_PGTABLE_LEVELS = 4static inline void -pgd_populate (struct mm_struct *mm,pgd_t * pgd_entry,pud_t * pud) +p4d_populate (struct mm_struct *mm,p4d_t * p4d_entry,pud_t * pud) { -pgd_val (*pgd_entry) =__pa (pud); +p4d_val (*p4d_entry) =__pa (pud); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)
arch/ia64/include/asm/pgtable.h +8 -9
@@ -283,12 +283,12 @@ extern unsigned long VMALLOC_END;#define pud_page (pud) virt_to_page((pud_val(pud) + PAGE_OFFSET)) #if CONFIG_PGTABLE_LEVELS = 4 -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (!ia64_phys_addr_valid(pgd_val(pgd))) -#define pgd_present (pgd) (pgd_val(pgd) != 0UL) -#define pgd_clear (pgdp) (pgd_val(*(pgdp)) = 0UL) -#define pgd_page_vaddr (pgd) ((unsigned long) __va(pgd_val(pgd) & _PFN_MASK)) -#define pgd_page (pgd) virt_to_page((pgd_val(pgd) + PAGE_OFFSET)) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (!ia64_phys_addr_valid(p4d_val(p4d))) +#define p4d_present (p4d) (p4d_val(p4d) != 0UL) +#define p4d_clear (p4dp) (p4d_val(*(p4dp)) = 0UL) +#define p4d_page_vaddr (p4d) ((unsigned long) __va(p4d_val(p4d) & _PFN_MASK)) +#define p4d_page (p4d) virt_to_page((p4d_val(p4d) + PAGE_OFFSET))#endif /*
@@ -388,7 +388,7 @@ pgd_offset (const struct mm_struct *mm, unsigned long address) #if CONFIG_PGTABLE_LEVELS = 4/* Find an entry in the second-level page table.. */ #define pud_offset (dir,addr) \ - ((pud_t *) pgd_page_vaddr(*(dir)) + (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))) + ((pud_t *) p4d_page_vaddr(*(dir)) + (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)))#endif /* Find an entry in the third-level page table.. */
@@ -582,10 +582,9 @@ extern struct page *zero_page_memmap_ptr; #if CONFIG_PGTABLE_LEVELS = 3 -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopud.h> #endif -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #include <asm-generic/pgtable.h> #endif /* _ASM_IA64_PGTABLE_H */
arch/ia64/mm/fault.c +6 -1
@@ -29,6 +29,7 @@ static int mapped_kernel_page_is_present (unsigned long address) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *ptep, pte;
@@ -37,7 +38,11 @@ mapped_kernel_page_is_present (unsigned long address)if (pgd_none (*pgd) ||pgd_bad (*pgd))return 0 ; - pud =pud_offset (pgd, address); + p4d =p4d_offset (pgd, address); +if (p4d_none (*p4d) ||p4d_bad (*p4d)) +return 0 ; + + pud =pud_offset (p4d, address);if (pud_none (*pud) ||pud_bad (*pud))return 0 ;
arch/ia64/mm/hugetlbpage.c +12 -6
@@ -30,12 +30,14 @@ huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz) {unsigned long taddr =htlbpage_to_page (addr);pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte =NULL ; pgd =pgd_offset (mm, taddr); - pud =pud_alloc (mm, pgd, taddr); + p4d =p4d_offset (pgd, taddr); + pud =pud_alloc (mm, p4d, taddr);if (pud) { pmd =pmd_alloc (mm, pud, taddr);if (pmd)
@@ -49,17 +51,21 @@ huge_pte_offset (struct mm_struct *mm, unsigned long addr, unsigned long sz) {unsigned long taddr =htlbpage_to_page (addr);pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte =NULL ; pgd =pgd_offset (mm, taddr);if (pgd_present (*pgd)) { - pud =pud_offset (pgd, taddr); -if (pud_present (*pud)) { - pmd =pmd_offset (pud, taddr); -if (pmd_present (*pmd)) - pte =pte_offset_map (pmd, taddr); + p4d =p4d_offset (pgd, addr); +if (p4d_present (*p4d)) { + pud =pud_offset (p4d, taddr); +if (pud_present (*pud)) { + pmd =pmd_offset (pud, taddr); +if (pmd_present (*pmd)) + pte =pte_offset_map (pmd, taddr); + } } }
arch/ia64/mm/init.c +24 -4
@@ -208,6 +208,7 @@ static struct page * __init put_kernel_page (struct page *page, unsigned long address, pgprot_t pgprot) { pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -215,7 +216,10 @@ put_kernel_page (struct page *page, unsigned long address, pgprot_t pgprot) pgd =pgd_offset_k (address);/* note: this is NOT pgd_offset()! */ { - pud =pud_alloc (&init_mm, pgd, address); + p4d =p4d_alloc (&init_mm, pgd, address); +if (!p4d) + goto out; + pud =pud_alloc (&init_mm, p4d, address);if (!pud) goto out; pmd =pmd_alloc (&init_mm, pud, address);
@@ -382,6 +386,7 @@ int vmemmap_find_next_valid_pfn(int node, int i) do { pgd_t *pgd; + p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte;
@@ -392,7 +397,13 @@ int vmemmap_find_next_valid_pfn(int node, int i)continue ; } - pud =pud_offset (pgd, end_address); + p4d =p4d_offset (pgd, end_address); +if (p4d_none (*p4d)) { + end_address +=P4D_SIZE ; +continue ; + } + + pud =pud_offset (p4d, end_address);if (pud_none (*pud)) { end_address +=PUD_SIZE ;continue ;
@@ -430,6 +441,7 @@ int __init create_mem_map_page_table(u64 start, u64 end, void *arg)struct page *map_start, *map_end;int node;pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -444,12 +456,20 @@ int __init create_mem_map_page_table(u64 start, u64 end, void *arg)for (address = start_page; address < end_page; address +=PAGE_SIZE ) { pgd =pgd_offset_k (address);if (pgd_none (*pgd)) { + p4d =memblock_alloc_node (PAGE_SIZE ,PAGE_SIZE , node); +if (!p4d) + goto err_alloc; +pgd_populate (&init_mm, pgd, p4d); + } + p4d =p4d_offset (pgd, address); + +if (p4d_none (*p4d)) { pud =memblock_alloc_node (PAGE_SIZE ,PAGE_SIZE , node);if (!pud) goto err_alloc; -pgd_populate (&init_mm, pgd, pud); +p4d_populate (&init_mm, p4d, pud); } - pud =pud_offset (pgd, address); + pud =pud_offset (p4d, address);if (pud_none (*pud)) { pmd =memblock_alloc_node (PAGE_SIZE ,PAGE_SIZE , node);
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 5/13 nios2: add support for folded p4d page tables
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and remove usage of __ARCH_USE_5LEVEL_HACK.
--- arch/nios2/include/asm/pgtable.h | 3 +-- arch/nios2/mm/fault.c | 9 +++++++-- arch/nios2/mm/ioremap.c | 6 +++++- 3 files changed, 13 insertions(+), 5 deletions(-)
arch/nios2/include/asm/pgtable.h +1 -2
@@ -22,7 +22,6 @@#include <asm/tlbflush.h> #include <asm/pgtable-bits.h> -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #define FIRST_USER_ADDRESS 0UL
@@ -100,7 +99,7 @@ extern pte_t invalid_pte_table[PAGE_SIZE/sizeof(pte_t)];*/ static inline void set_pmd (pmd_t *pmdptr,pmd_t pmdval) { - pmdptr->pud.pgd.pgd = pmdval.pud.pgd.pgd; + *pmdptr = pmdval; }/* to find an entry in a page-table-directory */
arch/nios2/mm/fault.c +7 -2
@@ -245,6 +245,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause,*/ int offset =pgd_index (address);pgd_t *pgd, *pgd_k; +p4d_t *p4d, *p4d_k;pud_t *pud, *pud_k;pmd_t *pmd, *pmd_k;pte_t *pte_k;
@@ -256,8 +257,12 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long cause, goto no_context;set_pgd (pgd, *pgd_k); - pud =pud_offset (pgd, address); - pud_k =pud_offset (pgd_k, address); + p4d =p4d_offset (pgd, address); + p4d_k =p4d_offset (pgd_k, address); +if (!p4d_present (*p4d_k)) + goto no_context; + pud =pud_offset (p4d, address); + pud_k =pud_offset (p4d_k, address);if (!pud_present (*pud_k)) goto no_context; pmd =pmd_offset (pud, address);
arch/nios2/mm/ioremap.c +5 -1
@@ -86,11 +86,15 @@ static int remap_area_pages(unsigned long address, unsigned long phys_addr,if (address >= end)BUG (); do { + p4d_t *p4d; pud_t *pud; pmd_t *pmd; error = -ENOMEM; - pud = pud_alloc(&init_mm, dir, address); + p4d = p4d_alloc(&init_mm, dir, address); + if (!p4d) + break; + pud = pud_alloc(&init_mm, p4d, address); if (!pud) break; pmd = pmd_alloc(&init_mm, pud, address);
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 6/13 openrisc: add support for folded p4d page tables
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and remove usage of __ARCH_USE_5LEVEL_HACK.
--- arch/openrisc/include/asm/pgtable.h | 1 - arch/openrisc/mm/fault.c | 10 ++++++++-- arch/openrisc/mm/init.c | 4 +++- 3 files changed, 11 insertions(+), 4 deletions(-)
arch/openrisc/include/asm/pgtable.h +0 -1
@@ -21,7 +21,6 @@#ifndef __ASM_OPENRISC_PGTABLE_H#define __ASM_OPENRISC_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #ifndef __ASSEMBLY__
arch/openrisc/mm/fault.c +8 -2
@@ -296,6 +296,7 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,int offset =pgd_index (address);pgd_t *pgd, *pgd_k; +p4d_t *p4d, *p4d_k;pud_t *pud, *pud_k;pmd_t *pmd, *pmd_k;pte_t *pte_k;
@@ -322,8 +323,13 @@ asmlinkage void do_page_fault(struct pt_regs *regs, unsigned long address,* it exists. */ - pud =pud_offset (pgd, address); - pud_k =pud_offset (pgd_k, address); + p4d =p4d_offset (pgd, address); + p4d_k =p4d_offset (pgd_k, address); +if (!p4d_present (*p4d_k)) + goto no_context; + + pud =pud_offset (p4d, address); + pud_k =pud_offset (p4d_k, address);if (!pud_present (*pud_k)) goto no_context;
arch/openrisc/mm/init.c +3 -1
@@ -71,6 +71,7 @@ static void __init map_ram(void)unsigned long v, p, e;pgprot_t prot;pgd_t *pge; +p4d_t *p4e;pud_t *pue;pmd_t *pme;pte_t *pte;
@@ -90,7 +91,8 @@ static void __init map_ram(void)while (p < e) {int j; - pue =pud_offset (pge, v); + p4e =p4d_offset (pge, v); + pue =pud_offset (p4e, v); pme =pmd_offset (pue, v);if ((u32 ) pue != (u32 ) pge || (u32 ) pme != (u32 ) pge) {
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 7/13 powerpc: add support for folded p4d page tables
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace 5level-fixup.h with pgtable-nop4d.h.
--- arch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 58 ++++++++++-------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 8 +++ arch/powerpc/kvm/book3s_64_mmu_radix.c | 59 ++++++++++++++++--- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 19 ++++-- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 +++-- arch/powerpc/mm/pgtable.c | 25 +++++++- arch/powerpc/mm/pgtable_32.c | 28 +++++---- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 22 ++++++- arch/powerpc/xmon/xmon.c | 17 +++++- 28 files changed, 284 insertions(+), 120 deletions(-)
arch/powerpc/include/asm/book3s/32/pgtable.h +0 -1
@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_BOOK3S_32_PGTABLE_H#define _ASM_POWERPC_BOOK3S_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/book3s/32/hash.h>
arch/powerpc/include/asm/book3s/64/hash.h +2 -2
@@ -134,9 +134,9 @@ static inline int get_region_id(unsigned long ea)#define hash__pmd_bad (pmd) (pmd_val(pmd) & H_PMD_BAD_BITS)#define hash__pud_bad (pud) (pud_val(pud) & H_PUD_BAD_BITS) -static inline int hash__pgd_bad (pgd_t pgd) +static inline int hash__p4d_bad (p4d_t p4d) { -return (pgd_val (pgd) =0 ); +return (p4d_val (p4d) =0 ); }#ifdef CONFIG_STRICT_KERNEL_RWX extern void hash__mark_rodata_ro (void );
arch/powerpc/include/asm/book3s/64/pgalloc.h +2 -2
@@ -85,9 +85,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)kmem_cache_free (PGT_CACHE (PGD_INDEX_SIZE ), pgd); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,pud_t *pud) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *pgd,pud_t *pud) { - *pgd =__pgd (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); + *pgd =__p4d (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)
arch/powerpc/include/asm/book3s/64/pgtable.h +35 -23
117 lines not on this page · raw message
arch/powerpc/include/asm/book3s/64/radix.h +3 -3
@@ -30,7 +30,7 @@/* Don't have anything in the reserved bits and leaf bits */ #define RADIX_PMD_BAD_BITS 0x60000000000000e0UL#define RADIX_PUD_BAD_BITS 0x60000000000000e0UL -#define RADIX_PGD_BAD_BITS 0x60000000000000e0UL +#define RADIX_P4D_BAD_BITS 0x60000000000000e0UL#define RADIX_PMD_SHIFT (PAGE_SHIFT + RADIX_PTE_INDEX_SIZE)#define RADIX_PUD_SHIFT (RADIX_PMD_SHIFT + RADIX_PMD_INDEX_SIZE)
@@ -227,9 +227,9 @@ static inline int radix__pud_bad(pud_t pud) } -static inline int radix__pgd_bad (pgd_t pgd) +static inline int radix__p4d_bad (p4d_t p4d) { -return !!(pgd_val (pgd) &RADIX_PGD_BAD_BITS ); +return !!(p4d_val (p4d) &RADIX_P4D_BAD_BITS ); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
arch/powerpc/include/asm/nohash/32/pgtable.h +0 -1
@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_32_PGTABLE_H#define _ASM_POWERPC_NOHASH_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #ifndef __ASSEMBLY__
arch/powerpc/include/asm/nohash/64/pgalloc.h +1 -1
@@ -15,7 +15,7 @@ struct vmemmap_backing { };extern struct vmemmap_backing *vmemmap_list; -#define pgd_populate (MM ,PGD ,PUD ) pgd_set(PGD, (unsigned long)PUD) +#define p4d_populate (MM ,P4D ,PUD ) p4d_set(P4D, (unsigned long)PUD)static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) {
arch/powerpc/include/asm/nohash/64/pgtable-4k.h +16 -16
@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H#define _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> /* * Entries per page directory level. The PTE level must use a 64b record
@@ -45,41 +45,41 @@#define PMD_MASKED_BITS 0/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0 -/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0 +/* Bits to mask out from a P4D to get to the PUD page */ +#define P4D_MASKED_BITS 0/* * 4-level page tables related bits */ -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (pgd_val(pgd) = 0) -#define pgd_present (pgd) (pgd_val(pgd) != 0) -#define pgd_page_vaddr (pgd) (pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (p4d_val(p4d) = 0) +#define p4d_present (p4d) (p4d_val(p4d) != 0) +#define p4d_page_vaddr (p4d) (p4d_val(p4d) & ~P4D_MASKED_BITS)#ifndef __ASSEMBLY__ -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte (pgd_val (pgd)); +return __pte (p4d_val (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd (pte_val (pte)); +return __p4d (pte_val (pte)); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);#endif /* !__ASSEMBLY__ */ -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + \ +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + \ (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)))#define pud_ERROR (e) \
arch/powerpc/include/asm/nohash/64/pgtable.h +3 -3
@@ -175,11 +175,11 @@ static inline pud_t pte_pud(pte_t pte)return __pud (pte_val (pte)); }#define pud_write (pud) pte_write(pud_pte(pud)) -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define p4d_write (pgd) pte_write(p4d_pte(p4d)) -static inline void pgd_set (pgd_t *pgdp,unsigned long val) +static inline void p4d_set (p4d_t *p4dp,unsigned long val) { - *pgdp =__pgd (val); + *p4dp =__p4d (val); }/*
arch/powerpc/include/asm/pgtable.h +8 -0
@@ -139,6 +139,14 @@ static inline bool pud_is_leaf(pud_t pud) }#endif +#ifndef p4d_is_leaf +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) +{ +return false; +} +#endif +#ifndef pgd_is_leaf#define pgd_is_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd)
arch/powerpc/kvm/book3s_64_mmu_radix.c +51 -8
108 lines not on this page · raw message
arch/powerpc/lib/code-patching.c +6 -1
@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;
arch/powerpc/mm/book3s32/mmu.c +1 -1
@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }
arch/powerpc/mm/book3s32/tlb.c +2 -2
@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)
@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }
arch/powerpc/mm/book3s64/hash_pgtable.c +3 -1
@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);
arch/powerpc/mm/book3s64/radix_pgtable.c +15 -4
@@ -64,17 +64,24 @@ static int early_map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep; pgdp =pgd_offset_k (ea);if (pgd_none (*pgdp)) { + p4dp =early_alloc_pgtable (PGD_TABLE_SIZE , nid, + region_start, region_end); +pgd_populate (&init_mm, pgdp, p4dp); + } + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { pudp =early_alloc_pgtable (PUD_TABLE_SIZE , nid, region_start, region_end); -pgd_populate (&init_mm, pgdp, pudp); +p4d_populate (&init_mm, p4dp, pudp); } - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (map_page_size =PUD_SIZE ) { ptep = (pte_t *)pudp; goto set_the_pte;
@@ -114,6 +121,7 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -136,7 +144,8 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa,* boot. */ pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ;if (map_page_size =PUD_SIZE ) {
@@ -173,6 +182,7 @@ void radix__change_memory_range(unsigned long start, unsigned long end, {unsigned long idx;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -185,7 +195,8 @@ void radix__change_memory_range(unsigned long start, unsigned long end,for (idx = start; idx < end; idx +=PAGE_SIZE ) { pgdp =pgd_offset_k (idx); - pudp =pud_alloc (&init_mm, pgdp, idx); + p4dp =p4d_offset (pgdp, idx); + pudp =pud_alloc (&init_mm, p4dp, idx);if (!pudp)continue ;if (pud_is_leaf (*pudp)) {
arch/powerpc/mm/book3s64/subpage_prot.c +5 -1
@@ -54,6 +54,7 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, int npages) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -62,7 +63,10 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, pgd =pgd_offset (mm, addr);if (pgd_none (*pgd))return ; - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d)) +return ; + pud =pud_offset (p4d, addr);if (pud_none (*pud))return ; pmd =pmd_offset (pud, addr);
arch/powerpc/mm/hugetlbpage.c +16 -12
@@ -112,6 +112,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {pgd_t *pg; +p4d_t *p4;pud_t *pu;pmd_t *pm;hugepd_t *hpdp =NULL ;
@@ -121,20 +122,21 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz addr &= ~(sz-1 ); pg =pgd_offset (mm, addr); + p4 =p4d_offset (pg, addr);#ifdef CONFIG_PPC_BOOK3S_64 if (pshift =PGDIR_SHIFT )/* 16GB huge page */ -return (pte_t *) pg; +return (pte_t *) p4;else if (pshift >PUD_SHIFT ) {/* * We need to use hugepd table */ ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift =PUD_SHIFT )
@@ -159,10 +161,10 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz#else if (pshift >=PGDIR_SHIFT ) { ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift >=PUD_SHIFT ) {
@@ -384,7 +386,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,mm_dec_nr_pmds (tlb->mm); } -static void hugetlb_free_pud_range (struct mmu_gather *tlb,pgd_t *pgd, +static void hugetlb_free_pud_range (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling) {
@@ -394,7 +396,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd, start = addr; do { - pud = pud_offset(pgd, addr); + pud = pud_offset(p4d, addr); next = pud_addr_end(addr, end); if (!is_hugepd(__hugepd(pud_val(*pud)))) { if (pud_none_or_clear_bad(pud))
@@ -429,8 +431,8 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,if (end -1 > ceiling -1 )return ; - pud =pud_offset (pgd, start); -pgd_clear (pgd); + pud =pud_offset (p4d, start); +p4d_clear (p4d);pud_free_tlb (tlb, pud, start);mm_dec_nr_puds (tlb->mm); }
@@ -443,6 +445,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long floor, unsigned long ceiling) {pgd_t *pgd; +p4d_t *p4d;unsigned long next;/*
@@ -465,10 +468,11 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, do { next = pgd_addr_end(addr, end); pgd = pgd_offset(tlb->mm, addr); + p4d = p4d_offset(pgd, addr); if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { - if (pgd_none_or_clear_bad(pgd)) + if (p4d_none_or_clear_bad(p4d)) continue; - hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling); + hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); } else { unsigned long more; /*
@@ -481,7 +485,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,if (more > next) next = more; -free_hugepd_range (tlb, (hugepd_t *)pgd,PGDIR_SHIFT , +free_hugepd_range (tlb, (hugepd_t *)p4d,PGDIR_SHIFT , addr, next, floor, ceiling); } }while (addr = next, addr != end);
arch/powerpc/mm/kasan/kasan_init_32.c +4 -4
@@ -36,7 +36,7 @@ static int __init kasan_init_shadow_page_tables(unsigned long k_start, unsignedunsigned long k_cur, k_next;pte_t *new =NULL ; - pmd =pmd_offset (pud_offset (pgd_offset_k (k_start), k_start), k_start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_start), k_start), k_start), k_start);for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++) { k_next =pgd_addr_end (k_cur, k_end);
@@ -78,7 +78,7 @@ static int __init kasan_init_region(void *start, size_t size) block =memblock_alloc (k_end - k_start,PAGE_SIZE );for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);void *va = block + k_cur - k_start;pte_t pte =pfn_pte (PHYS_PFN (__pa (va)),PAGE_KERNEL );
@@ -102,7 +102,7 @@ static void __init kasan_remap_early_shadow_ro(void)kasan_populate_pte (kasan_early_shadow_pte, prot);for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);pte_t *ptep =pte_offset_kernel (pmd, k_cur);if ((pte_val (*ptep) &PTE_RPN_MASK ) != pa)
@@ -202,7 +202,7 @@ void __init kasan_early_init(void)unsigned long addr =KASAN_SHADOW_START ;unsigned long end =KASAN_SHADOW_END ;unsigned long next; -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (addr), addr), addr); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (addr), addr), addr), addr);BUILD_BUG_ON (KASAN_SHADOW_START & ~PGDIR_MASK );
arch/powerpc/mm/mem.c +2 -2
@@ -69,8 +69,8 @@ EXPORT_SYMBOL(kmap_prot);static inline pte_t *virt_to_kpte (unsigned long vaddr) { -return pte_offset_kernel (pmd_offset (pud_offset (pgd_offset_k (vaddr), - vaddr), vaddr), vaddr); +return pte_offset_kernel (pmd_offset (pud_offset (p4d_offset (pgd_offset_k (vaddr), + vaddr), vaddr), vaddr), vaddr); }#endif
arch/powerpc/mm/nohash/40x.c +2 -2
@@ -104,7 +104,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_16M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val);
@@ -119,7 +119,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_4M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp =__pmd (val); v +=LARGE_PAGE_SIZE_4M ;
arch/powerpc/mm/nohash/book3e_pgtable.c +11 -4
@@ -73,6 +73,7 @@ static void __init *early_alloc_pgtable(unsigned long size)int __ref map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -80,7 +81,10 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_alloc (&init_mm, pgdp, ea); +if (!p4dp) +return -ENOMEM ; + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);
@@ -91,13 +95,16 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)return -ENOMEM ; }else { pgdp =pgd_offset_k (ea); -#ifndef __PAGETABLE_PUD_FOLDEDif (pgd_none (*pgdp)) { pudp =early_alloc_pgtable (PUD_TABLE_SIZE );pgd_populate (&init_mm, pgdp, pudp); } -#endif /* !__PAGETABLE_PUD_FOLDED */ - pudp =pud_offset (pgdp, ea); + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { + pmdp =early_alloc_pgtable (PMD_TABLE_SIZE ); +p4d_populate (&init_mm, p4dp, pmdp); + } + pudp =pud_offset (p4dp, ea);if (pud_none (*pudp)) { pmdp =early_alloc_pgtable (PMD_TABLE_SIZE );pud_populate (&init_mm, pudp, pmdp);
arch/powerpc/mm/pgtable.c +23 -2
@@ -265,6 +265,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,void assert_pte_locked (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;
@@ -272,7 +273,9 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr)return ; pgd = mm->pgd +pgd_index (addr);BUG_ON (pgd_none (*pgd)); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +BUG_ON (p4d_none (*p4d)); + pud =pud_offset (p4d, addr);BUG_ON (pud_none (*pud)); pmd =pmd_offset (pud, addr);/*
@@ -313,6 +316,7 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, bool *is_thp, unsigned *hpage_shift) { pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ret_pte;
@@ -346,13 +350,30 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea, goto out_huge; } + pdshift =P4D_SHIFT ; + p4dp =p4d_offset (&pgd, ea); + p4d =READ_ONCE (*p4dp); + +if (p4d_none (p4d)) +return NULL ; + +if (p4d_is_leaf (p4d)) { + ret_pte = (pte_t *)p4dp; + goto out; + } + +if (is_hugepd (__hugepd (p4d_val (p4d)))) { + hpdp = (hugepd_t *)&p4d; + goto out_huge; + } +/* * Even if we end up with an unmap, the pgtable will not * be freed, because we do an rcu free and here we are * irq disabled */ pdshift =PUD_SHIFT ; - pudp =pud_offset (&pgd, ea); + pudp =pud_offset (&p4d, ea); pud =READ_ONCE (*pudp);if (pud_none (pud))
arch/powerpc/mm/pgtable_32.c +16 -12
@@ -63,7 +63,7 @@ int __ref map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot)int err = -ENOMEM ;/* Use upper 10 bits of VA to index the first level map */ - pd =pmd_offset (pud_offset (pgd_offset_k (va), va), va); + pd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va);/* Use middle 10 bits of VA to index the second-level map */ if (likely (slab_is_available ())) pg =pte_alloc_kernel (pd, va);
@@ -130,6 +130,7 @@ static int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep, pmd_t **pmdp) { pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -137,17 +138,20 @@ get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep, pmd_t **pmdp) pgd =pgd_offset (mm, addr &PAGE_MASK );if (pgd) { - pud =pud_offset (pgd, addr &PAGE_MASK ); -if (pud &&pud_present (*pud)) { - pmd =pmd_offset (pud, addr &PAGE_MASK ); -if (pmd_present (*pmd)) { - pte =pte_offset_map (pmd, addr &PAGE_MASK ); -if (pte) { - retval =1 ; - *ptep = pte; -if (pmdp) - *pmdp = pmd; -/* XXX caller needs to do pte_unmap, yuck */ + p4d =p4d_offset (pgd, addr &PAGE_MASK ); +if (p4d &&p4d_present (*p4d)) { + pud =pud_offset (p4d, addr &PAGE_MASK ); +if (pud &&pud_present (*pud)) { + pmd =pmd_offset (pud, addr &PAGE_MASK ); +if (pmd_present (*pmd)) { + pte =pte_offset_map (pmd, addr &PAGE_MASK ); +if (pte) { + retval =1 ; + *ptep = pte; +if (pmdp) + *pmdp = pmd; +/* XXX caller needs to do pte_unmap, yuck */ + } } } }
arch/powerpc/mm/pgtable_64.c +5 -5
@@ -101,13 +101,13 @@ EXPORT_SYMBOL(__pte_frag_size_shift);#ifndef __PAGETABLE_PUD_FOLDED/* 4 level page table */ -struct page *pgd_page (pgd_t pgd) +struct page *p4d_page (p4d_t p4d) { -if (pgd_is_leaf (pgd)) { -VM_WARN_ON (!pgd_huge (pgd)); -return pte_page (pgd_pte (pgd)); +if (p4d_is_leaf (p4d)) { +VM_WARN_ON (!p4d_huge (p4d)); +return pte_page (p4d_pte (p4d)); } -return virt_to_page (pgd_page_vaddr (pgd)); +return virt_to_page (p4d_page_vaddr (p4d)); }#endif
arch/powerpc/mm/ptdump/hashpagetable.c +17 -3
@@ -417,9 +417,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;
@@ -431,6 +431,20 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); + } +} +static void walk_pagetables (struct pg_state *st) {pgd_t *pgd =pgd_offset_k (0UL );
@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr); } }
arch/powerpc/mm/ptdump/ptdump.c +19 -3
@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;
@@ -293,6 +293,22 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); +else +note_page (st, addr,2 ,p4d_val (*p4d)); + } +} +static void walk_pagetables (struct pg_state *st) {unsigned int i;
@@ -306,7 +322,7 @@ static void walk_pagetables(struct pg_state *st)for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) {if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr);else note_page (st, addr,1 ,pgd_val (*pgd)); }
arch/powerpc/xmon/xmon.c +16 -1
@@ -3130,6 +3130,7 @@ static void show_pte(unsigned long addr)struct task_struct *tsk =NULL ;struct mm_struct *mm;pgd_t *pgdp, *pgdir; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -3174,7 +3175,21 @@ static void show_pte(unsigned long addr) }printf ("pgdp @ 0x%px = 0x%016lx\n" , pgdp,pgd_val (*pgdp)); - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); + +if (p4d_none (*p4dp)) { +printf ("No valid P4D\n" ); +return ; + } + +if (p4d_is_leaf (*p4dp)) { +format_pte (p4dp,p4d_val (*p4dp)); +return ; + } + +printf ("p4dp @ 0x%px = 0x%016lx\n" , p4dp,p4d_val (*p4dp)); + + pudp =pud_offset (p4dp, addr);if (pud_none (*pudp)) {printf ("No valid PUD\n" );
Signature · 1 line
2.24.0
Re: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.
I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
- pud = pud_offset(pgd, gpa);
+ pud = pud_offset(p4d_offset(pgd, gpa), gpa);
The additionnal walks should be added through another patch the day powerpc need them.
See below for more comments.
Quoted 101 lines
Signed-off-by Mike Rapoport <[email protected]>--- arch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 58 ++++++++++-------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 8 +++ arch/powerpc/kvm/book3s_64_mmu_radix.c | 59 ++++++++++++++++--- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 19 ++++-- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 +++-- arch/powerpc/mm/pgtable.c | 25 +++++++- arch/powerpc/mm/pgtable_32.c | 28 +++++---- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 22 ++++++- arch/powerpc/xmon/xmon.c | 17 +++++- 28 files changed, 284 insertions(+), 120 deletions(-)arch/powerpc/include/asm/book3s/32/pgtable.h +0
-1@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_BOOK3S_32_PGTABLE_H#define _ASM_POWERPC_BOOK3S_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/book3s/32/hash.h> arch/powerpc/include/asm/book3s/64/hash.h +2
-2@@ -134,9 +134,9 @@ static inline int get_region_id(unsigned long ea)#define hash__pmd_bad (pmd) (pmd_val(pmd) & H_PMD_BAD_BITS)#define hash__pud_bad (pud) (pud_val(pud) & H_PUD_BAD_BITS) -static inline int hash__pgd_bad (pgd_t pgd) +static inline int hash__p4d_bad (p4d_t p4d) { -return (pgd_val (pgd) =0 ); +return (p4d_val (p4d) =0 ); }#ifdef CONFIG_STRICT_KERNEL_RWX extern void hash__mark_rodata_ro (void );arch/powerpc/include/asm/book3s/64/pgalloc.h +2
-2@@ -85,9 +85,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)kmem_cache_free (PGT_CACHE (PGD_INDEX_SIZE ), pgd); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,pud_t *pud) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *pgd,pud_t *pud) { - *pgd =__pgd (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); + *pgd =__p4d (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)arch/powerpc/include/asm/book3s/64/pgtable.h +2
-2@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_H_#define _ASM_POWERPC_BOOK3S_64_PGTABLE_H_ -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #ifndef __ASSEMBLY__#include <linux/mmdebug.h> @@ -251,7 +251,7 @@ extern unsigned long __pmd_frag_size_shift;/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0xc0000000000000ffUL/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0xc0000000000000ffUL +#define P4D_MASKED_BITS 0xc0000000000000ffUL/* * Used as an indicator for rcu callback functions
arch/powerpc/include/asm/book3s/64/pgtable.h static inline bool pud_access_permitted(pud_t pud, bool write) · line 957
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +6
-1@@ -949,54 +949,60 @@ static inline bool pud_access_permitted(pud_t pud, bool write)return pte_access_permitted (pud_pte (pud), write); } -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define __p4d_raw (x) ((p4d_t) { __pgd_raw(x) }) +static inline __be64 p4d_raw (p4d_t x) +{ +return pgd_raw (x.pgd); +} +
Shouldn't this be defined in asm/pgtable-be-types.h, just like other __pxx_raw() ?
Quoted 93 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +27
-20+#define p4d_write (p4d) pte_write(p4d_pte(p4d)) -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline int pgd_none (pgd_t pgd) +static inline int p4d_none (p4d_t p4d) { -return !pgd_raw (pgd); +return !p4d_raw (p4d); } -static inline int pgd_present (pgd_t pgd) +static inline int p4d_present (p4d_t p4d) { -return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PRESENT)); +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PRESENT)); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte_raw (pgd_raw (pgd)); +return __pte_raw (p4d_raw (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd_raw (pte_raw (pte)); +return __p4d_raw (pte_raw (pte)); } -static inline int pgd_bad (pgd_t pgd) +static inline int p4d_bad (p4d_t p4d) {if (radix_enabled ()) -return radix__pgd_bad (pgd); -return hash__pgd_bad (pgd); +return radix__p4d_bad (p4d); +return hash__p4d_bad (p4d); } -#define pgd_access_permitted pgd_access_permitted -static inline bool pgd_access_permitted (pgd_t pgd,bool write) +#define p4d_access_permitted p4d_access_permitted +static inline bool p4d_access_permitted (p4d_t p4d,bool write) { -return pte_access_permitted (pgd_pte (pgd), write); +return pte_access_permitted (p4d_pte (p4d), write); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);/* Pointers in the page table tree are physical addresses */ #define __pgtable_ptr_val (ptr) __pa(ptr)#define pmd_page_vaddr (pmd) __va(pmd_val(pmd) & ~PMD_MASKED_BITS)#define pud_page_vaddr (pud) __va(pud_val(pud) & ~PUD_MASKED_BITS) -#define pgd_page_vaddr (pgd) __va(pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_page_vaddr (p4d) __va(p4d_val(p4d) & ~P4D_MASKED_BITS)#define pgd_index (address) (((address) >> (PGDIR_SHIFT)) & (PTRS_PER_PGD - 1))#define pud_index (address) (((address) >> (PUD_SHIFT)) & (PTRS_PER_PUD - 1))@@ -1010,8 +1016,8 @@ extern struct page *pgd_page(pgd_t pgd);#define pgd_offset (mm, address) ((mm)->pgd + pgd_index(address)) -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + pud_index(addr)) +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + pud_index(addr))#define pmd_offset (pudp,addr) \ (((pmd_t *) pud_page_vaddr(*(pudp))) + pmd_index(addr))#define pte_offset_kernel (dir,addr) \@@ -1368,6 +1374,12 @@ static inline bool pud_is_leaf(pud_t pud)return !!(pud_raw (pud) &cpu_to_be64 (_PAGE_PTE)); } +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) +{ +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PTE)); +} +#define pgd_is_leaf pgd_is_leaf#define pgd_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd)
[...]
Quoted 79 lines
arch/powerpc/include/asm/pgtable.h +8
-0@@ -139,6 +139,14 @@ static inline bool pud_is_leaf(pud_t pud) }#endif +#ifndef p4d_is_leaf +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) +{ +return false; +} +#endif +#ifndef pgd_is_leaf#define pgd_is_leaf pgd_is_leafstatic inline bool pgd_is_leaf (pgd_t pgd)arch/powerpc/kvm/book3s_64_mmu_radix.c +26
-4@@ -494,17 +494,39 @@ static void kvmppc_unmap_free_pud(struct kvm *kvm, pud_t *pud,pud_free (kvm->mm, pud); } +static void kvmppc_unmap_free_p4d (struct kvm *kvm,p4d_t *p4d, +unsigned int lpid) +{ +unsigned long iu; +p4d_t *p = p4d; + +for (iu =0 ; iu <PTRS_PER_P4D ; ++iu, ++p) { +if (!p4d_present (*p)) +continue ; +if (p4d_is_leaf (*p)) { +p4d_clear (p); + }else { +pud_t *pud; + + pud =pud_offset (p,0 ); +kvmppc_unmap_free_pud (kvm, pud, lpid); +p4d_clear (p); + } + } +p4d_free (kvm->mm, p4d); +} +void kvmppc_free_pgtable_radix (struct kvm *kvm,pgd_t *pgd,unsigned int lpid) {unsigned long ig;for (ig =0 ; ig <PTRS_PER_PGD ; ++ig, ++pgd) { -pud_t *pud; +p4d_t *p4d;if (!pgd_present (*pgd))continue ; - pud =pud_offset (pgd,0 ); -kvmppc_unmap_free_pud (kvm, pud, lpid); + p4d =p4d_offset (pgd,0 ); +kvmppc_unmap_free_p4d (kvm, p4d, lpid);pgd_clear (pgd); } }@@ -566,6 +588,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, unsigned long *rmapp, struct rmap_nested **n_rmap) { pgd_t *pgd; +p4d_t *p4d, *new_p4d =NULL ;pud_t *pud, *new_pud =NULL ;pmd_t *pmd, *new_pmd =NULL ;pte_t *ptep, *new_ptep =NULL ;@@ -573,9 +596,15 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Traverse the guest's 2nd-level tree, allocate new levels needed */ pgd = pgtable +pgd_index (gpa); - pud =NULL ;
arch/powerpc/kvm/book3s_64_mmu_radix.c int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, · line 607
Quoted 10 lines
arch/powerpc/kvm/book3s_64_mmu_radix.c +8
-1+ p4d =NULL ;if (pgd_present (*pgd)) - pud =pud_offset (pgd, gpa); + p4d =p4d_offset (pgd, gpa); +else + new_p4d =p4d_alloc_one (kvm->mm, gpa); + + pud =NULL ; +if (p4d_present (*p4d)) + pud =pud_offset (p4d, gpa);
Is it worth adding all this new code ?
My understanding is that the series objective is to get rid of __ARCH_HAS_5LEVEL_HACK, to to add support for 5 levels to an architecture that not need it (at least for now).
If we want to add support for 5 levels, it can be done later in another patch.
Here I think your change could be limited to:
- pud = pud_offset(pgd, gpa);
+ pud = pud_offset(p4d_offset(pgd, gpa), gpa);
Quoted 35 lines
arch/powerpc/kvm/book3s_64_mmu_radix.c +9
-2else new_pud =pud_alloc_one (kvm->mm, gpa);@@ -597,12 +626,18 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Now traverse again under the lock and change the tree */ ret = -ENOMEM ;if (pgd_none (*pgd)) { +if (!new_p4d) + goto out_unlock; +pgd_populate (kvm->mm, pgd, new_p4d); + new_p4d =NULL ; + } +if (p4d_none (*p4d)) {if (!new_pud) goto out_unlock; -pgd_populate (kvm->mm, pgd, new_pud); +p4d_populate (kvm->mm, p4d, new_pud); new_pud =NULL ; } - pud =pud_offset (pgd, gpa); + pud =pud_offset (p4d, gpa);if (pud_is_leaf (*pud)) {unsigned long hgpa = gpa &PUD_MASK ;@@ -1220,6 +1255,7 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,pgd_t *pgt;struct kvm_nested_guest *nested;pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ptep;@@ -1298,7 +1334,14 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,continue ; }
arch/powerpc/kvm/book3s_64_mmu_radix.c static ssize_t debugfs_radix_read(struct file *file, char __user *buf, · line 1344
Quoted 10 lines
arch/powerpc/kvm/book3s_64_mmu_radix.c +8
-1- pudp =pud_offset (&pgd, gpa); + p4dp =p4d_offset (&pgd, gpa); + p4d =READ_ONCE (*p4dp); +if (!(p4d_val (p4d) & _PAGE_PRESENT)) { + gpa = (gpa &P4D_MASK ) +P4D_SIZE ; +continue ; + } + + pudp =pud_offset (&p4d, gpa);
Same, here you are forcing a useless read with READ_ONCE().
Your change could be limited to
- pudp = pud_offset(&pgd, gpa);
+ pudp = pud_offset(p4d_offset(&pgd, gpa), gpa);
This comment applies to many other places.
Quoted 32 lines
arch/powerpc/kvm/book3s_64_mmu_radix.c +0
-0pud =READ_ONCE (*pudp);if (!(pud_val (pud) & _PAGE_PRESENT)) { gpa = (gpa &PUD_MASK ) +PUD_SIZE ;arch/powerpc/lib/code-patching.c +6
-1@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;arch/powerpc/mm/book3s32/mmu.c +0
-0
arch/powerpc/mm/book3s32/mmu.c void hash_preload(struct mm_struct *mm, unsigned long ea) · line 315
Quoted 5 lines
arch/powerpc/mm/book3s32/mmu.c +1
-1@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);
If we continue like this, in ten years this like is going to be many kilometers long.
I think the above would be worth a generic helper.
Quoted 38 lines
arch/powerpc/mm/book3s32/mmu.c +0
-0if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }arch/powerpc/mm/book3s32/tlb.c +2
-2@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }arch/powerpc/mm/book3s64/hash_pgtable.c +1
-0@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
arch/powerpc/mm/book3s64/hash_pgtable.c int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot) · line 160
Quoted 6 lines
arch/powerpc/mm/book3s64/hash_pgtable.c +2
-1@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);
Could be a single line, without a new var.
- pudp = pud_alloc(&init_mm, pgdp, ea);
+ pudp = pud_alloc(&init_mm, p4d_offset(pgdp, ea), ea);
Same kind of comments as already done apply to the rest.
Christophe
Mike Rapoport
reviewerRe: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Quoted 14 lines · Christophe Leroy
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);
Quoted 2 lines
The additionnal walks should be added through another patch the day powerpc need them.
Ok, I'll update the patch to reduce walking the p4d.
Quoted 6 lines
See below for more comments.
Quoted 4 lines
Signed-off-by Mike Rapoport <[email protected]>---
...
Quoted 24 lines
Quoted 24 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +3
-3@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_H_#define _ASM_POWERPC_BOOK3S_64_PGTABLE_H_ -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #ifndef __ASSEMBLY__#include <linux/mmdebug.h> @@ -251,7 +251,7 @@ extern unsigned long __pmd_frag_size_shift;/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0xc0000000000000ffUL/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0xc0000000000000ffUL +#define P4D_MASKED_BITS 0xc0000000000000ffUL/* * Used as an indicator for rcu callback functions @@ -949,54 +949,60 @@ static inline bool pud_access_permitted(pud_t pud, bool write)return pte_access_permitted (pud_pte (pud), write); } -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define __p4d_raw (x) ((p4d_t) { __pgd_raw(x) })
Quoted 8 lines
arch/powerpc/include/asm/book3s/64/pgtable.h static inline bool pud_access_permitted(pud_t pud, bool write) · line 957
Quoted 5 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +5
-0+static inline __be64 p4d_raw (p4d_t x) +{ +return pgd_raw (x.pgd); +} +Shouldn't this be defined in
asm/pgtable-be-types.h, just like other__pxx_raw()?
Ideally yes, but this creates weird header file dependencies and untangling them would generate way too much churn.
Quoted 7 lines
Quoted 7 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +3
-2+#define p4d_write (p4d) pte_write(p4d_pte(p4d)) -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); }
...
Quoted 25 lines
Quoted 4 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +0
-1@@ -573,9 +596,15 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Traverse the guest's 2nd-level tree, allocate new levels needed */ pgd = pgtable +pgd_index (gpa); - pud =NULL ;arch/powerpc/include/asm/book3s/64/pgtable.h int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, · line 606
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +8
-1+ p4d =NULL ;if (pgd_present (*pgd)) - pud =pud_offset (pgd, gpa); + p4d =p4d_offset (pgd, gpa); +else + new_p4d =p4d_alloc_one (kvm->mm, gpa); + + pud =NULL ; +if (p4d_present (*p4d)) + pud =pud_offset (p4d, gpa);Is it worth adding all this new code ?
My understanding is that the series objective is to get rid of
__ARCH_HAS_5LEVEL_HACK, to to add support for 5 levels to an architecture that not need it (at least for now).
If we want to add support for 5 levels, it can be done later in another patch.Here I think your change could be limited to:
Review
Quoted 2 lines
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);
This won't work. Without __ARCH_USE_5LEVEL_HACK defined pgd_present() is hardwired to 1 and the actual check for the top level is performed with p4d_present(). The 'else' clause that allocates p4d will never be taken and it could be removed, but I prefer to keep it for consistency.
Quoted 47 lines
Quoted 32 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +9
-2else new_pud =pud_alloc_one (kvm->mm, gpa);@@ -597,12 +626,18 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Now traverse again under the lock and change the tree */ ret = -ENOMEM ;if (pgd_none (*pgd)) { +if (!new_p4d) + goto out_unlock; +pgd_populate (kvm->mm, pgd, new_p4d); + new_p4d =NULL ; + } +if (p4d_none (*p4d)) {if (!new_pud) goto out_unlock; -pgd_populate (kvm->mm, pgd, new_pud); +p4d_populate (kvm->mm, p4d, new_pud); new_pud =NULL ; } - pud =pud_offset (pgd, gpa); + pud =pud_offset (p4d, gpa);if (pud_is_leaf (*pud)) {unsigned long hgpa = gpa &PUD_MASK ;@@ -1220,6 +1255,7 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,pgd_t *pgt;struct kvm_nested_guest *nested;pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ptep;@@ -1298,7 +1334,14 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,continue ;arch/powerpc/include/asm/book3s/64/pgtable.h static ssize_t debugfs_radix_read(struct file *file, char __user *buf, · line 1343
Quoted 10 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +8
-1} - pudp =pud_offset (&pgd, gpa); + p4dp =p4d_offset (&pgd, gpa); + p4d =READ_ONCE (*p4dp); +if (!(p4d_val (p4d) & _PAGE_PRESENT)) { + gpa = (gpa &P4D_MASK ) +P4D_SIZE ; +continue ; + } + + pudp =pud_offset (&p4d, gpa);Same, here you are forcing a useless read with
READ_ONCE().Your change could be limited to
Review
Quoted 2 lines
(quoted diff) +1
-1- pudp = pud_offset(&pgd, gpa); + pudp = pud_offset(p4d_offset(&pgd, gpa), gpa);
Here again the actual check must be done against p4d rather than pgd. We could skip READ_ONCE() for pgd, but since it is a debugfs method I don't think it is more important than code consistency.
Quoted 1 line
This comment applies to many other places.
I'll make another pass to see where we can take the shortcut and use
pudp = pud_offset(p4d_offset(...))
Quoted 37 lines
Quoted 29 lines
arch/powerpc/include/asm/book3s/64/pgtable.h +0
-0pud =READ_ONCE (*pudp);if (!(pud_val (pud) & _PAGE_PRESENT)) { gpa = (gpa &PUD_MASK ) +PUD_SIZE ;arch/powerpc/lib/code-patching.c +6
-1@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;arch/powerpc/mm/book3s32/mmu.c +0
-0arch/powerpc/mm/book3s32/mmu.c void hash_preload(struct mm_struct *mm, unsigned long ea) · line 314
Quoted 4 lines
arch/powerpc/mm/book3s32/mmu.c +1
-1@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);If we continue like this, in ten years this like is going to be many kilometers long.
Quoted 1 line
I think the above would be worth a generic helper.
Agree. My plan was to first unify all the architectures and then start introducing the generic helpers, like e.g. pmd_offset_mm().
Quoted 54 lines
Quoted 38 lines
arch/powerpc/mm/book3s32/mmu.c +0
-0if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }arch/powerpc/mm/book3s32/tlb.c +2
-2@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }arch/powerpc/mm/book3s64/hash_pgtable.c +1
-0@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;arch/powerpc/mm/book3s64/hash_pgtable.c int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot) · line 160
Quoted 6 lines
arch/powerpc/mm/book3s64/hash_pgtable.c +2
-1@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);Could be a single line, without a new var.
(quoted diff) +1
-1- pudp = pud_alloc(&init_mm, pgdp, ea); + pudp = pud_alloc(&init_mm, p4d_offset(pgdp, ea), ea);Same kind of comments as already done apply to the rest.
Christophe
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Quoted 12 lines · Mike Rapoport
Quoted 11 lines · Christophe Leroy
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
Quoted 7 lines
Quoted 5 lines
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);The additionnal walks should be added through another patch the day powerpc need them.
Ok, I'll update the patch to reduce walking the p4d.
Here's what I have with more direct acceses from pgd to pud.
From 6c59a86ce8394fb6100e9b6ced2e346981fb0ce9 Mon Sep 17 00:00:00 2001
From: Mike Rapoport <[email protected]>
Date: Sun, 24 Nov 2019 15:38:00 +0200
Subject: [PATCH v3] powerpc: add support for folded p4d page tables
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace 5level-fixup.h with pgtable-nop4d.h.
---
v3:
* reduce amount of added p4d walks
* kill pgtable_32::get_pteptr and traverse page table in pgtable_32::__change_page_attr_noflush
arch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 60 ++++++++++--------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 6 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 30 ++++++---- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 26 +++++--- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 ++--- arch/powerpc/mm/pgtable.c | 30 ++++++---- arch/powerpc/mm/pgtable_32.c | 45 +++----------- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 14 +++-- arch/powerpc/xmon/xmon.c | 18 +++--- 28 files changed, 213 insertions(+), 184 deletions(-)
arch/powerpc/include/asm/book3s/32/pgtable.h +0 -1
@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_BOOK3S_32_PGTABLE_H#define _ASM_POWERPC_BOOK3S_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/book3s/32/hash.h>
arch/powerpc/include/asm/book3s/64/hash.h +2 -2
@@ -134,9 +134,9 @@ static inline int get_region_id(unsigned long ea)#define hash__pmd_bad (pmd) (pmd_val(pmd) & H_PMD_BAD_BITS)#define hash__pud_bad (pud) (pud_val(pud) & H_PUD_BAD_BITS) -static inline int hash__pgd_bad (pgd_t pgd) +static inline int hash__p4d_bad (p4d_t p4d) { -return (pgd_val (pgd) =0 ); +return (p4d_val (p4d) =0 ); }#ifdef CONFIG_STRICT_KERNEL_RWX extern void hash__mark_rodata_ro (void );
arch/powerpc/include/asm/book3s/64/pgalloc.h +2 -2
@@ -85,9 +85,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)kmem_cache_free (PGT_CACHE (PGD_INDEX_SIZE ), pgd); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,pud_t *pud) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *pgd,pud_t *pud) { - *pgd =__pgd (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); + *pgd =__p4d (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)
arch/powerpc/include/asm/book3s/64/pgtable.h +33 -27
120 lines not on this page · raw message
arch/powerpc/include/asm/book3s/64/radix.h +3 -3
@@ -30,7 +30,7 @@/* Don't have anything in the reserved bits and leaf bits */ #define RADIX_PMD_BAD_BITS 0x60000000000000e0UL#define RADIX_PUD_BAD_BITS 0x60000000000000e0UL -#define RADIX_PGD_BAD_BITS 0x60000000000000e0UL +#define RADIX_P4D_BAD_BITS 0x60000000000000e0UL#define RADIX_PMD_SHIFT (PAGE_SHIFT + RADIX_PTE_INDEX_SIZE)#define RADIX_PUD_SHIFT (RADIX_PMD_SHIFT + RADIX_PMD_INDEX_SIZE)
@@ -227,9 +227,9 @@ static inline int radix__pud_bad(pud_t pud) } -static inline int radix__pgd_bad (pgd_t pgd) +static inline int radix__p4d_bad (p4d_t p4d) { -return !!(pgd_val (pgd) &RADIX_PGD_BAD_BITS ); +return !!(p4d_val (p4d) &RADIX_P4D_BAD_BITS ); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE
arch/powerpc/include/asm/nohash/32/pgtable.h +0 -1
@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_32_PGTABLE_H#define _ASM_POWERPC_NOHASH_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #ifndef __ASSEMBLY__
arch/powerpc/include/asm/nohash/64/pgalloc.h +1 -1
@@ -15,7 +15,7 @@ struct vmemmap_backing { };extern struct vmemmap_backing *vmemmap_list; -#define pgd_populate (MM ,PGD ,PUD ) pgd_set(PGD, (unsigned long)PUD) +#define p4d_populate (MM ,P4D ,PUD ) p4d_set(P4D, (unsigned long)PUD)static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) {
arch/powerpc/include/asm/nohash/64/pgtable-4k.h +16 -16
@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H#define _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> /* * Entries per page directory level. The PTE level must use a 64b record
@@ -45,41 +45,41 @@#define PMD_MASKED_BITS 0/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0 -/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0 +/* Bits to mask out from a P4D to get to the PUD page */ +#define P4D_MASKED_BITS 0/* * 4-level page tables related bits */ -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (pgd_val(pgd) = 0) -#define pgd_present (pgd) (pgd_val(pgd) != 0) -#define pgd_page_vaddr (pgd) (pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (p4d_val(p4d) = 0) +#define p4d_present (p4d) (p4d_val(p4d) != 0) +#define p4d_page_vaddr (p4d) (p4d_val(p4d) & ~P4D_MASKED_BITS)#ifndef __ASSEMBLY__ -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte (pgd_val (pgd)); +return __pte (p4d_val (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd (pte_val (pte)); +return __p4d (pte_val (pte)); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);#endif /* !__ASSEMBLY__ */ -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + \ +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + \ (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)))#define pud_ERROR (e) \
arch/powerpc/include/asm/nohash/64/pgtable.h +3 -3
@@ -175,11 +175,11 @@ static inline pud_t pte_pud(pte_t pte)return __pud (pte_val (pte)); }#define pud_write (pud) pte_write(pud_pte(pud)) -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define p4d_write (pgd) pte_write(p4d_pte(p4d)) -static inline void pgd_set (pgd_t *pgdp,unsigned long val) +static inline void p4d_set (p4d_t *p4dp,unsigned long val) { - *pgdp =__pgd (val); + *p4dp =__p4d (val); }/*
arch/powerpc/include/asm/pgtable.h +3 -3
@@ -139,9 +139,9 @@ static inline bool pud_is_leaf(pud_t pud) }#endif -#ifndef pgd_is_leaf -#define pgd_is_leaf pgd_is_leaf -static inline bool pgd_is_leaf (pgd_t pgd) +#ifndef p4d_is_leaf +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) {return false; }
arch/powerpc/kvm/book3s_64_mmu_radix.c +18 -12
@@ -499,13 +499,14 @@ void kvmppc_free_pgtable_radix(struct kvm *kvm, pgd_t *pgd, unsigned int lpid)unsigned long ig;for (ig =0 ; ig <PTRS_PER_PGD ; ++ig, ++pgd) { +p4d_t *p4d =p4d_offset (pgd,0 );pud_t *pud; -if (!pgd_present (*pgd)) +if (!p4d_present (*p4d))continue ; - pud =pud_offset (pgd,0 ); + pud =pud_offset (p4d,0 );kvmppc_unmap_free_pud (kvm, pud, lpid); -pgd_clear (pgd); +p4d_clear (p4d); } }
@@ -566,6 +567,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, unsigned long *rmapp, struct rmap_nested **n_rmap) { pgd_t *pgd; +p4d_t *p4d;pud_t *pud, *new_pud =NULL ;pmd_t *pmd, *new_pmd =NULL ;pte_t *ptep, *new_ptep =NULL ;
@@ -573,9 +575,11 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Traverse the guest's 2nd-level tree, allocate new levels needed */ pgd = pgtable +pgd_index (gpa); + p4d =p4d_offset (pgd, gpa); + pud =NULL ; -if (pgd_present (*pgd)) - pud =pud_offset (pgd, gpa); +if (p4d_present (*p4d)) + pud =pud_offset (p4d, gpa);else new_pud =pud_alloc_one (kvm->mm, gpa);
@@ -596,13 +600,13 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Now traverse again under the lock and change the tree */ ret = -ENOMEM ; -if (pgd_none (*pgd)) { +if (p4d_none (*p4d)) {if (!new_pud) goto out_unlock; -pgd_populate (kvm->mm, pgd, new_pud); +p4d_populate (kvm->mm, p4d, new_pud); new_pud =NULL ; } - pud =pud_offset (pgd, gpa); + pud =pud_offset (p4d, gpa);if (pud_is_leaf (*pud)) {unsigned long hgpa = gpa &PUD_MASK ;
@@ -1220,6 +1224,7 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,pgd_t *pgt;struct kvm_nested_guest *nested;pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ptep;
@@ -1292,13 +1297,14 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf, } pgdp = pgt +pgd_index (gpa); - pgd =READ_ONCE (*pgdp); -if (!(pgd_val (pgd) & _PAGE_PRESENT)) { - gpa = (gpa &PGDIR_MASK ) +PGDIR_SIZE ; + p4dp =p4d_offset (pgdp, gpa); + p4d =READ_ONCE (*p4dp); +if (!(p4d_val (p4d) & _PAGE_PRESENT)) { + gpa = (gpa &P4D_MASK ) +P4D_SIZE ;continue ; } - pudp =pud_offset (&pgd, gpa); + pudp =pud_offset (&p4d, gpa); pud =READ_ONCE (*pudp);if (!(pud_val (pud) & _PAGE_PRESENT)) { gpa = (gpa &PUD_MASK ) +PUD_SIZE ;
arch/powerpc/lib/code-patching.c +6 -1
@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;
arch/powerpc/mm/book3s32/mmu.c +1 -1
@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }
arch/powerpc/mm/book3s32/tlb.c +2 -2
@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)
@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }
arch/powerpc/mm/book3s64/hash_pgtable.c +3 -1
@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);
arch/powerpc/mm/book3s64/radix_pgtable.c +17 -9
@@ -64,17 +64,19 @@ static int early_map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep; pgdp =pgd_offset_k (ea); -if (pgd_none (*pgdp)) { + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { pudp =early_alloc_pgtable (PUD_TABLE_SIZE , nid, region_start, region_end); -pgd_populate (&init_mm, pgdp, pudp); +p4d_populate (&init_mm, p4dp, pudp); } - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (map_page_size =PUD_SIZE ) { ptep = (pte_t *)pudp; goto set_the_pte;
@@ -114,6 +116,7 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -136,7 +139,8 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa,* boot. */ pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ;if (map_page_size =PUD_SIZE ) {
@@ -173,6 +177,7 @@ void radix__change_memory_range(unsigned long start, unsigned long end, {unsigned long idx;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -185,7 +190,8 @@ void radix__change_memory_range(unsigned long start, unsigned long end,for (idx = start; idx < end; idx +=PAGE_SIZE ) { pgdp =pgd_offset_k (idx); - pudp =pud_alloc (&init_mm, pgdp, idx); + p4dp =p4d_offset (pgdp, idx); + pudp =pud_alloc (&init_mm, p4dp, idx);if (!pudp)continue ;if (pud_is_leaf (*pudp)) {
@@ -847,6 +853,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)unsigned long addr, next;pud_t *pud_base;pgd_t *pgd; +p4d_t *p4d;spin_lock (&init_mm.page_table_lock);
@@ -854,15 +861,16 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end) next =pgd_addr_end (addr, end); pgd =pgd_offset_k (addr); -if (!pgd_present (*pgd)) + p4d =p4d_offset (pgd, addr); +if (!p4d_present (*p4d))continue ; -if (pgd_is_leaf (*pgd)) { -split_kernel_mapping (addr, end,PGDIR_SIZE , (pte_t *)pgd); +if (p4d_is_leaf (*p4d)) { +split_kernel_mapping (addr, end,P4D_SIZE , (pte_t *)p4d);continue ; } - pud_base = (pud_t *)pgd_page_vaddr (*pgd); + pud_base = (pud_t *)p4d_page_vaddr (*p4d);remove_pud_table (pud_base, addr, next); }
arch/powerpc/mm/book3s64/subpage_prot.c +4 -2
@@ -54,15 +54,17 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, int npages) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;spinlock_t *ptl; pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d))return ; - pud =pud_offset (pgd, addr); + pud =pud_offset (p4d, addr);if (pud_none (*pud))return ; pmd =pmd_offset (pud, addr);
arch/powerpc/mm/hugetlbpage.c +16 -12
@@ -119,6 +119,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {pgd_t *pg; +p4d_t *p4;pud_t *pu;pmd_t *pm;hugepd_t *hpdp =NULL ;
@@ -128,20 +129,21 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz addr &= ~(sz-1 ); pg =pgd_offset (mm, addr); + p4 =p4d_offset (pg, addr);#ifdef CONFIG_PPC_BOOK3S_64 if (pshift =PGDIR_SHIFT )/* 16GB huge page */ -return (pte_t *) pg; +return (pte_t *) p4;else if (pshift >PUD_SHIFT ) {/* * We need to use hugepd table */ ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift =PUD_SHIFT )
@@ -166,10 +168,10 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz#else if (pshift >=PGDIR_SHIFT ) { ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift >=PUD_SHIFT ) {
@@ -390,7 +392,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,mm_dec_nr_pmds (tlb->mm); } -static void hugetlb_free_pud_range (struct mmu_gather *tlb,pgd_t *pgd, +static void hugetlb_free_pud_range (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling) {
@@ -400,7 +402,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd, start = addr; do { - pud = pud_offset(pgd, addr); + pud = pud_offset(p4d, addr); next = pud_addr_end(addr, end); if (!is_hugepd(__hugepd(pud_val(*pud)))) { if (pud_none_or_clear_bad(pud))
@@ -435,8 +437,8 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,if (end -1 > ceiling -1 )return ; - pud =pud_offset (pgd, start); -pgd_clear (pgd); + pud =pud_offset (p4d, start); +p4d_clear (p4d);pud_free_tlb (tlb, pud, start);mm_dec_nr_puds (tlb->mm); }
@@ -449,6 +451,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long floor, unsigned long ceiling) {pgd_t *pgd; +p4d_t *p4d;unsigned long next;/*
@@ -471,10 +474,11 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, do { next = pgd_addr_end(addr, end); pgd = pgd_offset(tlb->mm, addr); + p4d = p4d_offset(pgd, addr); if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { - if (pgd_none_or_clear_bad(pgd)) + if (p4d_none_or_clear_bad(p4d)) continue; - hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling); + hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); } else { unsigned long more; /*
@@ -487,7 +491,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,if (more > next) next = more; -free_hugepd_range (tlb, (hugepd_t *)pgd,PGDIR_SHIFT , +free_hugepd_range (tlb, (hugepd_t *)p4d,PGDIR_SHIFT , addr, next, floor, ceiling); } }while (addr = next, addr != end);
arch/powerpc/mm/kasan/kasan_init_32.c +4 -4
@@ -36,7 +36,7 @@ static int __init kasan_init_shadow_page_tables(unsigned long k_start, unsignedunsigned long k_cur, k_next;pte_t *new =NULL ; - pmd =pmd_offset (pud_offset (pgd_offset_k (k_start), k_start), k_start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_start), k_start), k_start), k_start);for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++) { k_next =pgd_addr_end (k_cur, k_end);
@@ -78,7 +78,7 @@ static int __init kasan_init_region(void *start, size_t size) block =memblock_alloc (k_end - k_start,PAGE_SIZE );for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);void *va = block + k_cur - k_start;pte_t pte =pfn_pte (PHYS_PFN (__pa (va)),PAGE_KERNEL );
@@ -102,7 +102,7 @@ static void __init kasan_remap_early_shadow_ro(void)kasan_populate_pte (kasan_early_shadow_pte, prot);for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);pte_t *ptep =pte_offset_kernel (pmd, k_cur);if ((pte_val (*ptep) &PTE_RPN_MASK ) != pa)
@@ -201,7 +201,7 @@ void __init kasan_early_init(void)unsigned long addr =KASAN_SHADOW_START ;unsigned long end =KASAN_SHADOW_END ;unsigned long next; -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (addr), addr), addr); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (addr), addr), addr), addr);BUILD_BUG_ON (KASAN_SHADOW_START & ~PGDIR_MASK );
arch/powerpc/mm/mem.c +2 -2
@@ -69,8 +69,8 @@ EXPORT_SYMBOL(kmap_prot);static inline pte_t *virt_to_kpte (unsigned long vaddr) { -return pte_offset_kernel (pmd_offset (pud_offset (pgd_offset_k (vaddr), - vaddr), vaddr), vaddr); +return pte_offset_kernel (pmd_offset (pud_offset (p4d_offset (pgd_offset_k (vaddr), + vaddr), vaddr), vaddr), vaddr); }#endif
arch/powerpc/mm/nohash/40x.c +2 -2
@@ -104,7 +104,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_16M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val);
@@ -119,7 +119,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_4M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp =__pmd (val); v +=LARGE_PAGE_SIZE_4M ;
arch/powerpc/mm/nohash/book3e_pgtable.c +8 -7
@@ -73,6 +73,7 @@ static void __init *early_alloc_pgtable(unsigned long size)int __ref map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -80,7 +81,8 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);
@@ -91,13 +93,12 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)return -ENOMEM ; }else { pgdp =pgd_offset_k (ea); -#ifndef __PAGETABLE_PUD_FOLDED -if (pgd_none (*pgdp)) { - pudp =early_alloc_pgtable (PUD_TABLE_SIZE ); -pgd_populate (&init_mm, pgdp, pudp); + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { + pmdp =early_alloc_pgtable (PMD_TABLE_SIZE ); +p4d_populate (&init_mm, p4dp, pmdp); } -#endif /* !__PAGETABLE_PUD_FOLDED */ - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (pud_none (*pudp)) { pmdp =early_alloc_pgtable (PMD_TABLE_SIZE );pud_populate (&init_mm, pudp, pmdp);
arch/powerpc/mm/pgtable.c +19 -11
@@ -265,6 +265,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,void assert_pte_locked (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;
@@ -272,7 +273,9 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr)return ; pgd = mm->pgd +pgd_index (addr);BUG_ON (pgd_none (*pgd)); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +BUG_ON (p4d_none (*p4d)); + pud =pud_offset (p4d, addr);BUG_ON (pud_none (*pud)); pmd =pmd_offset (pud, addr);/*
@@ -312,12 +315,13 @@ EXPORT_SYMBOL_GPL(vmalloc_to_phys);pte_t *__find_linux_pte (pgd_t *pgdir,unsigned long ea,bool *is_thp,unsigned *hpage_shift) { -pgd_t pgd, *pgdp; +pgd_t *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ret_pte;hugepd_t *hpdp =NULL ; -unsigned pdshift =PGDIR_SHIFT ; +unsigned pdshift;if (hpage_shift) *hpage_shift =0 ;
@@ -325,24 +329,28 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,if (is_thp) *is_thp = false; - pgdp = pgdir +pgd_index (ea); - pgd =READ_ONCE (*pgdp);/* * Always operate on the local stack value. This make sure the * value don't get updated by a parallel THP split/collapse, * page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. +* Top level is an exception because it is folded into p4d. */ -if (pgd_none (pgd)) + pgdp = pgdir +pgd_index (ea); + p4dp =p4d_offset (pgdp, ea); + p4d =READ_ONCE (*p4dp); + pdshift =P4D_SHIFT ; + +if (p4d_none (p4d))return NULL ; -if (pgd_is_leaf (pgd)) { - ret_pte = (pte_t *)pgdp; +if (p4d_is_leaf (p4d)) { + ret_pte = (pte_t *)p4dp; goto out; } -if (is_hugepd (__hugepd (pgd_val (pgd)))) { - hpdp = (hugepd_t *)&pgd; +if (is_hugepd (__hugepd (p4d_val (p4d)))) { + hpdp = (hugepd_t *)&p4d; goto out_huge; }
@@ -352,7 +360,7 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,* irq disabled */ pdshift =PUD_SHIFT ; - pudp =pud_offset (&pgd, ea); + pudp =pud_offset (&p4d, ea); pud =READ_ONCE (*pudp);if (pud_none (pud))
arch/powerpc/mm/pgtable_32.c +8 -37
68 lines not on this page · raw message
arch/powerpc/mm/pgtable_64.c +5 -5
@@ -101,13 +101,13 @@ EXPORT_SYMBOL(__pte_frag_size_shift);#ifndef __PAGETABLE_PUD_FOLDED/* 4 level page table */ -struct page *pgd_page (pgd_t pgd) +struct page *p4d_page (p4d_t p4d) { -if (pgd_is_leaf (pgd)) { -VM_WARN_ON (!pgd_huge (pgd)); -return pte_page (pgd_pte (pgd)); +if (p4d_is_leaf (p4d)) { +VM_WARN_ON (!p4d_huge (p4d)); +return pte_page (p4d_pte (p4d)); } -return virt_to_page (pgd_page_vaddr (pgd)); +return virt_to_page (p4d_page_vaddr (p4d)); }#endif
arch/powerpc/mm/ptdump/hashpagetable.c +17 -3
@@ -417,9 +417,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;
@@ -431,6 +431,20 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); + } +} +static void walk_pagetables (struct pg_state *st) {pgd_t *pgd =pgd_offset_k (0UL );
@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr); } }
arch/powerpc/mm/ptdump/ptdump.c +8 -6
@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;
@@ -304,11 +304,13 @@ static void walk_pagetables(struct pg_state *st)* the hash pagetable. */ for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) { -if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd)) -/* pgd exists */ -walk_pud (st, pgd, addr); +p4d_t *p4d =p4d_offset (pgd,0 ); + +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr);else -note_page (st, addr,1 ,pgd_val (*pgd)); +note_page (st, addr,1 ,p4d_val (*p4d)); } }
arch/powerpc/xmon/xmon.c +10 -8
@@ -3130,6 +3130,7 @@ static void show_pte(unsigned long addr)struct task_struct *tsk =NULL ;struct mm_struct *mm;pgd_t *pgdp, *pgdir; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;
@@ -3161,20 +3162,21 @@ static void show_pte(unsigned long addr) pgdir =pgd_offset (mm,0 ); } -if (pgd_none (*pgdp)) { -printf ("no linux page table for address\n" ); + p4dp =p4d_offset (pgdp, addr); + +if (p4d_none (*p4dp)) { +printf ("No valid P4D\n" );return ; } -printf ("pgd @ 0x%px\n" , pgdir); - -if (pgd_is_leaf (*pgdp)) { -format_pte (pgdp,pgd_val (*pgdp)); +if (p4d_is_leaf (*p4dp)) { +format_pte (p4dp,p4d_val (*p4dp));return ; } -printf ("pgdp @ 0x%px = 0x%016lx\n" , pgdp,pgd_val (*pgdp)); - pudp =pud_offset (pgdp, addr); +printf ("p4dp @ 0x%px = 0x%016lx\n" , p4dp,p4d_val (*p4dp)); + + pudp =pud_offset (p4dp, addr);if (pud_none (*pudp)) {printf ("No valid PUD\n" );
Signature · 1 line
2.24.0
Re: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Le 26/02/2020 à 10:13, Mike Rapoport a écrit :
Quoted 16 lines · Mike Rapoport
Quoted 15 lines · Christophe Leroy
Quoted 14 lines
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The additionnal walks should be added through another patch the day powerpc need them.
Ok, I'll update the patch to reduce walking the p4d.
Here's what I have with more direct acceses from pgd to pud.
I went quickly through. This looks promising.
Do we need the walk_p4d() in arch/powerpc/mm/ptdump/hashpagetable.c ? Can't we just do
@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st)
addr = KERN_VIRT_START + i * PGDIR_SIZE; if (!pgd_none(*pgd))
/* pgd exists */
- walk_pud(st, pgd, addr);
+ walk_pud(st, p4d_offset(pgd, addr), addr);
}
}
Also, I think the removal of get_pteptr() should be a patch by itself.
You could include my patches in your series. See
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series2102
Christophe
Quoted 1,194 lines
From
6c59a86ce8394fb6100e9b6ced2e346981fb0ce9Mon Sep 17 00:00:00 2001
From: Mike Rapoport <[email protected]>
Date: Sun, 24 Nov 2019 15:38:00 +0200
Subject: [PATCH v3] powerpc: add support for folded p4d page tablesImplement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.Signed-off-by Mike Rapoport <[email protected]>---
v3:
* reduce amount of added p4d walks
* killpgtable_32::get_pteptrand traverse page table inpgtable_32::__change_page_attr_noflusharch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 60 ++++++++++--------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 6 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 30 ++++++---- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 26 +++++--- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 ++--- arch/powerpc/mm/pgtable.c | 30 ++++++---- arch/powerpc/mm/pgtable_32.c | 45 +++----------- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 14 +++-- arch/powerpc/xmon/xmon.c | 18 +++--- 28 files changed, 213 insertions(+), 184 deletions(-)arch/powerpc/include/asm/book3s/32/pgtable.h +0
-1@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_BOOK3S_32_PGTABLE_H#define _ASM_POWERPC_BOOK3S_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/book3s/32/hash.h> arch/powerpc/include/asm/book3s/64/hash.h +2
-2@@ -134,9 +134,9 @@ static inline int get_region_id(unsigned long ea)#define hash__pmd_bad (pmd) (pmd_val(pmd) & H_PMD_BAD_BITS)#define hash__pud_bad (pud) (pud_val(pud) & H_PUD_BAD_BITS) -static inline int hash__pgd_bad (pgd_t pgd) +static inline int hash__p4d_bad (p4d_t p4d) { -return (pgd_val (pgd) =0 ); +return (p4d_val (p4d) =0 ); }#ifdef CONFIG_STRICT_KERNEL_RWX extern void hash__mark_rodata_ro (void );arch/powerpc/include/asm/book3s/64/pgalloc.h +2
-2@@ -85,9 +85,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)kmem_cache_free (PGT_CACHE (PGD_INDEX_SIZE ), pgd); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,pud_t *pud) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *pgd,pud_t *pud) { - *pgd =__pgd (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); + *pgd =__p4d (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)arch/powerpc/include/asm/book3s/64/pgtable.h +33
-27@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_H_#define _ASM_POWERPC_BOOK3S_64_PGTABLE_H_ -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #ifndef __ASSEMBLY__#include <linux/mmdebug.h> @@ -251,7 +251,7 @@ extern unsigned long __pmd_frag_size_shift;/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0xc0000000000000ffUL/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0xc0000000000000ffUL +#define P4D_MASKED_BITS 0xc0000000000000ffUL/* * Used as an indicator for rcu callback functions @@ -949,54 +949,60 @@ static inline bool pud_access_permitted(pud_t pud, bool write)return pte_access_permitted (pud_pte (pud), write); } -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define __p4d_raw (x) ((p4d_t) { __pgd_raw(x) }) +static inline __be64 p4d_raw (p4d_t x) +{ +return pgd_raw (x.pgd); +} + +#define p4d_write (p4d) pte_write(p4d_pte(p4d)) -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline int pgd_none (pgd_t pgd) +static inline int p4d_none (p4d_t p4d) { -return !pgd_raw (pgd); +return !p4d_raw (p4d); } -static inline int pgd_present (pgd_t pgd) +static inline int p4d_present (p4d_t p4d) { -return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PRESENT)); +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PRESENT)); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte_raw (pgd_raw (pgd)); +return __pte_raw (p4d_raw (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd_raw (pte_raw (pte)); +return __p4d_raw (pte_raw (pte)); } -static inline int pgd_bad (pgd_t pgd) +static inline int p4d_bad (p4d_t p4d) {if (radix_enabled ()) -return radix__pgd_bad (pgd); -return hash__pgd_bad (pgd); +return radix__p4d_bad (p4d); +return hash__p4d_bad (p4d); } -#define pgd_access_permitted pgd_access_permitted -static inline bool pgd_access_permitted (pgd_t pgd,bool write) +#define p4d_access_permitted p4d_access_permitted +static inline bool p4d_access_permitted (p4d_t p4d,bool write) { -return pte_access_permitted (pgd_pte (pgd), write); +return pte_access_permitted (p4d_pte (p4d), write); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);/* Pointers in the page table tree are physical addresses */ #define __pgtable_ptr_val (ptr) __pa(ptr)#define pmd_page_vaddr (pmd) __va(pmd_val(pmd) & ~PMD_MASKED_BITS)#define pud_page_vaddr (pud) __va(pud_val(pud) & ~PUD_MASKED_BITS) -#define pgd_page_vaddr (pgd) __va(pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_page_vaddr (p4d) __va(p4d_val(p4d) & ~P4D_MASKED_BITS)#define pgd_index (address) (((address) >> (PGDIR_SHIFT)) & (PTRS_PER_PGD - 1))#define pud_index (address) (((address) >> (PUD_SHIFT)) & (PTRS_PER_PUD - 1))@@ -1010,8 +1016,8 @@ extern struct page *pgd_page(pgd_t pgd);#define pgd_offset (mm, address) ((mm)->pgd + pgd_index(address)) -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + pud_index(addr)) +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + pud_index(addr))#define pmd_offset (pudp,addr) \ (((pmd_t *) pud_page_vaddr(*(pudp))) + pmd_index(addr))#define pte_offset_kernel (dir,addr) \@@ -1368,11 +1374,11 @@ static inline bool pud_is_leaf(pud_t pud) return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); } -#define pgd_is_leaf pgd_is_leaf -#define pgd_leaf pgd_is_leaf -static inline bool pgd_is_leaf (pgd_t pgd) +#define p4d_is_leaf p4d_is_leaf +#define p4d_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) { -return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PTE)); +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PTE)); }#endif /* __ASSEMBLY__ */ arch/powerpc/include/asm/book3s/64/radix.h +3
-3@@ -30,7 +30,7 @@/* Don't have anything in the reserved bits and leaf bits */ #define RADIX_PMD_BAD_BITS 0x60000000000000e0UL#define RADIX_PUD_BAD_BITS 0x60000000000000e0UL -#define RADIX_PGD_BAD_BITS 0x60000000000000e0UL +#define RADIX_P4D_BAD_BITS 0x60000000000000e0UL#define RADIX_PMD_SHIFT (PAGE_SHIFT + RADIX_PTE_INDEX_SIZE)#define RADIX_PUD_SHIFT (RADIX_PMD_SHIFT + RADIX_PMD_INDEX_SIZE)@@ -227,9 +227,9 @@ static inline int radix__pud_bad(pud_t pud) } -static inline int radix__pgd_bad (pgd_t pgd) +static inline int radix__p4d_bad (p4d_t p4d) { -return !!(pgd_val (pgd) &RADIX_PGD_BAD_BITS ); +return !!(p4d_val (p4d) &RADIX_P4D_BAD_BITS ); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE arch/powerpc/include/asm/nohash/32/pgtable.h +0
-1@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_32_PGTABLE_H#define _ASM_POWERPC_NOHASH_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #ifndef __ASSEMBLY__arch/powerpc/include/asm/nohash/64/pgalloc.h +1
-1@@ -15,7 +15,7 @@ struct vmemmap_backing { };extern struct vmemmap_backing *vmemmap_list; -#define pgd_populate (MM ,PGD ,PUD ) pgd_set(PGD, (unsigned long)PUD) +#define p4d_populate (MM ,P4D ,PUD ) p4d_set(P4D, (unsigned long)PUD)static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) {arch/powerpc/include/asm/nohash/64/pgtable-4k.h +16
-16@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H#define _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> /* * Entries per page directory level. The PTE level must use a 64b record @@ -45,41 +45,41 @@#define PMD_MASKED_BITS 0/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0 -/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0 +/* Bits to mask out from a P4D to get to the PUD page */ +#define P4D_MASKED_BITS 0/* * 4-level page tables related bits */ -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (pgd_val(pgd) = 0) -#define pgd_present (pgd) (pgd_val(pgd) != 0) -#define pgd_page_vaddr (pgd) (pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (p4d_val(p4d) = 0) +#define p4d_present (p4d) (p4d_val(p4d) != 0) +#define p4d_page_vaddr (p4d) (p4d_val(p4d) & ~P4D_MASKED_BITS)#ifndef __ASSEMBLY__ -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte (pgd_val (pgd)); +return __pte (p4d_val (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd (pte_val (pte)); +return __p4d (pte_val (pte)); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);#endif /* !__ASSEMBLY__ */ -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + \ +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + \ (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)))#define pud_ERROR (e) \arch/powerpc/include/asm/nohash/64/pgtable.h +3
-3@@ -175,11 +175,11 @@ static inline pud_t pte_pud(pte_t pte)return __pud (pte_val (pte)); }#define pud_write (pud) pte_write(pud_pte(pud)) -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define p4d_write (pgd) pte_write(p4d_pte(p4d)) -static inline void pgd_set (pgd_t *pgdp,unsigned long val) +static inline void p4d_set (p4d_t *p4dp,unsigned long val) { - *pgdp =__pgd (val); + *p4dp =__p4d (val); }/* arch/powerpc/include/asm/pgtable.h +3
-3@@ -139,9 +139,9 @@ static inline bool pud_is_leaf(pud_t pud) }#endif -#ifndef pgd_is_leaf -#define pgd_is_leaf pgd_is_leaf -static inline bool pgd_is_leaf (pgd_t pgd) +#ifndef p4d_is_leaf +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) {return false; }arch/powerpc/kvm/book3s_64_mmu_radix.c +18
-12@@ -499,13 +499,14 @@ void kvmppc_free_pgtable_radix(struct kvm *kvm, pgd_t *pgd, unsigned int lpid)unsigned long ig;for (ig =0 ; ig <PTRS_PER_PGD ; ++ig, ++pgd) { +p4d_t *p4d =p4d_offset (pgd,0 );pud_t *pud; -if (!pgd_present (*pgd)) +if (!p4d_present (*p4d))continue ; - pud =pud_offset (pgd,0 ); + pud =pud_offset (p4d,0 );kvmppc_unmap_free_pud (kvm, pud, lpid); -pgd_clear (pgd); +p4d_clear (p4d); } }@@ -566,6 +567,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, unsigned long *rmapp, struct rmap_nested **n_rmap) { pgd_t *pgd; +p4d_t *p4d;pud_t *pud, *new_pud =NULL ;pmd_t *pmd, *new_pmd =NULL ;pte_t *ptep, *new_ptep =NULL ;@@ -573,9 +575,11 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Traverse the guest's 2nd-level tree, allocate new levels needed */ pgd = pgtable +pgd_index (gpa); + p4d =p4d_offset (pgd, gpa); + pud =NULL ; -if (pgd_present (*pgd)) - pud =pud_offset (pgd, gpa); +if (p4d_present (*p4d)) + pud =pud_offset (p4d, gpa);else new_pud =pud_alloc_one (kvm->mm, gpa);@@ -596,13 +600,13 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Now traverse again under the lock and change the tree */ ret = -ENOMEM ; -if (pgd_none (*pgd)) { +if (p4d_none (*p4d)) {if (!new_pud) goto out_unlock; -pgd_populate (kvm->mm, pgd, new_pud); +p4d_populate (kvm->mm, p4d, new_pud); new_pud =NULL ; } - pud =pud_offset (pgd, gpa); + pud =pud_offset (p4d, gpa);if (pud_is_leaf (*pud)) {unsigned long hgpa = gpa &PUD_MASK ;@@ -1220,6 +1224,7 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,pgd_t *pgt;struct kvm_nested_guest *nested;pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ptep;@@ -1292,13 +1297,14 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf, } pgdp = pgt +pgd_index (gpa); - pgd =READ_ONCE (*pgdp); -if (!(pgd_val (pgd) & _PAGE_PRESENT)) { - gpa = (gpa &PGDIR_MASK ) +PGDIR_SIZE ; + p4dp =p4d_offset (pgdp, gpa); + p4d =READ_ONCE (*p4dp); +if (!(p4d_val (p4d) & _PAGE_PRESENT)) { + gpa = (gpa &P4D_MASK ) +P4D_SIZE ;continue ; } - pudp =pud_offset (&pgd, gpa); + pudp =pud_offset (&p4d, gpa); pud =READ_ONCE (*pudp);if (!(pud_val (pud) & _PAGE_PRESENT)) { gpa = (gpa &PUD_MASK ) +PUD_SIZE ;arch/powerpc/lib/code-patching.c +6
-1@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;arch/powerpc/mm/book3s32/mmu.c +1
-1@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }arch/powerpc/mm/book3s32/tlb.c +2
-2@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }arch/powerpc/mm/book3s64/hash_pgtable.c +3
-1@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);arch/powerpc/mm/book3s64/radix_pgtable.c +17
-9@@ -64,17 +64,19 @@ static int early_map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep; pgdp =pgd_offset_k (ea); -if (pgd_none (*pgdp)) { + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { pudp =early_alloc_pgtable (PUD_TABLE_SIZE , nid, region_start, region_end); -pgd_populate (&init_mm, pgdp, pudp); +p4d_populate (&init_mm, p4dp, pudp); } - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (map_page_size =PUD_SIZE ) { ptep = (pte_t *)pudp; goto set_the_pte;@@ -114,6 +116,7 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -136,7 +139,8 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa,* boot. */ pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ;if (map_page_size =PUD_SIZE ) {@@ -173,6 +177,7 @@ void radix__change_memory_range(unsigned long start, unsigned long end, {unsigned long idx;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -185,7 +190,8 @@ void radix__change_memory_range(unsigned long start, unsigned long end,for (idx = start; idx < end; idx +=PAGE_SIZE ) { pgdp =pgd_offset_k (idx); - pudp =pud_alloc (&init_mm, pgdp, idx); + p4dp =p4d_offset (pgdp, idx); + pudp =pud_alloc (&init_mm, p4dp, idx);if (!pudp)continue ;if (pud_is_leaf (*pudp)) {@@ -847,6 +853,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)unsigned long addr, next;pud_t *pud_base;pgd_t *pgd; +p4d_t *p4d;spin_lock (&init_mm.page_table_lock);@@ -854,15 +861,16 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end) next =pgd_addr_end (addr, end); pgd =pgd_offset_k (addr); -if (!pgd_present (*pgd)) + p4d =p4d_offset (pgd, addr); +if (!p4d_present (*p4d))continue ; -if (pgd_is_leaf (*pgd)) { -split_kernel_mapping (addr, end,PGDIR_SIZE , (pte_t *)pgd); +if (p4d_is_leaf (*p4d)) { +split_kernel_mapping (addr, end,P4D_SIZE , (pte_t *)p4d);continue ; } - pud_base = (pud_t *)pgd_page_vaddr (*pgd); + pud_base = (pud_t *)p4d_page_vaddr (*p4d);remove_pud_table (pud_base, addr, next); }arch/powerpc/mm/book3s64/subpage_prot.c +4
-2@@ -54,15 +54,17 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, int npages) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;spinlock_t *ptl; pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d))return ; - pud =pud_offset (pgd, addr); + pud =pud_offset (p4d, addr);if (pud_none (*pud))return ; pmd =pmd_offset (pud, addr);arch/powerpc/mm/hugetlbpage.c +16
-12@@ -119,6 +119,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {pgd_t *pg; +p4d_t *p4;pud_t *pu;pmd_t *pm;hugepd_t *hpdp =NULL ;@@ -128,20 +129,21 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz addr &= ~(sz-1 ); pg =pgd_offset (mm, addr); + p4 =p4d_offset (pg, addr);#ifdef CONFIG_PPC_BOOK3S_64 if (pshift =PGDIR_SHIFT )/* 16GB huge page */ -return (pte_t *) pg; +return (pte_t *) p4;else if (pshift >PUD_SHIFT ) {/* * We need to use hugepd table */ ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift =PUD_SHIFT )@@ -166,10 +168,10 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz#else if (pshift >=PGDIR_SHIFT ) { ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift >=PUD_SHIFT ) {@@ -390,7 +392,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,mm_dec_nr_pmds (tlb->mm); } -static void hugetlb_free_pud_range (struct mmu_gather *tlb,pgd_t *pgd, +static void hugetlb_free_pud_range (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling) {@@ -400,7 +402,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd, start = addr; do { - pud = pud_offset(pgd, addr); + pud = pud_offset(p4d, addr); next = pud_addr_end(addr, end); if (!is_hugepd(__hugepd(pud_val(*pud)))) { if (pud_none_or_clear_bad(pud))@@ -435,8 +437,8 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,if (end -1 > ceiling -1 )return ; - pud =pud_offset (pgd, start); -pgd_clear (pgd); + pud =pud_offset (p4d, start); +p4d_clear (p4d);pud_free_tlb (tlb, pud, start);mm_dec_nr_puds (tlb->mm); }@@ -449,6 +451,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long floor, unsigned long ceiling) {pgd_t *pgd; +p4d_t *p4d;unsigned long next;/* @@ -471,10 +474,11 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, do { next = pgd_addr_end(addr, end); pgd = pgd_offset(tlb->mm, addr); + p4d = p4d_offset(pgd, addr); if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { - if (pgd_none_or_clear_bad(pgd)) + if (p4d_none_or_clear_bad(p4d)) continue; - hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling); + hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); } else { unsigned long more; /*@@ -487,7 +491,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,if (more > next) next = more; -free_hugepd_range (tlb, (hugepd_t *)pgd,PGDIR_SHIFT , +free_hugepd_range (tlb, (hugepd_t *)p4d,PGDIR_SHIFT , addr, next, floor, ceiling); } }while (addr = next, addr != end);arch/powerpc/mm/kasan/kasan_init_32.c +4
-4@@ -36,7 +36,7 @@ static int __init kasan_init_shadow_page_tables(unsigned long k_start, unsignedunsigned long k_cur, k_next;pte_t *new =NULL ; - pmd =pmd_offset (pud_offset (pgd_offset_k (k_start), k_start), k_start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_start), k_start), k_start), k_start);for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++) { k_next =pgd_addr_end (k_cur, k_end);@@ -78,7 +78,7 @@ static int __init kasan_init_region(void *start, size_t size) block =memblock_alloc (k_end - k_start,PAGE_SIZE );for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);void *va = block + k_cur - k_start;pte_t pte =pfn_pte (PHYS_PFN (__pa (va)),PAGE_KERNEL );@@ -102,7 +102,7 @@ static void __init kasan_remap_early_shadow_ro(void)kasan_populate_pte (kasan_early_shadow_pte, prot);for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);pte_t *ptep =pte_offset_kernel (pmd, k_cur);if ((pte_val (*ptep) &PTE_RPN_MASK ) != pa)@@ -201,7 +201,7 @@ void __init kasan_early_init(void)unsigned long addr =KASAN_SHADOW_START ;unsigned long end =KASAN_SHADOW_END ;unsigned long next; -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (addr), addr), addr); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (addr), addr), addr), addr);BUILD_BUG_ON (KASAN_SHADOW_START & ~PGDIR_MASK );arch/powerpc/mm/mem.c +2
-2@@ -69,8 +69,8 @@ EXPORT_SYMBOL(kmap_prot);static inline pte_t *virt_to_kpte (unsigned long vaddr) { -return pte_offset_kernel (pmd_offset (pud_offset (pgd_offset_k (vaddr), - vaddr), vaddr), vaddr); +return pte_offset_kernel (pmd_offset (pud_offset (p4d_offset (pgd_offset_k (vaddr), + vaddr), vaddr), vaddr), vaddr); }#endif arch/powerpc/mm/nohash/40x.c +2
-2@@ -104,7 +104,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_16M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val);@@ -119,7 +119,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_4M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp =__pmd (val); v +=LARGE_PAGE_SIZE_4M ;arch/powerpc/mm/nohash/book3e_pgtable.c +8
-7@@ -73,6 +73,7 @@ static void __init *early_alloc_pgtable(unsigned long size)int __ref map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -80,7 +81,8 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);@@ -91,13 +93,12 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)return -ENOMEM ; }else { pgdp =pgd_offset_k (ea); -#ifndef __PAGETABLE_PUD_FOLDED -if (pgd_none (*pgdp)) { - pudp =early_alloc_pgtable (PUD_TABLE_SIZE ); -pgd_populate (&init_mm, pgdp, pudp); + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { + pmdp =early_alloc_pgtable (PMD_TABLE_SIZE ); +p4d_populate (&init_mm, p4dp, pmdp); } -#endif /* !__PAGETABLE_PUD_FOLDED */ - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (pud_none (*pudp)) { pmdp =early_alloc_pgtable (PMD_TABLE_SIZE );pud_populate (&init_mm, pudp, pmdp);arch/powerpc/mm/pgtable.c +19
-11@@ -265,6 +265,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,void assert_pte_locked (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;@@ -272,7 +273,9 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr)return ; pgd = mm->pgd +pgd_index (addr);BUG_ON (pgd_none (*pgd)); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +BUG_ON (p4d_none (*p4d)); + pud =pud_offset (p4d, addr);BUG_ON (pud_none (*pud)); pmd =pmd_offset (pud, addr);/* @@ -312,12 +315,13 @@ EXPORT_SYMBOL_GPL(vmalloc_to_phys);pte_t *__find_linux_pte (pgd_t *pgdir,unsigned long ea,bool *is_thp,unsigned *hpage_shift) { -pgd_t pgd, *pgdp; +pgd_t *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ret_pte;hugepd_t *hpdp =NULL ; -unsigned pdshift =PGDIR_SHIFT ; +unsigned pdshift;if (hpage_shift) *hpage_shift =0 ;@@ -325,24 +329,28 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,if (is_thp) *is_thp = false; - pgdp = pgdir +pgd_index (ea); - pgd =READ_ONCE (*pgdp);/* * Always operate on the local stack value. This make sure the * value don't get updated by a parallel THP split/collapse, * page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. +* Top level is an exception because it is folded into p4d. */ -if (pgd_none (pgd)) + pgdp = pgdir +pgd_index (ea); + p4dp =p4d_offset (pgdp, ea); + p4d =READ_ONCE (*p4dp); + pdshift =P4D_SHIFT ; + +if (p4d_none (p4d))return NULL ; -if (pgd_is_leaf (pgd)) { - ret_pte = (pte_t *)pgdp; +if (p4d_is_leaf (p4d)) { + ret_pte = (pte_t *)p4dp; goto out; } -if (is_hugepd (__hugepd (pgd_val (pgd)))) { - hpdp = (hugepd_t *)&pgd; +if (is_hugepd (__hugepd (p4d_val (p4d)))) { + hpdp = (hugepd_t *)&p4d; goto out_huge; }@@ -352,7 +360,7 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,* irq disabled */ pdshift =PUD_SHIFT ; - pudp =pud_offset (&pgd, ea); + pudp =pud_offset (&p4d, ea); pud =READ_ONCE (*pudp);if (pud_none (pud))arch/powerpc/mm/pgtable_32.c +8
-37@@ -63,7 +63,7 @@ int __ref map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot)int err = -ENOMEM ;/* Use upper 10 bits of VA to index the first level map */ - pd =pmd_offset (pud_offset (pgd_offset_k (va), va), va); + pd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va);/* Use middle 10 bits of VA to index the second-level map */ if (likely (slab_is_available ())) pg =pte_alloc_kernel (pd, va);@@ -121,53 +121,24 @@ void __init mapin_ram(void) } } -/* Scan the real Linux page tables and return a PTE pointer for -* a virtual address in a context. -* Returns true (1) if PTE was found, zero otherwise. The pointer to -* the PTE pointer is unmodified if PTE is not found. -*/ -static int -get_pteptr (struct mm_struct *mm,unsigned long addr,pte_t **ptep,pmd_t **pmdp) -{ -pgd_t *pgd; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; -int retval =0 ; - - pgd =pgd_offset (mm, addr &PAGE_MASK ); -if (pgd) { - pud =pud_offset (pgd, addr &PAGE_MASK ); -if (pud &&pud_present (*pud)) { - pmd =pmd_offset (pud, addr &PAGE_MASK ); -if (pmd_present (*pmd)) { - pte =pte_offset_map (pmd, addr &PAGE_MASK ); -if (pte) { - retval =1 ; - *ptep = pte; -if (pmdp) - *pmdp = pmd; -/* XXX caller needs to do pte_unmap, yuck */ - } - } - } - } -return (retval); -} -static int __change_page_attr_noflush (struct page *page,pgprot_t prot) {pte_t *kpte;pmd_t *kpmd; -unsigned long address; +unsigned long address, va;BUG_ON (PageHighMem (page)); address = (unsigned long )page_address (page); + va = address &PAGE_MASK ;if (v_block_mapped (address))return 0 ; -if (!get_pteptr (&init_mm, address, &kpte, &kpmd)) + + kpmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va); +if (!pmd_present (*kpmd))return -EINVAL ; + + kpte =pte_offset_map (kpmd, va);__set_pte_at (&init_mm, address, kpte,mk_pte (page, prot),0 );pte_unmap (kpte);arch/powerpc/mm/pgtable_64.c +5
-5@@ -101,13 +101,13 @@ EXPORT_SYMBOL(__pte_frag_size_shift);#ifndef __PAGETABLE_PUD_FOLDED/* 4 level page table */ -struct page *pgd_page (pgd_t pgd) +struct page *p4d_page (p4d_t p4d) { -if (pgd_is_leaf (pgd)) { -VM_WARN_ON (!pgd_huge (pgd)); -return pte_page (pgd_pte (pgd)); +if (p4d_is_leaf (p4d)) { +VM_WARN_ON (!p4d_huge (p4d)); +return pte_page (p4d_pte (p4d)); } -return virt_to_page (pgd_page_vaddr (pgd)); +return virt_to_page (p4d_page_vaddr (p4d)); }#endif arch/powerpc/mm/ptdump/hashpagetable.c +17
-3@@ -417,9 +417,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -431,6 +431,20 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); + } +} +static void walk_pagetables (struct pg_state *st) {pgd_t *pgd =pgd_offset_k (0UL );@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr); } }arch/powerpc/mm/ptdump/ptdump.c +8
-6@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -304,11 +304,13 @@ static void walk_pagetables(struct pg_state *st)* the hash pagetable. */ for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) { -if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd)) -/* pgd exists */ -walk_pud (st, pgd, addr); +p4d_t *p4d =p4d_offset (pgd,0 ); + +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr);else -note_page (st, addr,1 ,pgd_val (*pgd)); +note_page (st, addr,1 ,p4d_val (*p4d)); } }arch/powerpc/xmon/xmon.c +10
-8@@ -3130,6 +3130,7 @@ static void show_pte(unsigned long addr)struct task_struct *tsk =NULL ;struct mm_struct *mm;pgd_t *pgdp, *pgdir; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -3161,20 +3162,21 @@ static void show_pte(unsigned long addr) pgdir =pgd_offset (mm,0 ); } -if (pgd_none (*pgdp)) { -printf ("no linux page table for address\n" ); + p4dp =p4d_offset (pgdp, addr); + +if (p4d_none (*p4dp)) { +printf ("No valid P4D\n" );return ; } -printf ("pgd @ 0x%px\n" , pgdir); - -if (pgd_is_leaf (*pgdp)) { -format_pte (pgdp,pgd_val (*pgdp)); +if (p4d_is_leaf (*p4dp)) { +format_pte (p4dp,p4d_val (*p4dp));return ; } -printf ("pgdp @ 0x%px = 0x%016lx\n" , pgdp,pgd_val (*pgdp)); - pudp =pud_offset (pgdp, addr); +printf ("p4dp @ 0x%px = 0x%016lx\n" , p4dp,p4d_val (*p4dp)); + + pudp =pud_offset (p4dp, addr);if (pud_none (*pudp)) {printf ("No valid PUD\n" );
Mike Rapoport
reviewerRe: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Quoted 32 lines · Christophe Leroy
Le 26/02/2020 à 10:13, Mike Rapoport a écrit :
Quoted 16 lines · Mike Rapoport
Quoted 15 lines · Christophe Leroy
Quoted 14 lines
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The additionnal walks should be added through another patch the day powerpc need them.
Ok, I'll update the patch to reduce walking the p4d.
Here's what I have with more direct acceses from pgd to pud.
I went quickly through. This looks promising.
Do we need the
walk_p4d()inarch/powerpc/mm/ptdump/hashpagetable.c? Can't we just do
arch/powerpc/mm/ptdump/hashpagetable.c static void walk_pagetables(struct pg_state *st) · line 462
Quoted 5 lines
arch/powerpc/mm/ptdump/hashpagetable.c +1
-1@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_pud (st,p4d_offset (pgd, addr), addr);
We can do
addr = KERN_VIRT_START + i * PGDIR_SIZE; p4d = p4d_offset(pgd, addr); if (!p4d_none(*pgd)) walk_pud()
But I don't think this is really essential. Again, we are trading off code consistency vs line count. I don't think line count is that important.
Quoted 1,096 lines
(quoted diff) +0
-0} }Also, I think the removal of
get_pteptr()should be a patch by itself.You could include my patches in your series. See
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series2102Christophe
Quoted 1,080 lines
From
6c59a86ce8394fb6100e9b6ced2e346981fb0ce9Mon Sep 17 00:00:00 2001
From: Mike Rapoport <[email protected]>
Date: Sun, 24 Nov 2019 15:38:00 +0200
Subject: [PATCH v3] powerpc: add support for folded p4d page tablesImplement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.Signed-off-by Mike Rapoport <[email protected]>---
v3:
* reduce amount of added p4d walks
* killpgtable_32::get_pteptrand traverse page table inpgtable_32::__change_page_attr_noflusharch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 60 ++++++++++--------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 6 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 30 ++++++---- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 26 +++++--- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 ++--- arch/powerpc/mm/pgtable.c | 30 ++++++---- arch/powerpc/mm/pgtable_32.c | 45 +++----------- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 14 +++-- arch/powerpc/xmon/xmon.c | 18 +++--- 28 files changed, 213 insertions(+), 184 deletions(-)arch/powerpc/include/asm/book3s/32/pgtable.h +0
-1@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_BOOK3S_32_PGTABLE_H#define _ASM_POWERPC_BOOK3S_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/book3s/32/hash.h> arch/powerpc/include/asm/book3s/64/hash.h +2
-2@@ -134,9 +134,9 @@ static inline int get_region_id(unsigned long ea)#define hash__pmd_bad (pmd) (pmd_val(pmd) & H_PMD_BAD_BITS)#define hash__pud_bad (pud) (pud_val(pud) & H_PUD_BAD_BITS) -static inline int hash__pgd_bad (pgd_t pgd) +static inline int hash__p4d_bad (p4d_t p4d) { -return (pgd_val (pgd) =0 ); +return (p4d_val (p4d) =0 ); }#ifdef CONFIG_STRICT_KERNEL_RWX extern void hash__mark_rodata_ro (void );arch/powerpc/include/asm/book3s/64/pgalloc.h +2
-2@@ -85,9 +85,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)kmem_cache_free (PGT_CACHE (PGD_INDEX_SIZE ), pgd); } -static inline void pgd_populate (struct mm_struct *mm,pgd_t *pgd,pud_t *pud) +static inline void p4d_populate (struct mm_struct *mm,p4d_t *pgd,pud_t *pud) { - *pgd =__pgd (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); + *pgd =__p4d (__pgtable_ptr_val (pud) |PGD_VAL_BITS ); }static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr)arch/powerpc/include/asm/book3s/64/pgtable.h +33
-27@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_BOOK3S_64_PGTABLE_H_#define _ASM_POWERPC_BOOK3S_64_PGTABLE_H_ -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> #ifndef __ASSEMBLY__#include <linux/mmdebug.h> @@ -251,7 +251,7 @@ extern unsigned long __pmd_frag_size_shift;/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0xc0000000000000ffUL/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0xc0000000000000ffUL +#define P4D_MASKED_BITS 0xc0000000000000ffUL/* * Used as an indicator for rcu callback functions @@ -949,54 +949,60 @@ static inline bool pud_access_permitted(pud_t pud, bool write)return pte_access_permitted (pud_pte (pud), write); } -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define __p4d_raw (x) ((p4d_t) { __pgd_raw(x) }) +static inline __be64 p4d_raw (p4d_t x) +{ +return pgd_raw (x.pgd); +} + +#define p4d_write (p4d) pte_write(p4d_pte(p4d)) -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline int pgd_none (pgd_t pgd) +static inline int p4d_none (p4d_t p4d) { -return !pgd_raw (pgd); +return !p4d_raw (p4d); } -static inline int pgd_present (pgd_t pgd) +static inline int p4d_present (p4d_t p4d) { -return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PRESENT)); +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PRESENT)); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte_raw (pgd_raw (pgd)); +return __pte_raw (p4d_raw (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd_raw (pte_raw (pte)); +return __p4d_raw (pte_raw (pte)); } -static inline int pgd_bad (pgd_t pgd) +static inline int p4d_bad (p4d_t p4d) {if (radix_enabled ()) -return radix__pgd_bad (pgd); -return hash__pgd_bad (pgd); +return radix__p4d_bad (p4d); +return hash__p4d_bad (p4d); } -#define pgd_access_permitted pgd_access_permitted -static inline bool pgd_access_permitted (pgd_t pgd,bool write) +#define p4d_access_permitted p4d_access_permitted +static inline bool p4d_access_permitted (p4d_t p4d,bool write) { -return pte_access_permitted (pgd_pte (pgd), write); +return pte_access_permitted (p4d_pte (p4d), write); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);/* Pointers in the page table tree are physical addresses */ #define __pgtable_ptr_val (ptr) __pa(ptr)#define pmd_page_vaddr (pmd) __va(pmd_val(pmd) & ~PMD_MASKED_BITS)#define pud_page_vaddr (pud) __va(pud_val(pud) & ~PUD_MASKED_BITS) -#define pgd_page_vaddr (pgd) __va(pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_page_vaddr (p4d) __va(p4d_val(p4d) & ~P4D_MASKED_BITS)#define pgd_index (address) (((address) >> (PGDIR_SHIFT)) & (PTRS_PER_PGD - 1))#define pud_index (address) (((address) >> (PUD_SHIFT)) & (PTRS_PER_PUD - 1))@@ -1010,8 +1016,8 @@ extern struct page *pgd_page(pgd_t pgd);#define pgd_offset (mm, address) ((mm)->pgd + pgd_index(address)) -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + pud_index(addr)) +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + pud_index(addr))#define pmd_offset (pudp,addr) \ (((pmd_t *) pud_page_vaddr(*(pudp))) + pmd_index(addr))#define pte_offset_kernel (dir,addr) \@@ -1368,11 +1374,11 @@ static inline bool pud_is_leaf(pud_t pud) return !!(pud_raw(pud) & cpu_to_be64(_PAGE_PTE)); } -#define pgd_is_leaf pgd_is_leaf -#define pgd_leaf pgd_is_leaf -static inline bool pgd_is_leaf (pgd_t pgd) +#define p4d_is_leaf p4d_is_leaf +#define p4d_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) { -return !!(pgd_raw (pgd) &cpu_to_be64 (_PAGE_PTE)); +return !!(p4d_raw (p4d) &cpu_to_be64 (_PAGE_PTE)); }#endif /* __ASSEMBLY__ */ arch/powerpc/include/asm/book3s/64/radix.h +3
-3@@ -30,7 +30,7 @@/* Don't have anything in the reserved bits and leaf bits */ #define RADIX_PMD_BAD_BITS 0x60000000000000e0UL#define RADIX_PUD_BAD_BITS 0x60000000000000e0UL -#define RADIX_PGD_BAD_BITS 0x60000000000000e0UL +#define RADIX_P4D_BAD_BITS 0x60000000000000e0UL#define RADIX_PMD_SHIFT (PAGE_SHIFT + RADIX_PTE_INDEX_SIZE)#define RADIX_PUD_SHIFT (RADIX_PMD_SHIFT + RADIX_PMD_INDEX_SIZE)@@ -227,9 +227,9 @@ static inline int radix__pud_bad(pud_t pud) } -static inline int radix__pgd_bad (pgd_t pgd) +static inline int radix__p4d_bad (p4d_t p4d) { -return !!(pgd_val (pgd) &RADIX_PGD_BAD_BITS ); +return !!(p4d_val (p4d) &RADIX_P4D_BAD_BITS ); }#ifdef CONFIG_TRANSPARENT_HUGEPAGE arch/powerpc/include/asm/nohash/32/pgtable.h +0
-1@@ -2,7 +2,6 @@#ifndef _ASM_POWERPC_NOHASH_32_PGTABLE_H#define _ASM_POWERPC_NOHASH_32_PGTABLE_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #ifndef __ASSEMBLY__arch/powerpc/include/asm/nohash/64/pgalloc.h +1
-1@@ -15,7 +15,7 @@ struct vmemmap_backing { };extern struct vmemmap_backing *vmemmap_list; -#define pgd_populate (MM ,PGD ,PUD ) pgd_set(PGD, (unsigned long)PUD) +#define p4d_populate (MM ,P4D ,PUD ) p4d_set(P4D, (unsigned long)PUD)static inline pud_t *pud_alloc_one (struct mm_struct *mm,unsigned long addr) {arch/powerpc/include/asm/nohash/64/pgtable-4k.h +16
-16@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H#define _ASM_POWERPC_NOHASH_64_PGTABLE_4K_H -#include <asm-generic/5level-fixup.h> +#include <asm-generic/pgtable-nop4d.h> /* * Entries per page directory level. The PTE level must use a 64b record @@ -45,41 +45,41 @@#define PMD_MASKED_BITS 0/* Bits to mask out from a PUD to get to the PMD page */ #define PUD_MASKED_BITS 0 -/* Bits to mask out from a PGD to get to the PUD page */ -#define PGD_MASKED_BITS 0 +/* Bits to mask out from a P4D to get to the PUD page */ +#define P4D_MASKED_BITS 0/* * 4-level page tables related bits */ -#define pgd_none (pgd) (!pgd_val(pgd)) -#define pgd_bad (pgd) (pgd_val(pgd) = 0) -#define pgd_present (pgd) (pgd_val(pgd) != 0) -#define pgd_page_vaddr (pgd) (pgd_val(pgd) & ~PGD_MASKED_BITS) +#define p4d_none (p4d) (!p4d_val(p4d)) +#define p4d_bad (p4d) (p4d_val(p4d) = 0) +#define p4d_present (p4d) (p4d_val(p4d) != 0) +#define p4d_page_vaddr (p4d) (p4d_val(p4d) & ~P4D_MASKED_BITS)#ifndef __ASSEMBLY__ -static inline void pgd_clear (pgd_t *pgdp) +static inline void p4d_clear (p4d_t *p4dp) { - *pgdp =__pgd (0 ); + *p4dp =__p4d (0 ); } -static inline pte_t pgd_pte (pgd_t pgd) +static inline pte_t p4d_pte (p4d_t p4d) { -return __pte (pgd_val (pgd)); +return __pte (p4d_val (p4d)); } -static inline pgd_t pte_pgd (pte_t pte) +static inline p4d_t pte_p4d (pte_t pte) { -return __pgd (pte_val (pte)); +return __p4d (pte_val (pte)); } -extern struct page *pgd_page (pgd_t pgd); +extern struct page *p4d_page (p4d_t p4d);#endif /* !__ASSEMBLY__ */ -#define pud_offset (pgdp, addr) \ - (((pud_t *) pgd_page_vaddr(*(pgdp))) + \ +#define pud_offset (p4dp, addr) \ + (((pud_t *) p4d_page_vaddr(*(p4dp))) + \ (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1)))#define pud_ERROR (e) \arch/powerpc/include/asm/nohash/64/pgtable.h +3
-3@@ -175,11 +175,11 @@ static inline pud_t pte_pud(pte_t pte)return __pud (pte_val (pte)); }#define pud_write (pud) pte_write(pud_pte(pud)) -#define pgd_write (pgd) pte_write(pgd_pte(pgd)) +#define p4d_write (pgd) pte_write(p4d_pte(p4d)) -static inline void pgd_set (pgd_t *pgdp,unsigned long val) +static inline void p4d_set (p4d_t *p4dp,unsigned long val) { - *pgdp =__pgd (val); + *p4dp =__p4d (val); }/* arch/powerpc/include/asm/pgtable.h +3
-3@@ -139,9 +139,9 @@ static inline bool pud_is_leaf(pud_t pud) }#endif -#ifndef pgd_is_leaf -#define pgd_is_leaf pgd_is_leaf -static inline bool pgd_is_leaf (pgd_t pgd) +#ifndef p4d_is_leaf +#define p4d_is_leaf p4d_is_leaf +static inline bool p4d_is_leaf (p4d_t p4d) {return false; }arch/powerpc/kvm/book3s_64_mmu_radix.c +18
-12@@ -499,13 +499,14 @@ void kvmppc_free_pgtable_radix(struct kvm *kvm, pgd_t *pgd, unsigned int lpid)unsigned long ig;for (ig =0 ; ig <PTRS_PER_PGD ; ++ig, ++pgd) { +p4d_t *p4d =p4d_offset (pgd,0 );pud_t *pud; -if (!pgd_present (*pgd)) +if (!p4d_present (*p4d))continue ; - pud =pud_offset (pgd,0 ); + pud =pud_offset (p4d,0 );kvmppc_unmap_free_pud (kvm, pud, lpid); -pgd_clear (pgd); +p4d_clear (p4d); } }@@ -566,6 +567,7 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte, unsigned long *rmapp, struct rmap_nested **n_rmap) { pgd_t *pgd; +p4d_t *p4d;pud_t *pud, *new_pud =NULL ;pmd_t *pmd, *new_pmd =NULL ;pte_t *ptep, *new_ptep =NULL ;@@ -573,9 +575,11 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Traverse the guest's 2nd-level tree, allocate new levels needed */ pgd = pgtable +pgd_index (gpa); + p4d =p4d_offset (pgd, gpa); + pud =NULL ; -if (pgd_present (*pgd)) - pud =pud_offset (pgd, gpa); +if (p4d_present (*p4d)) + pud =pud_offset (p4d, gpa);else new_pud =pud_alloc_one (kvm->mm, gpa);@@ -596,13 +600,13 @@ int kvmppc_create_pte(struct kvm *kvm, pgd_t *pgtable, pte_t pte,/* Now traverse again under the lock and change the tree */ ret = -ENOMEM ; -if (pgd_none (*pgd)) { +if (p4d_none (*p4d)) {if (!new_pud) goto out_unlock; -pgd_populate (kvm->mm, pgd, new_pud); +p4d_populate (kvm->mm, p4d, new_pud); new_pud =NULL ; } - pud =pud_offset (pgd, gpa); + pud =pud_offset (p4d, gpa);if (pud_is_leaf (*pud)) {unsigned long hgpa = gpa &PUD_MASK ;@@ -1220,6 +1224,7 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf,pgd_t *pgt;struct kvm_nested_guest *nested;pgd_t pgd, *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ptep;@@ -1292,13 +1297,14 @@ static ssize_t debugfs_radix_read(struct file *file, char __user *buf, } pgdp = pgt +pgd_index (gpa); - pgd =READ_ONCE (*pgdp); -if (!(pgd_val (pgd) & _PAGE_PRESENT)) { - gpa = (gpa &PGDIR_MASK ) +PGDIR_SIZE ; + p4dp =p4d_offset (pgdp, gpa); + p4d =READ_ONCE (*p4dp); +if (!(p4d_val (p4d) & _PAGE_PRESENT)) { + gpa = (gpa &P4D_MASK ) +P4D_SIZE ;continue ; } - pudp =pud_offset (&pgd, gpa); + pudp =pud_offset (&p4d, gpa); pud =READ_ONCE (*pudp);if (!(pud_val (pud) & _PAGE_PRESENT)) { gpa = (gpa &PUD_MASK ) +PUD_SIZE ;arch/powerpc/lib/code-patching.c +6
-1@@ -107,13 +107,18 @@ static inline int unmap_patch_area(unsigned long addr)pte_t *ptep;pmd_t *pmdp;pud_t *pudp; +p4d_t *p4dp;pgd_t *pgdp; pgdp =pgd_offset_k (addr);if (unlikely (!pgdp))return -EINVAL ; - pudp =pud_offset (pgdp, addr); + p4dp =p4d_offset (pgdp, addr); +if (unlikely (!p4dp)) +return -EINVAL ; + + pudp =pud_offset (p4dp, addr);if (unlikely (!pudp))return -EINVAL ;arch/powerpc/mm/book3s32/mmu.c +1
-1@@ -312,7 +312,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea)if (!Hash)return ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, ea), ea), ea); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, ea), ea), ea), ea);if (!pmd_none (*pmd))add_hash_page (mm->context.id, ea,pmd_val (*pmd)); }arch/powerpc/mm/book3s32/tlb.c +2
-2@@ -87,7 +87,7 @@ static void flush_range(struct mm_struct *mm, unsigned long start,if (start >= end)return ; end = (end -1 ) | ~PAGE_MASK ; - pmd =pmd_offset (pud_offset (pgd_offset (mm, start), start), start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, start), start), start), start);for (;;) { pmd_end = ((start +PGDIR_SIZE ) &PGDIR_MASK ) -1 ;if (pmd_end > end)@@ -145,7 +145,7 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)return ; } mm = (vmaddr <TASK_SIZE )? vma->vm_mm: &init_mm; - pmd =pmd_offset (pud_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset (mm, vmaddr), vmaddr), vmaddr), vmaddr);if (!pmd_none (*pmd))flush_hash_pages (mm->context.id, vmaddr,pmd_val (*pmd),1 ); }arch/powerpc/mm/book3s64/hash_pgtable.c +3
-1@@ -148,6 +148,7 @@ void hash__vmemmap_remove_mapping(unsigned long start,int hash__map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -155,7 +156,8 @@ int hash__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >H_PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);arch/powerpc/mm/book3s64/radix_pgtable.c +17
-9@@ -64,17 +64,19 @@ static int early_map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep; pgdp =pgd_offset_k (ea); -if (pgd_none (*pgdp)) { + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { pudp =early_alloc_pgtable (PUD_TABLE_SIZE , nid, region_start, region_end); -pgd_populate (&init_mm, pgdp, pudp); +p4d_populate (&init_mm, p4dp, pudp); } - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (map_page_size =PUD_SIZE ) { ptep = (pte_t *)pudp; goto set_the_pte;@@ -114,6 +116,7 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa, {unsigned long pfn = pa >>PAGE_SHIFT ;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -136,7 +139,8 @@ static int __map_kernel_page(unsigned long ea, unsigned long pa,* boot. */ pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ;if (map_page_size =PUD_SIZE ) {@@ -173,6 +177,7 @@ void radix__change_memory_range(unsigned long start, unsigned long end, {unsigned long idx;pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -185,7 +190,8 @@ void radix__change_memory_range(unsigned long start, unsigned long end,for (idx = start; idx < end; idx +=PAGE_SIZE ) { pgdp =pgd_offset_k (idx); - pudp =pud_alloc (&init_mm, pgdp, idx); + p4dp =p4d_offset (pgdp, idx); + pudp =pud_alloc (&init_mm, p4dp, idx);if (!pudp)continue ;if (pud_is_leaf (*pudp)) {@@ -847,6 +853,7 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end)unsigned long addr, next;pud_t *pud_base;pgd_t *pgd; +p4d_t *p4d;spin_lock (&init_mm.page_table_lock);@@ -854,15 +861,16 @@ static void __meminit remove_pagetable(unsigned long start, unsigned long end) next =pgd_addr_end (addr, end); pgd =pgd_offset_k (addr); -if (!pgd_present (*pgd)) + p4d =p4d_offset (pgd, addr); +if (!p4d_present (*p4d))continue ; -if (pgd_is_leaf (*pgd)) { -split_kernel_mapping (addr, end,PGDIR_SIZE , (pte_t *)pgd); +if (p4d_is_leaf (*p4d)) { +split_kernel_mapping (addr, end,P4D_SIZE , (pte_t *)p4d);continue ; } - pud_base = (pud_t *)pgd_page_vaddr (*pgd); + pud_base = (pud_t *)p4d_page_vaddr (*p4d);remove_pud_table (pud_base, addr, next); }arch/powerpc/mm/book3s64/subpage_prot.c +4
-2@@ -54,15 +54,17 @@ static void hpte_flush_range(struct mm_struct *mm, unsigned long addr, int npages) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;spinlock_t *ptl; pgd =pgd_offset (mm, addr); -if (pgd_none (*pgd)) + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d))return ; - pud =pud_offset (pgd, addr); + pud =pud_offset (p4d, addr);if (pud_none (*pud))return ; pmd =pmd_offset (pud, addr);arch/powerpc/mm/hugetlbpage.c +16
-12@@ -119,6 +119,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,pte_t *huge_pte_alloc (struct mm_struct *mm,unsigned long addr,unsigned long sz) {pgd_t *pg; +p4d_t *p4;pud_t *pu;pmd_t *pm;hugepd_t *hpdp =NULL ;@@ -128,20 +129,21 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz addr &= ~(sz-1 ); pg =pgd_offset (mm, addr); + p4 =p4d_offset (pg, addr);#ifdef CONFIG_PPC_BOOK3S_64 if (pshift =PGDIR_SHIFT )/* 16GB huge page */ -return (pte_t *) pg; +return (pte_t *) p4;else if (pshift >PUD_SHIFT ) {/* * We need to use hugepd table */ ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift =PUD_SHIFT )@@ -166,10 +168,10 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz#else if (pshift >=PGDIR_SHIFT ) { ptl = &mm->page_table_lock; - hpdp = (hugepd_t *)pg; + hpdp = (hugepd_t *)p4; }else { pdshift =PUD_SHIFT ; - pu =pud_alloc (mm, pg, addr); + pu =pud_alloc (mm, p4, addr);if (!pu)return NULL ;if (pshift >=PUD_SHIFT ) {@@ -390,7 +392,7 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,mm_dec_nr_pmds (tlb->mm); } -static void hugetlb_free_pud_range (struct mmu_gather *tlb,pgd_t *pgd, +static void hugetlb_free_pud_range (struct mmu_gather *tlb,p4d_t *p4d,unsigned long addr,unsigned long end,unsigned long floor,unsigned long ceiling) {@@ -400,7 +402,7 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd, start = addr; do { - pud = pud_offset(pgd, addr); + pud = pud_offset(p4d, addr); next = pud_addr_end(addr, end); if (!is_hugepd(__hugepd(pud_val(*pud)))) { if (pud_none_or_clear_bad(pud))@@ -435,8 +437,8 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,if (end -1 > ceiling -1 )return ; - pud =pud_offset (pgd, start); -pgd_clear (pgd); + pud =pud_offset (p4d, start); +p4d_clear (p4d);pud_free_tlb (tlb, pud, start);mm_dec_nr_puds (tlb->mm); }@@ -449,6 +451,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long floor, unsigned long ceiling) {pgd_t *pgd; +p4d_t *p4d;unsigned long next;/* @@ -471,10 +474,11 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb, do { next = pgd_addr_end(addr, end); pgd = pgd_offset(tlb->mm, addr); + p4d = p4d_offset(pgd, addr); if (!is_hugepd(__hugepd(pgd_val(*pgd)))) { - if (pgd_none_or_clear_bad(pgd)) + if (p4d_none_or_clear_bad(p4d)) continue; - hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling); + hugetlb_free_pud_range(tlb, p4d, addr, next, floor, ceiling); } else { unsigned long more; /*@@ -487,7 +491,7 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,if (more > next) next = more; -free_hugepd_range (tlb, (hugepd_t *)pgd,PGDIR_SHIFT , +free_hugepd_range (tlb, (hugepd_t *)p4d,PGDIR_SHIFT , addr, next, floor, ceiling); } }while (addr = next, addr != end);arch/powerpc/mm/kasan/kasan_init_32.c +4
-4@@ -36,7 +36,7 @@ static int __init kasan_init_shadow_page_tables(unsigned long k_start, unsignedunsigned long k_cur, k_next;pte_t *new =NULL ; - pmd =pmd_offset (pud_offset (pgd_offset_k (k_start), k_start), k_start); + pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_start), k_start), k_start), k_start);for (k_cur = k_start; k_cur != k_end; k_cur = k_next, pmd++) { k_next =pgd_addr_end (k_cur, k_end);@@ -78,7 +78,7 @@ static int __init kasan_init_region(void *start, size_t size) block =memblock_alloc (k_end - k_start,PAGE_SIZE );for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);void *va = block + k_cur - k_start;pte_t pte =pfn_pte (PHYS_PFN (__pa (va)),PAGE_KERNEL );@@ -102,7 +102,7 @@ static void __init kasan_remap_early_shadow_ro(void)kasan_populate_pte (kasan_early_shadow_pte, prot);for (k_cur = k_start &PAGE_MASK ; k_cur < k_end; k_cur +=PAGE_SIZE ) { -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (k_cur), k_cur), k_cur); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (k_cur), k_cur), k_cur), k_cur);pte_t *ptep =pte_offset_kernel (pmd, k_cur);if ((pte_val (*ptep) &PTE_RPN_MASK ) != pa)@@ -201,7 +201,7 @@ void __init kasan_early_init(void)unsigned long addr =KASAN_SHADOW_START ;unsigned long end =KASAN_SHADOW_END ;unsigned long next; -pmd_t *pmd =pmd_offset (pud_offset (pgd_offset_k (addr), addr), addr); +pmd_t *pmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (addr), addr), addr), addr);BUILD_BUG_ON (KASAN_SHADOW_START & ~PGDIR_MASK );arch/powerpc/mm/mem.c +2
-2@@ -69,8 +69,8 @@ EXPORT_SYMBOL(kmap_prot);static inline pte_t *virt_to_kpte (unsigned long vaddr) { -return pte_offset_kernel (pmd_offset (pud_offset (pgd_offset_k (vaddr), - vaddr), vaddr), vaddr); +return pte_offset_kernel (pmd_offset (pud_offset (p4d_offset (pgd_offset_k (vaddr), + vaddr), vaddr), vaddr), vaddr); }#endif arch/powerpc/mm/nohash/40x.c +2
-2@@ -104,7 +104,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_16M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val); *pmdp++ =__pmd (val);@@ -119,7 +119,7 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)pmd_t *pmdp;unsigned long val = p | _PMD_SIZE_4M | _PAGE_EXEC | _PAGE_HWWRITE; - pmdp =pmd_offset (pud_offset (pgd_offset_k (v), v), v); + pmdp =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (v), v), v), v); *pmdp =__pmd (val); v +=LARGE_PAGE_SIZE_4M ;arch/powerpc/mm/nohash/book3e_pgtable.c +8
-7@@ -73,6 +73,7 @@ static void __init *early_alloc_pgtable(unsigned long size)int __ref map_kernel_page (unsigned long ea,unsigned long pa,pgprot_t prot) {pgd_t *pgdp; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -80,7 +81,8 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)BUILD_BUG_ON (TASK_SIZE_USER64 >PGTABLE_RANGE );if (slab_is_available ()) { pgdp =pgd_offset_k (ea); - pudp =pud_alloc (&init_mm, pgdp, ea); + p4dp =p4d_offset (pgdp, ea); + pudp =pud_alloc (&init_mm, p4dp, ea);if (!pudp)return -ENOMEM ; pmdp =pmd_alloc (&init_mm, pudp, ea);@@ -91,13 +93,12 @@ int __ref map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t prot)return -ENOMEM ; }else { pgdp =pgd_offset_k (ea); -#ifndef __PAGETABLE_PUD_FOLDED -if (pgd_none (*pgdp)) { - pudp =early_alloc_pgtable (PUD_TABLE_SIZE ); -pgd_populate (&init_mm, pgdp, pudp); + p4dp =p4d_offset (pgdp, ea); +if (p4d_none (*p4dp)) { + pmdp =early_alloc_pgtable (PMD_TABLE_SIZE ); +p4d_populate (&init_mm, p4dp, pmdp); } -#endif /* !__PAGETABLE_PUD_FOLDED */ - pudp =pud_offset (pgdp, ea); + pudp =pud_offset (p4dp, ea);if (pud_none (*pudp)) { pmdp =early_alloc_pgtable (PMD_TABLE_SIZE );pud_populate (&init_mm, pudp, pmdp);arch/powerpc/mm/pgtable.c +19
-11@@ -265,6 +265,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,void assert_pte_locked (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;@@ -272,7 +273,9 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr)return ; pgd = mm->pgd +pgd_index (addr);BUG_ON (pgd_none (*pgd)); - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +BUG_ON (p4d_none (*p4d)); + pud =pud_offset (p4d, addr);BUG_ON (pud_none (*pud)); pmd =pmd_offset (pud, addr);/* @@ -312,12 +315,13 @@ EXPORT_SYMBOL_GPL(vmalloc_to_phys);pte_t *__find_linux_pte (pgd_t *pgdir,unsigned long ea,bool *is_thp,unsigned *hpage_shift) { -pgd_t pgd, *pgdp; +pgd_t *pgdp; +p4d_t p4d, *p4dp;pud_t pud, *pudp;pmd_t pmd, *pmdp;pte_t *ret_pte;hugepd_t *hpdp =NULL ; -unsigned pdshift =PGDIR_SHIFT ; +unsigned pdshift;if (hpage_shift) *hpage_shift =0 ;@@ -325,24 +329,28 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,if (is_thp) *is_thp = false; - pgdp = pgdir +pgd_index (ea); - pgd =READ_ONCE (*pgdp);/* * Always operate on the local stack value. This make sure the * value don't get updated by a parallel THP split/collapse, * page fault or a page unmap. The return pte_t * is still not * stable. So should be checked there for above conditions. +* Top level is an exception because it is folded into p4d. */ -if (pgd_none (pgd)) + pgdp = pgdir +pgd_index (ea); + p4dp =p4d_offset (pgdp, ea); + p4d =READ_ONCE (*p4dp); + pdshift =P4D_SHIFT ; + +if (p4d_none (p4d))return NULL ; -if (pgd_is_leaf (pgd)) { - ret_pte = (pte_t *)pgdp; +if (p4d_is_leaf (p4d)) { + ret_pte = (pte_t *)p4dp; goto out; } -if (is_hugepd (__hugepd (pgd_val (pgd)))) { - hpdp = (hugepd_t *)&pgd; +if (is_hugepd (__hugepd (p4d_val (p4d)))) { + hpdp = (hugepd_t *)&p4d; goto out_huge; }@@ -352,7 +360,7 @@ pte_t *__find_linux_pte(pgd_t *pgdir, unsigned long ea,* irq disabled */ pdshift =PUD_SHIFT ; - pudp =pud_offset (&pgd, ea); + pudp =pud_offset (&p4d, ea); pud =READ_ONCE (*pudp);if (pud_none (pud))arch/powerpc/mm/pgtable_32.c +8
-37@@ -63,7 +63,7 @@ int __ref map_kernel_page(unsigned long va, phys_addr_t pa, pgprot_t prot)int err = -ENOMEM ;/* Use upper 10 bits of VA to index the first level map */ - pd =pmd_offset (pud_offset (pgd_offset_k (va), va), va); + pd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va);/* Use middle 10 bits of VA to index the second-level map */ if (likely (slab_is_available ())) pg =pte_alloc_kernel (pd, va);@@ -121,53 +121,24 @@ void __init mapin_ram(void) } } -/* Scan the real Linux page tables and return a PTE pointer for -* a virtual address in a context. -* Returns true (1) if PTE was found, zero otherwise. The pointer to -* the PTE pointer is unmodified if PTE is not found. -*/ -static int -get_pteptr (struct mm_struct *mm,unsigned long addr,pte_t **ptep,pmd_t **pmdp) -{ -pgd_t *pgd; -pud_t *pud; -pmd_t *pmd; -pte_t *pte; -int retval =0 ; - - pgd =pgd_offset (mm, addr &PAGE_MASK ); -if (pgd) { - pud =pud_offset (pgd, addr &PAGE_MASK ); -if (pud &&pud_present (*pud)) { - pmd =pmd_offset (pud, addr &PAGE_MASK ); -if (pmd_present (*pmd)) { - pte =pte_offset_map (pmd, addr &PAGE_MASK ); -if (pte) { - retval =1 ; - *ptep = pte; -if (pmdp) - *pmdp = pmd; -/* XXX caller needs to do pte_unmap, yuck */ - } - } - } - } -return (retval); -} -static int __change_page_attr_noflush (struct page *page,pgprot_t prot) {pte_t *kpte;pmd_t *kpmd; -unsigned long address; +unsigned long address, va;BUG_ON (PageHighMem (page)); address = (unsigned long )page_address (page); + va = address &PAGE_MASK ;if (v_block_mapped (address))return 0 ; -if (!get_pteptr (&init_mm, address, &kpte, &kpmd)) + + kpmd =pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va); +if (!pmd_present (*kpmd))return -EINVAL ; + + kpte =pte_offset_map (kpmd, va);__set_pte_at (&init_mm, address, kpte,mk_pte (page, prot),0 );pte_unmap (kpte);arch/powerpc/mm/pgtable_64.c +5
-5@@ -101,13 +101,13 @@ EXPORT_SYMBOL(__pte_frag_size_shift);#ifndef __PAGETABLE_PUD_FOLDED/* 4 level page table */ -struct page *pgd_page (pgd_t pgd) +struct page *p4d_page (p4d_t p4d) { -if (pgd_is_leaf (pgd)) { -VM_WARN_ON (!pgd_huge (pgd)); -return pte_page (pgd_pte (pgd)); +if (p4d_is_leaf (p4d)) { +VM_WARN_ON (!p4d_huge (p4d)); +return pte_page (p4d_pte (p4d)); } -return virt_to_page (pgd_page_vaddr (pgd)); +return virt_to_page (p4d_page_vaddr (p4d)); }#endif arch/powerpc/mm/ptdump/hashpagetable.c +17
-3@@ -417,9 +417,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -431,6 +431,20 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 ); +unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); + } +} +static void walk_pagetables (struct pg_state *st) {pgd_t *pgd =pgd_offset_k (0UL );@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr); } }arch/powerpc/mm/ptdump/ptdump.c +8
-6@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -304,11 +304,13 @@ static void walk_pagetables(struct pg_state *st)* the hash pagetable. */ for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) { -if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd)) -/* pgd exists */ -walk_pud (st, pgd, addr); +p4d_t *p4d =p4d_offset (pgd,0 ); + +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr);else -note_page (st, addr,1 ,pgd_val (*pgd)); +note_page (st, addr,1 ,p4d_val (*p4d)); } }arch/powerpc/xmon/xmon.c +10
-8@@ -3130,6 +3130,7 @@ static void show_pte(unsigned long addr)struct task_struct *tsk =NULL ;struct mm_struct *mm;pgd_t *pgdp, *pgdir; +p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp;pte_t *ptep;@@ -3161,20 +3162,21 @@ static void show_pte(unsigned long addr) pgdir =pgd_offset (mm,0 ); } -if (pgd_none (*pgdp)) { -printf ("no linux page table for address\n" ); + p4dp =p4d_offset (pgdp, addr); + +if (p4d_none (*p4dp)) { +printf ("No valid P4D\n" );return ; } -printf ("pgd @ 0x%px\n" , pgdir); - -if (pgd_is_leaf (*pgdp)) { -format_pte (pgdp,pgd_val (*pgdp)); +if (p4d_is_leaf (*p4dp)) { +format_pte (p4dp,p4d_val (*p4dp));return ; } -printf ("pgdp @ 0x%px = 0x%016lx\n" , pgdp,pgd_val (*pgdp)); - pudp =pud_offset (pgdp, addr); +printf ("p4dp @ 0x%px = 0x%016lx\n" , p4dp,p4d_val (*p4dp)); + + pudp =pud_offset (p4dp, addr);if (pud_none (*pudp)) {printf ("No valid PUD\n" );
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Le 26/02/2020 à 11:56, Mike Rapoport a écrit :
Quoted 41 lines · Christophe Leroy
Quoted 32 lines
Le 26/02/2020 à 10:13, Mike Rapoport a écrit :
Quoted 16 lines · Mike Rapoport
Quoted 15 lines · Christophe Leroy
Quoted 14 lines
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 4 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.I don't think it is worth adding all this additionnals walks of p4d, this patch could be limited to changes like:
(quoted diff) +1
-1- pud = pud_offset(pgd, gpa); + pud = pud_offset(p4d_offset(pgd, gpa), gpa);Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The additionnal walks should be added through another patch the day powerpc need them.
Ok, I'll update the patch to reduce walking the p4d.
Here's what I have with more direct acceses from pgd to pud.
I went quickly through. This looks promising.
Do we need the
walk_p4d()inarch/powerpc/mm/ptdump/hashpagetable.c? Can't we just doarch/powerpc/mm/ptdump/hashpagetable.c static void walk_pagetables(struct pg_state *st) · line 462
Quoted 5 lines
arch/powerpc/mm/ptdump/hashpagetable.c +1
-1@@ -445,7 +459,7 @@ static void walk_pagetables(struct pg_state *st) addr =KERN_VIRT_START + i *PGDIR_SIZE ;if (!pgd_none (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_pud (st,p4d_offset (pgd, addr), addr);We can do
Quoted 7 lines
addr = KERN_VIRT_START + i * PGDIR_SIZE; p4d = p4d_offset(pgd, addr); if (!p4d_none(*pgd)) walk_pud()But I don't think this is really essential. Again, we are trading off code consistency vs line count. I don't think line count is that important.
Ok.
Christophe
Re: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
In reply to Mike RapoportLe 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 61 lines
From: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and replace
5level-fixup.hwithpgtable-nop4d.h.Signed-off-by Mike Rapoport <[email protected]>--- arch/powerpc/include/asm/book3s/32/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/hash.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 58 ++++++++++-------- arch/powerpc/include/asm/book3s/64/radix.h | 6 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 1 - arch/powerpc/include/asm/nohash/64/pgalloc.h | 2 +- .../include/asm/nohash/64/pgtable-4k.h | 32 +++++----- arch/powerpc/include/asm/nohash/64/pgtable.h | 6 +- arch/powerpc/include/asm/pgtable.h | 8 +++ arch/powerpc/kvm/book3s_64_mmu_radix.c | 59 ++++++++++++++++--- arch/powerpc/lib/code-patching.c | 7 ++- arch/powerpc/mm/book3s32/mmu.c | 2 +- arch/powerpc/mm/book3s32/tlb.c | 4 +- arch/powerpc/mm/book3s64/hash_pgtable.c | 4 +- arch/powerpc/mm/book3s64/radix_pgtable.c | 19 ++++-- arch/powerpc/mm/book3s64/subpage_prot.c | 6 +- arch/powerpc/mm/hugetlbpage.c | 28 +++++---- arch/powerpc/mm/kasan/kasan_init_32.c | 8 +-- arch/powerpc/mm/mem.c | 4 +- arch/powerpc/mm/nohash/40x.c | 4 +- arch/powerpc/mm/nohash/book3e_pgtable.c | 15 +++-- arch/powerpc/mm/pgtable.c | 25 +++++++- arch/powerpc/mm/pgtable_32.c | 28 +++++---- arch/powerpc/mm/pgtable_64.c | 10 ++-- arch/powerpc/mm/ptdump/hashpagetable.c | 20 ++++++- arch/powerpc/mm/ptdump/ptdump.c | 22 ++++++- arch/powerpc/xmon/xmon.c | 17 +++++- 28 files changed, 284 insertions(+), 120 deletions(-)arch/powerpc/mm/ptdump/ptdump.c +5
-2@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -293,6 +293,22 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 );
arch/powerpc/mm/ptdump/ptdump.c static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) · line 308
Quoted 10 lines
arch/powerpc/mm/ptdump/ptdump.c +10
-0+unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); +else +note_page (st, addr,2 ,p4d_val (*p4d));
Level 2 is already used by walk_pud().
I think you have to increment the level used in walk_pud() and walk_pmd() and walk_pte()
Quoted 7 lines
arch/powerpc/mm/ptdump/ptdump.c +3
-0+ } +} +static void walk_pagetables (struct pg_state *st) {unsigned int i;
arch/powerpc/mm/ptdump/ptdump.c static void walk_pagetables(struct pg_state *st) · line 328
Quoted 8 lines
arch/powerpc/mm/ptdump/ptdump.c +1
-1@@ -306,7 +322,7 @@ static void walk_pagetables(struct pg_state *st)for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) {if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr);else note_page (st, addr,1 ,pgd_val (*pgd)); }
Christophe
Mike Rapoport
reviewerRe: [PATCH v2 07/13] powerpc: add support for folded p4d page tables
Quoted 35 lines · Christophe Leroy
Le 16/02/2020 à 09:18, Mike Rapoport a écrit :
Quoted 20 lines
arch/powerpc/mm/ptdump/ptdump.c +5
-2@@ -277,9 +277,9 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) } } -static void walk_pud (struct pg_state *st,pgd_t *pgd,unsigned long start) +static void walk_pud (struct pg_state *st,p4d_t *p4d,unsigned long start) { -pud_t *pud =pud_offset (pgd,0 ); +pud_t *pud =pud_offset (p4d,0 );unsigned long addr;unsigned int i;@@ -293,6 +293,22 @@ static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) } } +static void walk_p4d (struct pg_state *st,pgd_t *pgd,unsigned long start) +{ +p4d_t *p4d =p4d_offset (pgd,0 );arch/powerpc/mm/ptdump/ptdump.c static void walk_pud(struct pg_state *st, pgd_t *pgd, unsigned long start) · line 307
Quoted 10 lines
arch/powerpc/mm/ptdump/ptdump.c +10
-0+unsigned long addr; +unsigned int i; + +for (i =0 ; i <PTRS_PER_P4D ; i++, p4d++) { + addr = start + i *P4D_SIZE ; +if (!p4d_none (*p4d) && !p4d_is_leaf (*p4d)) +/* p4d exists */ +walk_pud (st, p4d, addr); +else +note_page (st, addr,2 ,p4d_val (*p4d));Level 2 is already used by
walk_pud().
Quoted 2 lines
I think you have to increment the level used in
walk_pud()andwalk_pmd()andwalk_pte()
Thanks for catching this!
I'll fix the numbers in the next version.
Quoted 17 lines
Quoted 7 lines
arch/powerpc/mm/ptdump/ptdump.c +3
-0+ } +} +static void walk_pagetables (struct pg_state *st) {unsigned int i;arch/powerpc/mm/ptdump/ptdump.c static void walk_pagetables(struct pg_state *st) · line 328
Quoted 8 lines
arch/powerpc/mm/ptdump/ptdump.c +1
-1@@ -306,7 +322,7 @@ static void walk_pagetables(struct pg_state *st)for (i =pgd_index (addr); i <PTRS_PER_PGD ; i++, pgd++, addr +=PGDIR_SIZE ) {if (!pgd_none (*pgd) && !pgd_is_leaf (*pgd))/* pgd exists */ -walk_pud (st, pgd, addr); +walk_p4d (st, pgd, addr);else note_page (st, addr,1 ,pgd_val (*pgd)); }Christophe
Signature · 2 lines
Sincerely yours, Mike.
Mike Rapoport
reviewerPATCH v2 8/13 sh: fault: Modernize printing of kernel messages
In reply to Mike RapoportFrom: Geert Uytterhoeven <[email protected]>
- Convert from printk() to pr_*(),
- Add missing continuations,
- Use "%llx" to format u64,
- Join multiple prints in show_fault_oops() into a single print.
--- arch/sh/mm/fault.c | 39 ++++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 21 deletions(-)
arch/sh/mm/fault.c +18 -21
@@ -47,10 +47,10 @@ static void show_pte(struct mm_struct *mm, unsigned long addr) pgd = swapper_pg_dir; } -printk (KERN_ALERT "pgd = %p\n" , pgd); +pr_alert ("pgd = %p\n" , pgd); pgd +=pgd_index (addr); -printk (KERN_ALERT "[%08lx] *pgd=%0*Lx" , addr, - (u32)(sizeof (*pgd) *2 ), (u64 )pgd_val (*pgd)); +pr_alert ("[%08lx] *pgd=%0*llx" , addr, (u32)(sizeof (*pgd) *2 ), + (u64 )pgd_val (*pgd)); do { pud_t *pud;
@@ -61,33 +61,33 @@ static void show_pte(struct mm_struct *mm, unsigned long addr)break ;if (pgd_bad (*pgd)) { -printk ("(bad)" ); +pr_cont ("(bad)" );break ; } pud =pud_offset (pgd, addr);if (PTRS_PER_PUD !=1 ) -printk (", *pud=%0*Lx" , (u32)(sizeof (*pud) *2 ), - (u64 )pud_val (*pud)); +pr_cont (", *pud=%0*llx" , (u32)(sizeof (*pud) *2 ), + (u64 )pud_val (*pud));if (pud_none (*pud))break ;if (pud_bad (*pud)) { -printk ("(bad)" ); +pr_cont ("(bad)" );break ; } pmd =pmd_offset (pud, addr);if (PTRS_PER_PMD !=1 ) -printk (", *pmd=%0*Lx" , (u32)(sizeof (*pmd) *2 ), - (u64 )pmd_val (*pmd)); +pr_cont (", *pmd=%0*llx" , (u32)(sizeof (*pmd) *2 ), + (u64 )pmd_val (*pmd));if (pmd_none (*pmd))break ;if (pmd_bad (*pmd)) { -printk ("(bad)" ); +pr_cont ("(bad)" );break ; }
@@ -96,11 +96,11 @@ static void show_pte(struct mm_struct *mm, unsigned long addr)break ; pte =pte_offset_kernel (pmd, addr); -printk (", *pte=%0*Lx" , (u32)(sizeof (*pte) *2 ), - (u64 )pte_val (*pte)); +pr_cont (", *pte=%0*llx" , (u32)(sizeof (*pte) *2 ), + (u64 )pte_val (*pte)); }while (0 ); -printk ("\n" ); +pr_cont ("\n" ); }static inline pmd_t *vmalloc_sync_one (pgd_t *pgd,unsigned long address)
@@ -188,14 +188,11 @@ show_fault_oops(struct pt_regs *regs, unsigned long address)if (!oops_may_print ())return ; -printk (KERN_ALERT "BUG: unable to handle kernel " ); -if (address <PAGE_SIZE ) -printk (KERN_CONT "NULL pointer dereference" ); -else -printk (KERN_CONT "paging request" ); - -printk (KERN_CONT " at %08lx\n" , address); -printk (KERN_ALERT "PC:" ); +pr_alert ("BUG: unable to handle kernel %s at %08lx\n" , + address <PAGE_SIZE ?"NULL pointer dereference" + :"paging request" , + address); +pr_alert ("PC:" );printk_address (regs->pc,1 );show_pte (NULL , address);
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 9/13 sh: drop __pXd_offset() macros that duplicate pXd_index() ones
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
The __pXd_offset() macros are identical to the pXd_index() macros and there is no point to keep both of them. All architectures define and use pXd_index() so let's keep only those to make mips consistent with the rest of the kernel.
--- arch/sh/include/asm/pgtable_32.h | 5 ++--- arch/sh/include/asm/pgtable_64.h | 5 ++--- arch/sh/mm/init.c | 6 +++--- 3 files changed, 7 insertions(+), 9 deletions(-)
arch/sh/include/asm/pgtable_32.h +2 -3
@@ -407,13 +407,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)/* to find an entry in a page-table-directory. */ #define pgd_index (address) (((address) >> PGDIR_SHIFT) & (PTRS_PER_PGD-1))#define pgd_offset (mm, address) ((mm)->pgd + pgd_index(address)) -#define __pgd_offset (address) pgd_index(address)/* to find an entry in a kernel page-table-directory */ #define pgd_offset_k (address) pgd_offset(&init_mm, address) -#define __pud_offset (address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD-1)) -#define __pmd_offset (address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1)) +#define pud_index (address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD-1)) +#define pmd_index (address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1))/* Find an entry in the third-level page table.. */ #define pte_index (address) ((address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1))
arch/sh/include/asm/pgtable_64.h +2 -3
@@ -46,14 +46,13 @@ static __inline__ void set_pte(pte_t *pteptr, pte_t pteval)/* To find an entry in a generic PGD. */ #define pgd_index (address) (((address) >> PGDIR_SHIFT) & (PTRS_PER_PGD-1)) -#define __pgd_offset (address) pgd_index(address)#define pgd_offset (mm, address) ((mm)->pgd+pgd_index(address))/* To find an entry in a kernel PGD. */ #define pgd_offset_k (address) pgd_offset(&init_mm, address) -#define __pud_offset (address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD-1)) -#define __pmd_offset (address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1)) +#define pud_index (address) (((address) >> PUD_SHIFT) & (PTRS_PER_PUD-1)) +/* #define pmd_index(address) (((address) >> PMD_SHIFT) & (PTRS_PER_PMD-1)) */ /* * PMD level access routines. Same notes as above.
arch/sh/mm/init.c +3 -3
@@ -172,9 +172,9 @@ void __init page_table_range_init(unsigned long start, unsigned long end,unsigned long vaddr; vaddr = start; - i =__pgd_offset (vaddr); - j =__pud_offset (vaddr); - k =__pmd_offset (vaddr); + i =pgd_index (vaddr); + j =pud_index (vaddr); + k =pmd_index (vaddr); pgd = pgd_base + i;for ( ; (i <PTRS_PER_PGD ) && (vaddr != end); pgd++, i++) {
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 10/13 sh: add support for folded p4d page tables
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
Implement primitives necessary for the 4th level folding, add walks of p4d level where appropriate and remove usage of __ARCH_USE_5LEVEL_HACK.
--- arch/sh/include/asm/pgtable-2level.h | 1 - arch/sh/include/asm/pgtable-3level.h | 1 - arch/sh/kernel/io_trapped.c | 7 ++++++- arch/sh/mm/cache-sh4.c | 4 +++- arch/sh/mm/cache-sh5.c | 7 ++++++- arch/sh/mm/fault.c | 26 +++++++++++++++++++++++--- arch/sh/mm/hugetlbpage.c | 28 ++++++++++++++++++---------- arch/sh/mm/init.c | 9 ++++++++- arch/sh/mm/kmap.c | 2 +- arch/sh/mm/tlbex_32.c | 6 +++++- arch/sh/mm/tlbex_64.c | 7 ++++++- 11 files changed, 76 insertions(+), 22 deletions(-)
arch/sh/include/asm/pgtable-2level.h +0 -1
@@ -2,7 +2,6 @@#ifndef __ASM_SH_PGTABLE_2LEVEL_H#define __ASM_SH_PGTABLE_2LEVEL_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> /*
arch/sh/include/asm/pgtable-3level.h +0 -1
@@ -2,7 +2,6 @@#ifndef __ASM_SH_PGTABLE_3LEVEL_H#define __ASM_SH_PGTABLE_3LEVEL_H -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopud.h> /*
arch/sh/kernel/io_trapped.c +6 -1
@@ -136,6 +136,7 @@ EXPORT_SYMBOL_GPL(match_trapped_io_handler);static struct trapped_io *lookup_tiop (unsigned long address) {pgd_t *pgd_k; +p4d_t *p4d_k;pud_t *pud_k;pmd_t *pmd_k;pte_t *pte_k;
@@ -145,7 +146,11 @@ static struct trapped_io *lookup_tiop(unsigned long address)if (!pgd_present (*pgd_k))return NULL ; - pud_k =pud_offset (pgd_k, address); + p4d_k =p4d_offset (pgd_k, address); +if (!p4d_present (*p4d_k)) +return NULL ; + + pud_k =pud_offset (p4d_k, address);if (!pud_present (*pud_k))return NULL ;
arch/sh/mm/cache-sh4.c +3 -1
@@ -209,6 +209,7 @@ static void sh4_flush_cache_page(void *args)unsigned long address, pfn, phys;int map_coherent =0 ;pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -224,7 +225,8 @@ static void sh4_flush_cache_page(void *args)return ; pgd =pgd_offset (vma->vm_mm, address); - pud =pud_offset (pgd, address); + p4d =p4d_offset (pgd, address); + pud =pud_offset (p4d, address); pmd =pmd_offset (pud, address); pte =pte_offset_kernel (pmd, address);
arch/sh/mm/cache-sh5.c +6 -1
@@ -383,6 +383,7 @@ static void sh64_dcache_purge_user_pages(struct mm_struct *mm, unsigned long addr, unsigned long end) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -397,7 +398,11 @@ static void sh64_dcache_purge_user_pages(struct mm_struct *mm,if (pgd_bad (*pgd))return ; - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +if (p4d_none (*p4d) ||p4d_bad (*p4d)) +return ; + + pud =pud_offset (p4d, addr);if (pud_none (*pud) ||pud_bad (*pud))return ;
arch/sh/mm/fault.c +23 -3
@@ -53,6 +53,7 @@ static void show_pte(struct mm_struct *mm, unsigned long addr) (u64)pgd_val(*pgd)); do { + p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte;
@@ -65,7 +66,20 @@ static void show_pte(struct mm_struct *mm, unsigned long addr)break ; } - pud =pud_offset (pgd, addr); + p4d =p4d_offset (pgd, addr); +if (PTRS_PER_P4D !=1 ) +pr_cont (", *p4d=%0*Lx" , (u32)(sizeof (*p4d) *2 ), + (u64 )p4d_val (*p4d)); + +if (p4d_none (*p4d)) +break ; + +if (p4d_bad (*p4d)) { +pr_cont ("(bad)" ); +break ; + } + + pud =pud_offset (p4d, addr);if (PTRS_PER_PUD !=1 )pr_cont (", *pud=%0*llx" , (u32)(sizeof (*pud) *2 ), (u64 )pud_val (*pud));
@@ -107,6 +121,7 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address) {unsigned index =pgd_index (address);pgd_t *pgd_k; +p4d_t *p4d, *p4d_k;pud_t *pud, *pud_k;pmd_t *pmd, *pmd_k;
@@ -116,8 +131,13 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)if (!pgd_present (*pgd_k))return NULL ; - pud =pud_offset (pgd, address); - pud_k =pud_offset (pgd_k, address); + p4d =p4d_offset (pgd, address); + p4d_k =p4d_offset (pgd_k, address); +if (!p4d_present (*p4d_k)) +return NULL ; + + pud =pud_offset (p4d, address); + pud_k =pud_offset (p4d_k, address);if (!pud_present (*pud_k))return NULL ;
arch/sh/mm/hugetlbpage.c +18 -10
@@ -26,17 +26,21 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte =NULL ; pgd =pgd_offset (mm, addr);if (pgd) { - pud =pud_alloc (mm, pgd, addr); -if (pud) { - pmd =pmd_alloc (mm, pud, addr); -if (pmd) - pte =pte_alloc_map (mm, pmd, addr); + p4d =p4d_alloc (mm, pgd, addr); +if (p4d) { + pud =pud_alloc (mm, p4d, addr); +if (pud) { + pmd =pmd_alloc (mm, pud, addr); +if (pmd) + pte =pte_alloc_map (mm, pmd, addr); + } } }
@@ -47,17 +51,21 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte =NULL ; pgd =pgd_offset (mm, addr);if (pgd) { - pud =pud_offset (pgd, addr); -if (pud) { - pmd =pmd_offset (pud, addr); -if (pmd) - pte =pte_offset_map (pmd, addr); + p4d =p4d_offset (pgd, addr); +if (p4d) { + pud =pud_offset (p4d, addr); +if (pud) { + pmd =pmd_offset (pud, addr); +if (pmd) + pte =pte_offset_map (pmd, addr); + } } }
arch/sh/mm/init.c +8 -1
@@ -45,6 +45,7 @@ void __init __weak plat_mem_setup(void)static pte_t *__get_pte_phys (unsigned long addr) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;
@@ -54,7 +55,13 @@ static pte_t *__get_pte_phys(unsigned long addr)return NULL ; } - pud =pud_alloc (NULL , pgd, addr); + p4d =p4d_alloc (NULL , pgd, addr); +if (unlikely (!p4d)) { +p4d_ERROR (*p4d); +return NULL ; + } + + pud =pud_alloc (NULL , p4d, addr);if (unlikely (!pud)) {pud_ERROR (*pud);return NULL ;
arch/sh/mm/kmap.c +1 -1
@@ -15,7 +15,7 @@#include <asm/cacheflush.h> #define kmap_get_fixmap_pte (vaddr) \ - pte_offset_kernel(pmd_offset(pud_offset(pgd_offset_k(vaddr), (vaddr)), (vaddr)), (vaddr)) + pte_offset_kernel(pmd_offset(pud_offset(p4d_offset(pgd_offset_k(vaddr), (vaddr)), (vaddr)), (vaddr)), vaddr)static pte_t *kmap_coherent_pte;
arch/sh/mm/tlbex_32.c +5 -1
@@ -23,6 +23,7 @@ handle_tlbmiss(struct pt_regs *regs, unsigned long error_code, unsigned long address) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -42,7 +43,10 @@ handle_tlbmiss(struct pt_regs *regs, unsigned long error_code, pgd =pgd_offset (current->mm, address); } - pud =pud_offset (pgd, address); + p4d =p4d_offset (pgd, address); +if (p4d_none_or_clear_bad (p4d)) +return 1 ; + pud =pud_offset (p4d, address);if (pud_none_or_clear_bad (pud))return 1 ; pmd =pmd_offset (pud, address);
arch/sh/mm/tlbex_64.c +6 -1
@@ -44,6 +44,7 @@ static int handle_tlbmiss(unsigned long long protection_flags, unsigned long address) {pgd_t *pgd; +p4d_t *p4d;pud_t *pud;pmd_t *pmd;pte_t *pte;
@@ -58,7 +59,11 @@ static int handle_tlbmiss(unsigned long long protection_flags, pgd =pgd_offset (current->mm, address); } - pud =pud_offset (pgd, address); + p4d =p4d_offset (pgd, address); +if (p4d_none (*p4d) || !p4d_present (*p4d)) +return 1 ; + + pud =pud_offset (p4d, address);if (pud_none (*pud) || !pud_present (*pud))return 1 ;
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 11/13 unicore32: remove __ARCH_USE_5LEVEL_HACK
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
The unicore32 architecture has 2 level page tables and asm-generic/pgtable-nopmd.h and explicit casts from pud_t to pgd_t for page table folding.
Add p4d walk in the only place that actually unfolds the pud level and remove __ARCH_USE_5LEVEL_HACK.
--- arch/unicore32/include/asm/pgtable.h | 1 - arch/unicore32/kernel/hibernate.c | 4 +++- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/unicore32/include/asm/pgtable.h +0 -1
@@ -9,7 +9,6 @@#ifndef __UNICORE_PGTABLE_H__#define __UNICORE_PGTABLE_H__ -#define __ARCH_USE_5LEVEL_HACK#include <asm-generic/pgtable-nopmd.h> #include <asm/cpu-single.h>
arch/unicore32/kernel/hibernate.c +3 -1
@@ -33,9 +33,11 @@ struct swsusp_arch_regs swsusp_arch_regs_cpu0;static pmd_t *resume_one_md_table_init (pgd_t *pgd) {pud_t *pud; +p4d_t *p4d;pmd_t *pmd_table; - pud =pud_offset (pgd,0 ); + p4d =p4d_offset (pgd,0 ); + pud =pud_offset (p4d,0 ); pmd_table =pmd_offset (pud,0 );return pmd_table;
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 12/13 asm-generic: remove pgtable-nop4d-hack.h
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
No architecture defines __ARCH_USE_5LEVEL_HACK and therefore pgtable-nop4d-hack.h will be never actually included.
Remove it.
--- include/asm-generic/pgtable-nop4d-hack.h | 64 ------------------------ include/asm-generic/pgtable-nopud.h | 4 -- 2 files changed, 68 deletions(-) delete mode 100644 include/asm-generic/pgtable-nop4d-hack.h
include/asm-generic/pgtable-nop4d-hack.h deleted +0 -64
@@ -1,64 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _PGTABLE_NOP4D_HACK_H -#define _PGTABLE_NOP4D_HACK_H - -#ifndef __ASSEMBLY__ -#include <asm-generic/5level-fixup.h> - -#define __PAGETABLE_PUD_FOLDED 1 - -/* -* Having the pud type consist of a pgd gets the size right, and allows -* us to conceptually access the pgd entry that this pud is folded into -* without casting. -*/ -typedef struct {pgd_t pgd; }pud_t ; - -#define PUD_SHIFT PGDIR_SHIFT -#define PTRS_PER_PUD 1 -#define PUD_SIZE (1UL << PUD_SHIFT) -#define PUD_MASK (~(PUD_SIZE-1)) - -/* -* The "pgd_xxx()" functions here are trivial for a folded two-level -* setup: the pud is never bad, and a pud always exists (as it's folded -* into the pgd entry) -*/ -static inline int pgd_none (pgd_t pgd) {return 0 ; } -static inline int pgd_bad (pgd_t pgd) {return 0 ; } -static inline int pgd_present (pgd_t pgd) {return 1 ; } -static inline void pgd_clear (pgd_t *pgd) { } -#define pud_ERROR (pud) (pgd_ERROR((pud).pgd)) - -#define pgd_populate (mm, pgd, pud) do { } while (0) -#define pgd_populate_safe (mm, pgd, pud) do { } while (0) -/* -* (puds are folded into pgds so this doesn't get actually called, -* but the define is needed for a generic inline function.) -*/ -#define set_pgd (pgdptr, pgdval) set_pud((pud_t *)(pgdptr), (pud_t) { pgdval }) - -static inline pud_t *pud_offset (pgd_t *pgd,unsigned long address) -{ -return (pud_t *)pgd; -} - -#define pud_val (x) (pgd_val((x).pgd)) -#define __pud (x) ((pud_t) { __pgd(x) }) - -#define pgd_page (pgd) (pud_page((pud_t){ pgd })) -#define pgd_page_vaddr (pgd) (pud_page_vaddr((pud_t){ pgd })) - -/* -* allocating and freeing a pud is trivial: the 1-entry pud is -* inside the pgd, so has no extra memory associated with it. -*/ -#define pud_alloc_one (mm, address) NULL -#define pud_free (mm, x) do { } while (0) -#define __pud_free_tlb (tlb, x, a) do { } while (0) - -#undef pud_addr_end -#define pud_addr_end (addr, end) (end) - -#endif /* __ASSEMBLY__ */ -#endif /* _PGTABLE_NOP4D_HACK_H */
include/asm-generic/pgtable-nopud.h +0 -4
@@ -4,9 +4,6 @@#ifndef __ASSEMBLY__ -#ifdef __ARCH_USE_5LEVEL_HACK -#include <asm-generic/pgtable-nop4d-hack.h> -#else #include <asm-generic/pgtable-nop4d.h> #define __PAGETABLE_PUD_FOLDED 1
@@ -65,5 +62,4 @@ static inline pud_t *pud_offset(p4d_t *p4d, unsigned long address)#define pud_addr_end (addr, end) (end)#endif /* __ASSEMBLY__ */ -#endif /* !__ARCH_USE_5LEVEL_HACK */ #endif /* _PGTABLE_NOPUD_H */
Signature · 1 line
2.24.0
Mike Rapoport
reviewerPATCH v2 13/13 mm: remove __ARCH_HAS_5LEVEL_HACK and include/asm-generic/5level-fixup.h
In reply to Mike RapoportFrom: Mike Rapoport <[email protected]>
There are no architectures that use include/asm-generic/5level-fixup.h therefore it can be removed along with __ARCH_HAS_5LEVEL_HACK define and the code it surrounds
--- include/asm-generic/5level-fixup.h | 58 ------------------------------ include/linux/mm.h | 6 ---- mm/kasan/init.c | 11 ------ mm/memory.c | 8 ----- 4 files changed, 83 deletions(-) delete mode 100644 include/asm-generic/5level-fixup.h
include/asm-generic/5level-fixup.h deleted +0 -58
@@ -1,58 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _5LEVEL_FIXUP_H -#define _5LEVEL_FIXUP_H - -#define __ARCH_HAS_5LEVEL_HACK -#define __PAGETABLE_P4D_FOLDED 1 - -#define P4D_SHIFT PGDIR_SHIFT -#define P4D_SIZE PGDIR_SIZE -#define P4D_MASK PGDIR_MASK -#define MAX_PTRS_PER_P4D 1 -#define PTRS_PER_P4D 1 - -#define p4d_t pgd_t - -#define pud_alloc (mm, p4d, address) \ - ((unlikely(pgd_none(*(p4d))) && __pud_alloc(mm, p4d, address)) ? \ - NULL : pud_offset(p4d, address)) - -#define p4d_alloc (mm, pgd, address) (pgd) -#define p4d_offset (pgd, start) (pgd) - -#ifndef __ASSEMBLY__ -static inline int p4d_none (p4d_t p4d) -{ -return 0 ; -} - -static inline int p4d_bad (p4d_t p4d) -{ -return 0 ; -} - -static inline int p4d_present (p4d_t p4d) -{ -return 1 ; -} -#endif - -#define p4d_ERROR (p4d) do { } while (0) -#define p4d_clear (p4d) pgd_clear(p4d) -#define p4d_val (p4d) pgd_val(p4d) -#define p4d_populate (mm, p4d, pud) pgd_populate(mm, p4d, pud) -#define p4d_populate_safe (mm, p4d, pud) pgd_populate(mm, p4d, pud) -#define p4d_page (p4d) pgd_page(p4d) -#define p4d_page_vaddr (p4d) pgd_page_vaddr(p4d) - -#define __p4d (x) __pgd(x) -#define set_p4d (p4dp, p4d) set_pgd(p4dp, p4d) - -#undef p4d_free_tlb -#define p4d_free_tlb (tlb, x, addr) do { } while (0) -#define p4d_free (mm, x) do { } while (0) - -#undef p4d_addr_end -#define p4d_addr_end (addr, end) (end) - -#endif
include/linux/mm.h +0 -6
@@ -1841,11 +1841,6 @@ int __pte_alloc_kernel(pmd_t *pmd);#if defined(CONFIG_MMU ) -/* - * The following ifdef needed to get the 5level-fixup.h header to work. - * Remove it when 5level-fixup.h has been removed. - */ -#ifndef __ARCH_HAS_5LEVEL_HACKstatic inline p4d_t *p4d_alloc (struct mm_struct *mm,pgd_t *pgd,unsigned long address) {
@@ -1859,7 +1854,6 @@ static inline pud_t *pud_alloc(struct mm_struct *mm, p4d_t *p4d,return (unlikely (p4d_none (*p4d)) &&__pud_alloc (mm, p4d, address)) ?NULL :pud_offset (p4d, address); } -#endif /* !__ARCH_HAS_5LEVEL_HACK */ static inline pmd_t *pmd_alloc (struct mm_struct *mm,pud_t *pud,unsigned long address) {
mm/kasan/init.c +0 -11
@@ -250,20 +250,9 @@ int __ref kasan_populate_early_shadow(const void *shadow_start,* 3,2 - level page tables where we don't have * puds,pmds, so pgd_populate(), pud_populate() * is noops. -* -* The ifndef is required to avoid build breakage. -* -* With 5level-fixup.h, pgd_populate() is not nop and -* we reference kasan_early_shadow_p4d. It's not defined -* unless 5-level paging enabled. -* -* The ifndef can be dropped once all KASAN-enabled -* architectures will switch to pgtable-nop4d.h. */ -#ifndef __ARCH_HAS_5LEVEL_HACKpgd_populate (&init_mm, pgd,lm_alias (kasan_early_shadow_p4d)); -#endif p4d =p4d_offset (pgd, addr);p4d_populate (&init_mm, p4d,lm_alias (kasan_early_shadow_pud));
mm/memory.c +0 -8
@@ -4252,19 +4252,11 @@ int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address)smp_wmb ();/* See comment in __pte_alloc */ spin_lock (&mm->page_table_lock); -#ifndef __ARCH_HAS_5LEVEL_HACKif (!p4d_present (*p4d)) {mm_inc_nr_puds (mm);p4d_populate (mm, p4d, new); }else /* Another has populated it */ pud_free (mm, new); -#else -if (!pgd_present (*p4d)) { -mm_inc_nr_puds (mm); -pgd_populate (mm, p4d, new); - }else /* Another has populated it */ -pud_free (mm, new); -#endif /* __ARCH_HAS_5LEVEL_HACK */ spin_unlock (&mm->page_table_lock);return 0 ; }
Signature · 1 line
2.24.0
Russell King - ARM Linux admin
maintainerQuoted 4 lines · Mike Rapoport
From: Mike Rapoport <[email protected]>
Hi,
Quoted 8 lines
These patches convert several architectures to use page table folding and remove
__ARCH_HAS_5LEVEL_HACKalong withinclude/asm-generic/5level-fixup.h.The changes are mostly about mechanical replacement of pgd accessors with p4d ones and the addition of higher levels to page table traversals.
All the patches were sent separately to the respective arch lists and maintainers hence the "v2" prefix.
You fail to explain why this change which adds 488 additional lines of code is desirable.
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
Le 16/02/2020 à 09:22, Russell King - ARM Linux admin a écrit :
Quoted 8 lines · Mike Rapoport
Quoted 7 lines
From: Mike Rapoport <[email protected]>
Hi,
These patches convert several architectures to use page table folding and remove
__ARCH_HAS_5LEVEL_HACKalong withinclude/asm-generic/5level-fixup.h.
Quoted 9 lines
Quoted 5 lines
The changes are mostly about mechanical replacement of pgd accessors with p4d ones and the addition of higher levels to page table traversals.
All the patches were sent separately to the respective arch lists and maintainers hence the "v2" prefix.
You fail to explain why this change which adds 488 additional lines of code is desirable.
The purpose of the series, ie droping a HACK, is worth it.
However looking at the powerpc patch I have the feeling that this series goes behind its purpose.
The number additional lines could be deeply reduced I think if we limit the patches to the strict minimum, ie just do things like below instead of adding lots of handling of useless levels.
Instead of doing things like:
- pud = NULL;
+ p4d = NULL;
if (pgd_present(*pgd))
- pud = pud_offset(pgd, gpa);
+ p4d = p4d_offset(pgd, gpa);
+ else
+ new_p4d = p4d_alloc_one(kvm->mm, gpa);
+
+ pud = NULL;
+ if (p4d_present(*p4d))
+ pud = pud_offset(p4d, gpa);
else
new_pud = pud_alloc_one(kvm->mm, gpa);
It could be limited to:
if (pgd_present(*pgd))
- pud = pud_offset(pgd, gpa);
+ pud = pud_offset(p4d_offset(pgd, gpa), gpa);
else
new_pud = pud_alloc_one(kvm->mm, gpa);
Christophe
Mike Rapoport
reviewerQuoted 8 lines · Russell King - ARM Linux admin
Quoted 7 lines · Mike Rapoport
From: Mike Rapoport <[email protected]>
Hi,
These patches convert several architectures to use page table folding and remove
__ARCH_HAS_5LEVEL_HACKalong withinclude/asm-generic/5level-fixup.h.
Quoted 8 lines
Quoted 5 lines
The changes are mostly about mechanical replacement of pgd accessors with p4d ones and the addition of higher levels to page table traversals.
All the patches were sent separately to the respective arch lists and maintainers hence the "v2" prefix.
You fail to explain why this change which adds 488 additional lines of code is desirable.
Right, I should have been more explicit about it.
As Christophe mentioned in his reply, removing 'HACK' and 'fixup' is an improvement.
Another thing is that when all architectures behave the same it opens opportunities for cleaning up repeating definitions of page table manipulation primitives.
Quoted 4 lines
Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
Signature · 2 lines
Sincerely yours, Mike.