Implement use of HW assistance on TLB table walk on 8xx
Subsystems (2)
v8 cover Implement use of HW assistance on TLB table walk on 8xx
The purpose of this serie is to implement hardware assistance for TLB table walk on the 8xx.
First part prepares for using HW assistance in TLB routines:
- Trivial fixes:
- Remove CONFIG_BOOKE stuff from book3S headers.
- Removal of unneeded atomic PTE update requirement for 8xx.
- move book3s64 page fragment code in a common part for reusing it by the
8xx as 16k page size mode still uses 4k page tables.
- Fixing a bug in memcache handling when standard pages and hugepages share caches of the same size (see original discussion in https://patchwork.ozlabs.org/patch/957565/)
- Optimise access to 8xx perf counters (hence reducing number of registers used)
Second part implements HW assistance in TLB routines in the following steps:
- Disable 16k page size mode and 512k hugepages
- Switch 4k to HW assistance
- Bring back 512k hugepages
- Bring back 16k page size mode.
Last part cleans up:
- Take benefit of Miss handler size reduction to regroup related parts
- Reduce number of registers used in miss handlers, freeing them for future use.
Tested successfully on 8xx and 83xx (book3s/32)
Changes in v8:
- Moved definitions in pgalloc.h to avoid conflicting with memcache patches.
- Included the memcache bugfix serie in this serie to avoid conflicts between the two series when coming to the 512k pages patch.
- In the 512k HW assistance patch, reduced the #ifdef mess by using IS_ENABLED(CONFIG_PPC_8xx) instead.
Changes in v7:
- Reordered to get trivial and already reviewed patches in front.
- Reordered to regroup all HW assistance related patches together.
- Rebased on today merge branch (28 Nov)
- Added a helper for access to mm_context_t.frag
- Reduced the amount of changes in PPC32 to support pte_fragment
- Applied pte_fragment to both nohash/32 and book3s/32
Changes in v6:
- Droped the part related to handling GUARD attribute at PGD/PMD level.
- Moved the commonalisation of page_fragment in the begining (this part has been reviewed by Aneesh)
- Rebased on today merge branch (19 Oct)
Changes in v5:
- Also avoid useless lock in get_pmd_from_cache()
- A new patch to relocate mmu headers in platform specific directories
- A new patch to distribute pgtable_t typedefs in platform specific mmu headers instead of the uggly #ifdef
- Moved early_pte_alloc_kernel() in platform specific pgalloc
- Restricted definition of PTE_FRAG_SIZE and PTE_FRAG_NR to platforms using the pte fragmentation.
- arch_exit_mmap() and destroy_pagetable_cache() are now platform specific.
Changes in v4:
- Reordered the serie to put at the end the modifications which makes L1 and L2 entries independant.
- No modifications to ppc64 ioremap (we still have an opportunity to merge them, for a future patch serie)
- 8xx code modified to use patch_site instead of patch_instruction
to get a clearer code and avoid object pollution with global symbols
- Moved perf counters in first 32kb of memory to optimise access
- Split the big bang to HW assistance in several steps:
1. Temporarily removes support of 16k pages and 512k hugepages
2. Change TLB routines to use HW assistance for 4k pages and 8M hugepages
3. Add back support for 512k hugepages
4. Add back support for 16k pages (using pte_fragment as page tables are still 4k)
Changes in v3:
- Fixed an issue in the 09/14 when CONFIG_PIN_TLB_TEXT was not enabled
- Added performance measurement in the 09/14 commit log
- Rebased on latest 'powerpc/merge' tree, which conflicted with 13/14
Changes in v2:
- Removed the 3 first patchs which have been applied already
- Fixed compilation errors reported by Michael
- Squashed the commonalisation of ioremap functions into a single patch
- Fixed the use of pte_fragment
- Added a patch optimising perf counting of TLB misses and instructions
Christophe Leroy (20):
powerpc/book3s32: Remove CONFIG_BOOKE dependent code powerpc/8xx: Remove PTE_ATOMIC_UPDATES
powerpc/mm: Move pte_fragment_alloc() to a common location powerpc/mm: Avoid useless lock with single page fragments powerpc/mm: move platform specific mmu-xxx.h in platform directories powerpc/mm: Move pgtable_t into platform headers powerpc/mm: add helpers to get/set mm.context->pte_frag powerpc/mm: Extend pte_fragment functionality to PPC32 powerpc/mm: enable the use of page table cache of order 0 powerpc/mm: replace hugetlb_cache by PGT_CACHE(PTE_T_ORDER) powerpc/mm: fix a warning when a cache is common to PGD and hugepages powerpc/mm: remove unnecessary test in pgtable_cache_init() powerpc/8xx: Move SW perf counters in first 32kb of memory powerpc/8xx: Temporarily disable 16k pages and hugepages powerpc/8xx: Use hardware assistance in TLB handlers powerpc/8xx: Enable 8M hugepage support with HW assistance powerpc/8xx: Enable 512k hugepage support with HW assistance powerpc/8xx: reintroduce 16K pages with HW assistance powerpc/8xx: don't use r12/SPRN_SPRG_SCRATCH2 in TLB Miss handlers powerpc/8xx: regroup TLB handler routines
arch/powerpc/include/asm/book3s/32/mmu-hash.h | 5 +
arch/powerpc/include/asm/book3s/32/pgalloc.h | 40 +--
arch/powerpc/include/asm/book3s/32/pgtable.h | 19 +-
arch/powerpc/include/asm/book3s/64/mmu.h | 9 +
arch/powerpc/include/asm/book3s/64/pgalloc.h | 6 +-
arch/powerpc/include/asm/hugetlb.h | 6 +-
arch/powerpc/include/asm/mmu.h | 14 +-
arch/powerpc/include/asm/mmu_context.h | 2 +-
arch/powerpc/include/asm/{ => nohash/32}/mmu-40x.h | 0
arch/powerpc/include/asm/{ => nohash/32}/mmu-44x.h | 0
arch/powerpc/include/asm/{ => nohash/32}/mmu-8xx.h | 1 +
arch/powerpc/include/asm/nohash/32/mmu.h | 25 ++
arch/powerpc/include/asm/nohash/32/pgalloc.h | 27 +-
arch/powerpc/include/asm/nohash/32/pgtable.h | 27 +-
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 3 -
arch/powerpc/include/asm/nohash/64/mmu.h | 12 +
arch/powerpc/include/asm/nohash/64/pgalloc.h | 5 +-
arch/powerpc/include/asm/{ => nohash}/mmu-book3e.h | 0
arch/powerpc/include/asm/nohash/mmu.h | 11 +
arch/powerpc/include/asm/nohash/pgtable.h | 4 +
arch/powerpc/include/asm/page.h | 14 -
arch/powerpc/include/asm/pgtable-types.h | 4 +
arch/powerpc/include/asm/pgtable.h | 33 +-
arch/powerpc/kernel/cpu_setup_fsl_booke.S | 2 +-
arch/powerpc/kernel/head_8xx.S | 366 ++++++++-------------
arch/powerpc/kvm/e500.h | 2 +-
arch/powerpc/mm/8xx_mmu.c | 4 +-
arch/powerpc/mm/Makefile | 5 +-
arch/powerpc/mm/hugetlbpage.c | 40 +--
arch/powerpc/mm/init-common.c | 56 ++--
arch/powerpc/mm/mmu_context.c | 10 +
arch/powerpc/mm/mmu_context_book3s64.c | 15 -
arch/powerpc/mm/mmu_context_nohash.c | 2 +-
arch/powerpc/mm/pgtable-book3s64.c | 88 +----
arch/powerpc/mm/pgtable-frag.c | 119 +++++++
arch/powerpc/mm/pgtable_32.c | 25 +-
36 files changed, 497 insertions(+), 504 deletions(-)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-40x.h (100%)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-44x.h (100%)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-8xx.h (99%)
create mode 100644 arch/powerpc/include/asm/nohash/32/mmu.h
create mode 100644 arch/powerpc/include/asm/nohash/64/mmu.h
rename arch/powerpc/include/asm/{ => nohash}/mmu-book3e.h (100%)
create mode 100644 arch/powerpc/include/asm/nohash/mmu.h
create mode 100644 arch/powerpc/mm/pgtable-frag.cSignature · 1 line
2.13.3
PATCH v8 1/20 powerpc/book3s32: Remove CONFIG_BOOKE dependent code
BOOK3S/32 cannot be BOOKE, so remove useless code
--- arch/powerpc/include/asm/book3s/32/pgalloc.h | 18 ------------------ arch/powerpc/include/asm/book3s/32/pgtable.h | 14 -------------- 2 files changed, 32 deletions(-)
arch/powerpc/include/asm/book3s/32/pgalloc.h +0 -18
@@ -50,8 +50,6 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)#define __pmd_free_tlb (tlb,x,a) do { } while (0)/* #define pgd_populate(mm, pmd, pte) BUG() */ -#ifndef CONFIG_BOOKE -static inline void pmd_populate_kernel (struct mm_struct *mm,pmd_t *pmdp,pte_t *pte) {
@@ -65,22 +63,6 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, }#define pmd_pgtable (pmd) pmd_page(pmd) -#else - -static inline void pmd_populate_kernel (struct mm_struct *mm,pmd_t *pmdp, -pte_t *pte) -{ - *pmdp =__pmd ((unsigned long )pte | _PMD_PRESENT); -} - -static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmdp, -pgtable_t pte_page) -{ - *pmdp =__pmd ((unsigned long )lowmem_page_address (pte_page) | _PMD_PRESENT); -} - -#define pmd_pgtable (pmd) pmd_page(pmd) -#endif extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr);extern pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr);
arch/powerpc/include/asm/book3s/32/pgtable.h +0 -14
@@ -328,24 +328,10 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma,#define __HAVE_ARCH_PTE_SAME#define pte_same (A ,B ) (((pte_val(A) ^ pte_val(B)) & ~_PAGE_HASHPTE) == 0) -/* -* Note that on Book E processors, the pmd contains the kernel virtual -* (lowmem) address of the pte page. The physical address is less useful -* because everything runs with translation enabled (even the TLB miss -* handler). On everything else the pmd contains the physical address -* of the pte page. -- paulus -*/ -#ifndef CONFIG_BOOKE #define pmd_page_vaddr (pmd) \ ((unsigned long) __va(pmd_val(pmd) & PAGE_MASK))#define pmd_page (pmd) \ pfn_to_page(pmd_val(pmd) >> PAGE_SHIFT) -#else -#define pmd_page_vaddr (pmd) \ - ((unsigned long) (pmd_val(pmd) & PAGE_MASK)) -#define pmd_page (pmd) \ - pfn_to_page((__pa(pmd_val(pmd)) >> PAGE_SHIFT)) -#endif /* to find an entry in a kernel page-table-directory */ #define pgd_offset_k (address) pgd_offset(&init_mm, address)
Signature · 1 line
2.13.3
PATCH v8 2/20 powerpc/8xx: Remove PTE_ATOMIC_UPDATES
In reply to Christophe Leroycommit 1bc54c03117b9 ("powerpc: rework 4xx PTE access and TLB miss") introduced non atomic PTE updates and started the work of removing PTE updates in TLB miss handlers, but kept PTE_ATOMIC_UPDATES for the 8xx with the following comment:
/* Until my rework is finished, 8xx still needs atomic PTE updates */
commit fe11dc3f9628e ("powerpc/8xx: Update TLB asm so it behaves as linux mm expects") removed all PTE updates done in TLB miss handlers
Therefore, atomic PTE updates are not needed anymore for the 8xx
--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 3 --- 1 file changed, 3 deletions(-)
arch/powerpc/include/asm/nohash/32/pte-8xx.h +0 -3
@@ -65,9 +65,6 @@#define _PTE_NONE_MASK 0 -/* Until my rework is finished, 8xx still needs atomic PTE updates */ -#define PTE_ATOMIC_UPDATES 1 -#ifdef CONFIG_PPC_16K_PAGES #define _PAGE_PSIZE _PAGE_SPS#else
Signature · 1 line
2.13.3
PATCH v8 3/20 powerpc/mm: Move pte_fragment_alloc() to a common location
In reply to Christophe LeroyIn preparation of next patch which generalises the use of pte_fragment_alloc() for all, this patch moves the related functions in a place that is common to all subarches.
The 8xx will need that for supporting 16k pages, as in that mode page tables still have a size of 4k.
Since pte_fragment with only once fragment is not different from what is done in the general case, we can easily migrate all subarchs to pte fragments.
--- arch/powerpc/include/asm/book3s/64/pgalloc.h | 1 + arch/powerpc/mm/Makefile | 4 +- arch/powerpc/mm/mmu_context_book3s64.c | 15 ---- arch/powerpc/mm/pgtable-book3s64.c | 85 -------------------- arch/powerpc/mm/pgtable-frag.c | 116 +++++++++++++++++++++++++++ 5 files changed, 120 insertions(+), 101 deletions(-) create mode 100644 arch/powerpc/mm/pgtable-frag.c
arch/powerpc/include/asm/book3s/64/pgalloc.h +1 -0
@@ -50,6 +50,7 @@ extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift);#ifdef CONFIG_SMP extern void __tlb_remove_table (void *_table);#endif +void pte_frag_destroy (void *pte_frag);static inline pgd_t *radix__pgd_alloc (struct mm_struct *mm) {
arch/powerpc/mm/Makefile +3 -1
@@ -15,7 +15,9 @@ obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \ obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o hash64-$(CONFIG_PPC_NATIVE) := hash_native_64.o obj-$(CONFIG_PPC_BOOK3E_64) += pgtable-book3e.o -obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o +obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb.o \ + $(hash64-y) mmu_context_book3s64.o \ + pgtable-book3s64.o pgtable-frag.o obj-$(CONFIG_PPC_RADIX_MMU) += pgtable-radix.o tlb-radix.o obj-$(CONFIG_PPC_STD_MMU_32) += ppc_mmu_32.o hash_low_32.o mmu_context_hash32.o obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(BITS).o
arch/powerpc/mm/mmu_context_book3s64.c +0 -15
@@ -164,21 +164,6 @@ static void destroy_contexts(mm_context_t *ctx) } } -static void pte_frag_destroy (void *pte_frag) -{ -int count; -struct page *page; - - page =virt_to_page (pte_frag); -/* drop all the pending references */ - count = ((unsigned long )pte_frag & ~PAGE_MASK ) >>PTE_FRAG_SIZE_SHIFT ; -/* We allow PTE_FRAG_NR fragments from a PTE page */ -if (atomic_sub_and_test (PTE_FRAG_NR - count, &page->pt_frag_refcount)) { -pgtable_page_dtor (page); -__free_page (page); - } -} -static void pmd_frag_destroy (void *pmd_frag) {int count;
arch/powerpc/mm/pgtable-book3s64.c +0 -85
91 lines not on this page · raw message
arch/powerpc/mm/pgtable-frag.c new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.13.3
PATCH v8 4/20 powerpc/mm: Avoid useless lock with single page fragments
In reply to Christophe LeroyThere is no point in taking the page table lock as pte_frag or pmd_frag are always NULL when we have only one fragment.
--- arch/powerpc/mm/pgtable-book3s64.c | 3 +++ arch/powerpc/mm/pgtable-frag.c | 3 +++ 2 files changed, 6 insertions(+)
arch/powerpc/mm/pgtable-book3s64.c +3 -0
@@ -244,6 +244,9 @@ static pmd_t *get_pmd_from_cache(struct mm_struct *mm) {void *pmd_frag, *ret; +if (PMD_FRAG_NR ==1 ) +return NULL ; +spin_lock (&mm->page_table_lock); ret = mm->context.pmd_frag;if (ret) {
arch/powerpc/mm/pgtable-frag.c +3 -0
@@ -34,6 +34,9 @@ static pte_t *get_pte_from_cache(struct mm_struct *mm) {void *pte_frag, *ret; +if (PTE_FRAG_NR ==1 ) +return NULL ; +spin_lock (&mm->page_table_lock); ret = mm->context.pte_frag;if (ret) {
Signature · 1 line
2.13.3
PATCH v8 5/20 powerpc/mm: move platform specific mmu-xxx.h in platform directories
In reply to Christophe LeroyThe purpose of this patch is to move platform specific mmu-xxx.h files in platform directories like pte-xxx.h files.
In the meantime this patch creates common nohash and nohash/32 + nohash/64 mmu.h files for future common parts.
---
arch/powerpc/include/asm/mmu.h | 14 ++------------
arch/powerpc/include/asm/{ => nohash/32}/mmu-40x.h | 0
arch/powerpc/include/asm/{ => nohash/32}/mmu-44x.h | 0
arch/powerpc/include/asm/{ => nohash/32}/mmu-8xx.h | 0
arch/powerpc/include/asm/nohash/32/mmu.h | 19 +++++++++++++++++++
arch/powerpc/include/asm/nohash/64/mmu.h | 8 ++++++++
arch/powerpc/include/asm/{ => nohash}/mmu-book3e.h | 0
arch/powerpc/include/asm/nohash/mmu.h | 11 +++++++++++
arch/powerpc/kernel/cpu_setup_fsl_booke.S | 2 +-
arch/powerpc/kvm/e500.h | 2 +-
10 files changed, 42 insertions(+), 14 deletions(-)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-40x.h (100%)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-44x.h (100%)
rename arch/powerpc/include/asm/{ => nohash/32}/mmu-8xx.h (100%)
create mode 100644 arch/powerpc/include/asm/nohash/32/mmu.h
create mode 100644 arch/powerpc/include/asm/nohash/64/mmu.h
rename arch/powerpc/include/asm/{ => nohash}/mmu-book3e.h (100%)
create mode 100644 arch/powerpc/include/asm/nohash/mmu.harch/powerpc/include/asm/mmu.h +2 -12
@@ -341,18 +341,8 @@ static inline void mmu_early_init_devtree(void) { }#if defined(CONFIG_PPC_STD_MMU_32 )/* 32-bit classic hash table MMU */ #include <asm/book3s/32/mmu-hash.h> -#elif defined(CONFIG_40x) -/* 40x-style software loaded TLB */ -# include <asm/mmu-40x.h> -#elif defined(CONFIG_44x) -/* 44x-style software loaded TLB */ -# include <asm/mmu-44x.h> -#elif defined(CONFIG_PPC_BOOK3E_MMU ) -/* Freescale Book-E software loaded TLB or Book-3e (ISA 2.06+) MMU */ -# include <asm/mmu-book3e.h> -#elif defined (CONFIG_PPC_8xx) -/* Motorola/Freescale 8xx software loaded TLB */ -# include <asm/mmu-8xx.h> +#elif defined(CONFIG_PPC_MMU_NOHASH ) +#include <asm/nohash/mmu.h> #endif #endif /* __KERNEL__ */
arch/powerpc/include/asm/nohash/32/mmu-40x.h renamed from arch/powerpc/include/asm/mmu-40x.h +0 -0
arch/powerpc/include/asm/nohash/32/mmu-44x.h renamed from arch/powerpc/include/asm/mmu-44x.h +0 -0
arch/powerpc/include/asm/nohash/32/mmu-8xx.h renamed from arch/powerpc/include/asm/mmu-8xx.h +0 -0
arch/powerpc/include/asm/nohash/32/mmu.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_POWERPC_NOHASH_32_MMU_H_ +#define _ASM_POWERPC_NOHASH_32_MMU_H_ + +#if defined(CONFIG_40x) +/* 40x-style software loaded TLB */ +#include <asm/nohash/32/mmu-40x.h> +#elif defined(CONFIG_44x) +/* 44x-style software loaded TLB */ +#include <asm/nohash/32/mmu-44x.h> +#elif defined(CONFIG_PPC_BOOK3E_MMU ) +/* Freescale Book-E software loaded TLB or Book-3e (ISA 2.06+) MMU */ +#include <asm/nohash/mmu-book3e.h> +#elif defined (CONFIG_PPC_8xx) +/* Motorola/Freescale 8xx software loaded TLB */ +#include <asm/nohash/32/mmu-8xx.h> +#endif + +#endif /* _ASM_POWERPC_NOHASH_32_MMU_H_ */
arch/powerpc/include/asm/nohash/64/mmu.h new file +8 -0
@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_POWERPC_NOHASH_64_MMU_H_ +#define _ASM_POWERPC_NOHASH_64_MMU_H_ + +/* Freescale Book-E software loaded TLB or Book-3e (ISA 2.06+) MMU */ +#include <asm/nohash/mmu-book3e.h> + +#endif /* _ASM_POWERPC_NOHASH_64_MMU_H_ */
arch/powerpc/include/asm/nohash/mmu-book3e.h renamed from arch/powerpc/include/asm/mmu-book3e.h +0 -0
arch/powerpc/include/asm/nohash/mmu.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_POWERPC_NOHASH_MMU_H_ +#define _ASM_POWERPC_NOHASH_MMU_H_ + +#ifdef CONFIG_PPC64 +#include <asm/nohash/64/mmu.h> +#else +#include <asm/nohash/32/mmu.h> +#endif + +#endif /* _ASM_POWERPC_NOHASH_MMU_H_ */
arch/powerpc/kernel/cpu_setup_fsl_booke.S +1 -1
@@ -17,7 +17,7 @@ #include <asm/processor.h> #include <asm/cputable.h> #include <asm/ppc_asm.h> -#include <asm/mmu-book3e.h> +#include <asm/nohash/mmu-book3e.h> #include <asm/asm-offsets.h> #include <asm/mpc85xx.h>
arch/powerpc/kvm/e500.h +1 -1
@@ -20,7 +20,7 @@#define KVM_E500_H #include <linux/kvm_host.h> -#include <asm/mmu-book3e.h> +#include <asm/nohash/mmu-book3e.h> #include <asm/tlb.h> #include <asm/cputhreads.h>
Signature · 1 line
2.13.3
PATCH v8 6/20 powerpc/mm: Move pgtable_t into platform headers
In reply to Christophe LeroyThis patch move pgtable_t into platform headers.
It gets rid of the CONFIG_PPC_64K_PAGES case for PPC64 as nohash/64 doesn't support CONFIG_PPC_64K_PAGES.
--- arch/powerpc/include/asm/book3s/32/mmu-hash.h | 2 ++ arch/powerpc/include/asm/book3s/64/mmu.h | 9 +++++++++ arch/powerpc/include/asm/nohash/32/mmu.h | 4 ++++ arch/powerpc/include/asm/nohash/64/mmu.h | 4 ++++ arch/powerpc/include/asm/page.h | 14 -------------- 5 files changed, 19 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/book3s/32/mmu-hash.h +2 -0
@@ -42,6 +42,8 @@ struct ppc_bat {u32 batu; u32 batl; }; + +typedef struct page *pgtable_t ;#endif /* !__ASSEMBLY__ */ /*
arch/powerpc/include/asm/book3s/64/mmu.h +9 -0
@@ -2,6 +2,8 @@#ifndef _ASM_POWERPC_BOOK3S_64_MMU_H_#define _ASM_POWERPC_BOOK3S_64_MMU_H_ +#include <asm/page.h> +#ifndef __ASSEMBLY__/* * Page size definition
@@ -24,6 +26,13 @@ struct mmu_psize_def { };extern struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ]; +/* +* For BOOK3s 64 with 4k and 64K linux page size +* we want to use pointers, because the page table +* actually store pfn +*/ +typedef pte_t *pgtable_t ; +#endif /* __ASSEMBLY__ */ /* 64-bit classic hash table MMU */
arch/powerpc/include/asm/nohash/32/mmu.h +4 -0
@@ -16,4 +16,8 @@#include <asm/nohash/32/mmu-8xx.h> #endif +#ifndef __ASSEMBLY__ +typedef struct page *pgtable_t ; +#endif +#endif /* _ASM_POWERPC_NOHASH_32_MMU_H_ */
arch/powerpc/include/asm/nohash/64/mmu.h +4 -0
@@ -5,4 +5,8 @@/* Freescale Book-E software loaded TLB or Book-3e (ISA 2.06+) MMU */ #include <asm/nohash/mmu-book3e.h> +#ifndef __ASSEMBLY__ +typedef struct page *pgtable_t ; +#endif +#endif /* _ASM_POWERPC_NOHASH_64_MMU_H_ */
arch/powerpc/include/asm/page.h +0 -14
@@ -335,20 +335,6 @@ void arch_free_page(struct page *page, int order);#endif struct vm_area_struct ; -#ifdef CONFIG_PPC_BOOK3S_64 -/* -* For BOOK3s 64 with 4k and 64K linux page size -* we want to use pointers, because the page table -* actually store pfn -*/ -typedef pte_t *pgtable_t ; -#else -#if defined(CONFIG_PPC_64K_PAGES ) && defined(CONFIG_PPC64 ) -typedef pte_t *pgtable_t ; -#else -typedef struct page *pgtable_t ; -#endif -#endif #include <asm-generic/memory_model.h> #endif /* __ASSEMBLY__ */
Signature · 1 line
2.13.3
PATCH v8 7/20 powerpc/mm: add helpers to get/set mm.context->pte_frag
In reply to Christophe LeroyIn order to handle pte_fragment functions with single fragment without adding pte_frag in all mm_context_t, this patch creates two helpers which do nothing on platforms using a single fragment.
--- arch/powerpc/include/asm/pgtable.h | 31 +++++++++++++++++++++++++++++++ arch/powerpc/mm/pgtable-frag.c | 8 ++++---- 2 files changed, 35 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/pgtable.h +31 -0
@@ -110,6 +110,37 @@ void mark_initmem_nx(void);static inline void mark_initmem_nx (void ) { }#endif +/* +* When used, PTE_FRAG_NR is defined in subarch pgtable.h +* so we are sure it is included when arriving here. +*/ +#ifndef PTE_FRAG_NR +#define PTE_FRAG_NR 1 +#define PTE_FRAG_SIZE_SHIFT PAGE_SHIFT +#define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT) +#endif + +#if PTE_FRAG_NR !=1 +static inline void *pte_frag_get (mm_context_t *ctx) +{ +return ctx->pte_frag; +} + +static inline void pte_frag_set (mm_context_t *ctx,void *p) +{ + ctx->pte_frag = p; +} +#else +static inline void *pte_frag_get (mm_context_t *ctx) +{ +return NULL ; +} + +static inline void pte_frag_set (mm_context_t *ctx,void *p) +{ +} +#endif +#endif /* __ASSEMBLY__ */ #endif /* _ASM_POWERPC_PGTABLE_H */
arch/powerpc/mm/pgtable-frag.c +4 -4
@@ -38,7 +38,7 @@ static pte_t *get_pte_from_cache(struct mm_struct *mm)return NULL ;spin_lock (&mm->page_table_lock); - ret = mm->context.pte_frag; + ret =pte_frag_get (&mm->context);if (ret) { pte_frag = ret +PTE_FRAG_SIZE ;/*
@@ -46,7 +46,7 @@ static pte_t *get_pte_from_cache(struct mm_struct *mm)*/ if (((unsigned long )pte_frag & ~PAGE_MASK ) ==0 ) pte_frag =NULL ; - mm->context.pte_frag = pte_frag; +pte_frag_set (&mm->context, pte_frag); }spin_unlock (&mm->page_table_lock);return (pte_t *)ret;
@@ -86,9 +86,9 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)* the allocated page with single fragement * count. */ -if (likely (!mm->context.pte_frag)) { +if (likely (!pte_frag_get (&mm->context))) {atomic_set (&page->pt_frag_refcount,PTE_FRAG_NR ); - mm->context.pte_frag = ret +PTE_FRAG_SIZE ; +pte_frag_set (&mm->context, ret +PTE_FRAG_SIZE ); }spin_unlock (&mm->page_table_lock);
Signature · 1 line
2.13.3
kbuild test robot
build botRe: [PATCH v8 07/20] powerpc/mm: add helpers to get/set mm.context->pte_frag
Thank you for the patch! Perhaps something to improve:
Hi Christophe,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on powerpc/next]
[also build test WARNING on v4.20-rc4]
[cannot apply to next-20181129]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/powerpc-book3s32-Remove-CONFIG_BOOKE-dependent-code/20181129-210058 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next config: powerpc-allmodconfig (attached as .config) compiler: powerpc64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=powerpc
All warnings (new ones prefixed by >>):
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/hw_irq.h:64,
from arch/powerpc/include/asm/irqflags.h:12,
from include/linux/irqflags.h:16,
from include/linux/spinlock.h:54,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:6,
from include/linux/slab.h:15,
from drivers/usb/gadget/function/f_acm.c:14:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
--
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/current.h:16,
from include/linux/sched.h:12,
from drivers/usb/gadget/function/u_serial.c:18:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/current.h:16,
from include/linux/sched.h:12,
from drivers/usb/gadget/function/u_serial.c:18:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
--
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/current.h:16,
from include/linux/sched.h:12,
from drivers/staging/rtlwifi/rtl8822be/../wifi.h:20,
from drivers/staging/rtlwifi/rtl8822be/trx.c:15:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
In file included from drivers/staging/rtlwifi/rtl8822be/trx.c:26:0:
include/linux/vermagic.h:29:10: fatal error: generated/randomize_layout_hash.h: No such file or directory
#include <generated/randomize_layout_hash.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
--
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/smp.h:28,
from include/linux/smp.h:68,
from include/linux/percpu.h:7,
from arch/powerpc/include/asm/time.h:18,
from arch/powerpc/include/asm/delay.h:6,
from include/linux/delay.h:26,
from sound/aoa/codecs/onyx.c:34:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
sound/aoa/codecs/onyx.c: In function 'onyx_snd_single_bit_get':
sound/aoa/codecs/onyx.c:380:37: warning: 'c' may be used uninitialized in this function [-Wmaybe-uninitialized]
ucontrol->value.integer.value[0] = !!(c & mask) ^ polarity;
^~~~~~~~~~~~
--
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/smp.h:28,
from include/linux/smp.h:68,
from include/linux/percpu.h:7,
from arch/powerpc/include/asm/time.h:18,
from arch/powerpc/include/asm/delay.h:6,
from include/linux/delay.h:26,
from drivers/scsi/esas2r/esas2r.h:45,
from drivers/scsi/esas2r/esas2r_ioctl.c:44:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: warning: "__pte_frag_nr" is not defined, evaluates to 0 [-Wundef]
#define PTE_FRAG_NR __pte_frag_nr
^
>> arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
drivers/scsi/esas2r/esas2r_ioctl.c: In function 'esas2r_write_vda':
drivers/scsi/esas2r/esas2r_ioctl.c:1910:19: warning: 'dma_addr' may be used uninitialized in this function [-Wmaybe-uninitialized]
a->ppvda_buffer = dma_addr;
~~~~~~~~~~~~~~~~^~~~~~~~~~vim +/__pte_frag_nr +219 arch/powerpc/include/asm/book3s/64/pgtable.h
5ed7ecd0 Aneesh Kumar K.V 2016-04-29 217
5ed7ecd0 Aneesh Kumar K.V 2016-04-29 218 extern unsigned long __pte_frag_nr; 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 @219 #define PTE_FRAG_NR __pte_frag_nr 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 220 extern unsigned long __pte_frag_size_shift; 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 221 #define PTE_FRAG_SIZE_SHIFT __pte_frag_size_shift 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 222 #define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT)
dd1842a2 Aneesh Kumar K.V 2016-04-29 223
:::::: The code at line 219 was first introduced by commit
:::::: 5ed7ecd08a0807d6d616c3d958402f9c723bb048 powerpc/mm: pte_frag abstraction
:::::: TO: Aneesh Kumar K.V <[email protected]> :::::: CC: Michael Ellerman <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
kbuild test robot
build botRe: [PATCH v8 07/20] powerpc/mm: add helpers to get/set mm.context->pte_frag
In reply to Christophe LeroyThank you for the patch! Yet something to improve:
Hi Christophe,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on powerpc/next]
[also build test ERROR on v4.20-rc4]
[cannot apply to next-20181129]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/powerpc-book3s32-Remove-CONFIG_BOOKE-dependent-code/20181129-210058 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next config: powerpc-defconfig (attached as .config) compiler: powerpc64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=powerpc
All errors (new ones prefixed by >>):
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/hw_irq.h:64,
from arch/powerpc/include/asm/irqflags.h:12,
from include/linux/irqflags.h:16,
from include/linux/spinlock.h:54,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:6,
from include/linux/slab.h:15,
from include/linux/crypto.h:24,
from include/crypto/algapi.h:15,
from include/crypto/internal/hash.h:16,
from arch/powerpc/crypto/md5-glue.c:15:
>> arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: error: "__pte_frag_nr" is not defined, evaluates to 0 [-Werror=undef]
#define PTE_FRAG_NR __pte_frag_nr
^
arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~cc1: all warnings being treated as errors
vim +/__pte_frag_nr +219 arch/powerpc/include/asm/book3s/64/pgtable.h
5ed7ecd0 Aneesh Kumar K.V 2016-04-29 217
5ed7ecd0 Aneesh Kumar K.V 2016-04-29 218 extern unsigned long __pte_frag_nr; 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 @219 #define PTE_FRAG_NR __pte_frag_nr 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 220 extern unsigned long __pte_frag_size_shift; 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 221 #define PTE_FRAG_SIZE_SHIFT __pte_frag_size_shift 5ed7ecd0 Aneesh Kumar K.V 2016-04-29 222 #define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT)
dd1842a2 Aneesh Kumar K.V 2016-04-29 223
:::::: The code at line 219 was first introduced by commit
:::::: 5ed7ecd08a0807d6d616c3d958402f9c723bb048 powerpc/mm: pte_frag abstraction
:::::: TO: Aneesh Kumar K.V <[email protected]> :::::: CC: Michael Ellerman <[email protected]>
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v8 8/20 powerpc/mm: Extend pte_fragment functionality to PPC32
In reply to Christophe LeroyIn order to allow the 8xx to handle pte_fragments, this patch extends the use of pte_fragments to PPC32 platforms.
--- arch/powerpc/include/asm/book3s/32/mmu-hash.h | 5 ++++- arch/powerpc/include/asm/book3s/32/pgalloc.h | 17 +++++++++-------- arch/powerpc/include/asm/book3s/32/pgtable.h | 5 +++-- arch/powerpc/include/asm/mmu_context.h | 2 +- arch/powerpc/include/asm/nohash/32/mmu.h | 4 +++- arch/powerpc/include/asm/nohash/32/pgalloc.h | 22 +++++++++++----------- arch/powerpc/include/asm/nohash/32/pgtable.h | 8 +++++--- arch/powerpc/mm/Makefile | 1 + arch/powerpc/mm/mmu_context.c | 10 ++++++++++ arch/powerpc/mm/mmu_context_nohash.c | 2 +- arch/powerpc/mm/pgtable_32.c | 25 ++++--------------------- 11 files changed, 52 insertions(+), 49 deletions(-)
arch/powerpc/include/asm/book3s/32/mmu-hash.h +4 -1
@@ -1,6 +1,7 @@/* SPDX-License-Identifier: GPL-2.0 */ #ifndef _ASM_POWERPC_BOOK3S_32_MMU_HASH_H_#define _ASM_POWERPC_BOOK3S_32_MMU_HASH_H_ +/* * 32-bit hash table MMU support */
@@ -9,6 +10,8 @@* BATs */ +#include <asm/page.h> +/* Block size masks */ #define BL_128K 0x000#define BL_256K 0x001
@@ -43,7 +46,7 @@ struct ppc_bat { u32 batl; }; -typedef struct page *pgtable_t ; +typedef pte_t *pgtable_t ;#endif /* !__ASSEMBLY__ */ /*
arch/powerpc/include/asm/book3s/32/pgalloc.h +9 -8
@@ -59,30 +59,31 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp,static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pte_page) { - *pmdp =__pmd ((page_to_pfn (pte_page) <<PAGE_SHIFT ) | _PMD_PRESENT); + *pmdp =__pmd (__pa (pte_page) | _PMD_PRESENT); } -#define pmd_pgtable (pmd) pmd_page(pmd) +#define pmd_pgtable (pmd) ((pgtable_t)pmd_page_vaddr(pmd))extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr);extern pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr); +void pte_frag_destroy (void *pte_frag); +pte_t *pte_fragment_alloc (struct mm_struct *mm,unsigned long vmaddr,int kernel); +void pte_fragment_free (unsigned long *table,int kernel);static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) { -free_page ((unsigned long )pte); +pte_fragment_free ((unsigned long *)pte,1 ); }static inline void pte_free (struct mm_struct *mm,pgtable_t ptepage) { -pgtable_page_dtor (ptepage); -__free_page (ptepage); +pte_fragment_free ((unsigned long *)ptepage,0 ); }static inline void pgtable_free (void *table,unsigned index_size) {if (!index_size) { -pgtable_page_dtor (virt_to_page (table)); -free_page ((unsigned long )table); +pte_fragment_free ((unsigned long *)table,0 ); }else {BUG_ON (index_size >MAX_PGTABLE_INDEX_SIZE );kmem_cache_free (PGT_CACHE (index_size), table);
@@ -120,6 +121,6 @@ static inline void pgtable_free_tlb(struct mmu_gather *tlb,static inline void __pte_free_tlb (struct mmu_gather *tlb,pgtable_t table,unsigned long address) { -pgtable_free_tlb (tlb,page_address (table),0 ); +pgtable_free_tlb (tlb, table,0 ); }#endif /* _ASM_POWERPC_BOOK3S_32_PGALLOC_H */
arch/powerpc/include/asm/book3s/32/pgtable.h +3 -2
@@ -329,7 +329,7 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma,#define pte_same (A ,B ) (((pte_val(A) ^ pte_val(B)) & ~_PAGE_HASHPTE) == 0)#define pmd_page_vaddr (pmd) \ - ((unsigned long) __va(pmd_val(pmd) & PAGE_MASK)) + ((unsigned long)__va(pmd_val(pmd) & ~(PTE_TABLE_SIZE - 1)))#define pmd_page (pmd) \ pfn_to_page(pmd_val(pmd) >> PAGE_SHIFT)
@@ -346,7 +346,8 @@ static inline void __ptep_set_access_flags(struct vm_area_struct *vma,#define pte_offset_kernel (dir, addr) \ ((pte_t *) pmd_page_vaddr(*(dir)) + pte_index(addr))#define pte_offset_map (dir, addr) \ - ((pte_t *) kmap_atomic(pmd_page(*(dir))) + pte_index(addr)) + ((pte_t *)(kmap_atomic(pmd_page(*(dir))) + \ + (pmd_page_vaddr(*(dir)) & ~PAGE_MASK)) + pte_index(addr))#define pte_unmap (pte) kunmap_atomic(pte)/*
arch/powerpc/include/asm/mmu_context.h +1 -1
@@ -223,7 +223,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm,return 0 ; } -#ifndef CONFIG_PPC_BOOK3S_64 +#ifdef CONFIG_PPC_BOOK3E_64 static inline void arch_exit_mmap (struct mm_struct *mm) { }
arch/powerpc/include/asm/nohash/32/mmu.h +3 -1
@@ -2,6 +2,8 @@#ifndef _ASM_POWERPC_NOHASH_32_MMU_H_#define _ASM_POWERPC_NOHASH_32_MMU_H_ +#include <asm/page.h> +#if defined(CONFIG_40x)/* 40x-style software loaded TLB */ #include <asm/nohash/32/mmu-40x.h>
@@ -17,7 +19,7 @@#endif #ifndef __ASSEMBLY__ -typedef struct page *pgtable_t ; +typedef pte_t *pgtable_t ;#endif #endif /* _ASM_POWERPC_NOHASH_32_MMU_H_ */
arch/powerpc/include/asm/nohash/32/pgalloc.h +11 -11
@@ -61,11 +61,10 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp,static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pte_page) { - *pmdp =__pmd ((page_to_pfn (pte_page) <<PAGE_SHIFT ) | _PMD_USER | - _PMD_PRESENT); + *pmdp =__pmd (__pa (pte_page) | _PMD_USER | _PMD_PRESENT); } -#define pmd_pgtable (pmd) pmd_page(pmd) +#define pmd_pgtable (pmd) ((pgtable_t)pmd_page_vaddr(pmd))#else static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp,
@@ -77,31 +76,32 @@ static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp,static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pte_page) { - *pmdp =__pmd ((unsigned long )lowmem_page_address (pte_page) | _PMD_PRESENT); + *pmdp =__pmd ((unsigned long )pte_page | _PMD_PRESENT); } -#define pmd_pgtable (pmd) pmd_page(pmd) +#define pmd_pgtable (pmd) ((pgtable_t)pmd_page_vaddr(pmd))#endif extern pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long addr);extern pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long addr); +void pte_frag_destroy (void *pte_frag); +pte_t *pte_fragment_alloc (struct mm_struct *mm,unsigned long vmaddr,int kernel); +void pte_fragment_free (unsigned long *table,int kernel);static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) { -free_page ((unsigned long )pte); +pte_fragment_free ((unsigned long *)pte,1 ); }static inline void pte_free (struct mm_struct *mm,pgtable_t ptepage) { -pgtable_page_dtor (ptepage); -__free_page (ptepage); +pte_fragment_free ((unsigned long *)ptepage,0 ); }static inline void pgtable_free (void *table,unsigned index_size) {if (!index_size) { -pgtable_page_dtor (virt_to_page (table)); -free_page ((unsigned long )table); +pte_fragment_free ((unsigned long *)table,0 ); }else {BUG_ON (index_size >MAX_PGTABLE_INDEX_SIZE );kmem_cache_free (PGT_CACHE (index_size), table);
@@ -140,6 +140,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table, unsigned long address) {tlb_flush_pgtable (tlb, address); -pgtable_free_tlb (tlb,page_address (table),0 ); +pgtable_free_tlb (tlb, table,0 ); }#endif /* _ASM_POWERPC_PGALLOC_32_H */
arch/powerpc/include/asm/nohash/32/pgtable.h +5 -3
@@ -20,6 +20,7 @@ extern int icache_44x_need_flush;#endif /* __ASSEMBLY__ */ #define PTE_INDEX_SIZE PTE_SHIFT +#define PMD_INDEX_SIZE 0#define PUD_INDEX_SIZE 0#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
@@ -333,12 +334,12 @@ static inline int pte_young(pte_t pte)*/ #ifndef CONFIG_BOOKE #define pmd_page_vaddr (pmd) \ - ((unsigned long) __va(pmd_val(pmd) & PAGE_MASK)) + ((unsigned long)__va(pmd_val(pmd) & ~(PTE_TABLE_SIZE - 1)))#define pmd_page (pmd) \ pfn_to_page(pmd_val(pmd) >> PAGE_SHIFT)#else #define pmd_page_vaddr (pmd) \ - ((unsigned long) (pmd_val(pmd) & PAGE_MASK)) + ((unsigned long)(pmd_val(pmd) & ~(PTE_TABLE_SIZE - 1)))#define pmd_page (pmd) \ pfn_to_page((__pa(pmd_val(pmd)) >> PAGE_SHIFT))#endif
@@ -357,7 +358,8 @@ static inline int pte_young(pte_t pte) (pmd_bad (*(dir)) ?NULL : (pte_t *)pmd_page_vaddr (*(dir)) + \ pte_index(addr))#define pte_offset_map (dir, addr) \ - ((pte_t *) kmap_atomic(pmd_page(*(dir))) + pte_index(addr)) + ((pte_t *)(kmap_atomic(pmd_page(*(dir))) + \ + (pmd_page_vaddr(*(dir)) & ~PAGE_MASK)) + pte_index(addr))#define pte_unmap (pte) kunmap_atomic(pte)/*
arch/powerpc/mm/Makefile +1 -0
@@ -18,6 +18,7 @@ obj-$(CONFIG_PPC_BOOK3E_64) += pgtable-book3e.o obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb.o \ $(hash64-y) mmu_context_book3s64.o \ pgtable-book3s64.o pgtable-frag.o +obj-$(CONFIG_PPC32) += pgtable-frag.o obj-$(CONFIG_PPC_RADIX_MMU) += pgtable-radix.o tlb-radix.o obj-$(CONFIG_PPC_STD_MMU_32) += ppc_mmu_32.o hash_low_32.o mmu_context_hash32.o obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(BITS).o
arch/powerpc/mm/mmu_context.c +10 -0
@@ -15,6 +15,7 @@#include <linux/sched/mm.h> #include <asm/mmu_context.h> +#include <asm/pgalloc.h> #if defined(CONFIG_PPC32 ) static inline void switch_mm_pgdir(struct task_struct *tsk,
@@ -97,3 +98,12 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,switch_mmu_context (prev, next, tsk); } +#ifdef CONFIG_PPC32 +void arch_exit_mmap (struct mm_struct *mm) +{ +void *frag =pte_frag_get (&mm->context); + +if (frag) +pte_frag_destroy (frag); +} +#endif
arch/powerpc/mm/mmu_context_nohash.c +1 -1
@@ -385,6 +385,7 @@ int init_new_context(struct task_struct *t, struct mm_struct *mm)#endif mm->context.id =MMU_NO_CONTEXT ; mm->context.active =0 ; +pte_frag_set (&mm->context,NULL );return 0 ; }
@@ -487,4 +488,3 @@ void __init mmu_context_init(void) next_context =FIRST_CONTEXT ; nr_free_contexts =LAST_CONTEXT -FIRST_CONTEXT +1 ; } -
arch/powerpc/mm/pgtable_32.c +4 -21
@@ -45,32 +45,15 @@ extern char etext[], _stext[], _sinittext[], _einittext[];__ref pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) { -pte_t *pte; +if (!slab_is_available ()) +return memblock_alloc (PTE_FRAG_SIZE ,PTE_FRAG_SIZE ); -if (slab_is_available ()) { - pte = (pte_t *)__get_free_page (GFP_KERNEL |__GFP_ZERO); - }else { - pte =__va (memblock_phys_alloc (PAGE_SIZE ,PAGE_SIZE )); -if (pte) -clear_page (pte); - } -return pte; +return (pte_t *)pte_fragment_alloc (mm, address,1 ); }pgtable_t pte_alloc_one (struct mm_struct *mm,unsigned long address) { -struct page *ptepage; - -gfp_t flags =GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT; - - ptepage =alloc_pages (flags,0 ); -if (!ptepage) -return NULL ; -if (!pgtable_page_ctor (ptepage)) { -__free_page (ptepage); -return NULL ; - } -return ptepage; +return (pgtable_t )pte_fragment_alloc (mm, address,0 ); } void __iomem *
Signature · 1 line
2.13.3
PATCH v8 9/20 powerpc/mm: enable the use of page table cache of order 0
In reply to Christophe Leroyhugepages uses a cache of order 0. Lets allow page tables of order 0 in the common part in order to avoid open coding in hugetlb
--- arch/powerpc/include/asm/book3s/32/pgalloc.h | 5 +---- arch/powerpc/include/asm/book3s/64/pgalloc.h | 5 +---- arch/powerpc/include/asm/nohash/32/pgalloc.h | 5 +---- arch/powerpc/include/asm/nohash/64/pgalloc.h | 5 +---- arch/powerpc/mm/init-common.c | 6 +++--- 5 files changed, 7 insertions(+), 19 deletions(-)
arch/powerpc/include/asm/book3s/32/pgalloc.h +1 -4
@@ -25,10 +25,7 @@extern void __bad_pte (pmd_t *pmd);extern struct kmem_cache *pgtable_cache[]; -#define PGT_CACHE (shift) ({ \ - BUG_ON(!(shift)); \ - pgtable_cache[(shift) - 1]; \ - }) +#define PGT_CACHE (shift) pgtable_cache[shift]static inline pgd_t *pgd_alloc (struct mm_struct *mm) {
arch/powerpc/include/asm/book3s/64/pgalloc.h +1 -4
@@ -37,10 +37,7 @@ extern struct vmemmap_backing *vmemmap_list;#define MAX_PGTABLE_INDEX_SIZE 0xfextern struct kmem_cache *pgtable_cache[]; -#define PGT_CACHE (shift) ({ \ - BUG_ON(!(shift)); \ - pgtable_cache[(shift) - 1]; \ - }) +#define PGT_CACHE (shift) pgtable_cache[shift]extern pte_t *pte_fragment_alloc (struct mm_struct *,unsigned long ,int );extern pmd_t *pmd_fragment_alloc (struct mm_struct *,unsigned long );
arch/powerpc/include/asm/nohash/32/pgalloc.h +1 -4
@@ -25,10 +25,7 @@extern void __bad_pte (pmd_t *pmd);extern struct kmem_cache *pgtable_cache[]; -#define PGT_CACHE (shift) ({ \ - BUG_ON(!(shift)); \ - pgtable_cache[(shift) - 1]; \ - }) +#define PGT_CACHE (shift) pgtable_cache[shift]static inline pgd_t *pgd_alloc (struct mm_struct *mm) {
arch/powerpc/include/asm/nohash/64/pgalloc.h +1 -4
@@ -36,10 +36,7 @@ extern struct vmemmap_backing *vmemmap_list;#define MAX_PGTABLE_INDEX_SIZE 0xfextern struct kmem_cache *pgtable_cache[]; -#define PGT_CACHE (shift) ({ \ - BUG_ON(!(shift)); \ - pgtable_cache[(shift) - 1]; \ - }) +#define PGT_CACHE (shift) pgtable_cache[shift]static inline pgd_t *pgd_alloc (struct mm_struct *mm) {
arch/powerpc/mm/init-common.c +3 -3
@@ -40,7 +40,7 @@ static void pmd_ctor(void *addr)memset (addr,0 ,PMD_TABLE_SIZE ); } -struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE ]; +struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE +1 ];EXPORT_SYMBOL_GPL (pgtable_cache);/* used by kvm_hv module */ /*
@@ -71,7 +71,7 @@ void pgtable_cache_add(unsigned shift, void (*ctor)(void *))* moment, gcc doesn't seem to recognize is_power_of_2 as a * constant expression, so so much for that. */ BUG_ON (!is_power_of_2 (minalign)); -BUG_ON ((shift <1 ) || (shift >MAX_PGTABLE_INDEX_SIZE )); +BUG_ON (shift >MAX_PGTABLE_INDEX_SIZE );if (PGT_CACHE (shift))return ;/* Already have a cache of this size */
@@ -83,7 +83,7 @@ void pgtable_cache_add(unsigned shift, void (*ctor)(void *))panic ("Could not allocate pgtable cache for order %d" , shift);kfree (name); - pgtable_cache[shift -1 ] = new; + pgtable_cache[shift] = new;pr_debug ("Allocated pgtable cache for order %d\n" , shift); }
Signature · 1 line
2.13.3
PATCH v8 10/20 powerpc/mm: replace hugetlb_cache by PGT_CACHE(PTE_T_ORDER)
In reply to Christophe LeroyInstead of opencoding cache handling for the special case of hugepage tables having a single pte_t element, this patch makes use of the common pgtable_cache helpers
--- arch/powerpc/include/asm/hugetlb.h | 2 -- arch/powerpc/mm/hugetlbpage.c | 26 +++++++------------------- 2 files changed, 7 insertions(+), 21 deletions(-)
arch/powerpc/include/asm/hugetlb.h +0 -2
@@ -5,8 +5,6 @@#ifdef CONFIG_HUGETLB_PAGE #include <asm/page.h> -extern struct kmem_cache *hugepte_cache; -#ifdef CONFIG_PPC_BOOK3S_64 #include <asm/book3s/64/hugetlb.h>
arch/powerpc/mm/hugetlbpage.c +7 -19
@@ -42,6 +42,8 @@ EXPORT_SYMBOL(HPAGE_SHIFT);#define hugepd_none (hpd) (hpd_val(hpd) == 0) +#define PTE_T_ORDER (__builtin_ffs(sizeof(pte_t)) - __builtin_ffs(sizeof(void *))) +pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr,unsigned long sz) {/*
@@ -61,7 +63,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,int num_hugepd;if (pshift >= pdshift) { - cachep = hugepte_cache; + cachep =PGT_CACHE (PTE_T_ORDER ); num_hugepd =1 << (pshift - pdshift); }else { cachep =PGT_CACHE (pdshift - pshift);
@@ -264,7 +266,7 @@ static void hugepd_free_rcu_callback(struct rcu_head *head)unsigned int i;for (i =0 ; i < batch->index; i++) -kmem_cache_free (hugepte_cache, batch->ptes[i]); +kmem_cache_free (PGT_CACHE (PTE_T_ORDER ), batch->ptes[i]);free_page ((unsigned long )batch); }
@@ -277,7 +279,7 @@ static void hugepd_free(struct mmu_gather *tlb, void *hugepte)if (atomic_read (&tlb->mm->mm_users) <2 ||mm_is_thread_local (tlb->mm)) { -kmem_cache_free (hugepte_cache, hugepte); +kmem_cache_free (PGT_CACHE (PTE_T_ORDER ), hugepte);put_cpu_var (hugepd_freelist_cur);return ; }
@@ -652,7 +654,6 @@ static int __init hugepage_setup_sz(char *str) }__setup ("hugepagesz=" , hugepage_setup_sz); -struct kmem_cache *hugepte_cache;static int __init hugetlbpage_init (void ) {int psize;
@@ -702,21 +703,8 @@ static int __init hugetlbpage_init(void)if (pdshift > shift)pgtable_cache_add (pdshift - shift,NULL );#if defined(CONFIG_PPC_FSL_BOOK3E ) || defined(CONFIG_PPC_8xx) - else if (!hugepte_cache) { -/* -* Create a kmem cache for hugeptes. The bottom bits in -* the pte have size information encoded in them, so -* align them to allow this -*/ - hugepte_cache =kmem_cache_create ("hugepte-cache" , -sizeof (pte_t), -HUGEPD_SHIFT_MASK +1 , -0 ,NULL ); -if (hugepte_cache ==NULL ) -panic ("%s: Unable to create kmem cache " -"for hugeptes\n" , __func__); - - } + else +pgtable_cache_add (PTE_T_ORDER ,NULL );#endif }
Signature · 1 line
2.13.3
PATCH v8 11/20 powerpc/mm: fix a warning when a cache is common to PGD and hugepages
In reply to Christophe LeroyWhile implementing TLB miss HW assistance on the 8xx, the following warning was encountered:
[ 423.732965] WARNING: CPU: 0 PID: 345 at mm/slub.c:2412 ___slab_alloc.constprop.30+0x26c/0x46c [ 423.733033] CPU: 0 PID: 345 Comm: mmap Not tainted 4.18.0-rc8-00664-g2dfff9121c55 #671 [ 423.733075] NIP: c0108f90 LR: c0109ad0 CTR: 00000004 [ 423.733121] REGS: c455bba0 TRAP: 0700 Not tainted (4.18.0-rc8-00664-g2dfff9121c55) [ 423.733147] MSR: 00021032 <ME,IR,DR,RI> CR: 24224848 XER: 20000000 [ 423.733319] [ 423.733319] GPR00: c0109ad0 c455bc50 c4521910 c60053c0 007080c0 c0011b34 c7fa41e0 c455be30 [ 423.733319] GPR08: 00000001 c00103a0 c7fa41e0 c49afcc4 24282842 10018840 c079b37c 00000040 [ 423.733319] GPR16: 73f00000 00210d00 00000000 00000001 c455a000 00000100 00000200 c455a000 [ 423.733319] GPR24: c60053c0 c0011b34 007080c0 c455a000 c455a000 c7fa41e0 00000000 00009032 [ 423.734190] NIP [c0108f90] ___slab_alloc.constprop.30+0x26c/0x46c [ 423.734257] LR [c0109ad0] kmem_cache_alloc+0x210/0x23c [ 423.734283] Call Trace: [ 423.734326] [c455bc50] [00000100] 0x100 (unreliable) [ 423.734430] [c455bcc0] [c0109ad0] kmem_cache_alloc+0x210/0x23c [ 423.734543] [c455bcf0] [c0011b34] huge_pte_alloc+0xc0/0x1dc [ 423.734633] [c455bd20] [c01044dc] hugetlb_fault+0x408/0x48c [ 423.734720] [c455bdb0] [c0104b20] follow_hugetlb_page+0x14c/0x44c [ 423.734826] [c455be10] [c00e8e54] __get_user_pages+0x1c4/0x3dc [ 423.734919] [c455be80] [c00e9924] __mm_populate+0xac/0x140 [ 423.735020] [c455bec0] [c00db14c] vm_mmap_pgoff+0xb4/0xb8 [ 423.735127] [c455bf00] [c00f27c0] ksys_mmap_pgoff+0xcc/0x1fc [ 423.735222] [c455bf40] [c000e0f8] ret_from_syscall+0x0/0x38 [ 423.735271] Instruction dump: [ 423.735321] 7cbf482e 38fd0008 7fa6eb78 7fc4f378 4bfff5dd 7fe3fb78 4bfffe24 81370010 [ 423.735536] 71280004 41a2ff88 4840c571 4bffff80 <0fe00000> 4bfffeb8 81340010 712a0004 [ 423.735757] ---[ end trace e9b222919a470790 ]---
This warning occurs when calling kmem_cache_zalloc() on a cache having a constructor.
In this case it happens because PGD cache and 512k hugepte cache are the same size (4k). While a cache with constructor is created for the PGD, hugepages create cache without constructor and uses kmem_cache_zalloc(). As both expect a cache with the same size, the hugepages reuse the cache created for PGD, hence the conflict.
In order to avoid this conflict, this patch:
- modifies pgtable_cache_add() so that a zeroising constructor is added for any cache size.
- replaces calls to kmem_cache_zalloc() by kmem_cache_alloc()
---
see original discussion in https://patchwork.ozlabs.org/patch/957565/
arch/powerpc/include/asm/pgtable.h | 2 +- arch/powerpc/mm/hugetlbpage.c | 6 ++--- arch/powerpc/mm/init-common.c | 46 ++++++++++++++++++++++++++------------ 3 files changed, 36 insertions(+), 18 deletions(-)
arch/powerpc/include/asm/pgtable.h +1 -1
@@ -101,7 +101,7 @@ extern int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr,/* can we use this in kvm */ unsigned long vmalloc_to_phys (void *vmalloc_addr); -void pgtable_cache_add (unsigned shift,void (*ctor)(void *)); +void pgtable_cache_add (unsigned int shift);void pgtable_cache_init (void );#if defined(CONFIG_STRICT_KERNEL_RWX ) || defined(CONFIG_PPC32 )
arch/powerpc/mm/hugetlbpage.c +3 -3
@@ -70,7 +70,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, num_hugepd =1 ; } - new =kmem_cache_zalloc (cachep,pgtable_gfp_flags (mm,GFP_KERNEL )); + new =kmem_cache_alloc (cachep,pgtable_gfp_flags (mm,GFP_KERNEL ));BUG_ON (pshift >HUGEPD_SHIFT_MASK );BUG_ON ((unsigned long )new &HUGEPD_SHIFT_MASK );
@@ -701,10 +701,10 @@ static int __init hugetlbpage_init(void)* use pgt cache for hugepd. */ if (pdshift > shift) -pgtable_cache_add (pdshift - shift,NULL ); +pgtable_cache_add (pdshift - shift);#if defined(CONFIG_PPC_FSL_BOOK3E ) || defined(CONFIG_PPC_8xx) else -pgtable_cache_add (PTE_T_ORDER ,NULL ); +pgtable_cache_add (PTE_T_ORDER );#endif }
arch/powerpc/mm/init-common.c +32 -14
80 lines not on this page · raw message
Signature · 1 line
2.13.3
PATCH v8 12/20 powerpc/mm: remove unnecessary test in pgtable_cache_init()
In reply to Christophe Leroypgtable_cache_add() gracefully handles the case when a cache that size already exists by returning early with the following test:
if (PGT_CACHE(shift))
return; /* Already have a cache of this size */
It is then not needed to test the existence of the cache before.
--- arch/powerpc/mm/init-common.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/powerpc/mm/init-common.c +2 -2
@@ -111,13 +111,13 @@ void pgtable_cache_init(void) {pgtable_cache_add (PGD_INDEX_SIZE ); -if (PMD_CACHE_INDEX && !PGT_CACHE (PMD_CACHE_INDEX )) +if (PMD_CACHE_INDEX )pgtable_cache_add (PMD_CACHE_INDEX );/* * In all current configs, when the PUD index exists it's the * same size as either the pgd or pmd index except with THP enabled * on book3s 64 */ -if (PUD_CACHE_INDEX && !PGT_CACHE (PUD_CACHE_INDEX )) +if (PUD_CACHE_INDEX )pgtable_cache_add (PUD_CACHE_INDEX ); }
Signature · 1 line
2.13.3
PATCH v8 13/20 powerpc/8xx: Move SW perf counters in first 32kb of memory
In reply to Christophe LeroyIn order to simplify time critical exceptions handling 8xx specific SW perf counters, this patch moves the counters into the beginning of memory. This is possible because .text is readable and the counters are never modified outside of the handlers.
By doing this, we avoid having to set a second register with the upper part of the address of the counters.
--- arch/powerpc/kernel/head_8xx.S | 58 ++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 30 deletions(-)
arch/powerpc/kernel/head_8xx.S +28 -30
@@ -106,6 +106,23 @@ turn_on_mmu: mtspr SPRN_SRR0,r0 rfi /* enables MMU */ + +#ifdef CONFIG_PERF_EVENTS + .align 4 + + .globl itlb_miss_counter +itlb_miss_counter: + .space 4 + + .globl dtlb_miss_counter +dtlb_miss_counter: + .space 4 + + .globl instruction_counter +instruction_counter: + .space 4 +#endif + /* * Exception entry code. This code runs with address translation * turned off, i.e. using physical addresses.
@@ -384,17 +401,16 @@ InstructionTLBMiss: #ifdef CONFIG_PERF_EVENTS patch_site 0f, patch__itlbmiss_perf -0: lis r10, (itlb_miss_counter - PAGE_OFFSET)@ha - lwz r11, (itlb_miss_counter - PAGE_OFFSET)@l(r10) - addi r11, r11, 1 - stw r11, (itlb_miss_counter - PAGE_OFFSET)@l(r10) -#endif +0: lwz r10, (itlb_miss_counter - PAGE_OFFSET)@l(0) + addi r10, r10, 1 + stw r10, (itlb_miss_counter - PAGE_OFFSET)@l(0) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 #if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) mfspr r12, SPRN_SPRG_SCRATCH2 #endif rfi +#endif #ifdef CONFIG_HUGETLB_PAGE 10: /* 8M pages */
@@ -509,15 +525,14 @@ DataStoreTLBMiss: #ifdef CONFIG_PERF_EVENTS patch_site 0f, patch__dtlbmiss_perf -0: lis r10, (dtlb_miss_counter - PAGE_OFFSET)@ha - lwz r11, (dtlb_miss_counter - PAGE_OFFSET)@l(r10) - addi r11, r11, 1 - stw r11, (dtlb_miss_counter - PAGE_OFFSET)@l(r10) -#endif +0: lwz r10, (dtlb_miss_counter - PAGE_OFFSET)@l(0) + addi r10, r10, 1 + stw r10, (dtlb_miss_counter - PAGE_OFFSET)@l(0) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 mfspr r12, SPRN_SPRG_SCRATCH2 rfi +#endif #ifdef CONFIG_HUGETLB_PAGE 10: /* 8M pages */
@@ -625,16 +640,13 @@ DataBreakpoint: . = 0x1d00 InstructionBreakpoint: mtspr SPRN_SPRG_SCRATCH0, r10 - mtspr SPRN_SPRG_SCRATCH1, r11 - lis r10, (instruction_counter - PAGE_OFFSET)@ha - lwz r11, (instruction_counter - PAGE_OFFSET)@l(r10) - addi r11, r11, -1 - stw r11, (instruction_counter - PAGE_OFFSET)@l(r10) + lwz r10, (instruction_counter - PAGE_OFFSET)@l(0) + addi r10, r10, -1 + stw r10, (instruction_counter - PAGE_OFFSET)@l(0) lis r10, 0xffff ori r10, r10, 0x01 mtspr SPRN_COUNTA, r10 mfspr r10, SPRN_SPRG_SCRATCH0 - mfspr r11, SPRN_SPRG_SCRATCH1 rfi #else EXCEPTION(0x1d00, Trap_1d, unknown_exception, EXC_XFER_EE)
@@ -1065,17 +1077,3 @@ swapper_pg_dir: */ abatron_pteptrs: .space 8 - -#ifdef CONFIG_PERF_EVENTS - .globl itlb_miss_counter -itlb_miss_counter: - .space 4 - - .globl dtlb_miss_counter -dtlb_miss_counter: - .space 4 - - .globl instruction_counter -instruction_counter: - .space 4 -#endif
Signature · 1 line
2.13.3
PATCH v8 14/20 powerpc/8xx: Temporarily disable 16k pages and hugepages
In reply to Christophe LeroyIn preparation of making use of hardware assistance in TLB handlers, this patch temporarily disables 16K pages and hugepages. The reason is that when using HW assistance in 4K pages mode, the linux model fit with the HW model for 4K pages and 8M pages.
However for 16K pages and 512K mode some additional work is needed to get linux model fit with HW model.
For the 8M pages, they will naturaly come back when we switch to HW assistance, without any additional handling.
In order to keep the following patch smaller, the removal of the current special handling for 8M pages gets removed here as well.
Therefore the 4K pages mode will be implemented first and without support for 512k hugepages. Then the 512k hugepages will be brought back. And the 16K pages will be implemented in the following step.
--- arch/powerpc/Kconfig | 2 +- arch/powerpc/kernel/head_8xx.S | 74 +++--------------------------------------- arch/powerpc/mm/tlb_nohash.c | 6 ---- 3 files changed, 6 insertions(+), 76 deletions(-)
arch/powerpc/Kconfig +1 -1
@@ -689,7 +689,7 @@ config PPC_4K_PAGES config PPC_16K_PAGES bool "16k page size" - depends on 44x || PPC_8xx + depends on 44x config PPC_64K_PAGES bool "64k page size"
arch/powerpc/kernel/head_8xx.S +5 -69
131 lines not on this page · raw message
arch/powerpc/mm/tlb_nohash.c +0 -6
@@ -97,12 +97,6 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { .shift = 14, },#endif - [MMU_PAGE_512K] = { - .shift = 19, - }, - [MMU_PAGE_8M] = { - .shift = 23, - }, };#else struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = {
Signature · 1 line
2.13.3
PATCH v8 15/20 powerpc/8xx: Use hardware assistance in TLB handlers
In reply to Christophe LeroyToday, on the 8xx the TLB handlers do SW tablewalk by doing all the calculation in ASM, in order to match with the Linux page table structure.
The 8xx offers hardware assistance which allows significant size reduction of the TLB handlers, hence also reduces the time spent in the handlers.
However, using this HW assistance implies some constraints on the page table structure:
- Regardless of the main page size used (4k or 16k), the level 1 table (PGD) contains 1024 entries and each PGD entry covers a 4Mbytes area which is managed by a level 2 table (PTE) containing also 1024 entries each describing a 4k page.
- 16k pages require 4 identifical entries in the L2 table
- 512k pages PTE have to be spread every 128 bytes in the L2 table
- 8M pages PTE are at the address pointed by the L1 entry and each 8M page require 2 identical entries in the PGD.
This patch modifies the TLB handlers to use HW assistance for 4K PAGES.
Before that patch, the mean time spent in TLB miss handlers is:
- ITLB miss: 80 ticks
- DTLB miss: 62 ticks
After that patch, the mean time spent in TLB miss handlers is:
- ITLB miss: 72 ticks
- DTLB miss: 54 ticks
So the improvement is 10% for ITLB and 13% for DTLB misses
--- arch/powerpc/kernel/head_8xx.S | 58 +++++++++++++++++------------------------- arch/powerpc/mm/8xx_mmu.c | 4 +-- 2 files changed, 26 insertions(+), 36 deletions(-)
arch/powerpc/kernel/head_8xx.S +24 -34
@@ -292,7 +292,7 @@ SystemCall: . = 0x1100 /* * For the MPC8xx, this is a software tablewalk to load the instruction - * TLB. The task switch loads the M_TW register with the pointer to the first + * TLB. The task switch loads the M_TWB register with the pointer to the first * level table. * If we discover there is no second level table (value is zero) or if there * is an invalid pte, we load that into the TLB, which causes another fault
@@ -323,6 +323,7 @@ InstructionTLBMiss: */ mfspr r10, SPRN_SRR0 /* Get effective address of fault */ INVALIDATE_ADJACENT_PAGES_CPU15(r11, r10) + mtspr SPRN_MD_EPN, r10 /* Only modules will cause ITLB Misses as we always * pin the first 8MB of kernel memory */ #ifdef ITLB_MISS_KERNEL
@@ -339,7 +340,7 @@ InstructionTLBMiss: #endif #endif #endif - mfspr r11, SPRN_M_TW /* Get level 1 table */ + mfspr r11, SPRN_M_TWB /* Get level 1 table */ #ifdef ITLB_MISS_KERNEL #if defined(SIMPLE_KERNEL_ADDRESS) && defined(CONFIG_PIN_TLB_TEXT) beq+ 3f
@@ -349,16 +350,14 @@ InstructionTLBMiss: #ifndef CONFIG_PIN_TLB_TEXT blt cr7, ITLBMissLinear #endif - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha + rlwinm r11, r11, 0, 20, 31 + oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha 3: #endif - /* Insert level 1 index */ - rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 - rlwimi r10, r11, 0, 0, 32 - PAGE_SHIFT - 1 /* Add level 2 base */ + mtspr SPRN_MD_TWC, r11 + mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ #ifdef ITLB_MISS_KERNEL mtcr r12
@@ -417,7 +416,7 @@ DataStoreTLBMiss: mfspr r10, SPRN_MD_EPN rlwinm r11, r10, 16, 0xfff8 cmpli cr0, r11, PAGE_OFFSET@h - mfspr r11, SPRN_M_TW /* Get level 1 table */ + mfspr r11, SPRN_M_TWB /* Get level 1 table */ blt+ 3f rlwinm r11, r10, 16, 0xfff8 #ifndef CONFIG_PIN_TLB_IMMR
@@ -430,20 +429,16 @@ DataStoreTLBMiss: patch_site 0b, patch__dtlbmiss_immr_jmp #endif blt cr7, DTLBMissLinear - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha + mfspr r11, SPRN_M_TWB /* Get level 1 table */ + rlwinm r11, r11, 0, 20, 31 + oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha 3: - - /* Insert level 1 index */ - rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ - /* We have a pte table, so load fetch the pte from the table. - */ - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 - rlwimi r10, r11, 0, 0, 32 - PAGE_SHIFT - 1 /* Add level 2 base */ + mtspr SPRN_MD_TWC, r11 + mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ -4: + mtcr r12 /* Insert the Guarded flag into the TWC from the Linux PTE.
@@ -668,9 +663,10 @@ FixupDAR:/* Entry point for dcbx workaround. */ mtspr SPRN_SPRG_SCRATCH2, r10 /* fetch instruction from memory. */ mfspr r10, SPRN_SRR0 + mtspr SPRN_MD_EPN, r10 rlwinm r11, r10, 16, 0xfff8 cmpli cr0, r11, PAGE_OFFSET@h - mfspr r11, SPRN_M_TW /* Get level 1 table */ + mfspr r11, SPRN_M_TWB /* Get level 1 table */ blt+ 3f rlwinm r11, r10, 16, 0xfff8
@@ -680,17 +676,17 @@ FixupDAR:/* Entry point for dcbx workaround. */ /* create physical page address from effective address */ tophys(r11, r10) blt- cr7, 201f - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha - /* Insert level 1 index */ -3: rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 + mfspr r11, SPRN_M_TWB /* Get level 1 table */ + rlwinm r11, r11, 0, 20, 31 + oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha +3: lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + mtspr SPRN_MD_TWC, r11 mtcr r11 + mfspr r11, SPRN_MD_TWC + lwz r11, 0(r11) /* Get the pte */ bt 28,200f /* bit 28 = Large page (8M) */ bt 29,202f /* bit 29 = Large page (8M or 512K) */ - rlwinm r11, r11,0,0,19 /* Extract page descriptor page address */ - /* Insert level 2 index */ - rlwimi r11, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 201: lwz r11,0(r11)
@@ -712,18 +708,12 @@ FixupDAR:/* Entry point for dcbx workaround. */ 141: mfspr r10,SPRN_SPRG_SCRATCH2 b DARFixed /* Nope, go back to normal TLB processing */ - /* concat physical page address(r11) and page offset(r10) */ 200: - rlwinm r11, r10, 0, ~HUGEPD_SHIFT_MASK - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 b 201b 202: - rlwinm r11, r11, 0, 0, 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1) - 1 - rlwimi r11, r10, 32 - (PAGE_SHIFT_512K - 2), 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1), 29 - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT_512K, 31 b 201b
@@ -839,7 +829,7 @@ start_here: lis r6, swapper_pg_dir@ha tophys(r6,r6) - mtspr SPRN_M_TW, r6 + mtspr SPRN_M_TWB, r6 bl early_init /* We have to do this with MMU on */
arch/powerpc/mm/8xx_mmu.c +2 -2
@@ -175,12 +175,12 @@ void set_context(unsigned long id, pgd_t *pgd) *(ptr +1 ) = pgd;#endif -/* Register M_TW will contain base address of level 1 table minus the +/* Register M_TWB will contain base address of level 1 table minus the * lower part of the kernel PGDIR base address, so that all accesses to * level 1 table are done relative to lower part of kernel PGDIR base * address. */ -mtspr (SPRN_M_TW ,__pa (pgd) - offset); +mtspr (SPRN_M_TWB ,__pa (pgd) - offset);/* Update context */ mtspr (SPRN_M_CASID , id -1 );
Signature · 1 line
2.13.3
PATCH v8 16/20 powerpc/8xx: Enable 8M hugepage support with HW assistance
In reply to Christophe LeroyHW assistance naturally supports 8M huge pages without further modifications.
--- arch/powerpc/mm/tlb_nohash.c | 3 +++ 1 file changed, 3 insertions(+)
arch/powerpc/mm/tlb_nohash.c +3 -0
@@ -97,6 +97,9 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { .shift = 14, }, #endif + [MMU_PAGE_8M] = { + .shift = 23, + }, };#else struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT ] = {
Signature · 1 line
2.13.3
PATCH v8 17/20 powerpc/8xx: Enable 512k hugepage support with HW assistance
In reply to Christophe LeroyFor using 512k pages with hardware assistance, the PTEs have to be spread every 128 bytes in the L2 table.
--- arch/powerpc/include/asm/hugetlb.h | 4 +++- arch/powerpc/mm/hugetlbpage.c | 10 +++++++++- arch/powerpc/mm/tlb_nohash.c | 3 +++ 3 files changed, 15 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/hugetlb.h +3 -1
@@ -74,7 +74,9 @@ static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr,unsigned long idx =0 ;pte_t *dir =hugepd_page (hpd); -#ifndef CONFIG_PPC_FSL_BOOK3E +#ifdef CONFIG_PPC_8xx + idx = (addr & ((1UL << pdshift) -1 )) >>PAGE_SHIFT ; +#elif !defined(CONFIG_PPC_FSL_BOOK3E ) idx = (addr & ((1UL << pdshift) -1 )) >>hugepd_shift (hpd);#endif
arch/powerpc/mm/hugetlbpage.c +9 -1
@@ -65,6 +65,9 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,if (pshift >= pdshift) { cachep =PGT_CACHE (PTE_T_ORDER ); num_hugepd =1 << (pshift - pdshift); + }else if (IS_ENABLED (CONFIG_PPC_8xx)) { + cachep =PGT_CACHE (PTE_SHIFT ); + num_hugepd =1 ; }else { cachep =PGT_CACHE (pdshift - pshift); num_hugepd =1 ;
@@ -331,6 +334,9 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshifif (shift >= pdshift)hugepd_free (tlb, hugepte); +else if (IS_ENABLED (CONFIG_PPC_8xx)) +pgtable_free_tlb (tlb, hugepte, +get_hugepd_cache_index (PTE_SHIFT ));else pgtable_free_tlb (tlb, hugepte,get_hugepd_cache_index (pdshift - shift));
@@ -700,7 +706,9 @@ static int __init hugetlbpage_init(void)* if we have pdshift and shift value same, we don't * use pgt cache for hugepd. */ -if (pdshift > shift) +if (pdshift > shift &&IS_ENABLED (CONFIG_PPC_8xx)) +pgtable_cache_add (PTE_SHIFT ); +else if (pdshift > shift)pgtable_cache_add (pdshift - shift);#if defined(CONFIG_PPC_FSL_BOOK3E ) || defined(CONFIG_PPC_8xx) else
arch/powerpc/mm/tlb_nohash.c +3 -0
@@ -97,6 +97,9 @@ struct mmu_psize_def mmu_psize_defs[MMU_PAGE_COUNT] = { .shift = 14, }, #endif + [MMU_PAGE_512K] = { + .shift = 19, + }, [MMU_PAGE_8M] = { .shift = 23, },
Signature · 1 line
2.13.3
kbuild test robot
build botRe: [PATCH v8 17/20] powerpc/8xx: Enable 512k hugepage support with HW assistance
Thank you for the patch! Yet something to improve:
Hi Christophe,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on powerpc/next]
[also build test ERROR on v4.20-rc4]
[cannot apply to next-20181129]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Christophe-Leroy/powerpc-book3s32-Remove-CONFIG_BOOKE-dependent-code/20181129-210058 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next config: powerpc-defconfig (attached as .config) compiler: powerpc64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.2.0 make.cross ARCH=powerpc
All errors (new ones prefixed by >>):
In file included from arch/powerpc/include/asm/book3s/64/mmu-hash.h:24:0,
from arch/powerpc/include/asm/book3s/64/mmu.h:39,
from arch/powerpc/include/asm/mmu.h:328,
from arch/powerpc/include/asm/lppaca.h:36,
from arch/powerpc/include/asm/paca.h:21,
from arch/powerpc/include/asm/hw_irq.h:64,
from arch/powerpc/include/asm/irqflags.h:12,
from include/linux/irqflags.h:16,
from include/linux/spinlock.h:54,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:6,
from include/linux/mm.h:10,
from arch/powerpc//mm/hugetlbpage.c:11:
arch/powerpc/include/asm/book3s/64/pgtable.h:219:21: error: "__pte_frag_nr" is not defined, evaluates to 0 [-Werror=undef]
#define PTE_FRAG_NR __pte_frag_nr
^
arch/powerpc/include/asm/pgtable.h:123:5: note: in expansion of macro 'PTE_FRAG_NR'
#if PTE_FRAG_NR != 1
^~~~~~~~~~~
In file included from arch/powerpc/include/asm/book3s/pgalloc.h:10:0,
from arch/powerpc/include/asm/pgalloc.h:24,
from arch/powerpc//mm/hugetlbpage.c:23:
arch/powerpc//mm/hugetlbpage.c: In function '__hugepte_alloc':
>> arch/powerpc//mm/hugetlbpage.c:69:22: error: 'PTE_SHIFT' undeclared (first use in this function); did you mean 'PUD_SHIFT'?
cachep = PGT_CACHE(PTE_SHIFT);
^
arch/powerpc/include/asm/book3s/64/pgalloc.h:40:40: note: in definition of macro 'PGT_CACHE'
#define PGT_CACHE(shift) pgtable_cache[shift]
^~~~~
arch/powerpc//mm/hugetlbpage.c:69:22: note: each undeclared identifier is reported only once for each function it appears in
cachep = PGT_CACHE(PTE_SHIFT);
^
arch/powerpc/include/asm/book3s/64/pgalloc.h:40:40: note: in definition of macro 'PGT_CACHE'
#define PGT_CACHE(shift) pgtable_cache[shift]
^~~~~
arch/powerpc//mm/hugetlbpage.c: In function 'free_hugepd_range':
arch/powerpc//mm/hugetlbpage.c:339:29: error: 'PTE_SHIFT' undeclared (first use in this function); did you mean 'PUD_SHIFT'?
get_hugepd_cache_index(PTE_SHIFT));
^~~~~~~~~
PUD_SHIFT
arch/powerpc//mm/hugetlbpage.c: In function 'hugetlbpage_init':
arch/powerpc//mm/hugetlbpage.c:710:22: error: 'PTE_SHIFT' undeclared (first use in this function); did you mean 'PUD_SHIFT'?
pgtable_cache_add(PTE_SHIFT);
^~~~~~~~~
PUD_SHIFTcc1: all warnings being treated as errors
vim +69 arch/powerpc//mm/hugetlbpage.c
> 23 #include <asm/pgalloc.h>
24 #include <asm/tlb.h>
25 #include <asm/setup.h>
26 #include <asm/hugetlb.h>
27 #include <asm/pte-walk.h>
28
29
30 #ifdef CONFIG_HUGETLB_PAGE
31
32 #define PAGE_SHIFT_64K 16
33 #define PAGE_SHIFT_512K 19
34 #define PAGE_SHIFT_8M 23
35 #define PAGE_SHIFT_16M 24
36 #define PAGE_SHIFT_16G 34
37
38 bool hugetlb_disabled = false;
39
40 unsigned int HPAGE_SHIFT;
41 EXPORT_SYMBOL(HPAGE_SHIFT);
42
43 #define hugepd_none(hpd) (hpd_val(hpd) == 0)
44
45 #define PTE_T_ORDER (__builtin_ffs(sizeof(pte_t)) - __builtin_ffs(sizeof(void *)))
46
47 pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz)
48 {
49 /*
50 * Only called for hugetlbfs pages, hence can ignore THP and the
51 * irq disabled walk.
52 */
53 return __find_linux_pte(mm->pgd, addr, NULL, NULL);
54 }
55
56 static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
57 unsigned long address, unsigned int pdshift,
58 unsigned int pshift, spinlock_t *ptl)
59 {
60 struct kmem_cache *cachep;
61 pte_t *new;
62 int i;
63 int num_hugepd;
64
65 if (pshift >= pdshift) {
66 cachep = PGT_CACHE(PTE_T_ORDER);
67 num_hugepd = 1 << (pshift - pdshift);
68 } else if (IS_ENABLED(CONFIG_PPC_8xx)) {
> 69 cachep = PGT_CACHE(PTE_SHIFT);
70 num_hugepd = 1;
71 } else {
72 cachep = PGT_CACHE(pdshift - pshift);
73 num_hugepd = 1;
74 }
75
76 new = kmem_cache_alloc(cachep, pgtable_gfp_flags(mm, GFP_KERNEL));
77
78 BUG_ON(pshift > HUGEPD_SHIFT_MASK);
79 BUG_ON((unsigned long)new & HUGEPD_SHIFT_MASK);
80
81 if (! new)
82 return -ENOMEM;
83
84 /*
85 * Make sure other cpus find the hugepd set only after a
86 * properly initialized page table is visible to them.
87 * For more details look for comment in __pte_alloc().
88 */
89 smp_wmb();
90
91 spin_lock(ptl);
92 /*
93 * We have multiple higher-level entries that point to the same
94 * actual pte location. Fill in each as we go and backtrack on error.
95 * We need all of these so the DTLB pgtable walk code can find the
96 * right higher-level entry without knowing if it's a hugepage or not.
97 */
98 for (i = 0; i < num_hugepd; i++, hpdp++) {
99 if (unlikely(!hugepd_none(*hpdp))) 100 break;
101 else {
102 #ifdef CONFIG_PPC_BOOK3S_64
103 *hpdp = __hugepd(__pa(new) | HUGEPD_VAL_BITS |
104 (shift_to_mmu_psize(pshift) << 2));
105 #elif defined(CONFIG_PPC_8xx)
106 *hpdp = __hugepd(__pa(new) | _PMD_USER |
107 (pshift == PAGE_SHIFT_8M ? _PMD_PAGE_8M :
108 _PMD_PAGE_512K) | _PMD_PRESENT);
109 #else
110 /* We use the old format for PPC_FSL_BOOK3E */
111 *hpdp = __hugepd(((unsigned long)new & ~PD_HUGE) | pshift);
112 #endif
113 }
114 }
115 /* If we bailed from the for loop early, an error occurred, clean up */
116 if (i < num_hugepd) {
117 for (i = i - 1 ; i >= 0; i--, hpdp--)
118 *hpdp = __hugepd(0);
119 kmem_cache_free(cachep, new);
120 } else {
121 kmemleak_ignore(new);
122 }
123 spin_unlock(ptl);
124 return 0;
125 }
126
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH v8 18/20 powerpc/8xx: reintroduce 16K pages with HW assistance
In reply to Christophe LeroyUsing this HW assistance implies some constraints on the page table structure:
- Regardless of the main page size used (4k or 16k), the level 1 table (PGD) contains 1024 entries and each PGD entry covers a 4Mbytes area which is managed by a level 2 table (PTE) containing also 1024 entries each describing a 4k page.
- 16k pages require 4 identifical entries in the L2 table
- 512k pages PTE have to be spread every 128 bytes in the L2 table
- 8M pages PTE are at the address pointed by the L1 entry and each 8M page require 2 identical entries in the PGD.
In order to use hardware assistance with 16K pages, this patch does the following modifications:
- Make PGD size independent of the main page size
- In 16k pages mode, redefine pte_t as a struct with 4 elements, and populate those 4 elements in __set_pte_at() and pte_update()
- Adapt the size of the hugepage tables.
- Define a PTE_FRAGMENT_NB so that a 16k page contains 4 page tables.
--- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 1 + arch/powerpc/include/asm/nohash/32/pgtable.h | 19 ++++++++++++++++++- arch/powerpc/include/asm/nohash/pgtable.h | 4 ++++ arch/powerpc/include/asm/pgtable-types.h | 4 ++++ 5 files changed, 28 insertions(+), 2 deletions(-)
arch/powerpc/Kconfig +1 -1
@@ -689,7 +689,7 @@ config PPC_4K_PAGES config PPC_16K_PAGES bool "16k page size" - depends on 44x + depends on 44x || PPC_8xx config PPC_64K_PAGES bool "64k page size"
arch/powerpc/include/asm/nohash/32/mmu-8xx.h +1 -0
@@ -190,6 +190,7 @@ typedef struct {struct slice_mask mask_8m;# endif #endif +void *pte_frag; } mm_context_t;#define PHYS_IMMR_BASE (mfspr(SPRN_IMMR) & 0xfff80000)
arch/powerpc/include/asm/nohash/32/pgtable.h +18 -1
@@ -19,7 +19,14 @@ extern int icache_44x_need_flush;#endif /* __ASSEMBLY__ */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES ) +#define PTE_INDEX_SIZE (PTE_SHIFT - 2) +#define PTE_FRAG_NR 4 +#define PTE_FRAG_SIZE_SHIFT 12 +#define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT) +#else #define PTE_INDEX_SIZE PTE_SHIFT +#endif #define PMD_INDEX_SIZE 0#define PUD_INDEX_SIZE 0
@@ -49,7 +56,11 @@ extern int icache_44x_need_flush;* -Matt */ /* PGDIR_SHIFT determines what a top-level page table entry can map */ +#ifdef CONFIG_PPC_8xx +#define PGDIR_SHIFT 22 +#else #define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE) +#endif #define PGDIR_SIZE (1UL << PGDIR_SHIFT)#define PGDIR_MASK (~(PGDIR_SIZE-1))
@@ -233,7 +244,13 @@ static inline unsigned long pte_update(pte_t *p, : "cc" );#else /* PTE_ATOMIC_UPDATES */ unsigned long old =pte_val (*p); - *p =__pte ((old & ~clr) | set); +unsigned long new = (old & ~clr) | set; + +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES ) + p->pte = p->pte1 = p->pte2 = p->pte3 = new; +#else + *p =__pte (new); +#endif #endif /* !PTE_ATOMIC_UPDATES */ #ifdef CONFIG_44x
arch/powerpc/include/asm/nohash/pgtable.h +4 -0
@@ -209,7 +209,11 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,/* Anything else just stores the PTE normally. That covers all 64-bit * cases, and 32-bit non-hash with 32-bit PTEs. */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES ) + ptep->pte = ptep->pte1 = ptep->pte2 = ptep->pte3 =pte_val (pte); +#else *ptep = pte; +#endif /* * With hardware tablewalk, a sync is needed to ensure that
arch/powerpc/include/asm/pgtable-types.h +4 -0
@@ -3,7 +3,11 @@#define _ASM_POWERPC_PGTABLE_TYPES_H/* PTE level */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES ) +typedef struct {pte_basic_t pte, pte1, pte2, pte3; }pte_t ; +#else typedef struct {pte_basic_t pte; }pte_t ; +#endif #define __pte (x) ((pte_t) { (x) })static inline pte_basic_t pte_val (pte_t x) {
Signature · 1 line
2.13.3
PATCH v8 19/20 powerpc/8xx: don't use r12/SPRN_SPRG_SCRATCH2 in TLB Miss handlers
In reply to Christophe LeroyThis patch reworks the TLB Miss handler in order to not use r12 register, hence avoiding having to save it into SPRN_SPRG_SCRATCH2.
In the DAR Fixup code we can now use SPRN_M_TW, freeing SPRN_SPRG_SCRATCH2.
Then SPRN_SPRG_SCRATCH2 may be used for something else in the future.
--- arch/powerpc/kernel/head_8xx.S | 110 ++++++++++++++++++----------------------- 1 file changed, 49 insertions(+), 61 deletions(-)
arch/powerpc/kernel/head_8xx.S +49 -61
@@ -302,90 +302,87 @@ SystemCall: */ #ifdef CONFIG_8xx_CPU15 -#define INVALIDATE_ADJACENT_PAGES_CPU15(tmp, addr) \ - addi tmp, addr, PAGE_SIZE; \ - tlbie tmp; \ - addi tmp, addr, -PAGE_SIZE; \ - tlbie tmp +#define INVALIDATE_ADJACENT_PAGES_CPU15(addr) \ + addi addr, addr, PAGE_SIZE; \ + tlbie addr; \ + addi addr, addr, -(PAGE_SIZE << 1); \ + tlbie addr; \ + addi addr, addr, PAGE_SIZE #else -#define INVALIDATE_ADJACENT_PAGES_CPU15(tmp, addr) +#define INVALIDATE_ADJACENT_PAGES_CPU15(addr) #endif InstructionTLBMiss: mtspr SPRN_SPRG_SCRATCH0, r10 +#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_SWAP) mtspr SPRN_SPRG_SCRATCH1, r11 -#ifdef ITLB_MISS_KERNEL - mtspr SPRN_SPRG_SCRATCH2, r12 #endif /* If we are faulting a kernel address, we have to use the * kernel page tables. */ mfspr r10, SPRN_SRR0 /* Get effective address of fault */ - INVALIDATE_ADJACENT_PAGES_CPU15(r11, r10) + INVALIDATE_ADJACENT_PAGES_CPU15(r10) mtspr SPRN_MD_EPN, r10 /* Only modules will cause ITLB Misses as we always * pin the first 8MB of kernel memory */ #ifdef ITLB_MISS_KERNEL - mfcr r12 + mfcr r11 #if defined(SIMPLE_KERNEL_ADDRESS) && defined(CONFIG_PIN_TLB_TEXT) - andis. r11, r10, 0x8000 /* Address >= 0x80000000 */ + cmpi cr0, r10, 0 /* Address >= 0x80000000 */ #else - rlwinm r11, r10, 16, 0xfff8 - cmpli cr0, r11, PAGE_OFFSET@h + rlwinm r10, r10, 16, 0xfff8 + cmpli cr0, r10, PAGE_OFFSET@h #ifndef CONFIG_PIN_TLB_TEXT /* It is assumed that kernel code fits into the first 8M page */ -0: cmpli cr7, r11, (PAGE_OFFSET + 0x0800000)@h +0: cmpli cr7, r10, (PAGE_OFFSET + 0x0800000)@h patch_site 0b, patch__itlbmiss_linmem_top #endif #endif #endif - mfspr r11, SPRN_M_TWB /* Get level 1 table */ + mfspr r10, SPRN_M_TWB /* Get level 1 table */ #ifdef ITLB_MISS_KERNEL #if defined(SIMPLE_KERNEL_ADDRESS) && defined(CONFIG_PIN_TLB_TEXT) - beq+ 3f + bge+ 3f #else blt+ 3f #endif #ifndef CONFIG_PIN_TLB_TEXT blt cr7, ITLBMissLinear #endif - rlwinm r11, r11, 0, 20, 31 - oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha + rlwinm r10, r10, 0, 20, 31 + oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@ha 3: #endif - lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + lwz r10, (swapper_pg_dir-PAGE_OFFSET)@l(r10) /* Get level 1 entry */ + mtspr SPRN_MI_TWC, r10 /* Set segment attributes */ - mtspr SPRN_MD_TWC, r11 + mtspr SPRN_MD_TWC, r10 mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ #ifdef ITLB_MISS_KERNEL - mtcr r12 + mtcr r11 #endif - /* Load the MI_TWC with the attributes for this "segment." */ - mtspr SPRN_MI_TWC, r11 /* Set segment attributes */ - #ifdef CONFIG_SWAP rlwinm r11, r10, 32-5, _PAGE_PRESENT and r11, r11, r10 rlwimi r10, r11, 0, _PAGE_PRESENT #endif - li r11, RPN_PATTERN | 0x200 /* The Linux PTE won't go exactly into the MMU TLB. * Software indicator bits 20 and 23 must be clear. * Software indicator bits 22, 24, 25, 26, and 27 must be * set. All other Linux PTE bits control the behavior * of the MMU. */ - rlwimi r11, r10, 4, 0x0400 /* Copy _PAGE_EXEC into bit 21 */ - rlwimi r10, r11, 0, 0x0ff0 /* Set 22, 24-27, clear 20,23 */ + rlwimi r10, r10, 0, 0x0f00 /* Clear bits 20-23 */ + rlwimi r10, r10, 4, 0x0400 /* Copy _PAGE_EXEC into bit 21 */ + ori r10, r10, RPN_PATTERN | 0x200 /* Set 22 and 24-27 */ mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ /* Restore registers */ 0: mfspr r10, SPRN_SPRG_SCRATCH0 +#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_SWAP) mfspr r11, SPRN_SPRG_SCRATCH1 -#ifdef ITLB_MISS_KERNEL - mfspr r12, SPRN_SPRG_SCRATCH2 #endif rfi patch_site 0b, patch__itlbmiss_exit_1
@@ -396,9 +393,8 @@ InstructionTLBMiss: addi r10, r10, 1 stw r10, (itlb_miss_counter - PAGE_OFFSET)@l(0) mfspr r10, SPRN_SPRG_SCRATCH0 +#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_SWAP) mfspr r11, SPRN_SPRG_SCRATCH1 -#ifdef ITLB_MISS_KERNEL - mfspr r12, SPRN_SPRG_SCRATCH2 #endif rfi #endif
@@ -407,40 +403,37 @@ InstructionTLBMiss: DataStoreTLBMiss: mtspr SPRN_SPRG_SCRATCH0, r10 mtspr SPRN_SPRG_SCRATCH1, r11 - mtspr SPRN_SPRG_SCRATCH2, r12 - mfcr r12 + mfcr r11 /* If we are faulting a kernel address, we have to use the * kernel page tables. */ mfspr r10, SPRN_MD_EPN - rlwinm r11, r10, 16, 0xfff8 - cmpli cr0, r11, PAGE_OFFSET@h - mfspr r11, SPRN_M_TWB /* Get level 1 table */ - blt+ 3f - rlwinm r11, r10, 16, 0xfff8 + rlwinm r10, r10, 16, 0xfff8 + cmpli cr0, r10, PAGE_OFFSET@h #ifndef CONFIG_PIN_TLB_IMMR - cmpli cr0, r11, VIRT_IMMR_BASE@h + cmpli cr6, r10, VIRT_IMMR_BASE@h #endif -0: cmpli cr7, r11, (PAGE_OFFSET + 0x1800000)@h +0: cmpli cr7, r10, (PAGE_OFFSET + 0x1800000)@h patch_site 0b, patch__dtlbmiss_linmem_top + + mfspr r10, SPRN_M_TWB /* Get level 1 table */ + blt+ 3f #ifndef CONFIG_PIN_TLB_IMMR -0: beq- DTLBMissIMMR +0: beq- cr6, DTLBMissIMMR patch_site 0b, patch__dtlbmiss_immr_jmp #endif blt cr7, DTLBMissLinear - mfspr r11, SPRN_M_TWB /* Get level 1 table */ - rlwinm r11, r11, 0, 20, 31 - oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@ha + rlwinm r10, r10, 0, 20, 31 + oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@ha 3: - lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + mtcr r11 + lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r10) /* Get level 1 entry */ mtspr SPRN_MD_TWC, r11 mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ - mtcr r12 - /* Insert the Guarded flag into the TWC from the Linux PTE. * It is bit 27 of both the Linux PTE and the TWC (at least * I got that right :-). It will be better when we can put
@@ -478,7 +471,6 @@ DataStoreTLBMiss: 0: mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi patch_site 0b, patch__dtlbmiss_exit_1
@@ -489,7 +481,6 @@ DataStoreTLBMiss: stw r10, (dtlb_miss_counter - PAGE_OFFSET)@l(0) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi #endif
@@ -597,7 +588,7 @@ InstructionBreakpoint: * not enough space in the DataStoreTLBMiss area. */ DTLBMissIMMR: - mtcr r12 + mtcr r11 /* Set 512k byte guarded page and mark it valid */ li r10, MD_PS512K | MD_GUARDED | MD_SVALID mtspr SPRN_MD_TWC, r10
@@ -612,16 +603,15 @@ DTLBMissIMMR: 0: mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi patch_site 0b, patch__dtlbmiss_exit_2 DTLBMissLinear: - mtcr r12 + mtcr r11 /* Set 8M byte page and mark it valid */ li r11, MD_PS8MEG | MD_SVALID mtspr SPRN_MD_TWC, r11 - rlwinm r10, r10, 0, 0x0f800000 /* 8xx supports max 256Mb RAM */ + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ _PAGE_PRESENT mtspr SPRN_MD_RPN, r10 /* Update TLB entry */
@@ -631,24 +621,22 @@ DTLBMissLinear: 0: mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi patch_site 0b, patch__dtlbmiss_exit_3 #ifndef CONFIG_PIN_TLB_TEXT ITLBMissLinear: - mtcr r12 + mtcr r11 /* Set 8M byte page and mark it valid */ li r11, MI_PS8MEG | MI_SVALID mtspr SPRN_MI_TWC, r11 - rlwinm r10, r10, 0, 0x0f800000 /* 8xx supports max 256Mb RAM */ + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ ori r10, r10, 0xf0 | MI_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ _PAGE_PRESENT mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ 0: mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi patch_site 0b, patch__itlbmiss_exit_2 #endif
@@ -660,7 +648,7 @@ ITLBMissLinear: /* define if you don't want to use self modifying code */ #define NO_SELF_MODIFYING_CODE FixupDAR:/* Entry point for dcbx workaround. */ - mtspr SPRN_SPRG_SCRATCH2, r10 + mtspr SPRN_M_TW, r10 /* fetch instruction from memory. */ mfspr r10, SPRN_SRR0 mtspr SPRN_MD_EPN, r10
@@ -705,7 +693,7 @@ FixupDAR:/* Entry point for dcbx workaround. */ beq+ 142f cmpwi cr0, r10, 1964 /* Is icbi? */ beq+ 142f -141: mfspr r10,SPRN_SPRG_SCRATCH2 +141: mfspr r10,SPRN_M_TW b DARFixed /* Nope, go back to normal TLB processing */ 200:
@@ -740,7 +728,7 @@ modified_instr: bne+ 143f subf r10,r0,r10 /* r10=r10-r0, only if reg RA is r0 */ 143: mtdar r10 /* store faulting EA in DAR */ - mfspr r10,SPRN_SPRG_SCRATCH2 + mfspr r10,SPRN_M_TW b DARFixed /* Go back to normal TLB handling */ #else mfctr r10
@@ -794,7 +782,7 @@ modified_instr: mfdar r11 mtctr r11 /* restore ctr reg from DAR */ mtdar r10 /* save fault EA to DAR */ - mfspr r10,SPRN_SPRG_SCRATCH2 + mfspr r10,SPRN_M_TW b DARFixed /* Go back to normal TLB handling */ /* special handling for r10,r11 since these are modified already */
Signature · 1 line
2.13.3
PATCH v8 20/20 powerpc/8xx: regroup TLB handler routines
In reply to Christophe LeroyAs this is running with MMU off, the CPU only does speculative fetch for code in the same page.
Following the significant size reduction of TLB handler routines, the side handlers can be brought back close to the main part, ie in the same page.
--- arch/powerpc/kernel/head_8xx.S | 112 ++++++++++++++++++++--------------------- 1 file changed, 54 insertions(+), 58 deletions(-)
arch/powerpc/kernel/head_8xx.S +54 -58
@@ -399,6 +399,23 @@ InstructionTLBMiss: rfi #endif +#ifndef CONFIG_PIN_TLB_TEXT +ITLBMissLinear: + mtcr r11 + /* Set 8M byte page and mark it valid */ + li r11, MI_PS8MEG | MI_SVALID + mtspr SPRN_MI_TWC, r11 + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ + ori r10, r10, 0xf0 | MI_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ + _PAGE_PRESENT + mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ + +0: mfspr r10, SPRN_SPRG_SCRATCH0 + mfspr r11, SPRN_SPRG_SCRATCH1 + rfi + patch_site 0b, patch__itlbmiss_exit_2 +#endif + . = 0x1200 DataStoreTLBMiss: mtspr SPRN_SPRG_SCRATCH0, r10
@@ -484,6 +501,43 @@ DataStoreTLBMiss: rfi #endif +DTLBMissIMMR: + mtcr r11 + /* Set 512k byte guarded page and mark it valid */ + li r10, MD_PS512K | MD_GUARDED | MD_SVALID + mtspr SPRN_MD_TWC, r10 + mfspr r10, SPRN_IMMR /* Get current IMMR */ + rlwinm r10, r10, 0, 0xfff80000 /* Get 512 kbytes boundary */ + ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ + _PAGE_PRESENT | _PAGE_NO_CACHE + mtspr SPRN_MD_RPN, r10 /* Update TLB entry */ + + li r11, RPN_PATTERN + mtspr SPRN_DAR, r11 /* Tag DAR */ + +0: mfspr r10, SPRN_SPRG_SCRATCH0 + mfspr r11, SPRN_SPRG_SCRATCH1 + rfi + patch_site 0b, patch__dtlbmiss_exit_2 + +DTLBMissLinear: + mtcr r11 + /* Set 8M byte page and mark it valid */ + li r11, MD_PS8MEG | MD_SVALID + mtspr SPRN_MD_TWC, r11 + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ + ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ + _PAGE_PRESENT + mtspr SPRN_MD_RPN, r10 /* Update TLB entry */ + + li r11, RPN_PATTERN + mtspr SPRN_DAR, r11 /* Tag DAR */ + +0: mfspr r10, SPRN_SPRG_SCRATCH0 + mfspr r11, SPRN_SPRG_SCRATCH1 + rfi + patch_site 0b, patch__dtlbmiss_exit_3 + /* This is an instruction TLB error on the MPC8xx. This could be due * to many reasons, such as executing guarded memory or illegal instruction * addresses. There is nothing to do but handle a big time error fault.
@@ -583,64 +637,6 @@ InstructionBreakpoint: . = 0x2000 -/* - * Bottom part of DataStoreTLBMiss handlers for IMMR area and linear RAM. - * not enough space in the DataStoreTLBMiss area. - */ -DTLBMissIMMR: - mtcr r11 - /* Set 512k byte guarded page and mark it valid */ - li r10, MD_PS512K | MD_GUARDED | MD_SVALID - mtspr SPRN_MD_TWC, r10 - mfspr r10, SPRN_IMMR /* Get current IMMR */ - rlwinm r10, r10, 0, 0xfff80000 /* Get 512 kbytes boundary */ - ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ - _PAGE_PRESENT | _PAGE_NO_CACHE - mtspr SPRN_MD_RPN, r10 /* Update TLB entry */ - - li r11, RPN_PATTERN - mtspr SPRN_DAR, r11 /* Tag DAR */ - -0: mfspr r10, SPRN_SPRG_SCRATCH0 - mfspr r11, SPRN_SPRG_SCRATCH1 - rfi - patch_site 0b, patch__dtlbmiss_exit_2 - -DTLBMissLinear: - mtcr r11 - /* Set 8M byte page and mark it valid */ - li r11, MD_PS8MEG | MD_SVALID - mtspr SPRN_MD_TWC, r11 - rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ - ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ - _PAGE_PRESENT - mtspr SPRN_MD_RPN, r10 /* Update TLB entry */ - - li r11, RPN_PATTERN - mtspr SPRN_DAR, r11 /* Tag DAR */ - -0: mfspr r10, SPRN_SPRG_SCRATCH0 - mfspr r11, SPRN_SPRG_SCRATCH1 - rfi - patch_site 0b, patch__dtlbmiss_exit_3 - -#ifndef CONFIG_PIN_TLB_TEXT -ITLBMissLinear: - mtcr r11 - /* Set 8M byte page and mark it valid */ - li r11, MI_PS8MEG | MI_SVALID - mtspr SPRN_MI_TWC, r11 - rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ - ori r10, r10, 0xf0 | MI_SPS16K | _PAGE_SH | _PAGE_DIRTY | \ - _PAGE_PRESENT - mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ - -0: mfspr r10, SPRN_SPRG_SCRATCH0 - mfspr r11, SPRN_SPRG_SCRATCH1 - rfi - patch_site 0b, patch__itlbmiss_exit_2 -#endif - /* This is the procedure to calculate the data EA for buggy dcbx,dcbi instructions * by decoding the registers used by the dcbx instruction and adding them. * DAR is set to the calculated address.
Signature · 1 line
2.13.3