Reduce alignment constraint on STRICT_KERNEL_RWX and speed-up TLB misses on 8xx and 603
Subsystems (2)
cover Reduce alignment constraint on STRICT_KERNEL_RWX and speed-up TLB misses on 8xx and 603
This series does mainly two things:
- Remove the 8M alignment constraint on STRICT_KERNEL_RWX on 8xx to avoid wasting memory.
- Speed-up TLB misses by duplicating kernel PGD entries into each task's PGDIRs to avoid the address comparison in TLB miss handler.
On 8xx, the address comparison takes a significant part of CPU cycles as it requires saving/restoring CR, and because a taken branch requires 2 cycles.
On 603 it is less significant because CR is saved automatically and has to be restored anyway but it is still worth it.
For ITLB misses:
- Kernel PGD entries are setup for once during init, before creation of new PGDIRs.
- Module PGD entries are setup also at init by preallocating page tables for a very few number of pages
For DTLB misses:
- Some handling is added in 8xx DATA TLB error interrupt and in 603 DATA read and store TLB miss interrupts to copy missing PGD entries into child.
The cost of that additional handling on error paths is worth the gain on hot TLB miss pathes.
Because 8xx and 603 don't use leaf kernel pages at PGD level, there is no need to care about PGD entries cleanup, page tables are never freed.
Christophe Leroy (14):
powerpc/8xx: Fix initial memory mapping
powerpc/8xx: Fix kernel vs user address comparison powerpc/8xx: Copy kernel PGD entries into all PGDIRs Revert "powerpc/8xx: Always pin kernel text TLB" powerpc/8xx: Allow setting DATA alignment even with STRICT_KERNEL_RWX powerpc/8xx: Reduce default size of module/execmem area powerpc/8xx: Preallocate execmem page tables
powerpc/8xx: Inconditionally use task PGDIR in ITLB misses powerpc/8xx: Inconditionally use task PGDIR in DTLB misses powerpc/32s: Reduce default size of module/execmem area powerpc/603: Copy kernel PGD entries into all PGDIRs and preallocate
execmem page tables
powerpc/603: Switch r0 and r3 in TLB miss handlers powerpc/603: Inconditionally use task PGDIR in ITLB misses powerpc/603: Inconditionally use task PGDIR in DTLB misses
arch/powerpc/Kconfig | 31 +++- arch/powerpc/include/asm/book3s/32/pgtable.h | 3 +- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 3 +- arch/powerpc/include/asm/nohash/pgalloc.h | 8 +- arch/powerpc/kernel/head_8xx.S | 78 +++++----- arch/powerpc/kernel/head_book3s_32.S | 144 +++++++++---------- arch/powerpc/mm/book3s32/mmu.c | 2 + arch/powerpc/mm/mem.c | 14 ++ arch/powerpc/mm/nohash/8xx.c | 9 +- arch/powerpc/platforms/8xx/Kconfig | 7 + 10 files changed, 173 insertions(+), 126 deletions(-)
Signature · 1 line
2.44.0
PATCH 1/14 powerpc/8xx: Fix initial memory mapping
Commit cf209951fa7f ("powerpc/8xx: Map linear memory with huge pages") introduced an initial mapping of kernel TEXT using PAGE_KERNEL_TEXT, but the pages that contain kernel TEXT may also contain kernel RODATA, and depending on selected debug options PAGE_KERNEL_TEXT may be either RWX or ROX. RODATA must be writable during init because it also contains ro_after_init data.
So use PAGE_KERNEL_X instead to be sure it is RWX.
--- arch/powerpc/mm/nohash/8xx.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/powerpc/mm/nohash/8xx.c +2 -2
@@ -150,11 +150,11 @@ unsigned long __init mmu_mapin_ram(unsigned long base, unsigned long top)mmu_mapin_immr (); -mmu_mapin_ram_chunk (0 , boundary,PAGE_KERNEL_TEXT , true); +mmu_mapin_ram_chunk (0 , boundary,PAGE_KERNEL_X , true);if (debug_pagealloc_enabled_or_kfence ()) { top = boundary; }else { -mmu_mapin_ram_chunk (boundary, einittext8,PAGE_KERNEL_TEXT , true); +mmu_mapin_ram_chunk (boundary, einittext8,PAGE_KERNEL_X , true);mmu_mapin_ram_chunk (einittext8, top,PAGE_KERNEL , true); }
Signature · 1 line
2.44.0
PATCH 2/14 powerpc/8xx: Fix kernel vs user address comparison
In reply to Christophe LeroySince commit 9132a2e82adc ("powerpc/8xx: Define a MODULE area below kernel text"), module exec space is below PAGE_OFFSET so not only space above PAGE_OFFSET, but space above TASK_SIZE need to be seen as kernel space.
Until now the problem went undetected because by default TASK_SIZE is 0x8000000 which means address space is determined by just checking upper address bit. But when TASK_SIZE is over 0x80000000, PAGE_OFFSET is used for comparison, leading to thinking module addresses are part of user space.
Fix it by using TASK_SIZE instead of PAGE_OFFSET for address comparison.
--- arch/powerpc/kernel/head_8xx.S | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/powerpc/kernel/head_8xx.S +3 -3
@@ -41,12 +41,12 @@ #include "head_32.h" .macro compare_to_kernel_boundary scratch, addr -#if CONFIG_TASK_SIZE <= 0x80000000 && CONFIG_PAGE_OFFSET >= 0x80000000 +#if CONFIG_TASK_SIZE <= 0x80000000 && MODULES_VADDR >= 0x80000000 /* By simply checking Address >= 0x80000000, we know if its a kernel address */ not. \scratch, \addr #else rlwinm \scratch, \addr, 16, 0xfff8 - cmpli cr0, \scratch, PAGE_OFFSET@h + cmpli cr0, \scratch, TASK_SIZE@h #endif .endm
@@ -404,7 +404,7 @@ FixupDAR:/* Entry point for dcbx workaround. */ mfspr r10, SPRN_SRR0 mtspr SPRN_MD_EPN, r10 rlwinm r11, r10, 16, 0xfff8 - cmpli cr1, r11, PAGE_OFFSET@h + cmpli cr1, r11, TASK_SIZE@h mfspr r11, SPRN_M_TWB /* Get level 1 table */ blt+ cr1, 3f
Signature · 1 line
2.44.0
PATCH 3/14 powerpc/8xx: Copy kernel PGD entries into all PGDIRs
In reply to Christophe LeroyIn order to avoid having to select PGDIR at each TLB miss based on fault address, copy kernel PGD entries into all PGDIRs in pgd_alloc().
At first it will be used for ITLB misses for kernel TEXT, then for execmem then for kernel DATA.
--- arch/powerpc/include/asm/nohash/pgalloc.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/powerpc/include/asm/nohash/pgalloc.h +7 -1
@@ -19,8 +19,14 @@ static inline void tlb_flush_pgtable(struct mmu_gather *tlb,static inline pgd_t *pgd_alloc (struct mm_struct *mm) { -return kmem_cache_alloc (PGT_CACHE (PGD_INDEX_SIZE ), +pgd_t *pgd =kmem_cache_alloc (PGT_CACHE (PGD_INDEX_SIZE ),pgtable_gfp_flags (mm,GFP_KERNEL )); + +#ifdef CONFIG_PPC_8xx +memcpy (pgd +USER_PTRS_PER_PGD , swapper_pg_dir +USER_PTRS_PER_PGD , + (MAX_PTRS_PER_PGD -USER_PTRS_PER_PGD ) *sizeof (pgd_t)); +#endif +return pgd; }static inline void pgd_free (struct mm_struct *mm,pgd_t *pgd)
Signature · 1 line
2.44.0
PATCH 4/14 Revert "powerpc/8xx: Always pin kernel text TLB"
In reply to Christophe LeroyThis reverts commit bccc58986a2f98e3af349c85c5f49aac7fb19ef2.
When STRICT_KERNEL_RWX is selected, EXEC memory must stop where RW memory start. When pinning iTLBs it means an 8M alignment for RW data start. That may be acceptable on boards with a lot of memory but one of my supported boards only has 32 Mbytes and this forced alignment leads to a waste of almost 4 Mbytes with is more than 10% of the total memory.
So revert commit bccc58986a2f ("powerpc/8xx: Always pin kernel text TLB") but don't restore previous behaviour in ITLB miss handler as now kernel PGD entries are copied into each process PGDIR.
--- arch/powerpc/kernel/head_8xx.S | 8 ++++++++ arch/powerpc/mm/nohash/8xx.c | 3 ++- arch/powerpc/platforms/8xx/Kconfig | 7 +++++++ 3 files changed, 17 insertions(+), 1 deletion(-)
arch/powerpc/kernel/head_8xx.S +8 -0
@@ -587,6 +587,10 @@ start_here: lis r0, (MD_TWAM | MD_RSV4I)@h mtspr SPRN_MD_CTR, r0 #endif +#ifndef CONFIG_PIN_TLB_TEXT + li r0, 0 + mtspr SPRN_MI_CTR, r0 +#endif #if !defined(CONFIG_PIN_TLB_DATA) && !defined(CONFIG_PIN_TLB_IMMR) lis r0, MD_TWAM@h mtspr SPRN_MD_CTR, r0
@@ -683,6 +687,7 @@ SYM_FUNC_START_LOCAL(initial_mmu) blr SYM_FUNC_END(initial_mmu) +#ifdef CONFIG_PIN_TLB _GLOBAL(mmu_pin_tlb) lis r9, (1f - PAGE_OFFSET)@h ori r9, r9, (1f - PAGE_OFFSET)@l
@@ -704,6 +709,7 @@ _GLOBAL(mmu_pin_tlb) mtspr SPRN_MD_CTR, r6 tlbia +#ifdef CONFIG_PIN_TLB_TEXT LOAD_REG_IMMEDIATE(r5, 28 << 8) LOAD_REG_IMMEDIATE(r6, PAGE_OFFSET) LOAD_REG_IMMEDIATE(r7, MI_SVALID | MI_PS8MEG | _PMD_ACCESSED)
@@ -724,6 +730,7 @@ _GLOBAL(mmu_pin_tlb) bdnzt lt, 2b lis r0, MI_RSV4I@h mtspr SPRN_MI_CTR, r0 +#endif LOAD_REG_IMMEDIATE(r5, 28 << 8 | MD_TWAM) #ifdef CONFIG_PIN_TLB_DATA
@@ -783,3 +790,4 @@ _GLOBAL(mmu_pin_tlb) mtspr SPRN_SRR1, r10 mtspr SPRN_SRR0, r11 rfi +#endif
arch/powerpc/mm/nohash/8xx.c +2 -1
@@ -177,7 +177,8 @@ int mmu_mark_initmem_nx(void)if (!debug_pagealloc_enabled_or_kfence ()) err =mmu_mapin_ram_chunk (boundary, einittext8,PAGE_KERNEL , false); -mmu_pin_tlb (block_mapped_ram, false); +if (IS_ENABLED (CONFIG_PIN_TLB_TEXT )) +mmu_pin_tlb (block_mapped_ram, false);return err; }
arch/powerpc/platforms/8xx/Kconfig +7 -0
@@ -203,6 +203,13 @@ config PIN_TLB_IMMR CONFIG_PIN_TLB_DATA is also selected, it will reduce CONFIG_PIN_TLB_DATA to 24 Mbytes. +config PIN_TLB_TEXT + bool "Pinned TLB for TEXT" + depends on PIN_TLB + default y + help + This pins kernel text with 8M pages. + endmenu endmenu
Signature · 1 line
2.44.0
PATCH 5/14 powerpc/8xx: Allow setting DATA alignment even with STRICT_KERNEL_RWX
In reply to Christophe LeroyIt is now possible to not pin kernel text with a 8Mbytes TLB, so the alignment for STRICT_KERNEL_RWX can be relaxed.
--- arch/powerpc/Kconfig | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/powerpc/Kconfig +5 -5
@@ -853,8 +853,8 @@ config DATA_SHIFT_BOOL bool "Set custom data alignment" depends on ADVANCED_OPTIONS depends on STRICT_KERNEL_RWX || DEBUG_PAGEALLOC || KFENCE - depends on PPC_BOOK3S_32 || (PPC_8xx && !PIN_TLB_DATA && !STRICT_KERNEL_RWX) || \ - PPC_85xx + depends on (PPC_8xx && !PIN_TLB_DATA && (!STRICT_KERNEL_RWX || !PIN_TLB_TEXT)) || \ + PPC_BOOK3S_32 || PPC_85xx help This option allows you to set the kernel data alignment. When RAM is mapped by blocks, the alignment needs to fit the size and
@@ -870,9 +870,9 @@ config DATA_SHIFT range 20 24 if (STRICT_KERNEL_RWX || DEBUG_PAGEALLOC || KFENCE) && PPC_85xx default 22 if STRICT_KERNEL_RWX && PPC_BOOK3S_32 default 18 if (DEBUG_PAGEALLOC || KFENCE) && PPC_BOOK3S_32 - default 23 if STRICT_KERNEL_RWX && PPC_8xx - default 23 if (DEBUG_PAGEALLOC || KFENCE) && PPC_8xx && PIN_TLB_DATA - default 19 if (DEBUG_PAGEALLOC || KFENCE) && PPC_8xx + default 23 if (STRICT_KERNEL_RWX || DEBUG_PAGEALLOC || KFENCE) && PPC_8xx && \ + (PIN_TLB_DATA || PIN_TLB_TEXT) + default 19 if (STRICT_KERNEL_RWX || DEBUG_PAGEALLOC || KFENCE) && PPC_8xx default 24 if STRICT_KERNEL_RWX && PPC_85xx default PAGE_SHIFT help
Signature · 1 line
2.44.0
PATCH 6/14 powerpc/8xx: Reduce default size of module/execmem area
In reply to Christophe Leroy8xx boards don't have much memory, the two I know have respectively 32Mbytes and 128Mbytes, so there is no point in having 256 Mbytes of memory for module text.
Reduce it to 32Mbytes for 8xx, that's more than enough.
Nevertheless, make it a configurable value so that it can be customised if needed.
Also add a build verification for overlap of module execmem space with user PMD.
--- arch/powerpc/Kconfig | 18 ++++++++++++++++++ arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 3 ++- arch/powerpc/mm/nohash/8xx.c | 2 ++ 3 files changed, 22 insertions(+), 1 deletion(-)
arch/powerpc/Kconfig +18 -0
@@ -1271,6 +1271,24 @@ config TASK_SIZE default "0x80000000" if PPC_8xx default "0xb0000000" if PPC_BOOK3S_32 default "0xc0000000" + +config MODULES_SIZE_BOOL + bool "Set custom size for modules/execmem area" + depends on EXECMEM && ADVANCED_OPTIONS + depends on PPC_8xx + help + This option allows you to set the size of kernel virtual address + space dedicated for modules/execmem. + For the time being it is only for 8xx. + + Say N here unless you know what you are doing. + +config MODULES_SIZE + int "Size of modules/execmem area (In Mbytes)" if MODULES_SIZE_BOOL + range 1 256 if EXECMEM + default 32 if EXECMEM && PPC_8xx + default 0 + endmenu if PPC64
arch/powerpc/include/asm/nohash/32/mmu-8xx.h +2 -1
@@ -170,8 +170,9 @@#define mmu_linear_psize MMU_PAGE_8M -#define MODULES_VADDR (PAGE_OFFSET - SZ_256M)#define MODULES_END PAGE_OFFSET +#define MODULES_SIZE (CONFIG_MODULES_SIZE * SZ_1M) +#define MODULES_VADDR (MODULES_END - MODULES_SIZE)#ifndef __ASSEMBLY__
arch/powerpc/mm/nohash/8xx.c +2 -0
@@ -207,6 +207,8 @@ void __init setup_initial_memory_limit(phys_addr_t first_memblock_base,/* 8xx can only access 32MB at the moment */ memblock_set_current_limit (min_t (u64, first_memblock_size,SZ_32M )); + +BUILD_BUG_ON (ALIGN_DOWN (MODULES_VADDR ,PGDIR_SIZE ) <TASK_SIZE ); }int pud_clear_huge (pud_t *pud)
Signature · 1 line
2.44.0
PATCH 7/14 powerpc/8xx: Preallocate execmem page tables
In reply to Christophe LeroyPreallocate execmem page tables before creating new PGDs so that all PGD entries related to execmem can be copied in pgd_alloc().
On 8xx there are 32 Mbytes for execmem by default so this will use 32 kbytes.
--- arch/powerpc/mm/mem.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
arch/powerpc/mm/mem.c +14 -0
@@ -410,6 +410,18 @@ EXPORT_SYMBOL_GPL(walk_system_ram_range);#ifdef CONFIG_EXECMEM static struct execmem_info execmem_info__ro_after_init ; +#ifdef CONFIG_PPC_8xx +static void prealloc_execmem_pgtable (void ) +{ +unsigned long va; + +for (va =ALIGN_DOWN (MODULES_VADDR ,PGDIR_SIZE ); va <MODULES_END ; va +=PGDIR_SIZE ) +pte_alloc_kernel (pmd_off_k (va), va); +} +#else +static void prealloc_execmem_pgtable (void ) { } +#endif +struct execmem_info __init *execmem_arch_setup (void ) {pgprot_t kprobes_prot =strict_module_rwx_enabled () ?PAGE_KERNEL_ROX :PAGE_KERNEL_EXEC ;
@@ -441,6 +453,8 @@ struct execmem_info __init *execmem_arch_setup(void) end =VMALLOC_END ;#endif +prealloc_execmem_pgtable (); + execmem_info = (struct execmem_info ){ .ranges = { [EXECMEM_DEFAULT ] = {
Signature · 1 line
2.44.0
PATCH 8/14 powerpc/8xx: Inconditionally use task PGDIR in ITLB misses
In reply to Christophe LeroyNow that modules exec page tables are preallocated, the instruction TLBmiss handler can use task PGDIR inconditionally.
--- arch/powerpc/kernel/head_8xx.S | 11 ----------- 1 file changed, 11 deletions(-)
arch/powerpc/kernel/head_8xx.S +0 -11
@@ -199,18 +199,7 @@ instruction_counter: mfspr r10, SPRN_SRR0 /* Get effective address of fault */ INVALIDATE_ADJACENT_PAGES_CPU15(r10, r11) mtspr SPRN_MD_EPN, r10 -#ifdef CONFIG_EXECMEM - mfcr r11 - compare_to_kernel_boundary r10, r10 -#endif mfspr r10, SPRN_M_TWB /* Get level 1 table */ -#ifdef CONFIG_EXECMEM - blt+ 3f - rlwinm r10, r10, 0, 20, 31 - oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@ha -3: - mtcr r11 -#endif lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r10) /* Get level 1 entry */ mtspr SPRN_MD_TWC, r11 mfspr r10, SPRN_MD_TWC
Signature · 1 line
2.44.0
PATCH 9/14 powerpc/8xx: Inconditionally use task PGDIR in DTLB misses
In reply to Christophe LeroyAt the time being, DATA TLB miss handlers use task PGDIR for user addresses and swapper_pg_dir for kernel addresses.
Now that kernel part of swapper_pg_dir is copied into task PGDIR at PGD allocation, it is possible to avoid the above logic and always use task PGDIR.
But new kernel PGD entries can still be created after init, in which case those PGD entries may miss in task PGDIR. This can be handled in DATA TLB error handler.
However, it needs to be done in real mode because the missing entry might be related to the stack.
So implement copy of missing PGD entry in the prolog of DATA TLB ERROR handler just after the fixup of DAR.
Note that this is feasible because 8xx doesn't implement vmap or ioremap with 8Mbytes pages but only 512kbytes pages which are at PTE level.
--- arch/powerpc/kernel/head_8xx.S | 57 ++++++++++++++++++++-------------- 1 file changed, 34 insertions(+), 23 deletions(-)
arch/powerpc/kernel/head_8xx.S +34 -23
@@ -40,16 +40,6 @@ #include "head_32.h" -.macro compare_to_kernel_boundary scratch, addr -#if CONFIG_TASK_SIZE <= 0x80000000 && MODULES_VADDR >= 0x80000000 -/* By simply checking Address >= 0x80000000, we know if its a kernel address */ - not. \scratch, \addr -#else - rlwinm \scratch, \addr, 16, 0xfff8 - cmpli cr0, \scratch, TASK_SIZE@h -#endif -.endm - #define PAGE_SHIFT_512K 19 #define PAGE_SHIFT_8M 23
@@ -237,19 +227,12 @@ instruction_counter: START_EXCEPTION(INTERRUPT_DATA_TLB_MISS_8xx, DataStoreTLBMiss) mtspr SPRN_SPRG_SCRATCH2, r10 mtspr SPRN_M_TW, r11 - mfcr r11 /* If we are faulting a kernel address, we have to use the * kernel page tables. */ mfspr r10, SPRN_MD_EPN - compare_to_kernel_boundary r10, r10 mfspr r10, SPRN_M_TWB /* Get level 1 table */ - blt+ 3f - rlwinm r10, r10, 0, 20, 31 - oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@ha -3: - mtcr r11 lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r10) /* Get level 1 entry */ mtspr SPRN_MD_TWC, r11
@@ -321,15 +304,19 @@ instruction_counter: cmpwi cr1, r11, RPN_PATTERN beq- cr1, FixupDAR /* must be a buggy dcbX, icbi insn. */ DARFixed:/* Return from dcbx instruction bug workaround */ + mfspr r11, SPRN_DSISR + rlwinm r11, r11, 0, DSISR_NOHPTE + cmpwi cr1, r11, 0 + beq+ cr1, .Ldtlbie + mfspr r11, SPRN_DAR + tlbie r11 + rlwinm r11, r11, 16, 0xffff + cmplwi cr1, r11, TASK_SIZE@h + bge- cr1, FixupPGD +.Ldtlbie: EXCEPTION_PROLOG_1 /* 0x300 is DataAccess exception, needed by bad_page_fault() */ EXCEPTION_PROLOG_2 INTERRUPT_DATA_STORAGE DataTLBError handle_dar_dsisr=1 - lwz r4, _DAR(r11) - lwz r5, _DSISR(r11) - andis. r10,r5,DSISR_NOHPTE@h - beq+ .Ldtlbie - tlbie r4 -.Ldtlbie: prepare_transfer_to_handler bl do_page_fault b interrupt_return
@@ -383,6 +370,30 @@ DARFixed:/* Return from dcbx instruction bug workaround */ __HEAD . = 0x2000 +FixupPGD: + mtspr SPRN_M_TW, r10 + mfspr r10, SPRN_DAR + mtspr SPRN_MD_EPN, r10 + mfspr r11, SPRN_M_TWB /* Get level 1 table */ + lwz r10, (swapper_pg_dir - PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + cmpwi cr1, r10, 0 + bne cr1, 1f + + rlwinm r10, r11, 0, 20, 31 + oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@ha + lwz r10, (swapper_pg_dir - PAGE_OFFSET)@l(r10) /* Get the level 1 entry */ + cmpwi cr1, r10, 0 + beq cr1, 1f + stw r10, (swapper_pg_dir - PAGE_OFFSET)@l(r11) /* Set the level 1 entry */ + mfspr r10, SPRN_M_TW + mtcr r10 + mfspr r10, SPRN_SPRG_SCRATCH0 + mfspr r11, SPRN_SPRG_SCRATCH1 + rfi +1: + mfspr r10, SPRN_M_TW + b .Ldtlbie + /* This is the procedure to calculate the data EA for buggy dcbx,dcbi instructions * by decoding the registers used by the dcbx instruction and adding them. * DAR is set to the calculated address.
Signature · 1 line
2.44.0
PATCH 10/14 powerpc/32s: Reduce default size of module/execmem area
In reply to Christophe Leroybook3s/32 platforms have usually more memory than 8xx, but it is still not worth reserving a full segment (256 Mbytes) for module text. 64Mbytes should be far enough.
Also fix TASK_SIZE when EXECMEM is not selected, and add a build verification for overlap of module execmem space with user segments.
--- arch/powerpc/Kconfig | 7 ++++--- arch/powerpc/include/asm/book3s/32/pgtable.h | 3 ++- arch/powerpc/mm/book3s32/mmu.c | 2 ++ 3 files changed, 8 insertions(+), 4 deletions(-)
arch/powerpc/Kconfig +4 -3
@@ -1269,23 +1269,24 @@ config TASK_SIZE_BOOL config TASK_SIZE hex "Size of user task space" if TASK_SIZE_BOOL default "0x80000000" if PPC_8xx - default "0xb0000000" if PPC_BOOK3S_32 + default "0xb0000000" if PPC_BOOK3S_32 && EXECMEM default "0xc0000000" config MODULES_SIZE_BOOL bool "Set custom size for modules/execmem area" depends on EXECMEM && ADVANCED_OPTIONS - depends on PPC_8xx help This option allows you to set the size of kernel virtual address space dedicated for modules/execmem. - For the time being it is only for 8xx. + For the time being it is only for 8xx and book3s/32. Other + platform share it with vmalloc space. Say N here unless you know what you are doing. config MODULES_SIZE int "Size of modules/execmem area (In Mbytes)" if MODULES_SIZE_BOOL range 1 256 if EXECMEM + default 64 if EXECMEM && PPC_BOOK3S_32 default 32 if EXECMEM && PPC_8xx default 0
arch/powerpc/include/asm/book3s/32/pgtable.h +2 -1
@@ -196,7 +196,8 @@ void unmap_kernel_page(unsigned long va);#endif #define MODULES_END ALIGN_DOWN(PAGE_OFFSET, SZ_256M) -#define MODULES_VADDR (MODULES_END - SZ_256M) +#define MODULES_SIZE (CONFIG_MODULES_SIZE * SZ_1M) +#define MODULES_VADDR (MODULES_END - MODULES_SIZE)#ifndef __ASSEMBLY__#include <linux/sched.h>
arch/powerpc/mm/book3s32/mmu.c +2 -0
@@ -223,6 +223,8 @@ int mmu_mark_initmem_nx(void)update_bats (); +BUILD_BUG_ON (ALIGN_DOWN (MODULES_VADDR ,SZ_256M ) <TASK_SIZE ); +for (i =TASK_SIZE >>28 ; i <16 ; i++) {/* Do not set NX on VM space for modules */ if (is_module_segment (i <<28 ))
Signature · 1 line
2.44.0
PATCH 11/14 powerpc/603: Copy kernel PGD entries into all PGDIRs and preallocate execmem page tables
In reply to Christophe LeroyFor the same reason as 8xx, copy kernel PGD entries into all PGDIRs in pgd_alloc() and preallocate execmem page tables before creating new PGDs so that all PGD entries related to execmem are copied by pgd_alloc().
This will help reduce the fast-path in TLBmiss handlers.
--- arch/powerpc/include/asm/nohash/pgalloc.h | 2 +- arch/powerpc/mm/mem.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/nohash/pgalloc.h +1 -1
@@ -22,7 +22,7 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm)pgd_t *pgd =kmem_cache_alloc (PGT_CACHE (PGD_INDEX_SIZE ),pgtable_gfp_flags (mm,GFP_KERNEL )); -#ifdef CONFIG_PPC_8xx +#if defined(CONFIG_PPC_8xx) || defined(CONFIG_PPC_BOOK3S_603 )memcpy (pgd +USER_PTRS_PER_PGD , swapper_pg_dir +USER_PTRS_PER_PGD , (MAX_PTRS_PER_PGD -USER_PTRS_PER_PGD ) *sizeof (pgd_t));#endif
arch/powerpc/mm/mem.c +1 -1
@@ -410,7 +410,7 @@ EXPORT_SYMBOL_GPL(walk_system_ram_range); #ifdef CONFIG_EXECMEM static struct execmem_info execmem_info __ro_after_init; -#ifdef CONFIG_PPC_8xx +#if defined(CONFIG_PPC_8xx) || defined(CONFIG_PPC_BOOK3S_603) static void prealloc_execmem_pgtable(void) { unsigned long va;
Signature · 1 line
2.44.0
PATCH 12/14 powerpc/603: Switch r0 and r3 in TLB miss handlers
In reply to Christophe LeroyIn preparation of next patch that will perform some additional calculations to replace comparison, switch the use of r0 and r3 as r0 has some limitations in some instructions like 'addi/subi'.
Also remove outdated comments about the meaning of each register. The registers are used for many things and it would be difficult to accurately describe all things done with a given register. The function is now small enough to get a global view without much description.
--- arch/powerpc/kernel/head_book3s_32.S | 94 +++++++++++----------------- 1 file changed, 38 insertions(+), 56 deletions(-)
arch/powerpc/kernel/head_book3s_32.S +38 -56
@@ -411,39 +411,33 @@ END_FTR_SECTION_IFSET(CPU_FTR_FPU_UNAVAILABLE) */ . = INTERRUPT_INST_TLB_MISS_603 InstructionTLBMiss: -/* - * r0: userspace flag (later scratch) - * r1: linux style pte ( later becomes ppc hardware pte ) - * r2: ptr to linux-style pte - * r3: fault address - */ /* Get PTE (linux-style) and check access */ - mfspr r3,SPRN_IMISS + mfspr r0,SPRN_IMISS #ifdef CONFIG_EXECMEM lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r3 + cmplw 0,r1,r0 #endif mfspr r2, SPRN_SDR1 li r1,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC rlwinm r2, r2, 28, 0xfffff000 #ifdef CONFIG_EXECMEM - li r0, 3 + li r3, 3 bgt- 112f lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r0, 0 + li r3, 0 addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ #endif -112: rlwimi r2,r3,12,20,29 /* insert top 10 bits of address */ +112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ lwz r2,0(r2) /* get pmd entry */ rlwinm. r2,r2,0,0,19 /* extract address of pte page */ beq- InstructionAddressInvalid /* return if no mapping */ - rlwimi r2,r3,22,20,29 /* insert next 10 bits of address */ + rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ lwz r2,0(r2) /* get linux-style pte */ andc. r1,r1,r2 /* check access & ~permission */ bne- InstructionAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ #ifdef CONFIG_EXECMEM - rlwimi r2, r0, 0, 31, 31 /* userspace ? -> PP lsb */ + rlwimi r2, r3, 0, 31, 31 /* userspace ? -> PP lsb */ #endif ori r1, r1, 0xe06 /* clear out reserved bits */ andc r1, r2, r1 /* PP = user? 1 : 0 */
@@ -451,7 +445,7 @@ BEGIN_FTR_SECTION rlwinm r1,r1,0,~_PAGE_COHERENT /* clear M (coherence not required) */ END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) mtspr SPRN_RPA,r1 - tlbli r3 + tlbli r0 mfspr r3,SPRN_SRR1 /* Need to restore CR0 */ mtcrf 0x80,r3 rfi
@@ -480,35 +474,29 @@ InstructionAddressInvalid: */ . = INTERRUPT_DATA_LOAD_TLB_MISS_603 DataLoadTLBMiss: -/* - * r0: userspace flag (later scratch) - * r1: linux style pte ( later becomes ppc hardware pte ) - * r2: ptr to linux-style pte - * r3: fault address - */ /* Get PTE (linux-style) and check access */ - mfspr r3,SPRN_DMISS + mfspr r0,SPRN_DMISS lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r3 + cmplw 0,r1,r0 mfspr r2, SPRN_SDR1 li r1, _PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_READ rlwinm r2, r2, 28, 0xfffff000 - li r0, 3 + li r3, 3 bgt- 112f lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r0, 0 + li r3, 0 addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ -112: rlwimi r2,r3,12,20,29 /* insert top 10 bits of address */ +112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ lwz r2,0(r2) /* get pmd entry */ rlwinm. r2,r2,0,0,19 /* extract address of pte page */ beq- DataAddressInvalid /* return if no mapping */ - rlwimi r2,r3,22,20,29 /* insert next 10 bits of address */ + rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ lwz r2,0(r2) /* get linux-style pte */ andc. r1,r1,r2 /* check access & ~permission */ bne- DataAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ rlwinm r1,r2,32-9,30,30 /* _PAGE_WRITE -> PP msb */ - rlwimi r2,r0,0,30,31 /* userspace ? -> PP */ + rlwimi r2,r3,0,30,31 /* userspace ? -> PP */ rlwimi r1,r2,32-3,24,24 /* _PAGE_WRITE -> _PAGE_DIRTY */ xori r1,r1,_PAGE_DIRTY /* clear dirty when not rw */ ori r1,r1,0xe04 /* clear out reserved bits */
@@ -518,23 +506,23 @@ BEGIN_FTR_SECTION END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) mtspr SPRN_RPA,r1 BEGIN_MMU_FTR_SECTION - li r0,1 + li r3,1 mfspr r1,SPRN_SPRG_603_LRU - rlwinm r2,r3,20,27,31 /* Get Address bits 15:19 */ - slw r0,r0,r2 - xor r1,r0,r1 - srw r0,r1,r2 + rlwinm r2,r0,20,27,31 /* Get Address bits 15:19 */ + slw r3,r3,r2 + xor r1,r3,r1 + srw r3,r1,r2 mtspr SPRN_SPRG_603_LRU,r1 mfspr r2,SPRN_SRR1 - rlwimi r2,r0,31-14,14,14 + rlwimi r2,r3,31-14,14,14 mtspr SPRN_SRR1,r2 mtcrf 0x80,r2 - tlbld r3 + tlbld r0 rfi MMU_FTR_SECTION_ELSE mfspr r2,SPRN_SRR1 /* Need to restore CR0 */ mtcrf 0x80,r2 - tlbld r3 + tlbld r0 rfi ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_NEED_DTLB_SW_LRU) DataAddressInvalid:
@@ -560,34 +548,28 @@ DataAddressInvalid: */ . = INTERRUPT_DATA_STORE_TLB_MISS_603 DataStoreTLBMiss: -/* - * r0: userspace flag (later scratch) - * r1: linux style pte ( later becomes ppc hardware pte ) - * r2: ptr to linux-style pte - * r3: fault address - */ /* Get PTE (linux-style) and check access */ - mfspr r3,SPRN_DMISS + mfspr r0,SPRN_DMISS lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r3 + cmplw 0,r1,r0 mfspr r2, SPRN_SDR1 li r1, _PAGE_RW | _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED rlwinm r2, r2, 28, 0xfffff000 - li r0, 3 + li r3, 3 bgt- 112f lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r0, 0 + li r3, 0 addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ -112: rlwimi r2,r3,12,20,29 /* insert top 10 bits of address */ +112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ lwz r2,0(r2) /* get pmd entry */ rlwinm. r2,r2,0,0,19 /* extract address of pte page */ beq- DataAddressInvalid /* return if no mapping */ - rlwimi r2,r3,22,20,29 /* insert next 10 bits of address */ + rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ lwz r2,0(r2) /* get linux-style pte */ andc. r1,r1,r2 /* check access & ~permission */ bne- DataAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ - rlwimi r2,r0,0,31,31 /* userspace ? -> PP lsb */ + rlwimi r2,r3,0,31,31 /* userspace ? -> PP lsb */ li r1,0xe06 /* clear out reserved bits & PP msb */ andc r1,r2,r1 /* PP = user? 1: 0 */ BEGIN_FTR_SECTION
@@ -597,23 +579,23 @@ END_FTR_SECTION_IFCLR(CPU_FTR_NEED_COHERENT) mfspr r2,SPRN_SRR1 /* Need to restore CR0 */ mtcrf 0x80,r2 BEGIN_MMU_FTR_SECTION - li r0,1 + li r3,1 mfspr r1,SPRN_SPRG_603_LRU - rlwinm r2,r3,20,27,31 /* Get Address bits 15:19 */ - slw r0,r0,r2 - xor r1,r0,r1 - srw r0,r1,r2 + rlwinm r2,r0,20,27,31 /* Get Address bits 15:19 */ + slw r3,r3,r2 + xor r1,r3,r1 + srw r3,r1,r2 mtspr SPRN_SPRG_603_LRU,r1 mfspr r2,SPRN_SRR1 - rlwimi r2,r0,31-14,14,14 + rlwimi r2,r3,31-14,14,14 mtspr SPRN_SRR1,r2 mtcrf 0x80,r2 - tlbld r3 + tlbld r0 rfi MMU_FTR_SECTION_ELSE mfspr r2,SPRN_SRR1 /* Need to restore CR0 */ mtcrf 0x80,r2 - tlbld r3 + tlbld r0 rfi ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_NEED_DTLB_SW_LRU)
Signature · 1 line
2.44.0
PATCH 13/14 powerpc/603: Inconditionally use task PGDIR in ITLB misses
In reply to Christophe LeroyNow that modules exec page tables are preallocated, the instruction TLBmiss handler can use task PGDIR inconditionally.
Also revise the identification of user vs kernel user space by doing a calculation instead of a comparison: Get the segment number and subtract the number of the first kernel segment. The result is positive for kernel addresses and negative for user addresses, which means that upper 2 bits are 0 for kernel and 3 for user.
--- arch/powerpc/kernel/head_book3s_32.S | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
arch/powerpc/kernel/head_book3s_32.S +5 -12
@@ -413,22 +413,15 @@ END_FTR_SECTION_IFSET(CPU_FTR_FPU_UNAVAILABLE) InstructionTLBMiss: /* Get PTE (linux-style) and check access */ mfspr r0,SPRN_IMISS -#ifdef CONFIG_EXECMEM - lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r0 -#endif mfspr r2, SPRN_SDR1 li r1,_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_EXEC rlwinm r2, r2, 28, 0xfffff000 + rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ + lwz r2,0(r2) /* get pmd entry */ #ifdef CONFIG_EXECMEM - li r3, 3 - bgt- 112f - lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r3, 0 - addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ + rlwinm r3, r0, 4, 0xf + subi r3, r3, (TASK_SIZE >> 28) & 0xf #endif -112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ - lwz r2,0(r2) /* get pmd entry */ rlwinm. r2,r2,0,0,19 /* extract address of pte page */ beq- InstructionAddressInvalid /* return if no mapping */ rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */
@@ -437,7 +430,7 @@ InstructionTLBMiss: bne- InstructionAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ #ifdef CONFIG_EXECMEM - rlwimi r2, r3, 0, 31, 31 /* userspace ? -> PP lsb */ + rlwimi r2, r3, 1, 31, 31 /* userspace ? -> PP lsb */ #endif ori r1, r1, 0xe06 /* clear out reserved bits */ andc r1, r2, r1 /* PP = user? 1 : 0 */
Signature · 1 line
2.44.0
PATCH 14/14 powerpc/603: Inconditionally use task PGDIR in DTLB misses
In reply to Christophe LeroyAt the time being, DATA TLB miss handlers use task PGDIR for user addresses and swapper_pg_dir for kernel addresses.
Now that kernel part of swapper_pg_dir is copied into task PGDIR at PGD allocation, it is possible to avoid the above logic and always use task PGDIR.
But new kernel PGD entries can still be created after init, in which case those PGD entries may miss in task PGDIR. This can be handled in DATA TLB error handler.
However, it needs to be done in real mode because the missing entry might be related to the stack.
So implement copy of missing PGD entry in DATA TLB miss handler just after detection of invalid PGD entry.
Also replace comparison by same calculation as in previous patch to know if an address belongs to a kernel or user segment.
Note that as mentioned in platforms/Kconfig.cputype, SMP is not supported on 603 processors so there is no risk of the PGD entry be populated during the fault.
--- arch/powerpc/kernel/head_book3s_32.S | 65 ++++++++++++++++------------ 1 file changed, 38 insertions(+), 27 deletions(-)
arch/powerpc/kernel/head_book3s_32.S +38 -27
@@ -469,27 +469,22 @@ InstructionAddressInvalid: DataLoadTLBMiss: /* Get PTE (linux-style) and check access */ mfspr r0,SPRN_DMISS - lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r0 mfspr r2, SPRN_SDR1 - li r1, _PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_READ - rlwinm r2, r2, 28, 0xfffff000 - li r3, 3 - bgt- 112f - lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r3, 0 - addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ -112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ - lwz r2,0(r2) /* get pmd entry */ + rlwinm r1, r2, 28, 0xfffff000 + rlwimi r1,r0,12,20,29 /* insert top 10 bits of address */ + lwz r2,0(r1) /* get pmd entry */ + rlwinm r3, r0, 4, 0xf rlwinm. r2,r2,0,0,19 /* extract address of pte page */ - beq- DataAddressInvalid /* return if no mapping */ - rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ + subi r3, r3, (TASK_SIZE >> 28) & 0xf + beq- 2f /* bail if no mapping */ +1: rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ lwz r2,0(r2) /* get linux-style pte */ + li r1, _PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_READ andc. r1,r1,r2 /* check access & ~permission */ bne- DataAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ rlwinm r1,r2,32-9,30,30 /* _PAGE_WRITE -> PP msb */ - rlwimi r2,r3,0,30,31 /* userspace ? -> PP */ + rlwimi r2,r3,2,30,31 /* userspace ? -> PP */ rlwimi r1,r2,32-3,24,24 /* _PAGE_WRITE -> _PAGE_DIRTY */ xori r1,r1,_PAGE_DIRTY /* clear dirty when not rw */ ori r1,r1,0xe04 /* clear out reserved bits */
@@ -518,6 +513,16 @@ MMU_FTR_SECTION_ELSE tlbld r0 rfi ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_NEED_DTLB_SW_LRU) + +2: lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha + addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ + rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ + lwz r2,0(r2) /* get pmd entry */ + cmpwi cr0,r2,0 + beq- DataAddressInvalid /* return if no mapping */ + stw r2,0(r1) + rlwinm. r2,r2,0,0,19 /* extract address of pte page */ + b 1b DataAddressInvalid: mfspr r3,SPRN_SRR1 rlwinm r1,r3,9,6,6 /* Get load/store bit */
@@ -543,26 +548,22 @@ DataAddressInvalid: DataStoreTLBMiss: /* Get PTE (linux-style) and check access */ mfspr r0,SPRN_DMISS - lis r1, TASK_SIZE@h /* check if kernel address */ - cmplw 0,r1,r0 mfspr r2, SPRN_SDR1 - li r1, _PAGE_RW | _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED - rlwinm r2, r2, 28, 0xfffff000 - li r3, 3 - bgt- 112f - lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha /* if kernel address, use */ - li r3, 0 - addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ -112: rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ - lwz r2,0(r2) /* get pmd entry */ + rlwinm r1, r2, 28, 0xfffff000 + rlwimi r1,r0,12,20,29 /* insert top 10 bits of address */ + lwz r2,0(r1) /* get pmd entry */ + rlwinm r3, r0, 4, 0xf rlwinm. r2,r2,0,0,19 /* extract address of pte page */ - beq- DataAddressInvalid /* return if no mapping */ + subi r3, r3, (TASK_SIZE >> 28) & 0xf + beq- 2f /* bail if no mapping */ +1: rlwimi r2,r0,22,20,29 /* insert next 10 bits of address */ lwz r2,0(r2) /* get linux-style pte */ + li r1, _PAGE_RW | _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED andc. r1,r1,r2 /* check access & ~permission */ bne- DataAddressInvalid /* return if access not permitted */ /* Convert linux-style PTE to low word of PPC-style PTE */ - rlwimi r2,r3,0,31,31 /* userspace ? -> PP lsb */ + rlwimi r2,r3,1,31,31 /* userspace ? -> PP lsb */ li r1,0xe06 /* clear out reserved bits & PP msb */ andc r1,r2,r1 /* PP = user? 1: 0 */ BEGIN_FTR_SECTION
@@ -592,6 +593,16 @@ MMU_FTR_SECTION_ELSE rfi ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_NEED_DTLB_SW_LRU) +2: lis r2, (swapper_pg_dir - PAGE_OFFSET)@ha + addi r2, r2, (swapper_pg_dir - PAGE_OFFSET)@l /* kernel page table */ + rlwimi r2,r0,12,20,29 /* insert top 10 bits of address */ + lwz r2,0(r2) /* get pmd entry */ + cmpwi cr0,r2,0 + beq- DataAddressInvalid /* return if no mapping */ + stw r2,0(r1) + rlwinm r2,r2,0,0,19 /* extract address of pte page */ + b 1b + #ifndef CONFIG_ALTIVEC #define altivec_assist_exception unknown_exception #endif
Signature · 1 line
2.44.0
Michael Ellerman
AppliedQuoted 6 lines · Christophe Leroy
This series does mainly two things:
- Remove the 8M alignment constraint onSTRICT_KERNEL_RWXon 8xx to avoid wasting memory.
- Speed-up TLB misses by duplicating kernel PGD entries into each task's PGDIRs to avoid the address comparison in TLB miss handler.
Quoted 7 lines
On 8xx, the address comparison takes a significant part of CPU cycles as it requires saving/restoring CR, and because a taken branch requires 2 cycles.
On 603 it is less significant because CR is saved automatically and has to be restored anyway but it is still worth it.[...]
Applied to powerpc/next.
[01/14] powerpc/8xx: Fix initial memory mapping
https://git.kernel.org/powerpc/c/f9f2bff64c2f0dbee57be3d8c2741357ad3d05e6 [02/14] powerpc/8xx: Fix kernel vs user address comparison https://git.kernel.org/powerpc/c/65a82e117ffeeab0baf6f871a1cab11a28ace183 [03/14] powerpc/8xx: Copy kernel PGD entries into all PGDIRs https://git.kernel.org/powerpc/c/985db026c34dfc45213649023d5505822a5dcd78 [04/14] Revert "powerpc/8xx: Always pin kernel text TLB" https://git.kernel.org/powerpc/c/1a736d98c84acd38e40fff69528ce7aaa55dd22d [05/14] powerpc/8xx: Allow setting DATA alignment even with STRICT_KERNEL_RWX https://git.kernel.org/powerpc/c/bcf77a70c4ffc9b01044229de87f5b6f9c1f7913 [06/14] powerpc/8xx: Reduce default size of module/execmem area https://git.kernel.org/powerpc/c/c5eec4df25c34f4bee8c757ed157f5d96eaba554 [07/14] powerpc/8xx: Preallocate execmem page tables https://git.kernel.org/powerpc/c/16a71c045186a11c1c743934e330de78162b86dd [08/14] powerpc/8xx: Inconditionally use task PGDIR in ITLB misses https://git.kernel.org/powerpc/c/33c527522f394f63cc589a6f7af990b2232444c8 [09/14] powerpc/8xx: Inconditionally use task PGDIR in DTLB misses https://git.kernel.org/powerpc/c/ac9f97ff8b324905d457f2694490c63b9deccbc6 [10/14] powerpc/32s: Reduce default size of module/execmem area https://git.kernel.org/powerpc/c/2f2b9a3adc66e978a1248ffb38df8477e8e97c57 [11/14] powerpc/603: Copy kernel PGD entries into all PGDIRs and preallocate execmem page tables https://git.kernel.org/powerpc/c/82ef440f9a38a1fd7f4854397633a35af33840a5 [12/14] powerpc/603: Switch r0 and r3 in TLB miss handlers https://git.kernel.org/powerpc/c/31c0e137ec609f36877ea39cd343ef2476d080aa [13/14] powerpc/603: Inconditionally use task PGDIR in ITLB misses https://git.kernel.org/powerpc/c/3f57d90c231d3329aaed7079dd05b5a2f7692a58 [14/14] powerpc/603: Inconditionally use task PGDIR in DTLB misses https://git.kernel.org/powerpc/c/062e825a336017c0334c7497690826c95aa1a84f
cheers