powerpc: Optimise KUAP on book3s/32
Subsystems (2)
v2 cover powerpc: Optimise KUAP on book3s/32
This series is a rework of KUAP on book3s/32.
On book3s32, KUAP is heavier than on other platform because it can't be opened globaly at once, it must be done for each 256Mb segment.
Instead of opening access to all necessary segments via a heavy logic, only open access to the segment matching the start of the range.
99.999% of the time, the range doesn't cross segment limit so it is not worth checking and handling that corner case at the first place. If that happens, take a fault and do the additional segments opening in the fault handler. See patch 9 for details.
This series also adds the capability to disable KUAP and KUEP at boot time via the "nosmap" and "nosmep" kernel parameters as PPC64 does.
Last 2 patches are cleanup of generic KUAP code once book3s/32 has been simplified.
Changes in v2:
- Incorporated the conversion to C of switch_mmu_context()
- Converted initial KUAP/KUEP setup to C
- Added capability to disable KUAP and KUEP at boottime
- Enable KUAP and KUEP by default
Christophe Leroy (12):
powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c powerpc/32s: Refactor update of user segment registers powerpc/32s: move CTX_TO_VSID() into mmu-hash.h powerpc/32s: Convert switch_mmu_context() to C powerpc/32s: Simplify calculation of segment register content powerpc/32s: Initialise KUAP and KUEP in C
powerpc/32s: Allow disabling KUEP at boot time powerpc/32s: Allow disabling KUAP at boot time powerpc/32s: Rework Kernel Userspace Access Protection powerpc/32s: Activate KUAP and KUEP by default powerpc/kuap: Remove KUAP_CURRENT_XXX
powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
arch/powerpc/include/asm/book3s/32/kup.h | 199 ++++++++++++------ arch/powerpc/include/asm/book3s/32/mmu-hash.h | 41 ++++ arch/powerpc/include/asm/book3s/64/kup.h | 3 +- arch/powerpc/include/asm/kup.h | 28 +-- arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +- arch/powerpc/include/asm/processor.h | 10 +- arch/powerpc/kernel/asm-offsets.c | 5 - arch/powerpc/kernel/head_book3s_32.S | 64 ------ arch/powerpc/kernel/process.c | 3 + arch/powerpc/kernel/smp.c | 4 + arch/powerpc/kvm/book3s_32_mmu_host.c | 3 - arch/powerpc/mm/book3s32/Makefile | 1 + arch/powerpc/mm/book3s32/kuap.c | 30 +++ arch/powerpc/mm/book3s32/kuep.c | 42 +--- arch/powerpc/mm/book3s32/mmu.c | 20 -- arch/powerpc/mm/book3s32/mmu_context.c | 48 +++-- arch/powerpc/platforms/Kconfig.cputype | 4 +- 17 files changed, 276 insertions(+), 232 deletions(-) create mode 100644 arch/powerpc/mm/book3s32/kuap.c
Signature · 1 line
2.25.0
PATCH v2 1/12 powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
Avoids the #ifdef in mmu.c
--- arch/powerpc/mm/book3s32/Makefile | 1 + arch/powerpc/mm/book3s32/kuap.c | 11 +++++++++++ arch/powerpc/mm/book3s32/kuep.c | 8 ++++++++ arch/powerpc/mm/book3s32/mmu.c | 20 -------------------- 4 files changed, 20 insertions(+), 20 deletions(-) create mode 100644 arch/powerpc/mm/book3s32/kuap.c
arch/powerpc/mm/book3s32/Makefile +1 -0
@@ -10,3 +10,4 @@ obj-y += mmu.o mmu_context.o obj-$(CONFIG_PPC_BOOK3S_603) += nohash_low.o obj-$(CONFIG_PPC_BOOK3S_604) += hash_low.o tlb.o obj-$(CONFIG_PPC_KUEP) += kuep.o +obj-$(CONFIG_PPC_KUAP) += kuap.o
arch/powerpc/mm/book3s32/kuap.c new file +11 -0
@@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include <asm/kup.h> + +void __init setup_kuap (bool disabled) +{ +pr_info ("Activating Kernel Userspace Access Protection\n" ); + +if (disabled) +pr_warn ("KUAP cannot be disabled yet on 6xx when compiled in\n" ); +}
arch/powerpc/mm/book3s32/kuep.c +8 -0
@@ -38,3 +38,11 @@ void kuep_unlock(void) {kuep_update (mfsr (0 ) & ~SR_NX ); } + +void __init setup_kuep (bool disabled) +{ +pr_info ("Activating Kernel Userspace Execution Prevention\n" ); + +if (disabled) +pr_warn ("KUEP cannot be disabled yet on 6xx when compiled in\n" ); +}
arch/powerpc/mm/book3s32/mmu.c +0 -20
@@ -445,26 +445,6 @@ void __init print_system_hash_info(void)pr_info ("Hash_mask = 0x%lx\n" , Hash_mask); } -#ifdef CONFIG_PPC_KUEP -void __init setup_kuep (bool disabled) -{ -pr_info ("Activating Kernel Userspace Execution Prevention\n" ); - -if (disabled) -pr_warn ("KUEP cannot be disabled yet on 6xx when compiled in\n" ); -} -#endif - -#ifdef CONFIG_PPC_KUAP -void __init setup_kuap (bool disabled) -{ -pr_info ("Activating Kernel Userspace Access Protection\n" ); - -if (disabled) -pr_warn ("KUAP cannot be disabled yet on 6xx when compiled in\n" ); -} -#endif -void __init early_init_mmu (void ) { }
Signature · 1 line
2.25.0
PATCH v2 2/12 powerpc/32s: Refactor update of user segment registers
In reply to Christophe LeroyKUEP implements the update of user segment registers.
Move it into mmu-hash.h in order to use it from other places.
And inline kuep_lock() and kuep_unlock(). Inlining kuep_lock() is important for system_call_exception(), otherwise system_call_exception() has to save into stack the system call parameters that are used just after, and doing that takes more instructions than kuep_lock() itself.
--- arch/powerpc/include/asm/book3s/32/kup.h | 21 +++++++++++ arch/powerpc/include/asm/book3s/32/mmu-hash.h | 27 ++++++++++++++ arch/powerpc/include/asm/kup.h | 5 +-- arch/powerpc/mm/book3s32/kuep.c | 37 ------------------- 4 files changed, 49 insertions(+), 41 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +21 -0
@@ -7,6 +7,27 @@#ifndef __ASSEMBLY__ +static __always_inline bool kuep_is_disabled(void) +{ +return !IS_ENABLED (CONFIG_PPC_KUEP ); +} + +static inline void kuep_lock (void ) +{ +if (kuep_is_disabled ()) +return ; + +update_user_segments (mfsr (0 ) |SR_NX ); +} + +static inline void kuep_unlock (void ) +{ +if (kuep_is_disabled ()) +return ; + +update_user_segments (mfsr (0 ) & ~SR_NX ); +} +#ifdef CONFIG_PPC_KUAP #include <linux/sched.h>
arch/powerpc/include/asm/book3s/32/mmu-hash.h +27 -0
@@ -102,6 +102,33 @@ extern s32 patch__hash_page_B, patch__hash_page_C;extern s32 patch__flush_hash_A0, patch__flush_hash_A1, patch__flush_hash_A2;extern s32 patch__flush_hash_B; +#include <asm/reg.h> +#include <asm/task_size_32.h> + +#define UPDATE_TWO_USER_SEGMENTS (n) do { \ + if (TASK_SIZE > ((n) << 28)) \ + mtsr(val1, (n) << 28); \ + if (TASK_SIZE > (((n) + 1) << 28)) \ + mtsr(val2, ((n) + 1) << 28); \ + val1 = (val1 + 0x222) & 0xf0ffffff; \ + val2 = (val2 + 0x222) & 0xf0ffffff; \ +} while (0) + +static __always_inline void update_user_segments(u32 val) +{ +int val1 = val; +int val2 = (val +0x111 ) &0xf0ffffff ; + +UPDATE_TWO_USER_SEGMENTS (0 ); +UPDATE_TWO_USER_SEGMENTS (2 ); +UPDATE_TWO_USER_SEGMENTS (4 ); +UPDATE_TWO_USER_SEGMENTS (6 ); +UPDATE_TWO_USER_SEGMENTS (8 ); +UPDATE_TWO_USER_SEGMENTS (10 ); +UPDATE_TWO_USER_SEGMENTS (12 ); +UPDATE_TWO_USER_SEGMENTS (14 ); +} +#endif /* !__ASSEMBLY__ */ /* We happily ignore the smaller BATs on 601, we don't actually use
arch/powerpc/include/asm/kup.h +1 -4
@@ -46,10 +46,7 @@ void setup_kuep(bool disabled);static inline void setup_kuep (bool disabled) { }#endif /* CONFIG_PPC_KUEP */ -#if defined(CONFIG_PPC_KUEP ) && defined(CONFIG_PPC_BOOK3S_32 ) -void kuep_lock (void ); -void kuep_unlock (void ); -#else +#ifndef CONFIG_PPC_BOOK3S_32 static inline void kuep_lock (void ) { }static inline void kuep_unlock (void ) { }#endif
arch/powerpc/mm/book3s32/kuep.c +0 -37
@@ -1,43 +1,6 @@// SPDX-License-Identifier: GPL-2.0-or-later #include <asm/kup.h> -#include <asm/reg.h> -#include <asm/task_size_32.h> -#include <asm/mmu.h> - -#define KUEP_UPDATE_TWO_USER_SEGMENTS (n) do { \ - if (TASK_SIZE > ((n) << 28)) \ - mtsr(val1, (n) << 28); \ - if (TASK_SIZE > (((n) + 1) << 28)) \ - mtsr(val2, ((n) + 1) << 28); \ - val1 = (val1 + 0x222) & 0xf0ffffff; \ - val2 = (val2 + 0x222) & 0xf0ffffff; \ -} while (0) - -static __always_inline void kuep_update(u32 val) -{ -int val1 = val; -int val2 = (val +0x111 ) &0xf0ffffff ; - -KUEP_UPDATE_TWO_USER_SEGMENTS (0 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (2 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (4 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (6 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (8 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (10 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (12 ); -KUEP_UPDATE_TWO_USER_SEGMENTS (14 ); -} - -void kuep_lock (void ) -{ -kuep_update (mfsr (0 ) |SR_NX ); -} - -void kuep_unlock (void ) -{ -kuep_update (mfsr (0 ) & ~SR_NX ); -}void __init setup_kuep (bool disabled) {
Signature · 1 line
2.25.0
PATCH v2 3/12 powerpc/32s: move CTX_TO_VSID() into mmu-hash.h
In reply to Christophe LeroyFrom: Christophe Leroy <[email protected]>
In order to reuse it in switch_mmu_context(), this patch moves CTX_TO_VSID() macro into asm/book3s/32/mmu-hash.h
--- arch/powerpc/include/asm/book3s/32/mmu-hash.h | 10 ++++++++++ arch/powerpc/kvm/book3s_32_mmu_host.c | 3 --- arch/powerpc/mm/book3s32/mmu_context.c | 13 ------------- 3 files changed, 10 insertions(+), 16 deletions(-)
arch/powerpc/include/asm/book3s/32/mmu-hash.h +10 -0
@@ -66,6 +66,16 @@ struct ppc_bat {#ifndef __ASSEMBLY__ +/* +* This macro defines the mapping from contexts to VSIDs (virtual +* segment IDs). We use a skew on both the context and the high 4 bits +* of the 32-bit virtual address (the "effective segment ID") in order +* to spread out the entries in the MMU hash table. Note, if this +* function is changed then hash functions will have to be +* changed to correspond. +*/ +#define CTX_TO_VSID (c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff) +/* * Hardware Page Table Entry * Note that the xpn and x bitfields are used only by processors that
arch/powerpc/kvm/book3s_32_mmu_host.c +0 -3
@@ -353,9 +353,6 @@ void kvmppc_mmu_destroy_pr(struct kvm_vcpu *vcpu)preempt_enable (); } -/* From mm/mmu_context_hash32.c */ -#define CTX_TO_VSID (c, id) ((((c) * (897 * 16)) + (id * 0x111)) & 0xffffff) -int kvmppc_mmu_init_pr (struct kvm_vcpu *vcpu) {struct kvmppc_vcpu_book3s *vcpu3s =to_book3s (vcpu);
arch/powerpc/mm/book3s32/mmu_context.c +0 -13
@@ -39,19 +39,6 @@#define LAST_CONTEXT 32767#define FIRST_CONTEXT 1 -/* -* This function defines the mapping from contexts to VSIDs (virtual -* segment IDs). We use a skew on both the context and the high 4 bits -* of the 32-bit virtual address (the "effective segment ID") in order -* to spread out the entries in the MMU hash table. Note, if this -* function is changed then arch/ppc/mm/hashtable.S will have to be -* changed to correspond. -* -* -* CTX_TO_VSID(ctx, va) (((ctx) * (897 * 16) + ((va) >> 28) * 0x111) \ -* & 0xffffff) -*/ -static unsigned long next_mmu_context;static unsigned long context_map[LAST_CONTEXT /BITS_PER_LONG +1 ];
Signature · 1 line
2.25.0
PATCH v2 4/12 powerpc/32s: Convert switch_mmu_context() to C
In reply to Christophe Leroyswitch_mmu_context() does things that can easily be done in C.
For updating user segments, we have update_user_segments().
As mentionned in commit b5efec00b671 ("powerpc/32s: Move KUEP locking/unlocking in C"), update_user_segments() has the loop unrolled which is a significant performance gain.
--- arch/powerpc/include/asm/book3s/32/kup.h | 5 ++ arch/powerpc/kernel/asm-offsets.c | 5 -- arch/powerpc/kernel/head_book3s_32.S | 58 ------------------------ arch/powerpc/mm/book3s32/mmu_context.c | 35 ++++++++++++++ 4 files changed, 40 insertions(+), 63 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +5 -0
@@ -7,6 +7,11 @@#ifndef __ASSEMBLY__ +static __always_inline bool kuap_is_disabled(void) +{ +return !IS_ENABLED (CONFIG_PPC_KUAP ); +} + static __always_inline bool kuep_is_disabled(void) {return !IS_ENABLED (CONFIG_PPC_KUEP );
arch/powerpc/kernel/asm-offsets.c +0 -5
@@ -86,7 +86,6 @@ int main(void)OFFSET (PACA_CANARY , paca_struct, canary);#endif #endif -OFFSET (MMCONTEXTID , mm_struct, context.id);#ifdef CONFIG_PPC32 #ifdef CONFIG_PPC_RTAS OFFSET (RTAS_SP , thread_struct, rtas_sp);
@@ -325,10 +324,6 @@ int main(void)#endif #endif -#ifndef CONFIG_PPC64 -OFFSET (MM_PGD , mm_struct, pgd); -#endif /* ! CONFIG_PPC64 */ -/* About the CPU features table */ OFFSET (CPU_SPEC_FEATURES , cpu_spec, cpu_features);OFFSET (CPU_SPEC_SETUP , cpu_spec, cpu_setup);
arch/powerpc/kernel/head_book3s_32.S +0 -58
67 lines not on this page · raw message
arch/powerpc/mm/book3s32/mmu_context.c +35 -0
@@ -23,6 +23,12 @@#include <asm/mmu_context.h> +/* +* Room for two PTE pointers, usually the kernel and current user pointers +* to their respective root page table. +*/ +void *abatron_pteptrs[2 ]; +/* * On 32-bit PowerPC 6xx/7xx/7xxx CPUs, we use a set of 16 VSIDs * (virtual segment identifiers) for each context. Although the
@@ -98,3 +104,32 @@ void __init mmu_context_init(void) context_map[0 ] = (1 <<FIRST_CONTEXT ) -1 ; next_mmu_context =FIRST_CONTEXT ; } + +void switch_mmu_context (struct mm_struct *prev,struct mm_struct *next,struct task_struct *tsk) +{ +long id = next->context.id; +unsigned long val; + +if (id <0 ) +panic ("mm_struct %p has no context ID" , next); + +isync (); + + val =CTX_TO_VSID (id,0 ); +if (!kuep_is_disabled ()) + val |=SR_NX ; +if (!kuap_is_disabled ()) + val |=SR_KS ; + +update_user_segments (val); + +if (IS_ENABLED (CONFIG_BDI_SWITCH )) + abatron_pteptrs[1 ] = next->pgd; + +if (!mmu_has_feature (MMU_FTR_HPTE_TABLE )) +mtspr (SPRN_SDR1 ,rol32 (__pa (next->pgd),4 ) &0xffff01ff ); + +mb ();/* sync */ +isync (); +} +EXPORT_SYMBOL (switch_mmu_context);
Signature · 1 line
2.25.0
PATCH v2 5/12 powerpc/32s: Simplify calculation of segment register content
In reply to Christophe Leroysegment register has VSID on bits 8-31.
Bits 4-7 are reserved, there is no requirement to set them to 0.
VSIDs are calculated from VSID of SR0 by adding 0x111.
Even with highest possible VSID which would be 0xFFFFF0, adding 16 times 0x111 results in 0x1001100.
So, the reserved bits are never overflowed, no need to clear the reserved bits after each calculation.
--- arch/powerpc/include/asm/book3s/32/mmu-hash.h | 42 ++++++++++--------- 1 file changed, 23 insertions(+), 19 deletions(-)
arch/powerpc/include/asm/book3s/32/mmu-hash.h +23 -19
@@ -115,28 +115,32 @@ extern s32 patch__flush_hash_B;#include <asm/reg.h> #include <asm/task_size_32.h> -#define UPDATE_TWO_USER_SEGMENTS (n) do { \ - if (TASK_SIZE > ((n) << 28)) \ - mtsr(val1, (n) << 28); \ - if (TASK_SIZE > (((n) + 1) << 28)) \ - mtsr(val2, ((n) + 1) << 28); \ - val1 = (val1 + 0x222) & 0xf0ffffff; \ - val2 = (val2 + 0x222) & 0xf0ffffff; \ -} while (0) +static __always_inline void update_user_segment(u32 n, u32 val) +{ +if (n <<28 <TASK_SIZE ) +mtsr (val + n *0x111 , n <<28 ); +} static __always_inline void update_user_segments(u32 val) { -int val1 = val; -int val2 = (val +0x111 ) &0xf0ffffff ; - -UPDATE_TWO_USER_SEGMENTS (0 ); -UPDATE_TWO_USER_SEGMENTS (2 ); -UPDATE_TWO_USER_SEGMENTS (4 ); -UPDATE_TWO_USER_SEGMENTS (6 ); -UPDATE_TWO_USER_SEGMENTS (8 ); -UPDATE_TWO_USER_SEGMENTS (10 ); -UPDATE_TWO_USER_SEGMENTS (12 ); -UPDATE_TWO_USER_SEGMENTS (14 ); + val &=0xf0ffffff ; + +update_user_segment (0 , val); +update_user_segment (1 , val); +update_user_segment (2 , val); +update_user_segment (3 , val); +update_user_segment (4 , val); +update_user_segment (5 , val); +update_user_segment (6 , val); +update_user_segment (7 , val); +update_user_segment (8 , val); +update_user_segment (9 , val); +update_user_segment (10 , val); +update_user_segment (11 , val); +update_user_segment (12 , val); +update_user_segment (13 , val); +update_user_segment (14 , val); +update_user_segment (15 , val); }#endif /* !__ASSEMBLY__ */
Signature · 1 line
2.25.0
PATCH v2 6/12 powerpc/32s: Initialise KUAP and KUEP in C
In reply to Christophe LeroyIn order to selectively activate KUAP and KUEP in a following patch, perform KUAP and KUEP initialisation in C.
Unlike PPC64, PPC32 doesn't have an early_setup_secondary(), so do it in start_secondary().
--- arch/powerpc/kernel/head_book3s_32.S | 6 ------ arch/powerpc/kernel/smp.c | 4 ++++ arch/powerpc/mm/book3s32/kuap.c | 6 ++++++ arch/powerpc/mm/book3s32/kuep.c | 6 ++++++ 4 files changed, 16 insertions(+), 6 deletions(-)
arch/powerpc/kernel/head_book3s_32.S +0 -6
@@ -934,12 +934,6 @@ _GLOBAL(load_segment_registers) li r0, NUM_USER_SEGMENTS /* load up user segment register values */ mtctr r0 /* for context 0 */ li r3, 0 /* Kp = 0, Ks = 0, VSID = 0 */ -#ifdef CONFIG_PPC_KUEP - oris r3, r3, SR_NX@h /* Set Nx */ -#endif -#ifdef CONFIG_PPC_KUAP - oris r3, r3, SR_KS@h /* Set Ks */ -#endif li r4, 0 3: mtsrin r3, r4 addi r3, r3, 0x111 /* increment VSID */
arch/powerpc/kernel/smp.c +4 -0
@@ -1541,6 +1541,10 @@ void start_secondary(void *unused) {unsigned int cpu =raw_smp_processor_id (); +/* PPC64 calls setup_kup() in early_setup_secondary() */ +if (IS_ENABLED (CONFIG_PPC32 )) +setup_kup (); +mmgrab (&init_mm); current->active_mm = &init_mm;
arch/powerpc/mm/book3s32/kuap.c +6 -0
@@ -1,9 +1,15 @@// SPDX-License-Identifier: GPL-2.0-or-later #include <asm/kup.h> +#include <asm/smp.h> void __init setup_kuap (bool disabled) { +kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE ); + +if (smp_processor_id () != boot_cpuid) +return ; +pr_info ("Activating Kernel Userspace Access Protection\n" );if (disabled)
arch/powerpc/mm/book3s32/kuep.c +6 -0
@@ -1,9 +1,15 @@// SPDX-License-Identifier: GPL-2.0-or-later #include <asm/kup.h> +#include <asm/smp.h> void __init setup_kuep (bool disabled) { +kuep_lock (); + +if (smp_processor_id () != boot_cpuid) +return ; +pr_info ("Activating Kernel Userspace Execution Prevention\n" );if (disabled)
Signature · 1 line
2.25.0
PATCH v2 7/12 powerpc/32s: Allow disabling KUEP at boot time
In reply to Christophe LeroyPPC64 uses MMU features to enable/disable KUEP at boot time. But feature fixups are applied way too early on PPC32.
Now that all KUEP related actions are in C following the conversion of KUEP initial setup and context switch in C, static branches can be used to enable/disable KUEP.
--- arch/powerpc/include/asm/book3s/32/kup.h | 6 +++++- arch/powerpc/mm/book3s32/kuep.c | 11 +++++++---- 2 files changed, 12 insertions(+), 5 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +5 -1
@@ -7,6 +7,10 @@#ifndef __ASSEMBLY__ +#include <linux/jump_label.h> + +extern struct static_key_false disable_kuep_key; + static __always_inline bool kuap_is_disabled(void) {return !IS_ENABLED (CONFIG_PPC_KUAP );
@@ -14,7 +18,7 @@ static __always_inline bool kuap_is_disabled(void) static __always_inline bool kuep_is_disabled(void) { -return !IS_ENABLED (CONFIG_PPC_KUEP ); +return !IS_ENABLED (CONFIG_PPC_KUEP ) ||static_branch_unlikely (&disable_kuep_key); }static inline void kuep_lock (void )
arch/powerpc/mm/book3s32/kuep.c +7 -4
@@ -3,15 +3,18 @@#include <asm/kup.h> #include <asm/smp.h> +struct static_key_false disable_kuep_key; +void __init setup_kuep (bool disabled) { -kuep_lock (); +if (!disabled) +kuep_lock ();if (smp_processor_id () != boot_cpuid)return ; -pr_info ("Activating Kernel Userspace Execution Prevention\n" ); -if (disabled) -pr_warn ("KUEP cannot be disabled yet on 6xx when compiled in\n" ); +static_branch_enable (&disable_kuep_key); +else +pr_info ("Activating Kernel Userspace Execution Prevention\n" ); }
Signature · 1 line
2.25.0
PATCH v2 8/12 powerpc/32s: Allow disabling KUAP at boot time
In reply to Christophe LeroyPPC64 uses MMU features to enable/disable KUAP at boot time. But feature fixups are applied way too early on PPC32.
Now that all KUAP related actions are in C following the conversion of KUAP initial setup and context switch in C, static branches can be used to enable/disable KUAP.
--- arch/powerpc/include/asm/book3s/32/kup.h | 27 +++++++++++++++++++++++- arch/powerpc/mm/book3s32/kuap.c | 11 ++++++---- 2 files changed, 33 insertions(+), 5 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +26 -1
@@ -9,11 +9,12 @@#include <linux/jump_label.h> +extern struct static_key_false disable_kuap_key;extern struct static_key_false disable_kuep_key; static __always_inline bool kuap_is_disabled(void) { -return !IS_ENABLED (CONFIG_PPC_KUAP ); +return !IS_ENABLED (CONFIG_PPC_KUAP ) ||static_branch_unlikely (&disable_kuap_key); } static __always_inline bool kuep_is_disabled(void)
@@ -62,6 +63,9 @@ static inline void kuap_save_and_lock(struct pt_regs *regs)u32 addr = kuap &0xf0000000 ;u32 end = kuap <<28 ; +if (kuap_is_disabled ()) +return ; + regs->kuap = kuap;if (unlikely (!kuap))return ;
@@ -79,6 +83,9 @@ static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)u32 addr = regs->kuap &0xf0000000 ;u32 end = regs->kuap <<28 ; +if (kuap_is_disabled ()) +return ; + current->thread.kuap = regs->kuap;if (unlikely (regs->kuap == kuap))
@@ -91,6 +98,9 @@ static inline unsigned long kuap_get_and_assert_locked(void) {unsigned long kuap = current->thread.kuap; +if (kuap_is_disabled ()) +return 0 ; +WARN_ON_ONCE (IS_ENABLED (CONFIG_PPC_KUAP_DEBUG ) && kuap !=0 );return kuap;
@@ -106,6 +116,9 @@ static __always_inline void allow_user_access(void __user *to, const void __user {u32 addr, end; +if (kuap_is_disabled ()) +return ; +BUILD_BUG_ON (!__builtin_constant_p (dir));BUILD_BUG_ON (dir & ~KUAP_READ_WRITE );
@@ -128,6 +141,9 @@ static __always_inline void prevent_user_access(void __user *to, const void __us {u32 addr, end; +if (kuap_is_disabled ()) +return ; +BUILD_BUG_ON (!__builtin_constant_p (dir));if (dir &KUAP_CURRENT_WRITE ) {
@@ -159,6 +175,9 @@ static inline unsigned long prevent_user_access_return(void)unsigned long end = flags <<28 ;void __user *to = (__force void __user *)addr; +if (kuap_is_disabled ()) +return 0 ; +if (flags)prevent_user_access (to, to, end - addr,KUAP_READ_WRITE );
@@ -171,6 +190,9 @@ static inline void restore_user_access(unsigned long flags)unsigned long end = flags <<28 ;void __user *to = (__force void __user *)addr; +if (kuap_is_disabled ()) +return ; +if (flags)allow_user_access (to, to, end - addr,KUAP_READ_WRITE ); }
@@ -181,6 +203,9 @@ bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)unsigned long begin = regs->kuap &0xf0000000 ;unsigned long end = regs->kuap <<28 ; +if (kuap_is_disabled ()) +return false; +return is_write && (address < begin || address >= end); }
arch/powerpc/mm/book3s32/kuap.c +7 -4
@@ -3,15 +3,18 @@#include <asm/kup.h> #include <asm/smp.h> +struct static_key_false disable_kuap_key; +void __init setup_kuap (bool disabled) { -kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE ); +if (!disabled) +kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE );if (smp_processor_id () != boot_cpuid)return ; -pr_info ("Activating Kernel Userspace Access Protection\n" ); -if (disabled) -pr_warn ("KUAP cannot be disabled yet on 6xx when compiled in\n" ); +static_branch_enable (&disable_kuap_key); +else +pr_info ("Activating Kernel Userspace Access Protection\n" ); }
Signature · 1 line
2.25.0
PATCH v2 9/12 powerpc/32s: Rework Kernel Userspace Access Protection
In reply to Christophe LeroyOn book3s/32, KUAP is provided by toggling Ks bit in segment registers. One segment register addresses 256M of virtual memory.
At the time being, KUAP implements a complex logic to apply the unlock/lock on the exact number of segments covering the user range to access, with saving the boundaries of the range of segments in a member of thread struct.
But most if not all user accesses are within a single segment.
Rework KUAP with a different approach:
- Open only one segment, the one corresponding to the starting address of the range to be accessed.
- If a second segment is involved, it will generate a page fault. The segment will then be open by the page fault handler.
The kuap member of thread struct will now contain:
- The start address of the current on going user access, that will be used to know which segment to lock at the end of the user access.
- ~0 when no user access is open
- ~1 when additionnal segments are opened by a page fault.
Then, at lock time
- When only one segment is open, close it.
- When several segments are open, close all user segments.
Almost 100% of the time, only one segment will be involved.
In interrupts, inline the function that unlock/lock all segments, because not inlining them implies a lot of register save/restore.
With the patch, writing value 128 in userspace in perf_copy_attr() is done with 16 instructions:
3890: 93 82 04 dc stw r28,1244(r2)
3894: 7d 20 e5 26 mfsrin r9,r28
3898: 55 29 00 80 rlwinm r9,r9,0,2,0
389c: 7d 20 e1 e4 mtsrin r9,r28
38a0: 4c 00 01 2c isync
38a4: 39 20 00 80 li r9,128
38a8: 91 3c 00 00 stw r9,0(r28)
38ac: 81 42 04 dc lwz r10,1244(r2)
38b0: 39 00 ff ff li r8,-1
38b4: 91 02 04 dc stw r8,1244(r2)
38b8: 2c 0a ff fe cmpwi r10,-2
38bc: 41 82 00 88 beq 3944 <perf_copy_attr+0x36c>
38c0: 7d 20 55 26 mfsrin r9,r10
38c4: 65 29 40 00 oris r9,r9,16384
38c8: 7d 20 51 e4 mtsrin r9,r10
38cc: 4c 00 01 2c isync
...
3944: 48 00 00 01 bl 3944 <perf_copy_attr+0x36c>
3944: R_PPC_REL24 kuap_lock_all_oolBefore the patch it was 118 instructions. In reality only 42 are executed in most cases, but GCC is not able to see that a properly aligned user access cannot involve more than one segment.
5060: 39 1d 00 04 addi r8,r29,4
5064: 3d 20 b0 00 lis r9,-20480
5068: 7c 08 48 40 cmplw r8,r9
506c: 40 81 00 08 ble 5074 <perf_copy_attr+0x2cc>
5070: 3d 00 b0 00 lis r8,-20480
5074: 39 28 ff ff addi r9,r8,-1
5078: 57 aa 00 06 rlwinm r10,r29,0,0,3
507c: 55 29 27 3e rlwinm r9,r9,4,28,31
5080: 39 29 00 01 addi r9,r9,1
5084: 7d 29 53 78 or r9,r9,r10
5088: 91 22 04 dc stw r9,1244(r2)
508c: 7d 20 ed 26 mfsrin r9,r29
5090: 55 29 00 80 rlwinm r9,r9,0,2,0
5094: 7c 08 50 40 cmplw r8,r10
5098: 40 81 00 c0 ble 5158 <perf_copy_attr+0x3b0>
509c: 7d 46 50 f8 not r6,r10
50a0: 7c c6 42 14 add r6,r6,r8
50a4: 54 c6 27 be rlwinm r6,r6,4,30,31
50a8: 7d 20 51 e4 mtsrin r9,r10
50ac: 3c ea 10 00 addis r7,r10,4096
50b0: 39 29 01 11 addi r9,r9,273
50b4: 7f 88 38 40 cmplw cr7,r8,r7
50b8: 55 29 02 06 rlwinm r9,r9,0,8,3
50bc: 40 9d 00 9c ble cr7,5158 <perf_copy_attr+0x3b0> 50c0: 2f 86 00 00 cmpwi cr7,r6,0
50c4: 41 9e 00 4c beq cr7,5110 <perf_copy_attr+0x368>
50c8: 2f 86 00 01 cmpwi cr7,r6,1
50cc: 41 9e 00 2c beq cr7,50f8 <perf_copy_attr+0x350>
50d0: 2f 86 00 02 cmpwi cr7,r6,2
50d4: 41 9e 00 14 beq cr7,50e8 <perf_copy_attr+0x340>
50d8: 7d 20 39 e4 mtsrin r9,r7
50dc: 39 29 01 11 addi r9,r9,273
50e0: 3c e7 10 00 addis r7,r7,4096
50e4: 55 29 02 06 rlwinm r9,r9,0,8,3
50e8: 7d 20 39 e4 mtsrin r9,r7
50ec: 39 29 01 11 addi r9,r9,273
50f0: 3c e7 10 00 addis r7,r7,4096
50f4: 55 29 02 06 rlwinm r9,r9,0,8,3
50f8: 7d 20 39 e4 mtsrin r9,r7
50fc: 3c e7 10 00 addis r7,r7,4096
5100: 39 29 01 11 addi r9,r9,273
5104: 7f 88 38 40 cmplw cr7,r8,r7
5108: 55 29 02 06 rlwinm r9,r9,0,8,3
510c: 40 9d 00 4c ble cr7,5158 <perf_copy_attr+0x3b0>
5110: 7d 20 39 e4 mtsrin r9,r7
5114: 39 29 01 11 addi r9,r9,273
5118: 3c c7 10 00 addis r6,r7,4096
511c: 55 29 02 06 rlwinm r9,r9,0,8,3
5120: 7d 20 31 e4 mtsrin r9,r6
5124: 39 29 01 11 addi r9,r9,273
5128: 3c c6 10 00 addis r6,r6,4096
512c: 55 29 02 06 rlwinm r9,r9,0,8,3
5130: 7d 20 31 e4 mtsrin r9,r6
5134: 39 29 01 11 addi r9,r9,273
5138: 3c c7 30 00 addis r6,r7,12288
513c: 55 29 02 06 rlwinm r9,r9,0,8,3
5140: 7d 20 31 e4 mtsrin r9,r6
5144: 3c e7 40 00 addis r7,r7,16384
5148: 39 29 01 11 addi r9,r9,273
514c: 7f 88 38 40 cmplw cr7,r8,r7
5150: 55 29 02 06 rlwinm r9,r9,0,8,3
5154: 41 9d ff bc bgt cr7,5110 <perf_copy_attr+0x368> 5158: 4c 00 01 2c isync
515c: 39 20 00 80 li r9,128
5160: 91 3d 00 00 stw r9,0(r29)
5164: 38 e0 00 00 li r7,0
5168: 90 e2 04 dc stw r7,1244(r2)
516c: 7d 20 ed 26 mfsrin r9,r29
5170: 65 29 40 00 oris r9,r9,16384
5174: 40 81 00 c0 ble 5234 <perf_copy_attr+0x48c>
5178: 7d 47 50 f8 not r7,r10
517c: 7c e7 42 14 add r7,r7,r8
5180: 54 e7 27 be rlwinm r7,r7,4,30,31
5184: 7d 20 51 e4 mtsrin r9,r10
5188: 3d 4a 10 00 addis r10,r10,4096
518c: 39 29 01 11 addi r9,r9,273
5190: 7c 08 50 40 cmplw r8,r10
5194: 55 29 02 06 rlwinm r9,r9,0,8,3
5198: 40 81 00 9c ble 5234 <perf_copy_attr+0x48c>
519c: 2c 07 00 00 cmpwi r7,0
51a0: 41 82 00 4c beq 51ec <perf_copy_attr+0x444>
51a4: 2c 07 00 01 cmpwi r7,1
51a8: 41 82 00 2c beq 51d4 <perf_copy_attr+0x42c>
51ac: 2c 07 00 02 cmpwi r7,2
51b0: 41 82 00 14 beq 51c4 <perf_copy_attr+0x41c>
51b4: 7d 20 51 e4 mtsrin r9,r10
51b8: 39 29 01 11 addi r9,r9,273
51bc: 3d 4a 10 00 addis r10,r10,4096
51c0: 55 29 02 06 rlwinm r9,r9,0,8,3
51c4: 7d 20 51 e4 mtsrin r9,r10
51c8: 39 29 01 11 addi r9,r9,273
51cc: 3d 4a 10 00 addis r10,r10,4096
51d0: 55 29 02 06 rlwinm r9,r9,0,8,3
51d4: 7d 20 51 e4 mtsrin r9,r10
51d8: 3d 4a 10 00 addis r10,r10,4096
51dc: 39 29 01 11 addi r9,r9,273
51e0: 7c 08 50 40 cmplw r8,r10
51e4: 55 29 02 06 rlwinm r9,r9,0,8,3
51e8: 40 81 00 4c ble 5234 <perf_copy_attr+0x48c>
51ec: 7d 20 51 e4 mtsrin r9,r10
51f0: 39 29 01 11 addi r9,r9,273
51f4: 3c ea 10 00 addis r7,r10,4096
51f8: 55 29 02 06 rlwinm r9,r9,0,8,3
51fc: 7d 20 39 e4 mtsrin r9,r7
5200: 39 29 01 11 addi r9,r9,273
5204: 3c e7 10 00 addis r7,r7,4096
5208: 55 29 02 06 rlwinm r9,r9,0,8,3
520c: 7d 20 39 e4 mtsrin r9,r7
5210: 39 29 01 11 addi r9,r9,273
5214: 3c ea 30 00 addis r7,r10,12288
5218: 55 29 02 06 rlwinm r9,r9,0,8,3
521c: 7d 20 39 e4 mtsrin r9,r7
5220: 3d 4a 40 00 addis r10,r10,16384
5224: 39 29 01 11 addi r9,r9,273
5228: 7c 08 50 40 cmplw r8,r10
522c: 55 29 02 06 rlwinm r9,r9,0,8,3
5230: 41 81 ff bc bgt 51ec <perf_copy_attr+0x444>
5234: 4c 00 01 2c isync--- arch/powerpc/include/asm/book3s/32/kup.h | 150 ++++++++++++----------- arch/powerpc/include/asm/processor.h | 10 +- arch/powerpc/kernel/process.c | 3 + arch/powerpc/mm/book3s32/kuap.c | 12 +- 4 files changed, 102 insertions(+), 73 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +79 -71
@@ -42,36 +42,69 @@ static inline void kuep_unlock(void)#include <linux/sched.h> -static inline void kuap_update_sr (u32 sr,u32 addr,u32 end) -{ - addr &=0xf0000000 ;/* align addr to start of segment */ -barrier ();/* make sure thread.kuap is updated before playing with SRs */ -for (;;) { -mtsr (sr, addr); - addr +=0x10000000 ;/* address of next segment */ -if (addr >= end) -break ; - sr +=0x111 ;/* next VSID */ - sr &=0xf0ffffff ;/* clear VSID overflow */ - } +#define KUAP_NONE (~0UL) +#define KUAP_ALL (~1UL) + +static inline void kuap_lock_one (unsigned long addr) +{ +mtsr (mfsr (addr) |SR_KS , addr); +isync ();/* Context sync required after mtsr() */ +} + +static inline void kuap_unlock_one (unsigned long addr) +{ +mtsr (mfsr (addr) & ~SR_KS , addr); +isync ();/* Context sync required after mtsr() */ +} + +static inline void kuap_lock_all (void ) +{ +update_user_segments (mfsr (0 ) |SR_KS );isync ();/* Context sync required after mtsr() */ } +static inline void kuap_unlock_all (void ) +{ +update_user_segments (mfsr (0 ) & ~SR_KS ); +isync ();/* Context sync required after mtsr() */ +} + +void kuap_lock_all_ool (void ); +void kuap_unlock_all_ool (void ); + +static inline void kuap_lock (unsigned long addr,bool ool) +{ +if (likely (addr !=KUAP_ALL )) +kuap_lock_one (addr); +else if (!ool) +kuap_lock_all (); +else +kuap_lock_all_ool (); +} + +static inline void kuap_unlock (unsigned long addr,bool ool) +{ +if (likely (addr !=KUAP_ALL )) +kuap_unlock_one (addr); +else if (!ool) +kuap_unlock_all (); +else +kuap_unlock_all_ool (); +} +static inline void kuap_save_and_lock (struct pt_regs *regs) {unsigned long kuap = current->thread.kuap; -u32 addr = kuap &0xf0000000 ; -u32 end = kuap <<28 ;if (kuap_is_disabled ())return ; regs->kuap = kuap; -if (unlikely (!kuap)) +if (unlikely (kuap ==KUAP_NONE ))return ; - current->thread.kuap =0 ; -kuap_update_sr (mfsr (addr) |SR_KS , addr, end);/* Set Ks */ + current->thread.kuap =KUAP_NONE ; +kuap_lock (kuap, false); }static inline void kuap_user_restore (struct pt_regs *regs)
@@ -80,18 +113,12 @@ static inline void kuap_user_restore(struct pt_regs *regs)static inline void kuap_kernel_restore (struct pt_regs *regs,unsigned long kuap) { -u32 addr = regs->kuap &0xf0000000 ; -u32 end = regs->kuap <<28 ; -if (kuap_is_disabled ())return ; current->thread.kuap = regs->kuap; -if (unlikely (regs->kuap == kuap)) -return ; - -kuap_update_sr (mfsr (addr) & ~SR_KS , addr, end);/* Clear Ks */ +kuap_unlock (regs->kuap, false); }static inline unsigned long kuap_get_and_assert_locked (void )
@@ -99,9 +126,9 @@ static inline unsigned long kuap_get_and_assert_locked(void)unsigned long kuap = current->thread.kuap;if (kuap_is_disabled ()) -return 0 ; +return KUAP_NONE ; -WARN_ON_ONCE (IS_ENABLED (CONFIG_PPC_KUAP_DEBUG ) && kuap !=0 ); +WARN_ON_ONCE (IS_ENABLED (CONFIG_PPC_KUAP_DEBUG ) && kuap !=KUAP_NONE );return kuap; }
@@ -114,8 +141,6 @@ static inline void kuap_assert_locked(void) static __always_inline void allow_user_access(void __user *to, const void __user *from,u32 size,unsigned long dir) { -u32 addr, end; -if (kuap_is_disabled ())return ;
@@ -125,88 +150,71 @@ static __always_inline void allow_user_access(void __user *to, const void __userif (!(dir &KUAP_WRITE ))return ; - addr = (__force u32 )to; - -if (unlikely (addr >=TASK_SIZE || !size)) -return ; - - end =min (addr + size,TASK_SIZE ); - - current->thread.kuap = (addr &0xf0000000 ) | ((((end -1 ) >>28 ) +1 ) &0xf ); -kuap_update_sr (mfsr (addr) & ~SR_KS , addr, end);/* Clear Ks */ + current->thread.kuap = (__force u32 )to; +kuap_unlock_one ((__force u32 )to); } static __always_inline void prevent_user_access(void __user *to, const void __user *from,u32 size,unsigned long dir) { -u32 addr, end; +u32 kuap = current->thread.kuap;if (kuap_is_disabled ())return ;BUILD_BUG_ON (!__builtin_constant_p (dir)); -if (dir &KUAP_CURRENT_WRITE ) { -u32 kuap = current->thread.kuap; - -if (unlikely (!kuap)) -return ; - - addr = kuap &0xf0000000 ; - end = kuap <<28 ; - }else if (dir &KUAP_WRITE ) { - addr = (__force u32 )to; - end =min (addr + size,TASK_SIZE ); - -if (unlikely (addr >=TASK_SIZE || !size)) -return ; - }else { +if (!(dir &KUAP_WRITE ))return ; - } - current->thread.kuap =0 ; -kuap_update_sr (mfsr (addr) |SR_KS , addr, end);/* set Ks */ + current->thread.kuap =KUAP_NONE ; +kuap_lock (kuap, true); }static inline unsigned long prevent_user_access_return (void ) {unsigned long flags = current->thread.kuap; -unsigned long addr = flags &0xf0000000 ; -unsigned long end = flags <<28 ; -void __user *to = (__force void __user *)addr;if (kuap_is_disabled ()) -return 0 ; +return KUAP_NONE ; -if (flags) -prevent_user_access (to, to, end - addr,KUAP_READ_WRITE ); +if (flags !=KUAP_NONE ) { + current->thread.kuap =KUAP_NONE ; +kuap_lock (flags, true); + }return flags; }static inline void restore_user_access (unsigned long flags) { -unsigned long addr = flags &0xf0000000 ; -unsigned long end = flags <<28 ; -void __user *to = (__force void __user *)addr; -if (kuap_is_disabled ())return ; -if (flags) -allow_user_access (to, to, end - addr,KUAP_READ_WRITE ); +if (flags !=KUAP_NONE ) { + current->thread.kuap = flags; +kuap_unlock (flags, true); + } }static inline bool bad_kuap_fault (struct pt_regs *regs,unsigned long address,bool is_write) { -unsigned long begin = regs->kuap &0xf0000000 ; -unsigned long end = regs->kuap <<28 ; +unsigned long kuap = regs->kuap;if (kuap_is_disabled ())return false; -return is_write && (address < begin || address >= end); +if (!is_write || kuap ==KUAP_ALL ) +return false; +if (kuap ==KUAP_NONE ) +return true; + +/* If faulting address doesn't match unlocked segment, unlock all */ +if ((kuap ^ address) &0xf0000000 ) + regs->kuap =KUAP_ALL ; + +return false; }#endif /* CONFIG_PPC_KUAP */
arch/powerpc/include/asm/processor.h +9 -1
@@ -276,7 +276,15 @@ struct thread_struct {#define SPEFSCR_INIT #endif -#ifdef CONFIG_PPC32 +#if defined(CONFIG_PPC_BOOK3S_32 ) && defined(CONFIG_PPC_KUAP ) +#define INIT_THREAD { \ + .ksp = INIT_SP, \ + .pgdir = swapper_pg_dir, \ + .kuap = ~0UL, /* KUAP_NONE */ \ + .fpexc_mode = MSR_FE0 | MSR_FE1, \ + SPEFSCR_INIT \ +} +#elif defined(CONFIG_PPC32 )#define INIT_THREAD { \ .ksp = INIT_SP, \ .pgdir = swapper_pg_dir, \
arch/powerpc/kernel/process.c +3 -0
@@ -1736,6 +1736,9 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,#ifdef CONFIG_ALTIVEC p->thread.vr_save_area =NULL ;#endif +#if defined(CONFIG_PPC_BOOK3S_32 ) && defined(CONFIG_PPC_KUAP ) + p->thread.kuap =KUAP_NONE ; +#endif setup_ksp_vsid (p, sp);
arch/powerpc/mm/book3s32/kuap.c +11 -1
@@ -5,10 +5,20 @@struct static_key_false disable_kuap_key; +void kuap_lock_all_ool (void ) +{ +kuap_lock_all (); +} + +void kuap_unlock_all_ool (void ) +{ +kuap_unlock_all (); +} +void __init setup_kuap (bool disabled) {if (!disabled) -kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE ); +kuap_lock_all_ool ();if (smp_processor_id () != boot_cpuid)return ;
Signature · 1 line
2.25.0
PATCH v2 10/12 powerpc/32s: Activate KUAP and KUEP by default
In reply to Christophe LeroyNow that KUAP and KUEP have been significantly optimised and can be disabled at boot time using 'nosmap' and 'nosmep' kernel parameters, them can be active by default like in other powerpc platforms.
It is still possible to disable them completely in the configuration.
--- arch/powerpc/platforms/Kconfig.cputype | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/powerpc/platforms/Kconfig.cputype +2 -2
@@ -389,7 +389,7 @@ config PPC_HAVE_KUEP config PPC_KUEP bool "Kernel Userspace Execution Prevention" depends on PPC_HAVE_KUEP - default y if !PPC_BOOK3S_32 + default y help Enable support for Kernel Userspace Execution Prevention (KUEP)
@@ -401,7 +401,7 @@ config PPC_HAVE_KUAP config PPC_KUAP bool "Kernel Userspace Access Protection" depends on PPC_HAVE_KUAP - default y if !PPC_BOOK3S_32 + default y help Enable support for Kernel Userspace Access Protection (KUAP)
Signature · 1 line
2.25.0
PATCH v2 11/12 powerpc/kuap: Remove KUAP_CURRENT_XXX
In reply to Christophe Leroybook3s/32 was the only user of KUAP_CURRENT_XXX.
After rework of book3s/32 KUAP, it is not used anymore.
Remove them.
--- arch/powerpc/include/asm/book3s/32/kup.h | 1 - arch/powerpc/include/asm/kup.h | 14 +++----------- 2 files changed, 3 insertions(+), 12 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +0 -1
@@ -145,7 +145,6 @@ static __always_inline void allow_user_access(void __user *to, const void __userreturn ;BUILD_BUG_ON (!__builtin_constant_p (dir)); -BUILD_BUG_ON (dir & ~KUAP_READ_WRITE );if (!(dir &KUAP_WRITE ))return ;
arch/powerpc/include/asm/kup.h +3 -11
@@ -5,14 +5,6 @@#define KUAP_READ 1#define KUAP_WRITE 2#define KUAP_READ_WRITE (KUAP_READ | KUAP_WRITE) -/* -* For prevent_user_access() only. -* Use the current saved situation instead of the to/from/size params. -* Used on book3s/32 -*/ -#define KUAP_CURRENT_READ 4 -#define KUAP_CURRENT_WRITE 8 -#define KUAP_CURRENT (KUAP_CURRENT_READ | KUAP_CURRENT_WRITE)#ifdef CONFIG_PPC_BOOK3S_64 #include <asm/book3s/64/kup.h>
@@ -129,17 +121,17 @@ static inline void prevent_read_write_user(void __user *to, const void __user *fstatic inline void prevent_current_access_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_CURRENT ); +prevent_user_access (NULL ,NULL , ~0UL ,KUAP_READ_WRITE ); }static inline void prevent_current_read_from_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_CURRENT_READ ); +prevent_user_access (NULL ,NULL , ~0UL ,KUAP_READ ); }static inline void prevent_current_write_to_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_CURRENT_WRITE ); +prevent_user_access (NULL ,NULL , ~0UL ,KUAP_WRITE ); }#endif /* !__ASSEMBLY__ */
Signature · 1 line
2.25.0
PATCH v2 12/12 powerpc/kuap: Remove to/from/size parameters of prevent_user_access()
In reply to Christophe Leroyprevent_user_access() doesn't use anymore to/from/size parameters.
Remove them.
--- arch/powerpc/include/asm/book3s/32/kup.h | 3 +-- arch/powerpc/include/asm/book3s/64/kup.h | 3 +-- arch/powerpc/include/asm/kup.h | 15 +++++++-------- arch/powerpc/include/asm/nohash/32/kup-8xx.h | 3 +-- 4 files changed, 10 insertions(+), 14 deletions(-)
arch/powerpc/include/asm/book3s/32/kup.h +1 -2
@@ -153,8 +153,7 @@ static __always_inline void allow_user_access(void __user *to, const void __userkuap_unlock_one ((__force u32 )to); } -static __always_inline void prevent_user_access(void __user *to, const void __user *from, -u32 size,unsigned long dir) +static __always_inline void prevent_user_access(unsigned long dir) {u32 kuap = current->thread.kuap;
arch/powerpc/include/asm/book3s/64/kup.h +1 -2
@@ -398,8 +398,7 @@ static __always_inline void allow_user_access(void __user *to, const void __user#endif /* !CONFIG_PPC_KUAP */ -static inline void prevent_user_access (void __user *to,const void __user *from, -unsigned long size,unsigned long dir) +static inline void prevent_user_access (unsigned long dir) {set_kuap (AMR_KUAP_BLOCKED );if (static_branch_unlikely (&uaccess_flush_key))
arch/powerpc/include/asm/kup.h +7 -8
@@ -72,8 +72,7 @@ static inline unsigned long kuap_get_and_assert_locked(void)#ifndef CONFIG_PPC_BOOK3S_64 static inline void allow_user_access (void __user *to,const void __user *from,unsigned long size,unsigned long dir) { } -static inline void prevent_user_access (void __user *to,const void __user *from, -unsigned long size,unsigned long dir) { } +static inline void prevent_user_access (unsigned long dir) { }static inline unsigned long prevent_user_access_return (void ) {return 0UL ; }static inline void restore_user_access (unsigned long flags) { }#endif /* CONFIG_PPC_BOOK3S_64 */
@@ -105,33 +104,33 @@ static inline void allow_read_write_user(void __user *to, const void __user *frostatic inline void prevent_read_from_user (const void __user *from,unsigned long size) { -prevent_user_access (NULL , from, size,KUAP_READ ); +prevent_user_access (KUAP_READ ); }static inline void prevent_write_to_user (void __user *to,unsigned long size) { -prevent_user_access (to,NULL , size,KUAP_WRITE ); +prevent_user_access (KUAP_WRITE ); }static inline void prevent_read_write_user (void __user *to,const void __user *from,unsigned long size) { -prevent_user_access (to, from, size,KUAP_READ_WRITE ); +prevent_user_access (KUAP_READ_WRITE ); }static inline void prevent_current_access_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_READ_WRITE ); +prevent_user_access (KUAP_READ_WRITE ); }static inline void prevent_current_read_from_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_READ ); +prevent_user_access (KUAP_READ ); }static inline void prevent_current_write_to_user (void ) { -prevent_user_access (NULL ,NULL , ~0UL ,KUAP_WRITE ); +prevent_user_access (KUAP_WRITE ); }#endif /* !__ASSEMBLY__ */
arch/powerpc/include/asm/nohash/32/kup-8xx.h +1 -2
@@ -48,8 +48,7 @@ static inline void allow_user_access(void __user *to, const void __user *from,mtspr (SPRN_MD_AP ,MD_APG_INIT ); } -static inline void prevent_user_access (void __user *to,const void __user *from, -unsigned long size,unsigned long dir) +static inline void prevent_user_access (unsigned long dir) {mtspr (SPRN_MD_AP ,MD_APG_KUAP ); }
Signature · 1 line
2.25.0
Re: [PATCH v2 08/12] powerpc/32s: Allow disabling KUAP at boot time
In reply to Christophe LeroyLe 03/06/2021 à 10:41, Christophe Leroy a écrit :
Quoted 18 lines
PPC64 uses MMU features to enable/disable KUAP at boot time. But feature fixups are applied way too early on PPC32.
Now that all KUAP related actions are in C following the conversion of KUAP initial setup and context switch in C, static branches can be used to enable/disable KUAP.
Signed-off-by Christophe Leroy <[email protected]>--- arch/powerpc/include/asm/book3s/32/kup.h | 27 +++++++++++++++++++++++- arch/powerpc/mm/book3s32/kuap.c | 11 ++++++---- 2 files changed, 33 insertions(+), 5 deletions(-)arch/powerpc/include/asm/book3s/32/kup.h +0
-0
arch/powerpc/include/asm/book3s/32/kup.h line 12
Quoted 4 lines
arch/powerpc/include/asm/book3s/32/kup.h +1
-0@@ -9,11 +9,12 @@#include <linux/jump_label.h> +extern struct static_key_false disable_kuap_key;
Same as 8xx, this needs to be exported for modules.
Quoted 117 lines
arch/powerpc/include/asm/book3s/32/kup.h +25
-1extern struct static_key_false disable_kuep_key; static __always_inline bool kuap_is_disabled(void) { -return !IS_ENABLED (CONFIG_PPC_KUAP ); +return !IS_ENABLED (CONFIG_PPC_KUAP ) ||static_branch_unlikely (&disable_kuap_key); } static __always_inline bool kuep_is_disabled(void)@@ -62,6 +63,9 @@ static inline void kuap_save_and_lock(struct pt_regs *regs)u32 addr = kuap &0xf0000000 ;u32 end = kuap <<28 ; +if (kuap_is_disabled ()) +return ; + regs->kuap = kuap;if (unlikely (!kuap))return ;@@ -79,6 +83,9 @@ static inline void kuap_kernel_restore(struct pt_regs *regs, unsigned long kuap)u32 addr = regs->kuap &0xf0000000 ;u32 end = regs->kuap <<28 ; +if (kuap_is_disabled ()) +return ; + current->thread.kuap = regs->kuap;if (unlikely (regs->kuap == kuap))@@ -91,6 +98,9 @@ static inline unsigned long kuap_get_and_assert_locked(void) {unsigned long kuap = current->thread.kuap; +if (kuap_is_disabled ()) +return 0 ; +WARN_ON_ONCE (IS_ENABLED (CONFIG_PPC_KUAP_DEBUG ) && kuap !=0 );return kuap;@@ -106,6 +116,9 @@ static __always_inline void allow_user_access(void __user *to, const void __user {u32 addr, end; +if (kuap_is_disabled ()) +return ; +BUILD_BUG_ON (!__builtin_constant_p (dir));BUILD_BUG_ON (dir & ~KUAP_READ_WRITE );@@ -128,6 +141,9 @@ static __always_inline void prevent_user_access(void __user *to, const void __us {u32 addr, end; +if (kuap_is_disabled ()) +return ; +BUILD_BUG_ON (!__builtin_constant_p (dir));if (dir &KUAP_CURRENT_WRITE ) {@@ -159,6 +175,9 @@ static inline unsigned long prevent_user_access_return(void)unsigned long end = flags <<28 ;void __user *to = (__force void __user *)addr; +if (kuap_is_disabled ()) +return 0 ; +if (flags)prevent_user_access (to, to, end - addr,KUAP_READ_WRITE );@@ -171,6 +190,9 @@ static inline void restore_user_access(unsigned long flags)unsigned long end = flags <<28 ;void __user *to = (__force void __user *)addr; +if (kuap_is_disabled ()) +return ; +if (flags)allow_user_access (to, to, end - addr,KUAP_READ_WRITE ); }@@ -181,6 +203,9 @@ bad_kuap_fault(struct pt_regs *regs, unsigned long address, bool is_write)unsigned long begin = regs->kuap &0xf0000000 ;unsigned long end = regs->kuap <<28 ; +if (kuap_is_disabled ()) +return false; +return is_write && (address < begin || address >= end); }arch/powerpc/mm/book3s32/kuap.c +7
-4@@ -3,15 +3,18 @@#include <asm/kup.h> #include <asm/smp.h> +struct static_key_false disable_kuap_key; +void __init setup_kuap (bool disabled) { -kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE ); +if (!disabled) +kuap_update_sr (mfsr (0 ) |SR_KS ,0 ,TASK_SIZE );if (smp_processor_id () != boot_cpuid)return ; -pr_info ("Activating Kernel Userspace Access Protection\n" ); -if (disabled) -pr_warn ("KUAP cannot be disabled yet on 6xx when compiled in\n" ); +static_branch_enable (&disable_kuap_key); +else +pr_info ("Activating Kernel Userspace Access Protection\n" ); }
Michael Ellerman
AppliedQuoted 9 lines · Christophe Leroy
This series is a rework of KUAP on book3s/32.
On book3s32, KUAP is heavier than on other platform because it can't be opened globaly at once, it must be done for each 256Mb segment.
Instead of opening access to all necessary segments via a heavy logic, only open access to the segment matching the start of the range.
[...]
Applied to powerpc/next.
[01/12] powerpc/32s: Move setup_{kuep/kuap}() into {kuep/kuap}.c
https://git.kernel.org/powerpc/c/91ec66719d4c5c0e7b4e32585b01881660d1bc53 [02/12] powerpc/32s: Refactor update of user segment registers https://git.kernel.org/powerpc/c/91bb30822a2e1d7900f9f42e9e92647a9015f979 [03/12] powerpc/32s: move CTX_TO_VSID() into mmu-hash.h https://git.kernel.org/powerpc/c/7235bb3593781ed022d0714a73c2c0d8eb8a835f [04/12] powerpc/32s: Convert switch_mmu_context() to C https://git.kernel.org/powerpc/c/863771a28e27dc9eaeaa88cea300370d032f0e0f [05/12] powerpc/32s: Simplify calculation of segment register content https://git.kernel.org/powerpc/c/882136fb2f5208a35ddad9205b20e5791edd4782 [06/12] powerpc/32s: Initialise KUAP and KUEP in C https://git.kernel.org/powerpc/c/86f46f3432727933be82f64b739712a6edb9d704 [07/12] powerpc/32s: Allow disabling KUEP at boot time https://git.kernel.org/powerpc/c/50d2f104cd9572af476579eae9aa1b38de602ec7 [08/12] powerpc/32s: Allow disabling KUAP at boot time https://git.kernel.org/powerpc/c/6b4d630068b0c5cdd6d8e599182b131448e0cb06 [09/12] powerpc/32s: Rework Kernel Userspace Access Protection https://git.kernel.org/powerpc/c/16132529cee586ee9a058bb33cfbdcb5d884f6b3 [10/12] powerpc/32s: Activate KUAP and KUEP by default https://git.kernel.org/powerpc/c/9f5bd8f1471d7498c934c0a686fd0997cf872653 [11/12] powerpc/kuap: Remove KUAP_CURRENT_XXX https://git.kernel.org/powerpc/c/d008f8f8a0c3efe4fe1008a797f9497ea5965e27 [12/12] powerpc/kuap: Remove to/from/size parameters of prevent_user_access() https://git.kernel.org/powerpc/c/cb2f1fb205cc20695fcaef84baf80d9d3e54c88b
cheers