Context switch improvements
Subsystems (4)
cover Context switch improvements
Here are various improvements to our context switch path. Some of the highlights:
- Group all mfsprs and mtsprs in __switch_to(), which gives us a 10% improvement on POWER8.
- Create giveup_all() and flush_all_to_thread() so we only write the MSR once, which gives us a 3% improvement on POWER8.
- Create disable_kernel_{fp,altivec,vsx,spe}() and add a debug boot option (ppc_strict_facility_enable) to minimise kernel code running with floating point and vector bits enabled.
- Make giveup_vsx() and flush_vsx_to_thread() handle FP and Altivec state, so they behave more like their sister functions.
Scott: There are changes to the SPE code here which I have only been able to compile test.
Anton
--
Anton Blanchard (19):
powerpc: Don't disable kernel FP/VMX/VSX MSR bits on context switch
powerpc: Don't disable MSR bits in do_load_up_transact_*() functions
powerpc: Create context switch helpers save_sprs() and restore_sprs()
powerpc: Remove redundant mflr in _switch
powerpc: Remove UP only lazy floating point and vector optimisations
powerpc: Simplify TM restore checks
powerpc: Create mtmsrd_isync()
powerpc: Remove NULL task struct pointer checks in FP and vector code
powerpc: Move part of giveup_fpu,altivec,spe into c
powerpc: Move part of giveup_vsx into c
crypto: vmx: Only call enable_kernel_vsx()
powerpc: Create msr_check_and_{set,clear}()
powerpc: Create disable_kernel_{fp,altivec,vsx,spe}()
powerpc: Add ppc_strict_facility_enable boot option
powerpc: Remove fp_enable() and vec_enable(), use
msr_check_and_{set,clear}() powerpc: create giveup_all()
powerpc: create flush_all_to_thread()
powerpc: Rearrange __switch_to()
powerpc: clean up asm/switch_to.h
Documentation/kernel-parameters.txt | 6 + arch/powerpc/crypto/aes-spe-glue.c | 1 + arch/powerpc/crypto/sha1-spe-glue.c | 1 + arch/powerpc/crypto/sha256-spe-glue.c | 1 + arch/powerpc/include/asm/processor.h | 9 +- arch/powerpc/include/asm/reg.h | 17 ++ arch/powerpc/include/asm/switch_to.h | 65 ++--- arch/powerpc/kernel/align.c | 2 + arch/powerpc/kernel/entry_64.S | 75 +---- arch/powerpc/kernel/fpu.S | 73 +---- arch/powerpc/kernel/head_fsl_booke.S | 42 +-- arch/powerpc/kernel/idle_power7.S | 7 - arch/powerpc/kernel/ppc_ksyms.c | 6 - arch/powerpc/kernel/process.c | 471 ++++++++++++++++++------------- arch/powerpc/kernel/signal_32.c | 22 +- arch/powerpc/kernel/signal_64.c | 22 +- arch/powerpc/kernel/swsusp.c | 4 +- arch/powerpc/kernel/vector.S | 112 +------- arch/powerpc/kvm/book3s_hv.c | 5 +- arch/powerpc/kvm/book3s_paired_singles.c | 1 + arch/powerpc/kvm/book3s_pr.c | 21 +- arch/powerpc/kvm/booke.c | 4 + arch/powerpc/lib/vmx-helper.c | 2 + arch/powerpc/lib/xor_vmx.c | 4 + drivers/crypto/vmx/aes.c | 6 +- drivers/crypto/vmx/aes_cbc.c | 6 +- drivers/crypto/vmx/aes_ctr.c | 6 +- drivers/crypto/vmx/ghash.c | 12 +- lib/raid6/altivec.uc | 1 + 29 files changed, 376 insertions(+), 628 deletions(-)
Signature · 1 line
2.5.0
PATCH 1/19 powerpc: Don't disable kernel FP/VMX/VSX MSR bits on context switch
Writing the MSR is slow, so we want to avoid it whenever possible.
A subsequent patch will add a debug option that strictly manages the FP/VMX/VSX unavailable bits. For now just remove it, matching what we do in other areas of the kernel (eg enable_kernel_altivec()).
A context switch microbenchmark using yield():
http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield --fp 0 0
shows an improvement of almost 3% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/entry_64.S | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-)
arch/powerpc/kernel/entry_64.S +1 -14
@@ -453,26 +453,13 @@ _GLOBAL(_switch) SAVE_8GPRS(14, r1) SAVE_10GPRS(22, r1) mflr r20 /* Return to switch caller */ - mfmsr r22 - li r0, MSR_FP -#ifdef CONFIG_VSX -BEGIN_FTR_SECTION - oris r0,r0,MSR_VSX@h /* Disable VSX */ -END_FTR_SECTION_IFSET(CPU_FTR_VSX) -#endif /* CONFIG_VSX */ #ifdef CONFIG_ALTIVEC BEGIN_FTR_SECTION - oris r0,r0,MSR_VEC@h /* Disable altivec */ mfspr r24,SPRN_VRSAVE /* save vrsave register value */ std r24,THREAD_VRSAVE(r3) END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #endif /* CONFIG_ALTIVEC */ - and. r0,r0,r22 - beq+ 1f - andc r22,r22,r0 - MTMSRD(r22) - isync -1: std r20,_NIP(r1) + std r20,_NIP(r1) mfcr r23 std r23,_CCR(r1) std r1,KSP(r3) /* Set old stack pointer */
Signature · 1 line
2.5.0
Re: [01/19] powerpc: Don't disable kernel FP/VMX/VSX MSR bits on context switch
Quoted 8 lines · Anton Blanchard
Writing the MSR is slow, so we want to avoid it whenever possible.
A subsequent patch will add a debug option that strictly manages the FP/VMX/VSX unavailable bits. For now just remove it, matching what we do in other areas of the kernel (eg
enable_kernel_altivec()).A context switch microbenchmark using
yield():
Quoted 7 lines
http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield --fp 0 0
shows an improvement of almost 3% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>
Applied all 19 patches to powerpc next, thanks.
http://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git/log/?h=next
cheers
PATCH 2/19 powerpc: Don't disable MSR bits in do_load_up_transact_*() functions
In reply to Anton BlanchardSimilar to the non TM load_up_*() functions, don't disable the MSR bits on the way out.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/fpu.S | 4 ---- arch/powerpc/kernel/vector.S | 4 ---- 2 files changed, 8 deletions(-)
arch/powerpc/kernel/fpu.S +0 -4
@@ -73,10 +73,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) MTFSF_L(fr0) REST_32FPVSRS(0, R4, R7) - /* FP/VSX off again */ - MTMSRD(r6) - SYNC - blr #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */
arch/powerpc/kernel/vector.S +0 -4
@@ -29,10 +29,6 @@ _GLOBAL(do_load_up_transact_altivec) addi r10,r3,THREAD_TRANSACT_VRSTATE REST_32VRS(0,r4,r10) - /* Disable VEC again. */ - MTMSRD(r6) - isync - blr #endif
Signature · 1 line
2.5.0
PATCH 3/19 powerpc: Create context switch helpers save_sprs() and restore_sprs()
In reply to Anton BlanchardMove all our context switch SPR save and restore code into two helpers. We do a few optimisations:
- Group all mfsprs and all mtsprs. In many cases an mtspr sets a scoreboarding bit that an mfspr waits on, so the current practise of mfspr A; mtspr A; mfpsr B; mtspr B is the worst scheduling we can do.
- SPR writes are slow, so check that the value is changing before writing it.
A context switch microbenchmark using yield():
http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield 0 0
shows an improvement of almost 10% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/processor.h | 1 + arch/powerpc/include/asm/switch_to.h | 11 ----- arch/powerpc/kernel/entry_64.S | 60 +---------------------- arch/powerpc/kernel/process.c | 92 +++++++++++++++++++++++++++++++----- 4 files changed, 82 insertions(+), 82 deletions(-)
arch/powerpc/include/asm/processor.h +1 -0
@@ -294,6 +294,7 @@ struct thread_struct {#endif #ifdef CONFIG_PPC64 unsigned long dscr; +unsigned long fscr;/* * This member element dscr_inherit indicates that the process * has explicitly attempted and changed the DSCR register value
arch/powerpc/include/asm/switch_to.h +0 -11
@@ -15,17 +15,6 @@ extern struct task_struct *__switch_to(struct task_struct *,struct thread_struct ;extern struct task_struct *_switch (struct thread_struct *prev,struct thread_struct *next); -#ifdef CONFIG_PPC_BOOK3S_64 -static inline void save_early_sprs (struct thread_struct *prev) -{ -if (cpu_has_feature (CPU_FTR_ARCH_207S )) - prev->tar =mfspr (SPRN_TAR ); -if (cpu_has_feature (CPU_FTR_DSCR )) - prev->dscr =mfspr (SPRN_DSCR ); -} -#else -static inline void save_early_sprs (struct thread_struct *prev) {} -#endif extern void enable_kernel_fp (void );extern void enable_kernel_altivec (void );
arch/powerpc/kernel/entry_64.S +1 -59
77 lines not on this page · raw message
arch/powerpc/kernel/process.c +80 -12
125 lines not on this page · raw message
Signature · 1 line
2.5.0
Re: [PATCH 03/19] powerpc: Create context switch helpers save_sprs() and restore_sprs()
Quoted 2 lines · Anton Blanchard
Move all our context switch SPR save and restore code into two helpers. We do a few optimisations:
To avoid confusion with other places where we might save and restore SPRs for things like power management etc... can you name these save_task_sprs and restore_task_sprs or something similar ? At least for me it makes things a bit clearer :)
Quoted 275 lines
- Group all mfsprs and all mtsprs. In many cases an mtspr sets a scoreboarding bit that an mfspr waits on, so the current practise of mfspr A; mtspr A; mfpsr B; mtspr B is the worst scheduling we can do.
- SPR writes are slow, so check that the value is changing before writing it.
A context switch microbenchmark using
yield():http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield 0 0
shows an improvement of almost 10% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>--- arch/powerpc/include/asm/processor.h | 1 + arch/powerpc/include/asm/switch_to.h | 11 ----- arch/powerpc/kernel/entry_64.S | 60 +---------------------- arch/powerpc/kernel/process.c | 92 +++++++++++++++++++++++++++++++----- 4 files changed, 82 insertions(+), 82 deletions(-)arch/powerpc/include/asm/processor.h +1
-0@@ -294,6 +294,7 @@ struct thread_struct {#endif #ifdef CONFIG_PPC64 unsigned long dscr; +unsigned long fscr;/* * This member element dscr_inherit indicates that the process * has explicitly attempted and changed the DSCR register value arch/powerpc/include/asm/switch_to.h +0
-11@@ -15,17 +15,6 @@ extern struct task_struct *__switch_to(struct task_struct *,struct thread_struct ;extern struct task_struct *_switch (struct thread_struct *prev,struct thread_struct *next); -#ifdef CONFIG_PPC_BOOK3S_64 -static inline void save_early_sprs (struct thread_struct *prev) -{ -if (cpu_has_feature (CPU_FTR_ARCH_207S )) - prev->tar =mfspr (SPRN_TAR ); -if (cpu_has_feature (CPU_FTR_DSCR )) - prev->dscr =mfspr (SPRN_DSCR ); -} -#else -static inline void save_early_sprs (struct thread_struct *prev) {} -#endif extern void enable_kernel_fp (void );extern void enable_kernel_altivec (void );arch/powerpc/kernel/entry_64.S +1
-59@@ -453,29 +453,12 @@ _GLOBAL(_switch) SAVE_8GPRS(14, r1) SAVE_10GPRS(22, r1) mflr r20 /* Return to switch caller */ -#ifdef CONFIG_ALTIVEC -BEGIN_FTR_SECTION - mfspr r24,SPRN_VRSAVE /* save vrsave register value */ - std r24,THREAD_VRSAVE(r3) -END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) -#endif /* CONFIG_ALTIVEC */ + std r20,_NIP(r1) mfcr r23 std r23,_CCR(r1) std r1,KSP(r3) /* Set old stack pointer */ -#ifdef CONFIG_PPC_BOOK3S_64 -BEGIN_FTR_SECTION - /* Event based branch registers */ - mfspr r0, SPRN_BESCR - std r0, THREAD_BESCR(r3) - mfspr r0, SPRN_EBBHR - std r0, THREAD_EBBHR(r3) - mfspr r0, SPRN_EBBRR - std r0, THREAD_EBBRR(r3) -END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S) -#endif - #ifdef CONFIG_SMP /* We need a sync somewhere here to make sure that if the * previous task gets rescheduled on another CPU, it sees all@@ -563,47 +546,6 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) mr r1,r8 /* start using new stack pointer */ std r7,PACAKSAVE(r13) -#ifdef CONFIG_PPC_BOOK3S_64 -BEGIN_FTR_SECTION - /* Event based branch registers */ - ld r0, THREAD_BESCR(r4) - mtspr SPRN_BESCR, r0 - ld r0, THREAD_EBBHR(r4) - mtspr SPRN_EBBHR, r0 - ld r0, THREAD_EBBRR(r4) - mtspr SPRN_EBBRR, r0 - - ld r0,THREAD_TAR(r4) - mtspr SPRN_TAR,r0 -END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S) -#endif - -#ifdef CONFIG_ALTIVEC -BEGIN_FTR_SECTION - ld r0,THREAD_VRSAVE(r4) - mtspr SPRN_VRSAVE,r0 /* if G4, restore VRSAVE reg */ -END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) -#endif /* CONFIG_ALTIVEC */ -#ifdef CONFIG_PPC64 -BEGIN_FTR_SECTION - lwz r6,THREAD_DSCR_INHERIT(r4) - ld r0,THREAD_DSCR(r4) - cmpwi r6,0 - bne 1f - ld r0,PACA_DSCR_DEFAULT(r13) -1: -BEGIN_FTR_SECTION_NESTED(70) - mfspr r8, SPRN_FSCR - rldimi r8, r6, FSCR_DSCR_LG, (63 - FSCR_DSCR_LG) - mtspr SPRN_FSCR, r8 -END_FTR_SECTION_NESTED(CPU_FTR_ARCH_207S, CPU_FTR_ARCH_207S, 70) - cmpd r0,r25 - beq 2f - mtspr SPRN_DSCR,r0 -2: -END_FTR_SECTION_IFSET(CPU_FTR_DSCR) -#endif - ld r6,_CCR(r1) mtcrf 0xFF,r6arch/powerpc/kernel/process.c +80
-12@@ -742,6 +742,73 @@ void restore_tm_state(struct pt_regs *regs)#define __switch_to_tm (prev)#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ +static inline void save_sprs (struct thread_struct *t) +{ +#ifdef CONFIG_ALTIVEC +if (cpu_has_feature (cpu_has_feature (CPU_FTR_ALTIVEC ))) + t->vrsave =mfspr (SPRN_VRSAVE ); +#endif +#ifdef CONFIG_PPC_BOOK3S_64 +if (cpu_has_feature (CPU_FTR_DSCR )) + t->dscr =mfspr (SPRN_DSCR ); + +if (cpu_has_feature (CPU_FTR_ARCH_207S )) { + t->bescr =mfspr (SPRN_BESCR ); + t->ebbhr =mfspr (SPRN_EBBHR ); + t->ebbrr =mfspr (SPRN_EBBRR ); + + t->fscr =mfspr (SPRN_FSCR ); + +/* +* Note that the TAR is not available for use in the kernel. +* (To provide this, the TAR should be backed up/restored on +* exception entry/exit instead, and be in pt_regs. FIXME, +* this should be in pt_regs anyway (for debug).) +*/ + t->tar =mfspr (SPRN_TAR ); + } +#endif +} + +static inline void restore_sprs (struct thread_struct *old_thread, +struct thread_struct *new_thread) +{ +#ifdef CONFIG_ALTIVEC +if (cpu_has_feature (CPU_FTR_ALTIVEC ) && + old_thread->vrsave != new_thread->vrsave) +mtspr (SPRN_VRSAVE , new_thread->vrsave); +#endif +#ifdef CONFIG_PPC_BOOK3S_64 +if (cpu_has_feature (CPU_FTR_DSCR )) { +u64 dscr =get_paca ()->dscr_default; +u64 fscr = old_thread->fscr & ~FSCR_DSCR ; + +if (new_thread->dscr_inherit) { + dscr = new_thread->dscr; + fscr |=FSCR_DSCR ; + } + +if (old_thread->dscr != dscr) +mtspr (SPRN_DSCR , dscr); + +if (old_thread->fscr != fscr) +mtspr (SPRN_FSCR , fscr); + } + +if (cpu_has_feature (CPU_FTR_ARCH_207S )) { +if (old_thread->bescr != new_thread->bescr) +mtspr (SPRN_BESCR , new_thread->bescr); +if (old_thread->ebbhr != new_thread->ebbhr) +mtspr (SPRN_EBBHR , new_thread->ebbhr); +if (old_thread->ebbrr != new_thread->ebbrr) +mtspr (SPRN_EBBRR , new_thread->ebbrr); + +if (old_thread->tar != new_thread->tar) +mtspr (SPRN_TAR , new_thread->tar); + } +#endif +} +struct task_struct *__switch_to (struct task_struct *prev,struct task_struct *new) {@@ -751,17 +818,16 @@ struct task_struct *__switch_to(struct task_struct *prev,struct ppc64_tlb_batch *batch;#endif + new_thread = &new->thread; + old_thread = ¤t->thread; +WARN_ON (!irqs_disabled ()); -/* Back up the TAR and DSCR across context switches. -* Note that the TAR is not available for use in the kernel. (To -* provide this, the TAR should be backed up/restored on exception -* entry/exit instead, and be in pt_regs. FIXME, this should be in -* pt_regs anyway (for debug).) -* Save the TAR and DSCR here before we do treclaim/trecheckpoint as -* these will change them. +/* +* We need to save SPRs before treclaim/trecheckpoint as these will +* change a number of them. */ -save_early_sprs (&prev->thread); +save_sprs (&prev->thread);__switch_to_tm (prev);@@ -844,10 +910,6 @@ struct task_struct *__switch_to(struct task_struct *prev,#endif /* CONFIG_HAVE_HW_BREAKPOINT */ #endif - - new_thread = &new->thread; - old_thread = ¤t->thread; -#ifdef CONFIG_PPC64 /* * Collect processor utilization data per process @@ -883,6 +945,10 @@ struct task_struct *__switch_to(struct task_struct *prev, last =_switch (old_thread, new_thread); +/* Need to recalculate these after calling _switch() */ + old_thread = &last->thread; + new_thread = ¤t->thread; +#ifdef CONFIG_PPC_BOOK3S_64 if (current_thread_info ()->local_flags & _TLF_LAZY_MMU) {current_thread_info ()->local_flags &= ~_TLF_LAZY_MMU;@@ -891,6 +957,8 @@ struct task_struct *__switch_to(struct task_struct *prev, }#endif /* CONFIG_PPC_BOOK3S_64 */ +restore_sprs (old_thread, new_thread); +return last; }
PATCH 4/19 powerpc: Remove redundant mflr in _switch
In reply to Anton BlanchardNo need to execute mflr twice.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/entry_64.S | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
arch/powerpc/kernel/entry_64.S +1 -3
@@ -452,9 +452,7 @@ _GLOBAL(_switch) /* r3-r13 are caller saved -- Cort */ SAVE_8GPRS(14, r1) SAVE_10GPRS(22, r1) - mflr r20 /* Return to switch caller */ - - std r20,_NIP(r1) + std r0,_NIP(r1) /* Return to switch caller */ mfcr r23 std r23,_CCR(r1) std r1,KSP(r3) /* Set old stack pointer */
Signature · 1 line
2.5.0
PATCH 5/19 powerpc: Remove UP only lazy floating point and vector optimisations
In reply to Anton BlanchardThe UP only lazy floating point and vector optimisations were written back when SMP was not common, and neither glibc nor gcc used vector instructions. Now SMP is very common, glibc aggressively uses vector instructions and gcc autovectorises.
We want to add new optimisations that apply to both UP and SMP, but in preparation for that remove these UP only optimisations.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/processor.h | 6 -- arch/powerpc/include/asm/switch_to.h | 8 --- arch/powerpc/kernel/fpu.S | 35 ----------- arch/powerpc/kernel/head_fsl_booke.S | 32 ---------- arch/powerpc/kernel/idle_power7.S | 7 --- arch/powerpc/kernel/process.c | 113 +---------------------------------- arch/powerpc/kernel/signal_32.c | 18 ------ arch/powerpc/kernel/signal_64.c | 18 ------ arch/powerpc/kernel/vector.S | 68 --------------------- 9 files changed, 1 insertion(+), 304 deletions(-)
arch/powerpc/include/asm/processor.h +0 -6
@@ -88,12 +88,6 @@ struct task_struct;void start_thread (struct pt_regs *regs,unsigned long fdptr,unsigned long sp);void release_thread (struct task_struct *); -/* Lazy FPU handling on uni-processor */ -extern struct task_struct *last_task_used_math; -extern struct task_struct *last_task_used_altivec; -extern struct task_struct *last_task_used_vsx; -extern struct task_struct *last_task_used_spe; -#ifdef CONFIG_PPC32 #if CONFIG_TASK_SIZE >CONFIG_KERNEL_START
arch/powerpc/include/asm/switch_to.h +0 -8
@@ -27,14 +27,6 @@ extern void giveup_spe(struct task_struct *);extern void load_up_spe (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug); -#ifndef CONFIG_SMP -extern void discard_lazy_cpu_state (void ); -#else -static inline void discard_lazy_cpu_state (void ) -{ -} -#endif -#ifdef CONFIG_PPC_FPU extern void flush_fp_to_thread (struct task_struct *);extern void giveup_fpu (struct task_struct *);
arch/powerpc/kernel/fpu.S +0 -35
@@ -132,31 +132,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) SYNC MTMSRD(r5) /* enable use of fpu now */ isync -/* - * For SMP, we don't do lazy FPU switching because it just gets too - * horrendously complex, especially when a task switches from one CPU - * to another. Instead we call giveup_fpu in switch_to. - */ -#ifndef CONFIG_SMP - LOAD_REG_ADDRBASE(r3, last_task_used_math) - toreal(r3) - PPC_LL r4,ADDROFF(last_task_used_math)(r3) - PPC_LCMPI 0,r4,0 - beq 1f - toreal(r4) - addi r4,r4,THREAD /* want last_task_used_math->thread */ - addi r10,r4,THREAD_FPSTATE - SAVE_32FPVSRS(0, R5, R10) - mffs fr0 - stfd fr0,FPSTATE_FPSCR(r10) - PPC_LL r5,PT_REGS(r4) - toreal(r5) - PPC_LL r4,_MSR-STACK_FRAME_OVERHEAD(r5) - li r10,MSR_FP|MSR_FE0|MSR_FE1 - andc r4,r4,r10 /* disable FP for previous task */ - PPC_STL r4,_MSR-STACK_FRAME_OVERHEAD(r5) -1: -#endif /* CONFIG_SMP */ /* enable use of FP after return */ #ifdef CONFIG_PPC32 mfspr r5,SPRN_SPRG_THREAD /* current task's THREAD (phys) */
@@ -175,11 +150,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) lfd fr0,FPSTATE_FPSCR(r10) MTFSF_L(fr0) REST_32FPVSRS(0, R4, R10) -#ifndef CONFIG_SMP - subi r4,r5,THREAD - fromreal(r4) - PPC_STL r4,ADDROFF(last_task_used_math)(r3) -#endif /* CONFIG_SMP */ /* restore registers and return */ /* we haven't used ctr or xer or lr */ blr
@@ -226,11 +196,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) andc r4,r4,r3 /* disable FP for previous task */ PPC_STL r4,_MSR-STACK_FRAME_OVERHEAD(r5) 1: -#ifndef CONFIG_SMP - li r5,0 - LOAD_REG_ADDRBASE(r4,last_task_used_math) - PPC_STL r5,ADDROFF(last_task_used_math)(r4) -#endif /* CONFIG_SMP */ blr /*
arch/powerpc/kernel/head_fsl_booke.S +0 -32
@@ -857,29 +857,6 @@ _GLOBAL(load_up_spe) oris r5,r5,MSR_SPE@h mtmsr r5 /* enable use of SPE now */ isync -/* - * For SMP, we don't do lazy SPE switching because it just gets too - * horrendously complex, especially when a task switches from one CPU - * to another. Instead we call giveup_spe in switch_to. - */ -#ifndef CONFIG_SMP - lis r3,last_task_used_spe@ha - lwz r4,last_task_used_spe@l(r3) - cmpi 0,r4,0 - beq 1f - addi r4,r4,THREAD /* want THREAD of last_task_used_spe */ - SAVE_32EVRS(0,r10,r4,THREAD_EVR0) - evxor evr10, evr10, evr10 /* clear out evr10 */ - evmwumiaa evr10, evr10, evr10 /* evr10 <- ACC = 0 * 0 + ACC */ - li r5,THREAD_ACC - evstddx evr10, r4, r5 /* save off accumulator */ - lwz r5,PT_REGS(r4) - lwz r4,_MSR-STACK_FRAME_OVERHEAD(r5) - lis r10,MSR_SPE@h - andc r4,r4,r10 /* disable SPE for previous task */ - stw r4,_MSR-STACK_FRAME_OVERHEAD(r5) -1: -#endif /* !CONFIG_SMP */ /* enable use of SPE after return */ oris r9,r9,MSR_SPE@h mfspr r5,SPRN_SPRG_THREAD /* current task's THREAD (phys) */
@@ -889,10 +866,6 @@ _GLOBAL(load_up_spe) evlddx evr4,r10,r5 evmra evr4,evr4 REST_32EVRS(0,r10,r5,THREAD_EVR0) -#ifndef CONFIG_SMP - subi r4,r5,THREAD - stw r4,last_task_used_spe@l(r3) -#endif /* !CONFIG_SMP */ blr /*
@@ -1035,11 +1008,6 @@ _GLOBAL(giveup_spe) andc r4,r4,r3 /* disable SPE for previous task */ stw r4,_MSR-STACK_FRAME_OVERHEAD(r5) 1: -#ifndef CONFIG_SMP - li r5,0 - lis r4,last_task_used_spe@ha - stw r5,last_task_used_spe@l(r4) -#endif /* !CONFIG_SMP */ blr #endif /* CONFIG_SPE */
arch/powerpc/kernel/idle_power7.S +0 -7
@@ -89,13 +89,6 @@ _GLOBAL(power7_powersave_common) std r0,_LINK(r1) std r0,_NIP(r1) -#ifndef CONFIG_SMP - /* Make sure FPU, VSX etc... are flushed as we may lose - * state when going to nap mode - */ - bl discard_lazy_cpu_state -#endif /* CONFIG_SMP */ - /* Hard disable interrupts */ mfmsr r9 rldicl r9,r9,48,1
arch/powerpc/kernel/process.c +1 -112
234 lines not on this page · raw message
arch/powerpc/kernel/signal_32.c +0 -18
@@ -687,15 +687,6 @@ static long restore_user_regs(struct pt_regs *regs,if (sig) regs->msr = (regs->msr & ~MSR_LE ) | (msr &MSR_LE ); -/* -* Do this before updating the thread state in -* current->thread.fpr/vr/evr. That way, if we get preempted -* and another task grabs the FPU/Altivec/SPE, it won't be -* tempted to save the current CPU state into the thread_struct -* and corrupt what we are writing there. -*/ -discard_lazy_cpu_state (); -#ifdef CONFIG_ALTIVEC /* * Force the process to reload the altivec registers from
@@ -798,15 +789,6 @@ static long restore_tm_user_regs(struct pt_regs *regs,/* Restore the previous little-endian mode */ regs->msr = (regs->msr & ~MSR_LE ) | (msr &MSR_LE ); -/* -* Do this before updating the thread state in -* current->thread.fpr/vr/evr. That way, if we get preempted -* and another task grabs the FPU/Altivec/SPE, it won't be -* tempted to save the current CPU state into the thread_struct -* and corrupt what we are writing there. -*/ -discard_lazy_cpu_state (); -#ifdef CONFIG_ALTIVEC regs->msr &= ~MSR_VEC ;if (msr &MSR_VEC ) {
arch/powerpc/kernel/signal_64.c +0 -18
@@ -350,15 +350,6 @@ static long restore_sigcontext(struct pt_regs *regs, sigset_t *set, int sig, err |=__get_user (set->sig[0 ], &sc->oldmask);/* -* Do this before updating the thread state in -* current->thread.fpr/vr. That way, if we get preempted -* and another task grabs the FPU/Altivec, it won't be -* tempted to save the current CPU state into the thread_struct -* and corrupt what we are writing there. -*/ -discard_lazy_cpu_state (); - -/* * Force reload of FP/VEC. * This has to be done before copying stuff into current->thread.fpr/vr * for the reasons explained in the previous comment.
@@ -465,15 +456,6 @@ static long restore_tm_sigcontexts(struct pt_regs *regs, err |=__get_user (regs->result, &sc->gp_regs[PT_RESULT ]);/* -* Do this before updating the thread state in -* current->thread.fpr/vr. That way, if we get preempted -* and another task grabs the FPU/Altivec, it won't be -* tempted to save the current CPU state into the thread_struct -* and corrupt what we are writing there. -*/ -discard_lazy_cpu_state (); - -/* * Force reload of FP/VEC. * This has to be done before copying stuff into current->thread.fpr/vr * for the reasons explained in the previous comment.
arch/powerpc/kernel/vector.S +0 -68
104 lines not on this page · raw message
Signature · 1 line
2.5.0
PATCH 6/19 powerpc: Simplify TM restore checks
In reply to Anton BlanchardInstead of having multiple giveup_*_maybe_transactional() functions, separate out the TM check into a new function called check_if_tm_restore_required().
This will make it easier to optimise the giveup_*() functions in a subsequent patch.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/process.c | 53 ++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 34 deletions(-)
arch/powerpc/kernel/process.c +19 -34
@@ -68,7 +68,7 @@extern unsigned long _get_SP (void );#ifdef CONFIG_PPC_TRANSACTIONAL_MEM -void giveup_fpu_maybe_transactional (struct task_struct *tsk) +static void check_if_tm_restore_required (struct task_struct *tsk) {/* * If we are saving the current thread's registers, and the
@@ -82,31 +82,9 @@ void giveup_fpu_maybe_transactional(struct task_struct *tsk) tsk->thread.ckpt_regs.msr = tsk->thread.regs->msr; set_thread_flag(TIF_RESTORE_TM); } - -giveup_fpu (tsk); -} - -void giveup_altivec_maybe_transactional (struct task_struct *tsk) -{ -/* -* If we are saving the current thread's registers, and the -* thread is in a transactional state, set the TIF_RESTORE_TM -* bit so that we know to restore the registers before -* returning to userspace. -*/ -if (tsk == current && tsk->thread.regs && -MSR_TM_ACTIVE (tsk->thread.regs->msr) && - !test_thread_flag (TIF_RESTORE_TM )) { - tsk->thread.ckpt_regs.msr = tsk->thread.regs->msr; -set_thread_flag (TIF_RESTORE_TM ); - } - -giveup_altivec (tsk); } -#else -#define giveup_fpu_maybe_transactional (tsk) giveup_fpu(tsk) -#define giveup_altivec_maybe_transactional (tsk) giveup_altivec(tsk) +static inline void check_if_tm_restore_required (struct task_struct *tsk) { }#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ #ifdef CONFIG_PPC_FPU
@@ -135,7 +113,8 @@ void flush_fp_to_thread(struct task_struct *tsk)* to still have its FP state in the CPU registers. */ BUG_ON (tsk != current); -giveup_fpu_maybe_transactional (tsk); +check_if_tm_restore_required (tsk); +giveup_fpu (tsk); }preempt_enable (); }
@@ -147,10 +126,12 @@ void enable_kernel_fp(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_FP )) -giveup_fpu_maybe_transactional (current); -else +if (current->thread.regs && (current->thread.regs->msr &MSR_FP )) { +check_if_tm_restore_required (current); +giveup_fpu (current); + }else {giveup_fpu (NULL );/* just enables FP for kernel */ + } }EXPORT_SYMBOL (enable_kernel_fp);
@@ -159,10 +140,12 @@ void enable_kernel_altivec(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_VEC )) -giveup_altivec_maybe_transactional (current); -else +if (current->thread.regs && (current->thread.regs->msr &MSR_VEC )) { +check_if_tm_restore_required (current); +giveup_altivec (current); + }else {giveup_altivec_notask (); + } }EXPORT_SYMBOL (enable_kernel_altivec);
@@ -176,7 +159,8 @@ void flush_altivec_to_thread(struct task_struct *tsk)preempt_disable ();if (tsk->thread.regs->msr &MSR_VEC ) {BUG_ON (tsk != current); -giveup_altivec_maybe_transactional (tsk); +check_if_tm_restore_required (tsk); +giveup_altivec (tsk); }preempt_enable (); }
@@ -198,8 +182,9 @@ EXPORT_SYMBOL(enable_kernel_vsx);void giveup_vsx (struct task_struct *tsk) { -giveup_fpu_maybe_transactional (tsk); -giveup_altivec_maybe_transactional (tsk); +check_if_tm_restore_required (tsk); +giveup_fpu (tsk); +giveup_altivec (tsk);__giveup_vsx (tsk); }EXPORT_SYMBOL (giveup_vsx);
Signature · 1 line
2.5.0
PATCH 7/19 powerpc: Create mtmsrd_isync()
In reply to Anton Blanchardmtmsrd_isync() will do an mtmsrd followed by an isync on older processors. On newer processors we avoid the isync via a feature fixup.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/reg.h | 8 ++++++++ arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++-------- 2 files changed, 30 insertions(+), 8 deletions(-)
arch/powerpc/include/asm/reg.h +8 -0
@@ -1193,12 +1193,20 @@#define __mtmsrd (v, l) asm volatile("mtmsrd %0," __stringify(l) \ : : "r" (v) : "memory")#define mtmsr (v) __mtmsrd((v), 0) +#define __MTMSR "mtmsrd"#else #define mtmsr (v) asm volatile("mtmsr %0" : \ : "r" ((unsigned long)(v)) \ : "memory") +#define __MTMSR "mtmsr"#endif +static inline void mtmsr_isync (unsigned long val) +{ + asm volatile(__MTMSR " %0; " ASM_FTR_IFCLR("isync", "nop", %1) : : + "r" (val), "i" (CPU_FTR_ARCH_206) : "memory"); +} +#define mfspr (rn) ({unsigned long rval; \ asm volatile("mfspr %0," __stringify(rn) \ : "=r" (rval)); rval;})
arch/powerpc/kernel/process.c +22 -8
@@ -130,7 +130,10 @@ void enable_kernel_fp(void)check_if_tm_restore_required (current);giveup_fpu (current); }else { -giveup_fpu (NULL );/* just enables FP for kernel */ +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_FP )) +mtmsr_isync (oldmsr |MSR_FP ); } }EXPORT_SYMBOL (enable_kernel_fp);
@@ -144,7 +147,10 @@ void enable_kernel_altivec(void)check_if_tm_restore_required (current);giveup_altivec (current); }else { -giveup_altivec_notask (); +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_VEC )) +mtmsr_isync (oldmsr |MSR_VEC ); } }EXPORT_SYMBOL (enable_kernel_altivec);
@@ -173,10 +179,14 @@ void enable_kernel_vsx(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) +if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) {giveup_vsx (current); -else -giveup_vsx (NULL );/* just enable vsx for kernel - force */ + }else { +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_VSX )) +mtmsr_isync (oldmsr |MSR_VSX ); + } }EXPORT_SYMBOL (enable_kernel_vsx);
@@ -209,10 +219,14 @@ void enable_kernel_spe(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) +if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) {giveup_spe (current); -else -giveup_spe (NULL );/* just enable SPE for kernel - force */ + }else { +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_SPE )) +mtmsr_isync (oldmsr |MSR_SPE ); + } }EXPORT_SYMBOL (enable_kernel_spe);
Signature · 1 line
2.5.0
Michael Ellerman
reviewerRe: [PATCH 07/19] powerpc: Create mtmsrd_isync()
Quoted 8 lines · Anton Blanchard
mtmsrd_isync()will do an mtmsrd followed by an isync on older processors. On newer processors we avoid the isync via a feature fixup.arch/powerpc/kernel/process.c +0
-0
arch/powerpc/kernel/process.c void enable_kernel_fp(void) · line 136
Quoted 8 lines
arch/powerpc/kernel/process.c +4
-1@@ -130,7 +130,10 @@ void enable_kernel_fp(void)check_if_tm_restore_required (current);giveup_fpu (current); }else { -giveup_fpu (NULL );/* just enables FP for kernel */ +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_FP )) +mtmsr_isync (oldmsr |MSR_FP );
You seem to do this pattern at all call sites.
So should we instead have a helper that makes sure an MSR bit(s) is/are set?
Maybe:
static inline void msr_enable(unsigned long enable_bits)
{
u64 val, msr = mfmsr();
if (msr & enable_bits)
return;
val = msr | enable_bits;
asm volatile(__MTMSR " %0; " ASM_FTR_IFCLR("isync", "nop", %1) : :
"r" (val), "i" (CPU_FTR_ARCH_206) : "memory");
}Thoughts?
cheers
Michael Ellerman
reviewerRe: [PATCH 07/19] powerpc: Create mtmsrd_isync()
Quoted 27 lines · Michael Ellerman
Quoted 8 lines · Anton Blanchard
mtmsrd_isync()will do an mtmsrd followed by an isync on older processors. On newer processors we avoid the isync via a feature fixup.arch/powerpc/kernel/process.c +0
-0arch/powerpc/kernel/process.c void enable_kernel_fp(void) · line 136
Quoted 8 lines
arch/powerpc/kernel/process.c +4
-1@@ -130,7 +130,10 @@ void enable_kernel_fp(void)check_if_tm_restore_required (current);giveup_fpu (current); }else { -giveup_fpu (NULL );/* just enables FP for kernel */ +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_FP )) +mtmsr_isync (oldmsr |MSR_FP );You seem to do this pattern at all call sites.
So should we instead have a helper that makes sure an MSR bit(s) is/are set?
Maybe:
static inline void
msr_enable(unsigned long enable_bits)
Quoted 10 lines
{
u64 val, msr = mfmsr();
if (msr & enable_bits)
return;
val = msr | enable_bits;
asm volatile(__MTMSR " %0; " ASM_FTR_IFCLR("isync", "nop", %1) : :
"r" (val), "i" (CPU_FTR_ARCH_206) : "memory");
}Ugh, should have read the whole series. You do more or less this in patch 12.
cheers
Re: [PATCH 07/19] powerpc: Create mtmsrd_isync()
In reply to Anton BlanchardQuoted 2 lines · Anton Blanchard
mtmsrd_isync()will do an mtmsrd followed by an isync on older processors. On newer processors we avoid the isync via a feature fixup.
The isync is needed specifically when enabling/disable FP etc... right ?
I'd like to make the name a bit clearer. Maybe something like set_msr_fpvec() or maybe you can come up with something even better, ie
use a name that represents what it's for rather than what it does.
Quoted 95 lines
Signed-off-by Anton Blanchard <[email protected]>--- arch/powerpc/include/asm/reg.h | 8 ++++++++ arch/powerpc/kernel/process.c | 30 ++++++++++++++++++++++-------- 2 files changed, 30 insertions(+), 8 deletions(-)arch/powerpc/include/asm/reg.h +8
-0@@ -1193,12 +1193,20 @@#define __mtmsrd (v, l) asm volatile("mtmsrd %0," __stringify(l) \ : : "r" (v) : "memory")#define mtmsr (v) __mtmsrd((v), 0) +#define __MTMSR "mtmsrd"#else #define mtmsr (v) asm volatile("mtmsr %0" : \ : "r" ((unsigned long)(v)) \ : "memory") +#define __MTMSR "mtmsr"#endif +static inline void mtmsr_isync (unsigned long val) +{ + asm volatile(__MTMSR " %0; " ASM_FTR_IFCLR("isync", "nop", %1) : : + "r" (val), "i" (CPU_FTR_ARCH_206) : "memory"); +} +#define mfspr (rn) ({unsigned long rval; \ asm volatile("mfspr %0," __stringify(rn) \ : "=r" (rval)); rval;})arch/powerpc/kernel/process.c +22
-8@@ -130,7 +130,10 @@ void enable_kernel_fp(void)check_if_tm_restore_required (current);giveup_fpu (current); }else { -giveup_fpu (NULL );/* just enables FP for kernel */ +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_FP )) +mtmsr_isync (oldmsr |MSR_FP ); } }EXPORT_SYMBOL (enable_kernel_fp);@@ -144,7 +147,10 @@ void enable_kernel_altivec(void)check_if_tm_restore_required (current);giveup_altivec (current); }else { -giveup_altivec_notask (); +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_VEC )) +mtmsr_isync (oldmsr |MSR_VEC ); } }EXPORT_SYMBOL (enable_kernel_altivec);@@ -173,10 +179,14 @@ void enable_kernel_vsx(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) +if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) {giveup_vsx (current); -else -giveup_vsx (NULL );/* just enable vsx for kernel - force */ + }else { +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_VSX )) +mtmsr_isync (oldmsr |MSR_VSX ); + } }EXPORT_SYMBOL (enable_kernel_vsx);@@ -209,10 +219,14 @@ void enable_kernel_spe(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) +if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) {giveup_spe (current); -else -giveup_spe (NULL );/* just enable SPE for kernel - force */ + }else { +u64 oldmsr =mfmsr (); + +if (!(oldmsr &MSR_SPE )) +mtmsr_isync (oldmsr |MSR_SPE ); + } }EXPORT_SYMBOL (enable_kernel_spe);
PATCH 8/19 powerpc: Remove NULL task struct pointer checks in FP and vector code
In reply to Anton BlanchardWe used to allow giveup_*() to be called with a NULL task struct pointer. Now those cases are handled in the caller we can remove the checks. We can also remove giveup_altivec_notask() which is also unused.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/switch_to.h | 1 - arch/powerpc/kernel/fpu.S | 2 -- arch/powerpc/kernel/head_fsl_booke.S | 2 -- arch/powerpc/kernel/vector.S | 14 -------------- 4 files changed, 19 deletions(-)
arch/powerpc/include/asm/switch_to.h +0 -1
@@ -38,7 +38,6 @@ static inline void giveup_fpu(struct task_struct *t) { }#ifdef CONFIG_ALTIVEC extern void flush_altivec_to_thread (struct task_struct *);extern void giveup_altivec (struct task_struct *); -extern void giveup_altivec_notask (void );#else static inline void flush_altivec_to_thread (struct task_struct *t) {
arch/powerpc/kernel/fpu.S +0 -2
@@ -173,8 +173,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) MTMSRD(r5) /* enable use of fpu now */ SYNC_601 isync - PPC_LCMPI 0,r3,0 - beqlr- /* if no previous owner, done */ addi r3,r3,THREAD /* want THREAD of task */ PPC_LL r6,THREAD_FPSAVEAREA(r3) PPC_LL r5,PT_REGS(r3)
arch/powerpc/kernel/head_fsl_booke.S +0 -2
@@ -992,8 +992,6 @@ _GLOBAL(giveup_spe) oris r5,r5,MSR_SPE@h mtmsr r5 /* enable use of SPE now */ isync - cmpi 0,r3,0 - beqlr- /* if no previous owner, done */ addi r3,r3,THREAD /* want THREAD of task */ lwz r5,PT_REGS(r3) cmpi 0,r5,0
arch/powerpc/kernel/vector.S +0 -14
@@ -111,16 +111,6 @@ _GLOBAL(load_up_altivec) /* restore registers and return */ blr -_GLOBAL(giveup_altivec_notask) - mfmsr r3 - andis. r4,r3,MSR_VEC@h - bnelr /* Already enabled? */ - oris r3,r3,MSR_VEC@h - SYNC - MTMSRD(r3) /* enable use of VMX now */ - isync - blr - /* * giveup_altivec(tsk) * Disable VMX for the task given as the argument,
@@ -133,8 +123,6 @@ _GLOBAL(giveup_altivec) SYNC MTMSRD(r5) /* enable use of VMX now */ isync - PPC_LCMPI 0,r3,0 - beqlr /* if no previous owner, done */ addi r3,r3,THREAD /* want THREAD of task */ PPC_LL r7,THREAD_VRSAVEAREA(r3) PPC_LL r5,PT_REGS(r3)
@@ -203,8 +191,6 @@ _GLOBAL(__giveup_vsx) mtmsrd r5 /* enable use of VSX now */ isync - cmpdi 0,r3,0 - beqlr- /* if no previous owner, done */ addi r3,r3,THREAD /* want THREAD of task */ ld r5,PT_REGS(r3) cmpdi 0,r5,0
Signature · 1 line
2.5.0
PATCH 9/19 powerpc: Move part of giveup_fpu,altivec,spe into c
In reply to Anton BlanchardMove the MSR modification into new c functions. Removing it from the low level functions will allow us to avoid costly MSR writes by batching them up.
Move the check_if_tm_restore_required() check into these new functions.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/switch_to.h | 21 ++++++++------- arch/powerpc/kernel/fpu.S | 16 ++--------- arch/powerpc/kernel/head_fsl_booke.S | 8 ++---- arch/powerpc/kernel/ppc_ksyms.c | 6 ----- arch/powerpc/kernel/process.c | 52 +++++++++++++++++++++++++++++++++--- arch/powerpc/kernel/vector.S | 10 ++----- 6 files changed, 65 insertions(+), 48 deletions(-)
arch/powerpc/include/asm/switch_to.h +11 -10
@@ -23,28 +23,27 @@ extern int emulate_altivec(struct pt_regs *);extern void __giveup_vsx (struct task_struct *);extern void giveup_vsx (struct task_struct *);extern void enable_kernel_spe (void ); -extern void giveup_spe (struct task_struct *);extern void load_up_spe (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug);#ifdef CONFIG_PPC_FPU extern void flush_fp_to_thread (struct task_struct *);extern void giveup_fpu (struct task_struct *); +extern void __giveup_fpu (struct task_struct *);#else static inline void flush_fp_to_thread (struct task_struct *t) { }static inline void giveup_fpu (struct task_struct *t) { } +static inline void __giveup_fpu (struct task_struct *t) { }#endif #ifdef CONFIG_ALTIVEC extern void flush_altivec_to_thread (struct task_struct *);extern void giveup_altivec (struct task_struct *); +extern void __giveup_altivec (struct task_struct *);#else -static inline void flush_altivec_to_thread (struct task_struct *t) -{ -} -static inline void giveup_altivec (struct task_struct *t) -{ -} +static inline void flush_altivec_to_thread (struct task_struct *t) { } +static inline void giveup_altivec (struct task_struct *t) { } +static inline void __giveup_altivec (struct task_struct *t) { }#endif #ifdef CONFIG_VSX
@@ -57,10 +56,12 @@ static inline void flush_vsx_to_thread(struct task_struct *t)#ifdef CONFIG_SPE extern void flush_spe_to_thread (struct task_struct *); +extern void giveup_spe (struct task_struct *); +extern void __giveup_spe (struct task_struct *);#else -static inline void flush_spe_to_thread (struct task_struct *t) -{ -} +static inline void flush_spe_to_thread (struct task_struct *t) { } +static inline void giveup_spe (struct task_struct *t) { } +static inline void __giveup_spe (struct task_struct *t) { }#endif static inline void clear_task_ebb (struct task_struct *t)
arch/powerpc/kernel/fpu.S +2 -14
@@ -155,24 +155,12 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) blr /* - * giveup_fpu(tsk) + * __giveup_fpu(tsk) * Disable FP for the task given as the argument, * and save the floating-point registers in its thread_struct. * Enables the FPU for use in the kernel on return. */ -_GLOBAL(giveup_fpu) - mfmsr r5 - ori r5,r5,MSR_FP -#ifdef CONFIG_VSX -BEGIN_FTR_SECTION - oris r5,r5,MSR_VSX@h -END_FTR_SECTION_IFSET(CPU_FTR_VSX) -#endif - SYNC_601 - ISYNC_601 - MTMSRD(r5) /* enable use of fpu now */ - SYNC_601 - isync +_GLOBAL(__giveup_fpu) addi r3,r3,THREAD /* want THREAD of task */ PPC_LL r6,THREAD_FPSAVEAREA(r3) PPC_LL r5,PT_REGS(r3)
arch/powerpc/kernel/head_fsl_booke.S +2 -6
@@ -984,14 +984,10 @@ _GLOBAL(__setup_ehv_ivors) #ifdef CONFIG_SPE /* - * extern void giveup_spe(struct task_struct *prev) + * extern void __giveup_spe(struct task_struct *prev) * */ -_GLOBAL(giveup_spe) - mfmsr r5 - oris r5,r5,MSR_SPE@h - mtmsr r5 /* enable use of SPE now */ - isync +_GLOBAL(__giveup_spe) addi r3,r3,THREAD /* want THREAD of task */ lwz r5,PT_REGS(r3) cmpi 0,r5,0
arch/powerpc/kernel/ppc_ksyms.c +0 -6
@@ -19,13 +19,11 @@ EXPORT_SYMBOL(_mcount);#endif #ifdef CONFIG_PPC_FPU -EXPORT_SYMBOL (giveup_fpu);EXPORT_SYMBOL (load_fp_state);EXPORT_SYMBOL (store_fp_state);#endif #ifdef CONFIG_ALTIVEC -EXPORT_SYMBOL (giveup_altivec);EXPORT_SYMBOL (load_vr_state);EXPORT_SYMBOL (store_vr_state);#endif
@@ -34,10 +32,6 @@ EXPORT_SYMBOL(store_vr_state);EXPORT_SYMBOL_GPL (__giveup_vsx);#endif -#ifdef CONFIG_SPE -EXPORT_SYMBOL (giveup_spe); -#endif -#ifdef CONFIG_EPAPR_PARAVIRT EXPORT_SYMBOL (epapr_hypercall_start);#endif
arch/powerpc/kernel/process.c +48 -4
93 lines not on this page · raw message
arch/powerpc/kernel/vector.S +2 -8
@@ -112,17 +112,11 @@ _GLOBAL(load_up_altivec) blr /* - * giveup_altivec(tsk) + * __giveup_altivec(tsk) * Disable VMX for the task given as the argument, * and save the vector registers in its thread_struct. - * Enables the VMX for use in the kernel on return. */ -_GLOBAL(giveup_altivec) - mfmsr r5 - oris r5,r5,MSR_VEC@h - SYNC - MTMSRD(r5) /* enable use of VMX now */ - isync +_GLOBAL(__giveup_altivec) addi r3,r3,THREAD /* want THREAD of task */ PPC_LL r7,THREAD_VRSAVEAREA(r3) PPC_LL r5,PT_REGS(r3)
Signature · 1 line
2.5.0
Michael Ellerman
reviewerRe: [PATCH 09/19] powerpc: Move part of giveup_fpu,altivec,spe into c
Quoted 11 lines · Anton Blanchard
Move the MSR modification into new c functions. Removing it from the low level functions will allow us to avoid costly MSR writes by batching them up.
Move the
check_if_tm_restore_required()check into these new functions.arch/powerpc/kernel/process.c +0
-0
arch/powerpc/kernel/process.c EXPORT_SYMBOL_GPL(flush_vsx_to_thread); · line 252
Quoted 9 lines
arch/powerpc/kernel/process.c +6
-0@@ -214,6 +244,20 @@ EXPORT_SYMBOL_GPL(flush_vsx_to_thread);#endif /* CONFIG_VSX */ #ifdef CONFIG_SPE +void giveup_spe (struct task_struct *tsk) +{ +u64 oldmsr =mfmsr (); +u64 newmsr; + +check_if_tm_restore_required (tsk);
We can't be building with SPE and also TM can we? I guess this is harmless as it will compile to nothing, but it might be confusing to readers.
arch/powerpc/kernel/process.c EXPORT_SYMBOL_GPL(flush_vsx_to_thread); · line 263
Quoted 10 lines
arch/powerpc/kernel/process.c +7
-0+ newmsr = oldmsr |MSR_SPE ; +if (oldmsr != newmsr) +mtmsr_isync (newmsr); + +__giveup_spe (tsk); +} +EXPORT_SYMBOL (giveup_spe);void enable_kernel_spe (void ) {
cheers
PATCH 10/19 powerpc: Move part of giveup_vsx into c
In reply to Anton BlanchardMove the MSR modification into c. Removing it from the assembly function will allow us to avoid costly MSR writes by batching them up.
Check the FP and VMX bits before calling the relevant giveup_*() function. This makes giveup_vsx() and flush_vsx_to_thread() perform more like their sister functions, and allows us to use flush_vsx_to_thread() in the signal code.
Move the check_if_tm_restore_required() check in.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/process.c | 28 +++++++++++++++++++--------- arch/powerpc/kernel/signal_32.c | 4 ++-- arch/powerpc/kernel/signal_64.c | 4 ++-- arch/powerpc/kernel/vector.S | 6 ------ 4 files changed, 23 insertions(+), 19 deletions(-)
arch/powerpc/kernel/process.c +19 -9
@@ -205,6 +205,25 @@ EXPORT_SYMBOL_GPL(flush_altivec_to_thread);#endif /* CONFIG_ALTIVEC */ #ifdef CONFIG_VSX +void giveup_vsx (struct task_struct *tsk) +{ +u64 oldmsr =mfmsr (); +u64 newmsr; + +check_if_tm_restore_required (tsk); + + newmsr = oldmsr | (MSR_FP |MSR_VEC |MSR_VSX ); +if (oldmsr != newmsr) +mtmsr_isync (newmsr); + +if (tsk->thread.regs->msr &MSR_FP ) +__giveup_fpu (tsk); +if (tsk->thread.regs->msr &MSR_VEC ) +__giveup_altivec (tsk); +__giveup_vsx (tsk); +} +EXPORT_SYMBOL (giveup_vsx); +void enable_kernel_vsx (void ) {WARN_ON (preemptible ());
@@ -220,15 +239,6 @@ void enable_kernel_vsx(void) }EXPORT_SYMBOL (enable_kernel_vsx); -void giveup_vsx (struct task_struct *tsk) -{ -check_if_tm_restore_required (tsk); -giveup_fpu (tsk); -giveup_altivec (tsk); -__giveup_vsx (tsk); -} -EXPORT_SYMBOL (giveup_vsx); -void flush_vsx_to_thread (struct task_struct *tsk) {if (tsk->thread.regs) {
arch/powerpc/kernel/signal_32.c +2 -2
@@ -458,7 +458,7 @@ static int save_user_regs(struct pt_regs *regs, struct mcontext __user *frame,* contains valid data */ if (current->thread.used_vsr && ctx_has_vsx_region) { -__giveup_vsx (current); +flush_vsx_to_thread (current);if (copy_vsx_to_user (&frame->mc_vsregs, current))return 1 ; msr |=MSR_VSX ;
@@ -606,7 +606,7 @@ static int save_tm_user_regs(struct pt_regs *regs,* contains valid data */ if (current->thread.used_vsr) { -__giveup_vsx (current); +flush_vsx_to_thread (current);if (copy_vsx_to_user (&frame->mc_vsregs, current))return 1 ;if (msr &MSR_VSX ) {
arch/powerpc/kernel/signal_64.c +2 -2
@@ -147,7 +147,7 @@ static long setup_sigcontext(struct sigcontext __user *sc, struct pt_regs *regs,* VMX data. */ if (current->thread.used_vsr && ctx_has_vsx_region) { -__giveup_vsx (current); +flush_vsx_to_thread (current); v_regs +=ELF_NVRREG ; err |=copy_vsx_to_user (v_regs, current);/* set MSR_VSX in the MSR value in the frame to
@@ -270,7 +270,7 @@ static long setup_tm_sigcontexts(struct sigcontext __user *sc,* VMX data. */ if (current->thread.used_vsr) { -__giveup_vsx (current); +flush_vsx_to_thread (current); v_regs +=ELF_NVRREG ; tm_v_regs +=ELF_NVRREG ;
arch/powerpc/kernel/vector.S +0 -6
@@ -177,14 +177,8 @@ _GLOBAL(load_up_vsx) * __giveup_vsx(tsk) * Disable VSX for the task given as the argument. * Does NOT save vsx registers. - * Enables the VSX for use in the kernel on return. */ _GLOBAL(__giveup_vsx) - mfmsr r5 - oris r5,r5,MSR_VSX@h - mtmsrd r5 /* enable use of VSX now */ - isync - addi r3,r3,THREAD /* want THREAD of task */ ld r5,PT_REGS(r3) cmpdi 0,r5,0
Signature · 1 line
2.5.0
PATCH 11/19 crypto: vmx: Only call enable_kernel_vsx()
In reply to Anton BlanchardWith the recent change to enable_kernel_vsx(), we no longer need to call enable_kernel_fp() and enable_kernel_altivec().
Signed-off-by Anton Blanchard <[email protected]>
--- drivers/crypto/vmx/aes.c | 3 --- drivers/crypto/vmx/aes_cbc.c | 3 --- drivers/crypto/vmx/aes_ctr.c | 3 --- drivers/crypto/vmx/ghash.c | 8 -------- 4 files changed, 17 deletions(-)
drivers/crypto/vmx/aes.c +0 -3
@@ -83,7 +83,6 @@ static int p8_aes_setkey(struct crypto_tfm *tfm, const u8 *key,preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key); ret +=aes_p8_set_decrypt_key (key, keylen *8 , &ctx->dec_key);
@@ -103,7 +102,6 @@ static void p8_aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) }else {preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx ();aes_p8_encrypt (src, dst, &ctx->enc_key);pagefault_enable ();
@@ -120,7 +118,6 @@ static void p8_aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src) }else {preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx ();aes_p8_decrypt (src, dst, &ctx->dec_key);pagefault_enable ();
drivers/crypto/vmx/aes_cbc.c +0 -3
@@ -84,7 +84,6 @@ static int p8_aes_cbc_setkey(struct crypto_tfm *tfm, const u8 *key,preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key); ret +=aes_p8_set_decrypt_key (key, keylen *8 , &ctx->dec_key);
@@ -115,7 +114,6 @@ static int p8_aes_cbc_encrypt(struct blkcipher_desc *desc, }else {preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx ();blkcipher_walk_init (&walk, dst, src, nbytes);
@@ -156,7 +154,6 @@ static int p8_aes_cbc_decrypt(struct blkcipher_desc *desc, }else {preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx ();blkcipher_walk_init (&walk, dst, src, nbytes);
drivers/crypto/vmx/aes_ctr.c +0 -3
@@ -81,7 +81,6 @@ static int p8_aes_ctr_setkey(struct crypto_tfm *tfm, const u8 *key,struct p8_aes_ctr_ctx *ctx =crypto_tfm_ctx (tfm);pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key);pagefault_enable ();
@@ -100,7 +99,6 @@ static void p8_aes_ctr_final(struct p8_aes_ctr_ctx *ctx,unsigned int nbytes = walk->nbytes;pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx ();aes_p8_encrypt (ctrblk, keystream, &ctx->enc_key);pagefault_enable ();
@@ -133,7 +131,6 @@ static int p8_aes_ctr_crypt(struct blkcipher_desc *desc, ret =blkcipher_walk_virt_block (desc, &walk,AES_BLOCK_SIZE );while ((nbytes = walk.nbytes) >=AES_BLOCK_SIZE ) {pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); aes_p8_ctr32_encrypt_blocks(walk.src.virt.addr, walk.dst.virt.addr,
drivers/crypto/vmx/ghash.c +0 -8
@@ -118,9 +118,7 @@ static int p8_ghash_setkey(struct crypto_shash *tfm, const u8 *key,preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); -enable_kernel_fp ();gcm_init_p8 (ctx->htable, (const u64 *) key);pagefault_enable ();preempt_enable ();
@@ -149,9 +147,7 @@ static int p8_ghash_update(struct shash_desc *desc, GHASH_DIGEST_SIZE - dctx->bytes);preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); -enable_kernel_fp ();gcm_ghash_p8 (dctx->shash, ctx->htable, dctx->buffer,GHASH_DIGEST_SIZE );pagefault_enable ();
@@ -164,9 +160,7 @@ static int p8_ghash_update(struct shash_desc *desc,if (len) {preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); -enable_kernel_fp ();gcm_ghash_p8 (dctx->shash, ctx->htable, src, len);pagefault_enable ();preempt_enable ();
@@ -195,9 +189,7 @@ static int p8_ghash_final(struct shash_desc *desc, u8 *out) dctx->buffer[i] =0 ;preempt_disable ();pagefault_disable (); -enable_kernel_altivec ();enable_kernel_vsx (); -enable_kernel_fp ();gcm_ghash_p8 (dctx->shash, ctx->htable, dctx->buffer,GHASH_DIGEST_SIZE );pagefault_enable ();
Signature · 1 line
2.5.0
PATCH 12/19 powerpc: Create msr_check_and_{set,clear}()
In reply to Anton BlanchardCreate helper functions to set and clear MSR bits after first checking if they are already set. Grouping them will make it easy to avoid the MSR writes in a subsequent optimisation.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/process.c | 107 ++++++++++++++++++++---------------------- 1 file changed, 52 insertions(+), 55 deletions(-)
arch/powerpc/kernel/process.c +52 -55
@@ -87,23 +87,46 @@ static void check_if_tm_restore_required(struct task_struct *tsk)static inline void check_if_tm_restore_required (struct task_struct *tsk) { }#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ -#ifdef CONFIG_PPC_FPU -void giveup_fpu (struct task_struct *tsk) +static void msr_check_and_set (unsigned long bits) { -u64 oldmsr =mfmsr (); -u64 newmsr; +unsigned long oldmsr =mfmsr (); +unsigned long newmsr; -check_if_tm_restore_required (tsk); + newmsr = oldmsr | bits; - newmsr = oldmsr |MSR_FP ;#ifdef CONFIG_VSX -if (cpu_has_feature (CPU_FTR_VSX )) +if (cpu_has_feature (CPU_FTR_VSX ) && (bits &MSR_FP )) newmsr |=MSR_VSX ;#endif +if (oldmsr != newmsr)mtmsr_isync (newmsr); +} + +static void msr_check_and_clear (unsigned long bits) +{ +unsigned long oldmsr =mfmsr (); +unsigned long newmsr; + + newmsr = oldmsr & ~bits; + +#ifdef CONFIG_VSX +if (cpu_has_feature (CPU_FTR_VSX ) && (bits &MSR_FP )) + newmsr &= ~MSR_VSX ; +#endif +if (oldmsr != newmsr) +mtmsr_isync (newmsr); +} + +#ifdef CONFIG_PPC_FPU +void giveup_fpu (struct task_struct *tsk) +{ +check_if_tm_restore_required (tsk); + +msr_check_and_set (MSR_FP );__giveup_fpu (tsk); +msr_check_and_clear (MSR_FP ); }EXPORT_SYMBOL (giveup_fpu);
@@ -144,30 +167,21 @@ void enable_kernel_fp(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_FP )) { -giveup_fpu (current); - }else { -u64 oldmsr =mfmsr (); +msr_check_and_set (MSR_FP ); -if (!(oldmsr &MSR_FP )) -mtmsr_isync (oldmsr |MSR_FP ); - } +if (current->thread.regs && (current->thread.regs->msr &MSR_FP )) +__giveup_fpu (current); }EXPORT_SYMBOL (enable_kernel_fp);#ifdef CONFIG_ALTIVEC void giveup_altivec (struct task_struct *tsk) { -u64 oldmsr =mfmsr (); -u64 newmsr; -check_if_tm_restore_required (tsk); - newmsr = oldmsr |MSR_VEC ; -if (oldmsr != newmsr) -mtmsr_isync (newmsr); - +msr_check_and_set (MSR_VEC );__giveup_altivec (tsk); +msr_check_and_clear (MSR_VEC ); }EXPORT_SYMBOL (giveup_altivec);
@@ -175,14 +189,10 @@ void enable_kernel_altivec(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_VEC )) { -giveup_altivec (current); - }else { -u64 oldmsr =mfmsr (); +msr_check_and_set (MSR_VEC ); -if (!(oldmsr &MSR_VEC )) -mtmsr_isync (oldmsr |MSR_VEC ); - } +if (current->thread.regs && (current->thread.regs->msr &MSR_VEC )) +__giveup_altivec (current); }EXPORT_SYMBOL (enable_kernel_altivec);
@@ -207,20 +217,15 @@ EXPORT_SYMBOL_GPL(flush_altivec_to_thread);#ifdef CONFIG_VSX void giveup_vsx (struct task_struct *tsk) { -u64 oldmsr =mfmsr (); -u64 newmsr; -check_if_tm_restore_required (tsk); - newmsr = oldmsr | (MSR_FP |MSR_VEC |MSR_VSX ); -if (oldmsr != newmsr) -mtmsr_isync (newmsr); - +msr_check_and_set (MSR_FP |MSR_VEC |MSR_VSX );if (tsk->thread.regs->msr &MSR_FP )__giveup_fpu (tsk);if (tsk->thread.regs->msr &MSR_VEC )__giveup_altivec (tsk);__giveup_vsx (tsk); +msr_check_and_clear (MSR_FP |MSR_VEC |MSR_VSX ); }EXPORT_SYMBOL (giveup_vsx);
@@ -228,13 +233,14 @@ void enable_kernel_vsx(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) { -giveup_vsx (current); - }else { -u64 oldmsr =mfmsr (); +msr_check_and_set (MSR_FP |MSR_VEC |MSR_VSX ); -if (!(oldmsr &MSR_VSX )) -mtmsr_isync (oldmsr |MSR_VSX ); +if (current->thread.regs && (current->thread.regs->msr &MSR_VSX )) { +if (current->thread.regs->msr &MSR_FP ) +__giveup_fpu (current); +if (current->thread.regs->msr &MSR_VEC ) +__giveup_altivec (current); +__giveup_vsx (current); } }EXPORT_SYMBOL (enable_kernel_vsx);
@@ -256,16 +262,11 @@ EXPORT_SYMBOL_GPL(flush_vsx_to_thread);#ifdef CONFIG_SPE void giveup_spe (struct task_struct *tsk) { -u64 oldmsr =mfmsr (); -u64 newmsr; -check_if_tm_restore_required (tsk); - newmsr = oldmsr |MSR_SPE ; -if (oldmsr != newmsr) -mtmsr_isync (newmsr); - +msr_check_and_set (MSR_SPE );__giveup_spe (tsk); +msr_check_and_clear (MSR_SPE ); }EXPORT_SYMBOL (giveup_spe);
@@ -273,14 +274,10 @@ void enable_kernel_spe(void) {WARN_ON (preemptible ()); -if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) { -giveup_spe (current); - }else { -u64 oldmsr =mfmsr (); +msr_check_and_set (MSR_SPE ); -if (!(oldmsr &MSR_SPE )) -mtmsr_isync (oldmsr |MSR_SPE ); - } +if (current->thread.regs && (current->thread.regs->msr &MSR_SPE )) +__giveup_spe (current); }EXPORT_SYMBOL (enable_kernel_spe);
Signature · 1 line
2.5.0
kbuild test robot
build botRe: [PATCH 12/19] powerpc: Create msr_check_and_{set,clear}()
reproduce: wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross # save the attached .config to linux build tree…
Hi Anton,
[auto build test ERROR on powerpc/next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]
url: https://github.com/0day-ci/linux/commits/Anton-Blanchard/Context-switch-improvements/20151029-092229 config: powerpc-virtex_defconfig (attached as .config)
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=powerpc
Note: the linux-review/Anton-Blanchard/Context-switch-improvements/20151029-092229 HEAD 13bdf12302e872951c7835777c51a7e04f784fb0 builds fine.
It only hurts bisectibility.
All errors (new ones prefixed by >>):
>> arch/powerpc/kernel/process.c:106:13: error: 'msr_check_and_clear' defined but not used [-Werror=unused-function]
static void msr_check_and_clear(unsigned long bits)
^
cc1: all warnings being treated as errors
vim +/msr_check_and_clear +106 arch/powerpc/kernel/process.c
100 #endif
101
102 if (oldmsr != newmsr)
103 mtmsr_isync(newmsr);
104 }
105
> 106 static void msr_check_and_clear(unsigned long bits)
107 {
108 unsigned long oldmsr = mfmsr();
109 unsigned long newmsr;
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH 13/19 powerpc: Create disable_kernel_{fp,altivec,vsx,spe}()
In reply to Anton BlanchardThe enable_kernel_*() functions leave the relevant MSR bits enabled until we exit the kernel sometime later. Create disable versions that wrap the kernel use of FP, Altivec VSX or SPE.
While we don't want to disable it normally for performance reasons (MSR writes are slow), it will be used for a debug boot option that does this and catches bad uses in other areas of the kernel.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/crypto/aes-spe-glue.c | 1 + arch/powerpc/crypto/sha1-spe-glue.c | 1 + arch/powerpc/crypto/sha256-spe-glue.c | 1 + arch/powerpc/include/asm/switch_to.h | 5 +++++ arch/powerpc/kernel/align.c | 2 ++ arch/powerpc/kvm/book3s_paired_singles.c | 1 + arch/powerpc/kvm/book3s_pr.c | 4 ++++ arch/powerpc/kvm/booke.c | 4 ++++ arch/powerpc/lib/vmx-helper.c | 2 ++ arch/powerpc/lib/xor_vmx.c | 4 ++++ drivers/crypto/vmx/aes.c | 3 +++ drivers/crypto/vmx/aes_cbc.c | 3 +++ drivers/crypto/vmx/aes_ctr.c | 3 +++ drivers/crypto/vmx/ghash.c | 4 ++++ lib/raid6/altivec.uc | 1 + 15 files changed, 39 insertions(+)
arch/powerpc/crypto/aes-spe-glue.c +1 -0
@@ -85,6 +85,7 @@ static void spe_begin(void)static void spe_end (void ) { +disable_kernel_spe ();/* reenable preemption */ preempt_enable (); }
arch/powerpc/crypto/sha1-spe-glue.c +1 -0
@@ -46,6 +46,7 @@ static void spe_begin(void)static void spe_end (void ) { +disable_kernel_spe ();/* reenable preemption */ preempt_enable (); }
arch/powerpc/crypto/sha256-spe-glue.c +1 -0
@@ -47,6 +47,7 @@ static void spe_begin(void)static void spe_end (void ) { +disable_kernel_spe ();/* reenable preemption */ preempt_enable (); }
arch/powerpc/include/asm/switch_to.h +5 -0
@@ -26,6 +26,11 @@ extern void enable_kernel_spe(void);extern void load_up_spe (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug); +static inline void disable_kernel_fp (void ) { } +static inline void disable_kernel_altivec (void ) { } +static inline void disable_kernel_spe (void ) { } +static inline void disable_kernel_vsx (void ) { } +#ifdef CONFIG_PPC_FPU extern void flush_fp_to_thread (struct task_struct *);extern void giveup_fpu (struct task_struct *);
arch/powerpc/kernel/align.c +2 -0
@@ -960,6 +960,7 @@ int fix_alignment(struct pt_regs *regs)preempt_disable ();enable_kernel_fp ();cvt_df (&data.dd, (float *)&data.x32.low32); +disable_kernel_fp ();preempt_enable ();#else return 0 ;
@@ -1000,6 +1001,7 @@ int fix_alignment(struct pt_regs *regs)preempt_disable ();enable_kernel_fp ();cvt_fd ((float *)&data.x32.low32, &data.dd); +disable_kernel_fp ();preempt_enable ();#else return 0 ;
arch/powerpc/kvm/book3s_paired_singles.c +1 -0
@@ -1265,6 +1265,7 @@ int kvmppc_emulate_paired_single(struct kvm_run *run, struct kvm_vcpu *vcpu)if (rcomp)kvmppc_set_cr (vcpu, cr); +disable_kernel_fp ();preempt_enable ();return emulated;
arch/powerpc/kvm/book3s_pr.c +4 -0
@@ -751,6 +751,7 @@ static int kvmppc_handle_ext(struct kvm_vcpu *vcpu, unsigned int exit_nr,preempt_disable ();enable_kernel_fp ();load_fp_state (&vcpu->arch.fp); +disable_kernel_fp (); t->fp_save_area = &vcpu->arch.fp;preempt_enable (); }
@@ -760,6 +761,7 @@ static int kvmppc_handle_ext(struct kvm_vcpu *vcpu, unsigned int exit_nr,preempt_disable ();enable_kernel_altivec ();load_vr_state (&vcpu->arch.vr); +disable_kernel_altivec (); t->vr_save_area = &vcpu->arch.vr;preempt_enable ();#endif
@@ -788,6 +790,7 @@ static void kvmppc_handle_lost_ext(struct kvm_vcpu *vcpu)preempt_disable ();enable_kernel_fp ();load_fp_state (&vcpu->arch.fp); +disable_kernel_fp ();preempt_enable (); }#ifdef CONFIG_ALTIVEC
@@ -795,6 +798,7 @@ static void kvmppc_handle_lost_ext(struct kvm_vcpu *vcpu)preempt_disable ();enable_kernel_altivec ();load_vr_state (&vcpu->arch.vr); +disable_kernel_altivec (); preempt_enable(); }#endif
arch/powerpc/kvm/booke.c +4 -0
@@ -98,6 +98,7 @@ void kvmppc_vcpu_disable_spe(struct kvm_vcpu *vcpu)preempt_disable ();enable_kernel_spe ();kvmppc_save_guest_spe (vcpu); +disable_kernel_spe (); vcpu->arch.shadow_msr &= ~MSR_SPE ;preempt_enable (); }
@@ -107,6 +108,7 @@ static void kvmppc_vcpu_enable_spe(struct kvm_vcpu *vcpu)preempt_disable ();enable_kernel_spe ();kvmppc_load_guest_spe (vcpu); +disable_kernel_spe (); vcpu->arch.shadow_msr |=MSR_SPE ;preempt_enable (); }
@@ -141,6 +143,7 @@ static inline void kvmppc_load_guest_fp(struct kvm_vcpu *vcpu)if (!(current->thread.regs->msr &MSR_FP )) {enable_kernel_fp ();load_fp_state (&vcpu->arch.fp); +disable_kernel_fp (); current->thread.fp_save_area = &vcpu->arch.fp; current->thread.regs->msr |=MSR_FP ; }
@@ -182,6 +185,7 @@ static inline void kvmppc_load_guest_altivec(struct kvm_vcpu *vcpu)if (!(current->thread.regs->msr &MSR_VEC )) {enable_kernel_altivec ();load_vr_state (&vcpu->arch.vr); +disable_kernel_altivec (); current->thread.vr_save_area = &vcpu->arch.vr; current->thread.regs->msr |=MSR_VEC ; }
arch/powerpc/lib/vmx-helper.c +2 -0
@@ -46,6 +46,7 @@ int enter_vmx_usercopy(void)*/ int exit_vmx_usercopy (void ) { +disable_kernel_altivec ();pagefault_enable ();preempt_enable ();return 0 ;
@@ -70,6 +71,7 @@ int enter_vmx_copy(void)*/ void *exit_vmx_copy (void *dest) { +disable_kernel_altivec ();preempt_enable ();return dest; }
arch/powerpc/lib/xor_vmx.c +4 -0
@@ -74,6 +74,7 @@ void xor_altivec_2(unsigned long bytes, unsigned long *v1_in, v2 +=4 ; }while (--lines >0 ); +disable_kernel_altivec ();preempt_enable (); }EXPORT_SYMBOL (xor_altivec_2);
@@ -102,6 +103,7 @@ void xor_altivec_3(unsigned long bytes, unsigned long *v1_in, v3 +=4 ; }while (--lines >0 ); +disable_kernel_altivec ();preempt_enable (); }EXPORT_SYMBOL (xor_altivec_3);
@@ -135,6 +137,7 @@ void xor_altivec_4(unsigned long bytes, unsigned long *v1_in, v4 +=4 ; }while (--lines >0 ); +disable_kernel_altivec ();preempt_enable (); }EXPORT_SYMBOL (xor_altivec_4);
@@ -172,6 +175,7 @@ void xor_altivec_5(unsigned long bytes, unsigned long *v1_in, v5 +=4 ; }while (--lines >0 ); +disable_kernel_altivec ();preempt_enable (); }EXPORT_SYMBOL (xor_altivec_5);
drivers/crypto/vmx/aes.c +3 -0
@@ -86,6 +86,7 @@ static int p8_aes_setkey(struct crypto_tfm *tfm, const u8 *key,enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key); ret +=aes_p8_set_decrypt_key (key, keylen *8 , &ctx->dec_key); +disable_kernel_vsx ();pagefault_enable ();preempt_enable ();
@@ -104,6 +105,7 @@ static void p8_aes_encrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)pagefault_disable ();enable_kernel_vsx ();aes_p8_encrypt (src, dst, &ctx->enc_key); +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); }
@@ -120,6 +122,7 @@ static void p8_aes_decrypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)pagefault_disable ();enable_kernel_vsx ();aes_p8_decrypt (src, dst, &ctx->dec_key); +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); }
drivers/crypto/vmx/aes_cbc.c +3 -0
@@ -87,6 +87,7 @@ static int p8_aes_cbc_setkey(struct crypto_tfm *tfm, const u8 *key,enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key); ret +=aes_p8_set_decrypt_key (key, keylen *8 , &ctx->dec_key); +disable_kernel_vsx ();pagefault_enable ();preempt_enable ();
@@ -127,6 +128,7 @@ static int p8_aes_cbc_encrypt(struct blkcipher_desc *desc, ret =blkcipher_walk_done (desc, &walk, nbytes); } +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); }
@@ -167,6 +169,7 @@ static int p8_aes_cbc_decrypt(struct blkcipher_desc *desc, ret =blkcipher_walk_done (desc, &walk, nbytes); } +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); }
drivers/crypto/vmx/aes_ctr.c +3 -0
@@ -83,6 +83,7 @@ static int p8_aes_ctr_setkey(struct crypto_tfm *tfm, const u8 *key,pagefault_disable ();enable_kernel_vsx (); ret =aes_p8_set_encrypt_key (key, keylen *8 , &ctx->enc_key); +disable_kernel_vsx ();pagefault_enable (); ret +=crypto_blkcipher_setkey (ctx->fallback, key, keylen);
@@ -101,6 +102,7 @@ static void p8_aes_ctr_final(struct p8_aes_ctr_ctx *ctx,pagefault_disable ();enable_kernel_vsx ();aes_p8_encrypt (ctrblk, keystream, &ctx->enc_key); +disable_kernel_vsx ();pagefault_enable ();crypto_xor (keystream, src, nbytes);
@@ -139,6 +141,7 @@ static int p8_aes_ctr_crypt(struct blkcipher_desc *desc,AES_BLOCK_SIZE , &ctx->enc_key, walk.iv); +disable_kernel_vsx ();pagefault_enable ();/* We need to update IV mostly for last bytes/round */
drivers/crypto/vmx/ghash.c +4 -0
@@ -120,6 +120,7 @@ static int p8_ghash_setkey(struct crypto_shash *tfm, const u8 *key,pagefault_disable ();enable_kernel_vsx ();gcm_init_p8 (ctx->htable, (const u64 *) key); +disable_kernel_vsx ();pagefault_enable ();preempt_enable ();return crypto_shash_setkey (ctx->fallback, key, keylen);
@@ -150,6 +151,7 @@ static int p8_ghash_update(struct shash_desc *desc,enable_kernel_vsx ();gcm_ghash_p8 (dctx->shash, ctx->htable, dctx->buffer,GHASH_DIGEST_SIZE ); +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); src +=GHASH_DIGEST_SIZE - dctx->bytes;
@@ -162,6 +164,7 @@ static int p8_ghash_update(struct shash_desc *desc,pagefault_disable ();enable_kernel_vsx ();gcm_ghash_p8 (dctx->shash, ctx->htable, src, len); +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); src += len;
@@ -192,6 +195,7 @@ static int p8_ghash_final(struct shash_desc *desc, u8 *out)enable_kernel_vsx ();gcm_ghash_p8 (dctx->shash, ctx->htable, dctx->buffer,GHASH_DIGEST_SIZE ); +disable_kernel_vsx ();pagefault_enable ();preempt_enable (); dctx->bytes =0 ;
lib/raid6/altivec.uc +1 -0
@@ -101,6 +101,7 @@ static void raid6_altivec$#_gen_syndrome(int disks, size_t bytes, void **ptrs) raid6_altivec$#_gen_syndrome_real(disks, bytes, ptrs); + disable_kernel_altivec(); preempt_enable(); }
Signature · 1 line
2.5.0
PATCH 14/19 powerpc: Add ppc_strict_facility_enable boot option
In reply to Anton BlanchardAdd a boot option that strictly manages the MSR unavailable bits. This catches kernel uses of FP/Altivec/SPE that would otherwise corrupt user state.
Signed-off-by Anton Blanchard <[email protected]>
--- Documentation/kernel-parameters.txt | 6 ++++++ arch/powerpc/include/asm/reg.h | 9 +++++++++ arch/powerpc/include/asm/switch_to.h | 24 +++++++++++++++++++----- arch/powerpc/kernel/process.c | 17 +++++++++++++++-- 4 files changed, 49 insertions(+), 7 deletions(-)
Documentation/kernel-parameters.txt +6 -0
@@ -2939,6 +2939,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted. may be specified. Format: <port>,<port>.... + ppc_strict_facility_enable + [PPC] This option catches any kernel floating point, + Altivec, VSX and SPE outside of regions specifically + allowed (eg kernel_enable_fpu()/kernel_disable_fpu()). + There is some performance impact when enabling this. + print-fatal-signals= [KNL] debug: print fatal signals
arch/powerpc/include/asm/reg.h +9 -0
@@ -1214,6 +1214,15 @@ static inline void mtmsr_isync(unsigned long val) : "r" ((unsigned long)(v)) \ : "memory") +extern void msr_check_and_set (unsigned long bits); +extern bool strict_msr_control; +extern void __msr_check_and_clear (unsigned long bits); +static inline void msr_check_and_clear (unsigned long bits) +{ +if (strict_msr_control) +__msr_check_and_clear (bits); +} +static inline unsigned long mfvtb (void ) {#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/include/asm/switch_to.h +19 -5
@@ -4,6 +4,8 @@#ifndef _ASM_POWERPC_SWITCH_TO_H#define _ASM_POWERPC_SWITCH_TO_H +#include <asm/reg.h> +struct thread_struct ;struct task_struct ;struct pt_regs ;
@@ -26,15 +28,15 @@ extern void enable_kernel_spe(void);extern void load_up_spe (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug); -static inline void disable_kernel_fp (void ) { } -static inline void disable_kernel_altivec (void ) { } -static inline void disable_kernel_spe (void ) { } -static inline void disable_kernel_vsx (void ) { } -#ifdef CONFIG_PPC_FPU extern void flush_fp_to_thread (struct task_struct *);extern void giveup_fpu (struct task_struct *);extern void __giveup_fpu (struct task_struct *); +static inline void disable_kernel_fp (void ) +{ +msr_check_and_clear (MSR_FP ); +} +#else static inline void flush_fp_to_thread (struct task_struct *t) { }static inline void giveup_fpu (struct task_struct *t) { }
@@ -45,6 +47,10 @@ static inline void __giveup_fpu(struct task_struct *t) { }extern void flush_altivec_to_thread (struct task_struct *);extern void giveup_altivec (struct task_struct *);extern void __giveup_altivec (struct task_struct *); +static inline void disable_kernel_altivec (void ) +{ +msr_check_and_clear (MSR_VEC ); +}#else static inline void flush_altivec_to_thread (struct task_struct *t) { }static inline void giveup_altivec (struct task_struct *t) { }
@@ -53,6 +59,10 @@ static inline void __giveup_altivec(struct task_struct *t) { }#ifdef CONFIG_VSX extern void flush_vsx_to_thread (struct task_struct *); +static inline void disable_kernel_vsx (void ) +{ +msr_check_and_clear (MSR_FP |MSR_VEC |MSR_VSX ); +}#else static inline void flush_vsx_to_thread (struct task_struct *t) {
@@ -63,6 +73,10 @@ static inline void flush_vsx_to_thread(struct task_struct *t)extern void flush_spe_to_thread (struct task_struct *);extern void giveup_spe (struct task_struct *);extern void __giveup_spe (struct task_struct *); +static inline void disable_kernel_spe (void ) +{ +msr_check_and_clear (MSR_SPE ); +}#else static inline void flush_spe_to_thread (struct task_struct *t) { }static inline void giveup_spe (struct task_struct *t) { }
arch/powerpc/kernel/process.c +15 -2
@@ -87,7 +87,19 @@ static void check_if_tm_restore_required(struct task_struct *tsk)static inline void check_if_tm_restore_required (struct task_struct *tsk) { }#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ -static void msr_check_and_set (unsigned long bits) +bool strict_msr_control; +EXPORT_SYMBOL (strict_msr_control); + +static int __init enable_strict_msr_control (char *str) +{ + strict_msr_control = true; +pr_info ("Enabling strict facility control\n" ); + +return 0 ; +} +early_param ("ppc_strict_facility_enable" , enable_strict_msr_control); + +void msr_check_and_set (unsigned long bits) {unsigned long oldmsr =mfmsr ();unsigned long newmsr;
@@ -103,7 +115,7 @@ static void msr_check_and_set(unsigned long bits)mtmsr_isync (newmsr); } -static void msr_check_and_clear (unsigned long bits) +void __msr_check_and_clear (unsigned long bits) {unsigned long oldmsr =mfmsr ();unsigned long newmsr;
@@ -118,6 +130,7 @@ static void msr_check_and_clear(unsigned long bits)if (oldmsr != newmsr)mtmsr_isync (newmsr); } +EXPORT_SYMBOL (__msr_check_and_clear);#ifdef CONFIG_PPC_FPU void giveup_fpu (struct task_struct *tsk)
Signature · 1 line
2.5.0
Re: [PATCH 14/19] powerpc: Add ppc_strict_facility_enable boot option
arch/powerpc/include/asm/reg.h static inline void mtmsr_isync(unsigned long val) · line 1226
Quoted 11 lines · Anton Blanchard
arch/powerpc/include/asm/reg.h +9
-0+extern void msr_check_and_set (unsigned long bits); +extern bool strict_msr_control; +extern void __msr_check_and_clear (unsigned long bits); +static inline void msr_check_and_clear (unsigned long bits) +{ +if (strict_msr_control) +__msr_check_and_clear (bits); +} +static inline unsigned long mfvtb (void )
Sounds like a good candidate for a static key...
Cheers,
Ben.
PATCH 15/19 powerpc: Remove fp_enable() and vec_enable(), use msr_check_and_{set, clear}()
In reply to Anton BlanchardMore consolidation of our MSR available bit handling.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/processor.h | 2 -- arch/powerpc/kernel/fpu.S | 16 ---------------- arch/powerpc/kernel/process.c | 6 ++++-- arch/powerpc/kernel/vector.S | 10 ---------- 4 files changed, 4 insertions(+), 30 deletions(-)
arch/powerpc/include/asm/processor.h +0 -2
@@ -380,8 +380,6 @@ extern int set_endian(struct task_struct *tsk, unsigned int val);extern int get_unalign_ctl (struct task_struct *tsk,unsigned long adr);extern int set_unalign_ctl (struct task_struct *tsk,unsigned int val); -extern void fp_enable (void ); -extern void vec_enable (void );extern void load_fp_state (struct thread_fp_state *fp);extern void store_fp_state (struct thread_fp_state *fp);extern void load_vr_state (struct thread_vr_state *vr);
arch/powerpc/kernel/fpu.S +0 -16
@@ -77,22 +77,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_VSX) #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ /* - * Enable use of the FPU, and VSX if possible, for the caller. - */ -_GLOBAL(fp_enable) - mfmsr r3 - ori r3,r3,MSR_FP -#ifdef CONFIG_VSX -BEGIN_FTR_SECTION - oris r3,r3,MSR_VSX@h -END_FTR_SECTION_IFSET(CPU_FTR_VSX) -#endif - SYNC - MTMSRD(r3) - isync /* (not necessary for arch 2.02 and later) */ - blr - -/* * Load state from memory into FP registers including FPSCR. * Assumes the caller has enabled FP in the MSR. */
arch/powerpc/kernel/process.c +4 -2
@@ -732,13 +732,15 @@ void restore_tm_state(struct pt_regs *regs) msr_diff = current->thread.ckpt_regs.msr & ~regs->msr; msr_diff &=MSR_FP |MSR_VEC |MSR_VSX ;if (msr_diff &MSR_FP ) { -fp_enable (); +msr_check_and_set (MSR_FP );load_fp_state (¤t->thread.fp_state); +msr_check_and_clear (MSR_FP ); regs->msr |= current->thread.fpexc_mode; }if (msr_diff &MSR_VEC ) { -vec_enable (); +msr_check_and_set (MSR_VEC );load_vr_state (¤t->thread.vr_state); +msr_check_and_clear (MSR_VEC ); } regs->msr |= msr_diff; }
arch/powerpc/kernel/vector.S +0 -10
@@ -33,16 +33,6 @@ _GLOBAL(do_load_up_transact_altivec) #endif /* - * Enable use of VMX/Altivec for the caller. - */ -_GLOBAL(vec_enable) - mfmsr r3 - oris r3,r3,MSR_VEC@h - MTMSRD(r3) - isync - blr - -/* * Load state from memory into VMX registers including VSCR. * Assumes the caller has enabled VMX in the MSR. */
Signature · 1 line
2.5.0
PATCH 16/19 powerpc: create giveup_all()
In reply to Anton BlanchardCreate a single function that gives everything up (FP, VMX, VSX, SPE). Doing this all at once means we only do one MSR write.
A context switch microbenchmark using yield():
http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield --fp --altivec --vector 0 0
shows an improvement of 3% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/switch_to.h | 1 + arch/powerpc/kernel/process.c | 75 ++++++++++++++++++++++++++++-------- arch/powerpc/kvm/book3s_pr.c | 17 +------- 3 files changed, 63 insertions(+), 30 deletions(-)
arch/powerpc/include/asm/switch_to.h +1 -0
@@ -26,6 +26,7 @@ extern void __giveup_vsx(struct task_struct *);extern void giveup_vsx (struct task_struct *);extern void enable_kernel_spe (void );extern void load_up_spe (struct task_struct *); +extern void giveup_all (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug);#ifdef CONFIG_PPC_FPU
arch/powerpc/kernel/process.c +60 -15
87 lines not on this page · raw message
arch/powerpc/kvm/book3s_pr.c +2 -15
@@ -1490,21 +1490,8 @@ static int kvmppc_vcpu_run_pr(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu) goto out;/* interrupts now hard-disabled */ -/* Save FPU state in thread_struct */ -if (current->thread.regs->msr &MSR_FP ) -giveup_fpu (current); - -#ifdef CONFIG_ALTIVEC -/* Save Altivec state in thread_struct */ -if (current->thread.regs->msr &MSR_VEC ) -giveup_altivec (current); -#endif - -#ifdef CONFIG_VSX -/* Save VSX state in thread_struct */ -if (current->thread.regs->msr &MSR_VSX ) -__giveup_vsx (current); -#endif +/* Save FPU, Altivec and VSX state */ +giveup_all (current);/* Preload FPU if it's enabled */ if (kvmppc_get_msr (vcpu) &MSR_FP )
Signature · 1 line
2.5.0
Michael Ellerman
reviewerRe: [PATCH 16/19] powerpc: create giveup_all()
Quoted 9 lines · Anton Blanchard
Create a single function that gives everything up (FP, VMX, VSX, SPE). Doing this all at once means we only do one MSR write.
A context switch microbenchmark using
yield():http://ozlabs.org/~anton/junkcode/context_switch2.c
./context_switch2 --test=yield --fp --altivec --vector 0 0
Quoted 7 lines
shows an improvement of 3% on POWER8.
Signed-off-by Anton Blanchard <[email protected]>--- arch/powerpc/include/asm/switch_to.h | 1 + arch/powerpc/kernel/process.c | 75 ++++++++++++++++++++++++++++-------- arch/powerpc/kvm/book3s_pr.c | 17 +-------
books3s_pr.c can be a module, so we need giveup_all() to be EXPORT_SYMBOL.
I'll fix it up on my end.
cheers
PATCH 17/19 powerpc: create flush_all_to_thread()
In reply to Anton BlanchardCreate a single function that flushes everything (FP, VMX, VSX, SPE). Doing this all at once means we only do one MSR write.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/switch_to.h | 1 + arch/powerpc/kernel/process.c | 22 ++++++++++++++++++---- arch/powerpc/kernel/swsusp.c | 4 +--- arch/powerpc/kvm/book3s_hv.c | 5 ++--- 4 files changed, 22 insertions(+), 10 deletions(-)
arch/powerpc/include/asm/switch_to.h +1 -0
@@ -27,6 +27,7 @@ extern void giveup_vsx(struct task_struct *);extern void enable_kernel_spe (void );extern void load_up_spe (struct task_struct *);extern void giveup_all (struct task_struct *); +extern void flush_all_to_thread (struct task_struct *);extern void switch_booke_debug_regs (struct debug_reg *new_debug);#ifdef CONFIG_PPC_FPU
arch/powerpc/kernel/process.c +18 -4
@@ -366,6 +366,23 @@ void giveup_all(struct task_struct *tsk)msr_check_and_clear (msr_all_available); } +void flush_all_to_thread (struct task_struct *tsk) +{ +if (tsk->thread.regs) { +preempt_disable (); +BUG_ON (tsk != current); +giveup_all (tsk); + +#ifdef CONFIG_SPE +if (tsk->thread.regs->msr &MSR_SPE ) + tsk->thread.spefscr =mfspr (SPRN_SPEFSCR ); +#endif + +preempt_enable (); + } +} +EXPORT_SYMBOL (flush_all_to_thread); +#ifdef CONFIG_PPC_ADV_DEBUG_REGS void do_send_trap (struct pt_regs *regs,unsigned long address,unsigned long error_code,int signal_code,int breakpt)
@@ -1136,10 +1153,7 @@ release_thread(struct task_struct *t)*/ int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src) { -flush_fp_to_thread (src); -flush_altivec_to_thread (src); -flush_vsx_to_thread (src); -flush_spe_to_thread (src); +flush_all_to_thread (src);/* * Flush TM state out so we can copy it. __switch_to_tm() does this * flush but it removes the checkpointed state from the current CPU and
arch/powerpc/kernel/swsusp.c +1 -3
@@ -20,9 +20,7 @@ void save_processor_state(void)* flush out all the special registers so we don't need * to save them in the snapshot */ -flush_fp_to_thread (current); -flush_altivec_to_thread (current); -flush_spe_to_thread (current); +flush_all_to_thread (current);#ifdef CONFIG_PPC64 hard_irq_disable ();
arch/powerpc/kvm/book3s_hv.c +2 -3
@@ -2700,9 +2700,8 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu) goto out; } -flush_fp_to_thread (current); -flush_altivec_to_thread (current); -flush_vsx_to_thread (current); +flush_all_to_thread (current); + vcpu->arch.wqp = &vcpu->arch.vcore->wq; vcpu->arch.pgdir = current->mm->pgd; vcpu->arch.state =KVMPPC_VCPU_BUSY_IN_HOST ;
Signature · 1 line
2.5.0
PATCH 18/19 powerpc: Rearrange __switch_to()
In reply to Anton BlanchardMost of __switch_to() is housekeeping, TLB batching, timekeeping etc. Move these away from the more complex and critical context switching code.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/kernel/process.c | 52 +++++++++++++++++++++---------------------- 1 file changed, 26 insertions(+), 26 deletions(-)
arch/powerpc/kernel/process.c +26 -26
@@ -906,30 +906,6 @@ struct task_struct *__switch_to(struct task_struct *prev,WARN_ON (!irqs_disabled ()); -/* -* We need to save SPRs before treclaim/trecheckpoint as these will -* change a number of them. -*/ -save_sprs (&prev->thread); - -__switch_to_tm (prev); - -/* Save FPU, Altivec, VSX and SPE state */ -giveup_all (prev); - -#ifdef CONFIG_PPC_ADV_DEBUG_REGS -switch_booke_debug_regs (&new->thread.debug); -#else -/* -* For PPC_BOOK3S_64, we use the hw-breakpoint interfaces that would -* schedule DABR -*/ -#ifndef CONFIG_HAVE_HW_BREAKPOINT -if (unlikely (!hw_brk_match (this_cpu_ptr (¤t_brk), &new->thread.hw_brk))) -__set_breakpoint (&new->thread.hw_brk); -#endif /* CONFIG_HAVE_HW_BREAKPOINT */ -#endif -#ifdef CONFIG_PPC64 /* * Collect processor utilization data per process
@@ -954,6 +930,30 @@ struct task_struct *__switch_to(struct task_struct *prev, }#endif /* CONFIG_PPC_BOOK3S_64 */ +#ifdef CONFIG_PPC_ADV_DEBUG_REGS +switch_booke_debug_regs (&new->thread.debug); +#else +/* +* For PPC_BOOK3S_64, we use the hw-breakpoint interfaces that would +* schedule DABR +*/ +#ifndef CONFIG_HAVE_HW_BREAKPOINT +if (unlikely (!hw_brk_match (this_cpu_ptr (¤t_brk), &new->thread.hw_brk))) +__set_breakpoint (&new->thread.hw_brk); +#endif /* CONFIG_HAVE_HW_BREAKPOINT */ +#endif + +/* +* We need to save SPRs before treclaim/trecheckpoint as these will +* change a number of them. +*/ +save_sprs (&prev->thread); + +__switch_to_tm (prev); + +/* Save FPU, Altivec, VSX and SPE state */ +giveup_all (prev); +/* * We can't take a PMU exception inside _switch() since there is a * window where the kernel stack SLB and the kernel stack are out
@@ -969,6 +969,8 @@ struct task_struct *__switch_to(struct task_struct *prev, old_thread = &last->thread; new_thread = ¤t->thread; +restore_sprs (old_thread, new_thread); +#ifdef CONFIG_PPC_BOOK3S_64 if (current_thread_info ()->local_flags & _TLF_LAZY_MMU) {current_thread_info ()->local_flags &= ~_TLF_LAZY_MMU;
@@ -977,8 +979,6 @@ struct task_struct *__switch_to(struct task_struct *prev, }#endif /* CONFIG_PPC_BOOK3S_64 */ -restore_sprs (old_thread, new_thread); -return last; }
Signature · 1 line
2.5.0
PATCH 19/19 powerpc: clean up asm/switch_to.h
In reply to Anton BlanchardRemove a bunch of unnecessary fallback functions and group things in a more logical way.
Signed-off-by Anton Blanchard <[email protected]>
--- arch/powerpc/include/asm/switch_to.h | 35 ++++++++++------------------------- arch/powerpc/kernel/process.c | 2 +- 2 files changed, 11 insertions(+), 26 deletions(-)
arch/powerpc/include/asm/switch_to.h +10 -25
@@ -14,23 +14,18 @@ extern struct task_struct *__switch_to(struct task_struct *, struct task_struct *);#define switch_to (prev, next, last) ((last) = __switch_to((prev), (next))) -struct thread_struct ;extern struct task_struct *_switch (struct thread_struct *prev,struct thread_struct *next); -extern void enable_kernel_fp (void ); -extern void enable_kernel_altivec (void ); -extern void enable_kernel_vsx (void ); +extern void switch_booke_debug_regs (struct debug_reg *new_debug); +extern int emulate_altivec (struct pt_regs *); -extern void __giveup_vsx (struct task_struct *); -extern void giveup_vsx (struct task_struct *); -extern void enable_kernel_spe (void ); -extern void load_up_spe (struct task_struct *); -extern void giveup_all (struct task_struct *); +extern void flush_all_to_thread (struct task_struct *); -extern void switch_booke_debug_regs (struct debug_reg *new_debug); +extern void giveup_all (struct task_struct *);#ifdef CONFIG_PPC_FPU +extern void enable_kernel_fp (void );extern void flush_fp_to_thread (struct task_struct *);extern void giveup_fpu (struct task_struct *);extern void __giveup_fpu (struct task_struct *);
@@ -38,14 +33,12 @@ static inline void disable_kernel_fp(void) {msr_check_and_clear (MSR_FP ); } -#else static inline void flush_fp_to_thread (struct task_struct *t) { } -static inline void giveup_fpu (struct task_struct *t) { } -static inline void __giveup_fpu (struct task_struct *t) { }#endif #ifdef CONFIG_ALTIVEC +extern void enable_kernel_altivec (void );extern void flush_altivec_to_thread (struct task_struct *);extern void giveup_altivec (struct task_struct *);extern void __giveup_altivec (struct task_struct *);
@@ -53,25 +46,21 @@ static inline void disable_kernel_altivec(void) {msr_check_and_clear (MSR_VEC ); } -#else -static inline void flush_altivec_to_thread (struct task_struct *t) { } -static inline void giveup_altivec (struct task_struct *t) { } -static inline void __giveup_altivec (struct task_struct *t) { }#endif #ifdef CONFIG_VSX +extern void enable_kernel_vsx (void );extern void flush_vsx_to_thread (struct task_struct *); +extern void giveup_vsx (struct task_struct *); +extern void __giveup_vsx (struct task_struct *);static inline void disable_kernel_vsx (void ) {msr_check_and_clear (MSR_FP |MSR_VEC |MSR_VSX ); } -#else -static inline void flush_vsx_to_thread (struct task_struct *t) -{ -}#endif #ifdef CONFIG_SPE +extern void enable_kernel_spe (void );extern void flush_spe_to_thread (struct task_struct *);extern void giveup_spe (struct task_struct *);extern void __giveup_spe (struct task_struct *);
@@ -79,10 +68,6 @@ static inline void disable_kernel_spe(void) {msr_check_and_clear (MSR_SPE ); } -#else -static inline void flush_spe_to_thread (struct task_struct *t) { } -static inline void giveup_spe (struct task_struct *t) { } -static inline void __giveup_spe (struct task_struct *t) { }#endif static inline void clear_task_ebb (struct task_struct *t)
arch/powerpc/kernel/process.c +1 -1
@@ -174,7 +174,6 @@ void flush_fp_to_thread(struct task_struct *tsk) } }EXPORT_SYMBOL_GPL (flush_fp_to_thread); -#endif /* CONFIG_PPC_FPU */ void enable_kernel_fp (void ) {
@@ -186,6 +185,7 @@ void enable_kernel_fp(void) __giveup_fpu(current); }EXPORT_SYMBOL (enable_kernel_fp); +#endif /* CONFIG_PPC_FPU */ #ifdef CONFIG_ALTIVEC void giveup_altivec (struct task_struct *tsk)
Signature · 1 line
2.5.0
Quoted 13 lines · Anton Blanchard
Here are various improvements to our context switch path. Some of the highlights:
- Group all mfsprs and mtsprs in
__switch_to(), which gives us a 10% improvement on POWER8.- Create
giveup_all()andflush_all_to_thread()so we only write the MSR once, which gives us a 3% improvement on POWER8.- Create disable_kernel_{fp,altivec,vsx,spe}() and add a debug boot option (
ppc_strict_facility_enable) to minimise kernel code running with floating point and vector bits enabled.
Quoted 5 lines
- Make
giveup_vsx()andflush_vsx_to_thread()handle FP and Altivec state, so they behave more like their sister functions.Scott: There are changes to the SPE code here which I have only been able to compile test.
Hey Anton,
Wondering if you have a git tree hosted some where to pull these patches ?
-- Anshuman
Quoted 23 lines · Anshuman Khandual
Quoted 14 lines · Anton Blanchard
Here are various improvements to our context switch path. Some of the
highlights:
=20
- Group all mfsprs and mtsprs in__switch_to(), which gives us a 10% improvement on POWER8.
=20
- Creategiveup_all()andflush_all_to_thread()so we only write the
MSR once, which gives us a 3% improvement on POWER8.
=20
- Create disable_kernel_{fp,altivec,vsx,spe}() and add a debug boot option (ppc_strict_facility_enable) to minimise kernel code
runningQuoted 8 lines
with floating point and vector bits enabled.
=20
- Makegiveup_vsx()andflush_vsx_to_thread()handle FP and Altivec state, so they behave more like their sister functions.
=20
Scott: There are changes to the SPE code here which I have only been
able to compile test.
Quoted 5 lines
=20
Hey Anton,
=20
Wondering if you have a git tree hosted some where to pull these patches ?
The 0day bot does this automatically for you...
https://github.com/0day-ci/linux/tree/Anton-Blanchard/Context-switch-improv=
ements/20151029-092229
Mikey