Skip to contentSkip to conversation

powerpc: Implement lazy save of FP, VMX and VSX state in SMP

PATCH New Michael Neuling 1 participant 8 messages Dec 6, 2010 PowerPC

lore
Share & tools
lore
#0 lore raw

cover powerpc: Implement lazy save of FP, VMX and VSX state in SMP

This implements lazy save of FP, VMX and VSX state on SMP 64bit and 32 bit powerpc.

Currently we only do lazy save in UP, but this patch set extends this to SMP. We always do lazy restore.

For VMX, on a context switch we do the following:
- if we are switching to a CPU that currently holds the new processes state, just turn on VMX in the MSR (this is the lazy/quick case)
- if the new processes state is in the thread_struct, turn VMX off.
- if the new processes state is in someone else's CPU, IPI that CPU to giveup it's state and turn VMX off in the MSR (slow IPI case).
We always start the new process at this point, irrespective of if we have the state or not in the thread struct or current CPU.

So in the slow case, we attempt to avoid the IPI latency by starting the process immediately and only waiting for the state to be flushed when the process actually needs VMX. ie. when we take the VMX unavailable exception after the context switch.

FP is implemented in a similar way. VSX reuses the FP and VMX code as it doesn't have any additional state over what FP and VMX used.

I've been benchmarking with Anton Blanchard's context_switch.c benchmark found here:
http://ozlabs.org/~anton/junkcode/context_switch.c
Using this benchmark as is gives no degradation in performance with these patches applied.

Inserting a simple FP instruction into one of the threads (gives the nice save/restore lazy case), I get about a 4% improvement in context switching rates with my patches applied. I get similar results VMX. With a simple VSX instruction (VSX state is 64x128bit registers) in 1 thread I get an 8% bump in performance with these patches.

With FP/VMX/VSX instructions in both threads, I get no degradation in performance.

Running lmbench doesn't have any degradation in performance.

Most of my benchmarking and testing has been done on 64 bit systems. I've tested 32 bit FP but I've not tested 32 bit VMX at all.

There is probably some optimisations to my asm code that can also be made. I've been concentrating on correctness, as opposed to speed with the asm code, since if you get a lazy context switch, you skip all the asm now anyway.

Whole series is bisectable to compile with various 64/32bit SMP/UP FPU/VMX/VSX config options on and off.

I really hate the include file changes in this series. Getting the call_single_data in the powerpc threads_struct was a PITA :-)

Mikey

#1 lore raw

PATCH 1/7 Add csd_locked function

Add csd_locked function to determine if a struct call_single_data is currently locked. This can be used to see if an IPI can be called again using this call_single_data.

---
 kernel/smp.c |   17 +++++++++++++++++
 1 file changed, 17 insertions(+)
kernel/smp.c +17 -0
Index: linux-lazy/kernel/smp.c
===================================================================
@@ -12,6 +12,7 @@
 #include <linux/gfp.h>
 #include <linux/smp.h>
 #include <linux/cpu.h>
+#include <linux/hardirq.h>
 
 static struct {
 	struct list_head	queue;
@@ -131,6 +132,22 @@
 }
 
 /*
+ * Determine if a csd is currently locked.  This can be used to
+ * determine if an IPI is currently pending using this csd already.
+ */
+int csd_locked(struct call_single_data *data)
+{
+	WARN_ON(preemptible());
+
+	/* Ensure flags have propagated */
+	smp_mb();
+
+	if (data->flags & CSD_FLAG_LOCK)
+		return 1;
+	return 0;
+}
+
+/*
  * Insert a previously allocated call_single_data element
  * for execution on the given CPU. data must already have
  * ->func, ->info, and ->flags set.
#2 lore raw

PATCH 2/7 Rearrange include files to make struct call_single_data usable in more places

In reply to Michael Neuling

We need to put struct call_single_data in the powerpc thread_struct, but can't without this.

The thread_struct is in processor.h. To add a struct call_single_data to the thread_struct asm/processor.h must include linux/smp.h. When linux/smp.h is added to processor.h this creates an include loop via with list.h via:

linux/list.h includes:
linux/prefetch.h includes:
asm/processor.h (for powerpc) includes:
linux/smp.h includes:
linux/list.h

This loops results in an "incomplete list type" compile when using struct list_head as used in struct call_single_data.

This patch rearanges some include files to avoid this loop.

---
 include/linux/call_single_data.h |   14 ++++++++++++++
 include/linux/list.h             |    4 +++-
 include/linux/smp.h              |    8 +-------
 3 files changed, 18 insertions(+), 8 deletions(-)
include/linux/call_single_data.h new file +14 -0
Index: linux-lazy/include/linux/call_single_data.h
===================================================================
@@ -0,0 +1,14 @@
+#ifndef __LINUX_CALL_SINGLE_DATA_H
+#define __LINUX_CALL_SINGLE_DATA_H
+
+#include <linux/list.h>
+
+struct call_single_data {
+	struct list_head list;
+	void (*func) (void *info);
+	void *info;
+	u16 flags;
+	u16 priv;
+};
+
+#endif /* __LINUX_CALL_SINGLE_DATA_H */
include/linux/list.h +3 -1
Index: linux-lazy/include/linux/list.h
===================================================================
@@ -4,7 +4,6 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/poison.h>
-#include <linux/prefetch.h>
 
 /*
  * Simple doubly linked list implementation.
@@ -16,6 +15,9 @@
  * using the generic single-entry routines.
  */
 
+#include <linux/prefetch.h>
+#include <asm/system.h>
+
 #define LIST_HEAD_INIT(name) { &(name), &(name) }
 
 #define LIST_HEAD(name) \
include/linux/smp.h +1 -7
Index: linux-lazy/include/linux/smp.h
===================================================================
@@ -9,18 +9,12 @@
 #include <linux/errno.h>
 #include <linux/types.h>
 #include <linux/list.h>
+#include <linux/call_single_data.h>
 #include <linux/cpumask.h>
 
 extern void cpu_idle(void);
 
 typedef void (*smp_call_func_t)(void *info);
-struct call_single_data {
-	struct list_head list;
-	smp_call_func_t func;
-	void *info;
-	u16 flags;
-	u16 priv;
-};
 
 /* total number of cpus in this system (may exceed NR_CPUS) */
 extern unsigned int total_cpus;
#3 lore raw

PATCH 3/7 powerpc: Reorganise powerpc include files to make call_single_data

In reply to Michael Neuling

We need to put struct call_single_data in the powerpc thread_struct, but can't without this.

In processor.h this moves up the prefetch() functions before the #include of types.h to ensure __builtin_prefetch doesn't get defined twice.

Similarly in hw_irq.h move arch_irqs_disabled_flags() to before the #include of processor.h to ensure it's correctly found when hw_irq.h has already been included somewhere earlier

---
 arch/powerpc/include/asm/hw_irq.h    |   21 ++++++++------
 arch/powerpc/include/asm/processor.h |   52 ++++++++++++++++++-----------------
 2 files changed, 40 insertions(+), 33 deletions(-)
arch/powerpc/include/asm/hw_irq.h +12 -9
Index: linux-lazy/arch/powerpc/include/asm/hw_irq.h
===================================================================
@@ -8,10 +8,6 @@
 
 #include <linux/errno.h>
 #include <linux/compiler.h>
-#include <asm/ptrace.h>
-#include <asm/processor.h>
-
-extern void timer_interrupt(struct pt_regs *);
 
 #ifdef CONFIG_PPC64
 #include <asm/paca.h>
@@ -81,6 +77,8 @@
 
 #else /* CONFIG_PPC64 */
 
+#include <asm/reg.h>
+
 #define SET_MSR_EE(x)	mtmsr(x)
 
 static inline unsigned long arch_local_save_flags(void)
@@ -108,6 +106,16 @@
 	return flags;
 }
 
+static inline bool arch_irqs_disabled_flags(unsigned long flags)
+{
+	return (flags & MSR_EE) == 0;
+}
+
+#include <asm/ptrace.h>
+#include <asm/processor.h>
+
+extern void timer_interrupt(struct pt_regs *);
+
 static inline void arch_local_irq_disable(void)
 {
 #ifdef CONFIG_BOOKE
@@ -127,11 +135,6 @@
 #endif
 }
 
-static inline bool arch_irqs_disabled_flags(unsigned long flags)
-{
-	return (flags & MSR_EE) == 0;
-}
-
 static inline bool arch_irqs_disabled(void)
 {
 	return arch_irqs_disabled_flags(arch_local_save_flags());
arch/powerpc/include/asm/processor.h +28 -24

64 lines not on this page · raw message

#4 lore raw

PATCH 4/7 powerpc: Change fast_exception_return to restore r0, r7. r8, and CTR

In reply to Michael Neuling

Currently we don't restore r0, r7, r8 and CTR in fast_exception_return. This changes fast_exception_return to restore these, which were saved anyway on exception entry.

This seems like a bug waiting to happen, plus we do it in hash_page for 32bit anyway.

---
 arch/powerpc/kernel/entry_32.S |    4 ++++
 arch/powerpc/mm/hash_low_32.S  |    7 -------
 2 files changed, 4 insertions(+), 7 deletions(-)
arch/powerpc/kernel/entry_32.S +4 -0
Index: linux-lazy/arch/powerpc/kernel/entry_32.S
===================================================================
@@ -720,6 +720,10 @@
 #endif
 
 2:	REST_4GPRS(3, r11)
+	REST_2GPRS(7, r11)
+	REST_GPR(0, r11)
+ 	lwz	r10,_CTR(r11)
+ 	mtctr	r10
 	lwz	r10,_CCR(r11)
 	REST_GPR(1, r11)
 	mtcr	r10
arch/powerpc/mm/hash_low_32.S +0 -7
Index: linux-lazy/arch/powerpc/mm/hash_low_32.S
===================================================================
@@ -146,13 +146,6 @@
 	li	r0,0
 	stw	r0,mmu_hash_lock@l(r8)
 #endif
-
-	/* Return from the exception */
-	lwz	r5,_CTR(r11)
-	mtctr	r5
-	lwz	r0,GPR0(r11)
-	lwz	r7,GPR7(r11)
-	lwz	r8,GPR8(r11)
 	b	fast_exception_return
 
 #ifdef CONFIG_SMP
#5 lore raw

PATCH 5/7 powerpc: Enable lazy save VMX registers for SMP

In reply to Michael Neuling

This enables lazy save of VMX registers for SMP configurations.

This adds a pointer to the thread struct to say which CPU holds this processes VMX register state. On 64 bit, this points to the paca of the CPU holding the state or NULL if it's in the thread_struct. On 32 bit, this is the CPU number of the CPU holding the state or -1 if it's in the thread_struct.

It also adds a per cpu pointer (paca on 64bit), which points to the thread_struct of the process who's state we currently own.

On a context switch we do the following:
- if we are switching to a CPU that currently holds the new processes state, just turn on VMX in the MSR (this is the lazy/quick case)
- if the new processes state is in the thread_struct, turn VMX off.
- if the new processes state is in someone else's CPU, IPI that CPU to giveup it's state and turn VMX off.
We always start the new process at this point, irrespective of if we have the state or not in the thread struct or current CPU.

When we take the VMX unavailable, load_up_altivec checks to see if the state is now in the thread_struct. If it is, we restore the VMX registers and start the process. If it's not, we need to wait for the IPI to finish. Unfortunately, IRQs are off on the current CPU at this point, so we must turn IRQs on (to avoid a deadlock) before we block waiting for the IPI to finished on the other CPU.

We also change load_up_altivec to call giveup_altivec to save it's state rather than duplicating this code. This means that giveup_altivec can now be called with the MMU on or off, hence we pass in an offset, which gets subtracted on 32 bit systems on loads and stores.

For 32bit it's be nice to have last_used_altivec cacheline aligned or as per_cpu variables but we can't access per_cpu vars in asm

---
 arch/powerpc/include/asm/paca.h              |    5 
 arch/powerpc/include/asm/processor.h         |   22 ++
 arch/powerpc/include/asm/reg.h               |    9 +
 arch/powerpc/include/asm/system.h            |   10 -
 arch/powerpc/kernel/asm-offsets.c            |    4 
 arch/powerpc/kernel/paca.c                   |    3 
 arch/powerpc/kernel/process.c                |  138 ++++++++++++------
 arch/powerpc/kernel/vector.S                 |  199 ++++++++++++++++++++-------
 arch/powerpc/platforms/pseries/hotplug-cpu.c |    1 
 9 files changed, 288 insertions(+), 103 deletions(-)
arch/powerpc/include/asm/paca.h +5 -0
Index: linux-lazy/arch/powerpc/include/asm/paca.h
===================================================================
@@ -145,6 +145,11 @@
 	u64 dtl_ridx;			/* read index in dispatch log */
 	struct dtl_entry *dtl_curr;	/* pointer corresponding to dtl_ridx */
 
+#ifdef CONFIG_ALTIVEC
+	/* lazy save pointers */
+	struct task_struct *last_used_altivec;
+#endif
+
 #ifdef CONFIG_KVM_BOOK3S_HANDLER
 	/* We use this to store guest state in */
 	struct kvmppc_book3s_shadow_vcpu shadow_vcpu;
arch/powerpc/include/asm/processor.h +21 -1
Index: linux-lazy/arch/powerpc/include/asm/processor.h
===================================================================
@@ -18,6 +18,16 @@
 #define TS_FPRWIDTH 1
 #endif
 
+#ifdef CONFIG_PPC64
+#ifdef __ASSEMBLY__
+#define TS_LAZY_STATE_INVALID 0
+#else
+#define TS_LAZY_STATE_INVALID NULL
+#endif
+#else
+#define TS_LAZY_STATE_INVALID -1
+#endif
+
 #ifndef __ASSEMBLY__
 #include <linux/compiler.h>
 
@@ -48,6 +58,7 @@
 
 #define spin_lock_prefetch(x)	prefetchw(x)
 
+#include <linux/call_single_data.h>
 #include <asm/ptrace.h>
 #include <asm/types.h>
 
@@ -109,7 +120,6 @@
 
 /* Lazy FPU handling on uni-processor */
 extern struct task_struct *last_task_used_math;
-extern struct task_struct *last_task_used_altivec;
 extern struct task_struct *last_task_used_vsx;
 extern struct task_struct *last_task_used_spe;
 
@@ -177,6 +187,14 @@
 #define TS_VSRLOWOFFSET 1
 #define TS_FPR(i) fpr[i][TS_FPROFFSET]
 
+#ifdef CONFIG_PPC64
+#define TS_LAZY_STATE_TYPE struct paca_struct *
+#define TS_LAZY_STATE_INVALID NULL
+#else
+#define TS_LAZY_STATE_TYPE unsigned long
+#define TS_LAZY_STATE_INVALID -1
+#endif
+
 struct thread_struct {
 	unsigned long	ksp;		/* Kernel stack pointer */
 	unsigned long	ksp_limit;	/* if ksp <= ksp_limit stack overflow */
@@ -253,6 +271,8 @@
 	/* AltiVec status */
 	vector128	vscr __attribute__((aligned(16)));
 	unsigned long	vrsave;
+	TS_LAZY_STATE_TYPE vr_state;	/* where my vr state is */
+	struct call_single_data vr_csd;	/* IPI data structure */
 	int		used_vr;	/* set if process has used altivec */
 #endif /* CONFIG_ALTIVEC */
 #ifdef CONFIG_VSX
arch/powerpc/include/asm/reg.h +9 -0
Index: linux-lazy/arch/powerpc/include/asm/reg.h
===================================================================
@@ -808,6 +808,15 @@
 
 #define __is_processor(pv)	(PVR_VER(mfspr(SPRN_PVR)) == (pv))
 
+#ifdef CONFIG_PPC64
+#define GET_CURRENT_THREAD(reg)			\
+	ld	(reg),PACACURRENT(r13) ;	\
+	addi    (reg),(reg),THREAD
+#else
+#define GET_CURRENT_THREAD(reg)			\
+	mfspr	(reg),SPRN_SPRG_THREAD
+#endif
+
 /*
  * IBM has further subdivided the standard PowerPC 16-bit version and
  * revision subfields of the PVR for the PowerPC 403s into the following:
arch/powerpc/include/asm/system.h +3 -7
Index: linux-lazy/arch/powerpc/include/asm/system.h
===================================================================
@@ -7,6 +7,7 @@
 #include <linux/kernel.h>
 #include <linux/irqflags.h>
 
+#include <asm/irqflags.h>
 #include <asm/hw_irq.h>
 
 /*
@@ -145,7 +146,8 @@
 extern void enable_kernel_fp(void);
 extern void flush_fp_to_thread(struct task_struct *);
 extern void enable_kernel_altivec(void);
-extern void giveup_altivec(struct task_struct *);
+extern void giveup_altivec(unsigned long offset);
+extern void giveup_altivec_ipi(void *);
 extern void load_up_altivec(struct task_struct *);
 extern int emulate_altivec(struct pt_regs *);
 extern void __giveup_vsx(struct task_struct *);
@@ -157,13 +159,7 @@
 extern void cvt_fd(float *from, double *to);
 extern void cvt_df(double *from, float *to);
 
-#ifndef CONFIG_SMP
 extern void discard_lazy_cpu_state(void);
-#else
-static inline void discard_lazy_cpu_state(void)
-{
-}
-#endif
 
 #ifdef CONFIG_ALTIVEC
 extern void flush_altivec_to_thread(struct task_struct *);
arch/powerpc/kernel/asm-offsets.c +4 -0
Index: linux-lazy/arch/powerpc/kernel/asm-offsets.c
===================================================================
@@ -89,6 +89,7 @@
 	DEFINE(THREAD_VRSAVE, offsetof(struct thread_struct, vrsave));
 	DEFINE(THREAD_VSCR, offsetof(struct thread_struct, vscr));
 	DEFINE(THREAD_USED_VR, offsetof(struct thread_struct, used_vr));
+	DEFINE(THREAD_VR_STATE, offsetof(struct thread_struct, vr_state));
 #endif /* CONFIG_ALTIVEC */
 #ifdef CONFIG_VSX
 	DEFINE(THREAD_VSR0, offsetof(struct thread_struct, fpr));
@@ -197,6 +198,9 @@
 	DEFINE(PACA_USER_TIME, offsetof(struct paca_struct, user_time));
 	DEFINE(PACA_SYSTEM_TIME, offsetof(struct paca_struct, system_time));
 	DEFINE(PACA_TRAP_SAVE, offsetof(struct paca_struct, trap_save));
+#ifdef CONFIG_ALTIVEC
+	DEFINE(PACA_LAST_USED_ALTIVEC, offsetof(struct paca_struct, last_used_altivec));
+#endif
 #ifdef CONFIG_KVM_BOOK3S_64_HANDLER
 	DEFINE(PACA_KVM_SVCPU, offsetof(struct paca_struct, shadow_vcpu));
 	DEFINE(SVCPU_SLB, offsetof(struct kvmppc_book3s_shadow_vcpu, slb));
arch/powerpc/kernel/paca.c +3 -0
Index: linux-lazy/arch/powerpc/kernel/paca.c
===================================================================
@@ -162,6 +162,9 @@
 	new_paca->hw_cpu_id = 0xffff;
 	new_paca->kexec_state = KEXEC_STATE_NONE;
 	new_paca->__current = &init_task;
+#ifdef CONFIG_ALTIVEC
+	new_paca->last_used_altivec = NULL;
+#endif
 #ifdef CONFIG_PPC_STD_MMU_64
 	new_paca->slb_shadow_ptr = &slb_shadow[cpu];
 #endif /* CONFIG_PPC_STD_MMU_64 */
arch/powerpc/kernel/process.c +90 -48

214 lines not on this page · raw message

arch/powerpc/kernel/vector.S +152 -47

257 lines not on this page · raw message

arch/powerpc/platforms/pseries/hotplug-cpu.c +1 -0
Index: linux-lazy/arch/powerpc/platforms/pseries/hotplug-cpu.c
===================================================================
@@ -112,6 +112,7 @@
 
 	local_irq_disable();
 	idle_task_exit();
+	discard_lazy_cpu_state();
 	xics_teardown_cpu();
 
 	if (get_preferred_offline_state(cpu) == CPU_STATE_INACTIVE) {
#6 lore raw

PATCH 6/7 powerpc: Enable lazy save FP registers for SMP

In reply to Michael Neuling

This enables lazy save of FP registers for SMP configurations.

This adds a pointer to the thread struct to say which CPU holds this processes FP register state. On 64 bit, this points to the paca of the CPU holding the state or NULL if it's in the thread_struct. On 32 bit, this is the CPU number of the CPU holding the state or -1 if it's in the thread_struct.

It also adds a per cpu pointer (paca on 64bit), which points to the thread_struct of the process who's state we currently own.

On a context switch we do the following:
- if we are switching to a CPU that currently holds the new processes state, just turn on FP in the MSR (this is the lazy/quick case)
- if the new processes state is in the thread_struct, turn FP off.
- if the new processes state is in someone else's CPU, IPI that CPU to giveup it's state and turn FP off.
We always start the new process at this point, irrespective of if we have the state or not in the thread struct or current CPU.

When we take the FP unavailable, load_up_altivec checks to see if the state is now in the thread_struct. If it is, we restore the FP registers and start the process. If it's not, we need to wait for the IPI to finish. Unfortunately, IRQs are off on the current CPU at this point, so we must turn IRQs on (to avoid a deadlock) before we block waiting for the IPI to finished on the other CPU.

We also change load_up_fpu to call giveup_fpu to save it's state rather than duplicating this code. This means that giveup_altivec can now be called with the MMU on or off, hence we pass in an offset, which gets subtracted on 32 bit systems on loads and stores.

For 32bit it's be nice to have last_used_fp cacheline aligned or as per_cpu variables but we can't access per_cpu vars in asm.

---
 arch/powerpc/include/asm/paca.h      |    1 
 arch/powerpc/include/asm/processor.h |    8 +
 arch/powerpc/include/asm/system.h    |    3 
 arch/powerpc/kernel/asm-offsets.c    |    3 
 arch/powerpc/kernel/fpu.S            |  198 +++++++++++++++++++++++++++--------
 arch/powerpc/kernel/paca.c           |    1 
 arch/powerpc/kernel/process.c        |  114 +++++++++++---------
 7 files changed, 237 insertions(+), 91 deletions(-)
arch/powerpc/include/asm/paca.h +1 -0
Index: linux-lazy/arch/powerpc/include/asm/paca.h
===================================================================
@@ -145,6 +145,7 @@
 	u64 dtl_ridx;			/* read index in dispatch log */
 	struct dtl_entry *dtl_curr;	/* pointer corresponding to dtl_ridx */
 
+	struct task_struct *last_used_fp;
 #ifdef CONFIG_ALTIVEC
 	/* lazy save pointers */
 	struct task_struct *last_used_altivec;
arch/powerpc/include/asm/processor.h +7 -1
Index: linux-lazy/arch/powerpc/include/asm/processor.h
===================================================================
@@ -120,7 +120,6 @@
 extern long kernel_thread(int (*fn)(void *), void *arg, unsigned long flags);
 
 /* Lazy FPU handling on uni-processor */
-extern struct task_struct *last_task_used_math;
 extern struct task_struct *last_task_used_vsx;
 extern struct task_struct *last_task_used_spe;
 
@@ -253,6 +252,13 @@
 	} fpscr;
 	int		fpexc_mode;	/* floating-point exception mode */
 	unsigned int	align_ctl;	/* alignment handling control */
+	int		used_fp;	/* set if process has used fp */
+#ifdef CONFIG_PPC64
+	struct paca_struct *fp_state;	/* paca where my fp state could be? */
+#else
+	unsigned long	fp_state;	/* paca where my fp state could be? */
+#endif
+	struct call_single_data fp_csd;	/* IPI data structure */
 #ifdef CONFIG_PPC64
 	unsigned long	start_tb;	/* Start purr when proc switched in */
 	unsigned long	accum_tb;	/* Total accumilated purr for process */
arch/powerpc/include/asm/system.h +2 -1
Index: linux-lazy/arch/powerpc/include/asm/system.h
===================================================================
@@ -140,7 +140,8 @@
 extern void via_cuda_init(void);
 extern void read_rtc_time(void);
 extern void pmac_find_display(void);
-extern void giveup_fpu(struct task_struct *);
+extern void giveup_fpu(unsigned long offset);
+extern void giveup_fpu_ipi(void *);
 extern void disable_kernel_fp(void);
 extern void enable_kernel_fp(void);
 extern void flush_fp_to_thread(struct task_struct *);
arch/powerpc/kernel/asm-offsets.c +3 -0
Index: linux-lazy/arch/powerpc/kernel/asm-offsets.c
===================================================================
@@ -84,6 +84,8 @@
 	DEFINE(THREAD_FPEXC_MODE, offsetof(struct thread_struct, fpexc_mode));
 	DEFINE(THREAD_FPR0, offsetof(struct thread_struct, fpr[0]));
 	DEFINE(THREAD_FPSCR, offsetof(struct thread_struct, fpscr));
+	DEFINE(THREAD_FP_STATE, offsetof(struct thread_struct, fp_state));
+	DEFINE(THREAD_USED_FP, offsetof(struct thread_struct, used_fp));
 #ifdef CONFIG_ALTIVEC
 	DEFINE(THREAD_VR0, offsetof(struct thread_struct, vr[0]));
 	DEFINE(THREAD_VRSAVE, offsetof(struct thread_struct, vrsave));
@@ -198,6 +200,7 @@
 	DEFINE(PACA_USER_TIME, offsetof(struct paca_struct, user_time));
 	DEFINE(PACA_SYSTEM_TIME, offsetof(struct paca_struct, system_time));
 	DEFINE(PACA_TRAP_SAVE, offsetof(struct paca_struct, trap_save));
+	DEFINE(PACA_LAST_USED_FP, offsetof(struct paca_struct, last_used_fp));
 #ifdef CONFIG_ALTIVEC
 	DEFINE(PACA_LAST_USED_ALTIVEC, offsetof(struct paca_struct, last_used_altivec));
 #endif
arch/powerpc/kernel/fpu.S +157 -41

252 lines not on this page · raw message

arch/powerpc/kernel/paca.c +1 -0
Index: linux-lazy/arch/powerpc/kernel/paca.c
===================================================================
@@ -162,6 +162,7 @@
 	new_paca->hw_cpu_id = 0xffff;
 	new_paca->kexec_state = KEXEC_STATE_NONE;
 	new_paca->__current = &init_task;
+	new_paca->last_used_fp = NULL;
 #ifdef CONFIG_ALTIVEC
 	new_paca->last_used_altivec = NULL;
 #endif
arch/powerpc/kernel/process.c +66 -48

172 lines not on this page · raw message

#7 lore raw

PATCH 7/7 powerpc: Enable lazy save VSX for SMP

In reply to Michael Neuling

This enables lazy save of VSX state for SMP configurations.

Most of the logic for this is in the FP and VMX code, since VSX has no additional state over these.

When context switching to a new process:
- if both VMX and FP state are on the CPU we are switch to, turn VSX on also.
- if either FP or VMX are not on the CPU we are switch on, do not turn VSX on in the MSR.
We always start the new process at this point, irrespective of if we have the FP and/or VMX state in the thread struct or current CPU.

When we take the vsx_unavailable exception, we first run load_up_fpu and load_up_altivec to enable our state. If either of these fail to enable their respective MSR bits, the state has not arrived from the IPI and hence we should bail to userspace with VSX off. This will enable IRQs, while we are waiting for the FP and VMX state.

---
 arch/powerpc/include/asm/system.h |    4 +-
 arch/powerpc/kernel/process.c     |   72 ++++++++++++++++++++------------------
 arch/powerpc/kernel/signal_32.c   |    2 -
 arch/powerpc/kernel/signal_64.c   |    2 -
 arch/powerpc/kernel/vector.S      |   48 +++----------------------
 5 files changed, 49 insertions(+), 79 deletions(-)
arch/powerpc/include/asm/system.h +2 -2
Index: linux-lazy/arch/powerpc/include/asm/system.h
===================================================================
@@ -150,8 +150,8 @@
 extern void giveup_altivec_ipi(void *);
 extern void load_up_altivec(struct task_struct *);
 extern int emulate_altivec(struct pt_regs *);
-extern void __giveup_vsx(struct task_struct *);
-extern void giveup_vsx(struct task_struct *);
+extern void __giveup_vsx(void);
+extern void giveup_vsx(void);
 extern void enable_kernel_spe(void);
 extern void giveup_spe(struct task_struct *);
 extern void load_up_spe(struct task_struct *);
arch/powerpc/kernel/process.c +38 -34

129 lines not on this page · raw message

arch/powerpc/kernel/signal_32.c +1 -1
Index: linux-lazy/arch/powerpc/kernel/signal_32.c
===================================================================
@@ -452,7 +452,7 @@
 	 * contains valid data
 	 */
 	if (current->thread.used_vsr && ctx_has_vsx_region) {
-		__giveup_vsx(current);
+		__giveup_vsx();
 		if (copy_vsx_to_user(&frame->mc_vsregs, current))
 			return 1;
 		msr |= MSR_VSX;
arch/powerpc/kernel/signal_64.c +1 -1
Index: linux-lazy/arch/powerpc/kernel/signal_64.c
===================================================================
@@ -123,7 +123,7 @@
 	 * VMX data.
 	 */
 	if (current->thread.used_vsr && ctx_has_vsx_region) {
-		__giveup_vsx(current);
+		__giveup_vsx();
 		v_regs += ELF_NVRREG;
 		err |= copy_vsx_to_user(v_regs, current);
 		/* set MSR_VSX in the MSR value in the frame to
arch/powerpc/kernel/vector.S +7 -41

73 lines not on this page · raw message