Skip to contentSkip to conversation

i386: inline restore_fpu

PATCH In review Chuck Ebbert 6 participants 8 messages Jul 22, 2005 – Jul 25, 2005

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i386: inline restore_fpu

This patch makes restore_fpu() an inline. When L1/L2 cache are saturated
it makes a measurable difference.

Results from profiling Volanomark follow. Sample rate was 2000 samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.

Before:

10680 restore_fpu 333.7500
8351 device_not_available 203.6829
3823 math_state_restore 59.7344
-----
22854

After:

12534 math_state_restore 130.5625
8354 device_not_available 203.7561
-----
20888

Patch is "obviously correct" and cuts 9% of the overhead. Please apply.

Next step should be to physically place math_state_restore() after device_not_available(). Would such a patch be accepted? (Yes it would be ugly and require linker script changes.)

arch/i386/kernel/i387.c +0 -11
Index: 2.6.13-rc3a/arch/i386/kernel/i387.c
===================================================================
@@ -82,17 +82,6 @@
 }
 EXPORT_SYMBOL_GPL(kernel_fpu_begin);
 
-void restore_fpu( struct task_struct *tsk )
-{
-	if ( cpu_has_fxsr ) {
-		asm volatile( "fxrstor %0"
-			      : : "m" (tsk->thread.i387.fxsave) );
-	} else {
-		asm volatile( "frstor %0"
-			      : : "m" (tsk->thread.i387.fsave) );
-	}
-}
-
 /*
  * FPU tag word conversions.
  */
include/asm-i386/i387.h +11 -2
Index: 2.6.13-rc3a/include/asm-i386/i387.h
===================================================================
@@ -22,11 +22,20 @@
 /*
  * FPU lazy state save handling...
  */
-extern void restore_fpu( struct task_struct *tsk );
-
 extern void kernel_fpu_begin(void);
 #define kernel_fpu_end() do { stts(); preempt_enable(); } while(0)
 
+static inline void restore_fpu( struct task_struct *tsk )
+{
+	if ( cpu_has_fxsr ) {
+		asm volatile( "fxrstor %0"
+			      : : "m" (tsk->thread.i387.fxsave) );
+	} else {
+		asm volatile( "frstor %0"
+			      : : "m" (tsk->thread.i387.fsave) );
+	}
+}
+
 /*
  * These must be called with preempt disabled
  */

__
Chuck

#1 lore raw
Quoted 20 lines · Chuck Ebbert

This patch makes restore_fpu() an inline. When L1/L2 cache are saturated
it makes a measurable difference.

Results from profiling Volanomark follow. Sample rate was 2000 samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.

Before:

10680 restore_fpu 333.7500
8351 device_not_available 203.6829
3823 math_state_restore 59.7344
-----
22854

After:

Quoted 7 lines

12534 math_state_restore 130.5625
8354 device_not_available 203.7561
-----
20888

Patch is "obviously correct" and cuts 9% of the overhead. Please apply.

hm. What context switch rate is that thing doing?

Is the benchmark actually doing floating point stuff?

We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math. But <ancient recollections> there's code in glibc startup which always does a bit of float, so that optimisation is always defeated. There was some discussion about periodically setting tasks back into !used_math state to try to restore the optimisation for tasks which only do a little bit of FP, but nothing actually got done.

Quoted 3 lines

Next step should be to physically place math_state_restore() after device_not_available(). Would such a patch be accepted? (Yes it would be ugly and require linker script changes.)

Depends on the benefit/ugly ratio ;)

#2 lore raw
Quoted 2 lines · Andrew Morton

Is the benchmark actually doing floating point stuff?

It must be. We still do lazy FP saves.

Quoted 2 lines

We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math.

No, it's more than that. There's a per-processor "used_math" flag to determine if we need to _initialize_ the FPU, but on context switches we always assume the program we're switching to will _not_ use FP, and we just set the "fault on FP" flag and do not normally restore FP state.

It seems volanomark will always use FP, if this is the hot path.

We'll only save the FP context if the thread has used the FP in _that_ particular time-slice (TS_USEDFPU).

As to why volanomark also uses FP, I don't know. I wouldn't be surprised if the benchmark was designed by somebody to not benefit from the x87 state save optimization.

On the other hand, I also wouldn't be surprised if glibc (or similar system libraries) is over-eagerly using things like SSE instructions for memcopies etc, not realizing that they can have serious downsides. I don't see why volanomark would use FP, but hey, it's billed as a java VM and thread torture test for "chatrooms". Whatever.

Linus

#3 lore raw
Quoted 8 lines
Quoted 3 lines

We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math.

No, it's more than that. There's a per-processor "used_math" flag to determine if we need to _initialize_ the FPU, but on context switches we always assume the program we're switching to will _not_ use FP, and we just set the "fault on FP" flag and do not normally restore FP state.

This shows room for optimization; if an app is consistently faulting to use FP after a context switch, in principle the kernel could start to assume that it will in the next timeslice as well.

Quoted 1 line

On the other hand, I also wouldn't be surprised if glibc (or similar

I doubt glibc is normally, at least most distros don't ship an SSE enabled glibc, only an "i686" one.

#4 lore raw
In reply to Chuck Ebbert
Quoted 19 lines · Chuck Ebbert

This patch makes restore_fpu() an inline. When L1/L2 cache are saturated
it makes a measurable difference.

Results from profiling Volanomark follow. Sample rate was 2000 samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.

Before:

10680 restore_fpu 333.7500
8351 device_not_available 203.6829
3823 math_state_restore 59.7344
-----
22854

After:

Quoted 8 lines

12534 math_state_restore 130.5625
8354 device_not_available 203.7561
-----
20888

Patch is "obviously correct" and cuts 9% of the overhead. Please apply.
...

If this patch makes a difference, could you do me a favour and check whether replacing the current cpu_has_fxsr #define in include/asm-i386/cpufeature.h with

  #define cpu_has_fxsr           1

on top of your patch brings an additional improvement?

Quoted 1 line

Chuck

TIA
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#5 lore raw
Quoted 8 lines · Adrian Bunk

If this patch makes a difference, could you do me a favour and check whether replacing the current cpu_has_fxsr #define in include/asm-i386/cpufeature.h with

  #define cpu_has_fxsr           1

on top of your patch brings an additional improvement?

It would be really sad if it made a difference. There might be a branch mispredict, but the real expense of the fnsave/fxsave will be that instruction itself, and any cache misses associated with it. The 9% performace difference would almost have to be due to a memory bank conflict or something (likely some unnecessary I$ prefetching that interacts badly with the writeback needed for the _big_ memory write forced by the fxsave).

I can't see any way that a single branch mispredict could make that big of a difference, but I _can_ see how bad memory access patterns could do it.

Btw, the switch from fnsave to fxsave (and thus the change from a 112-byte save area to a 512-byte one, or whatever the exact details are) caused _huge_ performance degradation for various context switching benchmarks. I really hated that, but obviously the need to support SSE2 made it non-optional. The point being that the real overhead is that big memory read/write in fxrestor/fxsave.

What _could_ make a bigger difference is not doing the lazy FPU at all. That lazy FPU is a huge optimization on 99.9% of all loads, but it sounds like java/volanomark are broken and always use the FPU, and then we take a big hit on doing the FP restore exception (an exception is a lot more expensive than a mispredict).

Something like the following (totally untested) should make it be non-lazy. It's going to slow down normal task switches, but might speed up the "restoring FP context all the time" case.

Chuck? This should work fine with or without your inline thing. Does it make any difference?

Linus

-----

arch/i386/kernel/process.c +16 -8
@@ -678,8 +678,16 @@ struct task_struct fastcall * __switch_t
 	struct tss_struct *tss = &per_cpu(init_tss, cpu);
 
 	/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */
-
-	__unlazy_fpu(prev_p);
+	if (prev_p->thread_info->status & TS_USEDFPU) {
+		__save_init_fpu(prev_p);
+		if (tsk_used_math(next_p))
+			init_fpu(next_p);
+		restore_fpu(next_p);
+		next_p->thread_info->status |= TS_USEDFPU;
+	} else {
+		stts();
+		next_p->thread_info->status &= ~TS_USEDFPU;
+	}
 
 	/*
 	 * Reload esp0, LDT and the page table pointer:
@@ -710,13 +718,13 @@ struct task_struct fastcall * __switch_t
 	 * Now maybe reload the debug registers
 	 */
 	if (unlikely(next->debugreg[7])) {
-		set_debugreg(current->thread.debugreg[0], 0);
-		set_debugreg(current->thread.debugreg[1], 1);
-		set_debugreg(current->thread.debugreg[2], 2);
-		set_debugreg(current->thread.debugreg[3], 3);
+		set_debugreg(next->debugreg[0], 0);
+		set_debugreg(next->debugreg[1], 1);
+		set_debugreg(next->debugreg[2], 2);
+		set_debugreg(next->debugreg[3], 3);
 		/* no 4 and 5 */
-		set_debugreg(current->thread.debugreg[6], 6);
-		set_debugreg(current->thread.debugreg[7], 7);
+		set_debugreg(next->debugreg[6], 6);
+		set_debugreg(next->debugreg[7], 7);
 	}
 
 	if (unlikely(prev->io_bitmap_ptr || next->io_bitmap_ptr))
#6 lore raw
Quoted 27 lines · Linus Torvalds
Quoted 6 lines · Adrian Bunk

If this patch makes a difference, could you do me a favour and check whether replacing the current cpu_has_fxsr #define in include/asm-i386/cpufeature.h with

#define cpu_has_fxsr 1

on top of your patch brings an additional improvement?

It would be really sad if it made a difference. There might be a branch mispredict, but the real expense of the fnsave/fxsave will be that instruction itself, and any cache misses associated with it. The 9% performace difference would almost have to be due to a memory bank conflict or something (likely some unnecessary I$ prefetching that interacts badly with the writeback needed for the _big_ memory write forced by the fxsave).

I can't see any way that a single branch mispredict could make that big of a difference, but I _can_ see how bad memory access patterns could do it.

Btw, the switch from fnsave to fxsave (and thus the change from a 112-byte
save area to a 512-byte one, or whatever the exact details are) caused
_huge_ performance degradation for various context switching benchmarks. I
really hated that, but obviously the need to support SSE2 made it
non-optional. The point being that the real overhead is that big memory read/write in fxrestor/fxsave.

Quoted 4 lines

What _could_ make a bigger difference is not doing the lazy FPU at all. That lazy FPU is a huge optimization on 99.9% of all loads, but it sounds like java/volanomark are broken and always use the FPU, and then we take a
big hit on doing the FP restore exception (an exception is a lot more
expensive than a mispredict).

It seems expensive to do the save/restore when it isn't needed, that's why the code got lazy. Would it be useful to have a small flag or count field and start by assuming that FPU is not used, and if the exception takes place set the count to unconditionally save the FP state for some number of context switches and then try reverting to lazy save?

That 99.9% may be a guess, but I suspect that there are a lot of applications which alternate between using FPU and not, even if they do use FPU for some parts of the application. That way the performance of lazy save would be realized for the common applications, and the overhead of exception would be greatly reduced for both the ill-behaved and legitimately FPU intensive application.

Quoted 5 lines

Something like the following (totally untested) should make it be
non-lazy. It's going to slow down normal task switches, but might speed up the "restoring FP context all the time" case.

Chuck? This should work fine with or without your inline thing. Does it make any difference?

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#7 lore raw
In reply to Chuck Ebbert
Quoted 3 lines · Chuck Ebbert

This patch makes restore_fpu() an inline. When L1/L2 cache are saturated
it makes a measurable difference.

I've now pushed out an alternative fix for this, which imho is much cleaner.

We've long had infrastructure for "alternative asm instructions" that are conditional on CPU features, and I just made restore_fpu() use that instead:

	/*
	 * The "nop" is needed to make the instructions the same
	 * length.
	 */
	#define restore_fpu(tsk)                        \
	        alternative_input(                      \
	                "nop ; frstor %1",              \
	                "fxrstor %1",                   \
	                X86_FEATURE_FXSR,               \
	                "m" ((tsk)->thread.i387.fsave))

which not only makes it inline, but makes it a single instruction instead of the mess it was before.

Now, we should do the same for "fnsave ; fwait" vs "fxsave ; fnclex" too, but I was lazy. If somebody wants to try that, it would need an "alternative_output()" define but should otherwise be trivial too.

Linus