i386: inline restore_fpu
PATCH 1/1 i386: inline restore_fpu
This patch makes restore_fpu() an inline. When L1/L2 cache are saturated
it makes a measurable difference.
Results from profiling Volanomark follow. Sample rate was 2000 samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.
Before:
10680 restore_fpu 333.7500
8351 device_not_available 203.6829
3823 math_state_restore 59.7344
-----
22854
After:
12534 math_state_restore 130.5625
8354 device_not_available 203.7561
-----
20888
Patch is "obviously correct" and cuts 9% of the overhead. Please apply.
Next step should be to physically place math_state_restore() after device_not_available(). Would such a patch be accepted? (Yes it would be ugly and require linker script changes.)
arch/i386/kernel/i387.c +0 -11
@@ -82,17 +82,6 @@ }EXPORT_SYMBOL_GPL (kernel_fpu_begin); -void restore_fpu (struct task_struct *tsk ) -{ -if ( cpu_has_fxsr ) { - asmvolatile ("fxrstor %0" - : :"m" (tsk->thread.i387.fxsave) ); - }else { - asmvolatile ("frstor %0" - : :"m" (tsk->thread.i387.fsave) ); - } -} -/* * FPU tag word conversions. */
include/asm-i386/i387.h +11 -2
@@ -22,11 +22,20 @@/* * FPU lazy state save handling... */ -extern void restore_fpu (struct task_struct *tsk ); -extern void kernel_fpu_begin (void );#define kernel_fpu_end () do { stts(); preempt_enable(); } while(0) +static inline void restore_fpu (struct task_struct *tsk ) +{ +if ( cpu_has_fxsr ) { + asmvolatile ("fxrstor %0" + : :"m" (tsk->thread.i387.fxsave) ); + }else { + asmvolatile ("frstor %0" + : :"m" (tsk->thread.i387.fsave) ); + } +} +/* * These must be called with preempt disabled */
__
Chuck
Quoted 20 lines · Chuck Ebbert
This patch makes
restore_fpu()an inline. When L1/L2 cache are saturated
it makes a measurable difference.Results from profiling Volanomark follow. Sample rate was 2000
samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.Before:
10680
restore_fpu333.7500
8351device_not_available203.6829
3823math_state_restore59.7344
-----
22854After:
Quoted 7 lines
12534
math_state_restore130.5625
8354device_not_available203.7561
-----
20888Patch is "obviously correct" and cuts 9% of the overhead. Please apply.
hm. What context switch rate is that thing doing?
Is the benchmark actually doing floating point stuff?
We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math. But <ancient recollections> there's code in glibc startup which always does a bit of float, so that optimisation is always defeated. There was some discussion about periodically setting tasks back into !used_math state to try to restore the optimisation for tasks which only do a little bit of FP, but nothing actually got done.
Quoted 3 lines
Next step should be to physically place
math_state_restore()afterdevice_not_available(). Would such a patch be accepted? (Yes it would be ugly and require linker script changes.)
Depends on the benefit/ugly ratio ;)
Quoted 2 lines · Andrew Morton
Is the benchmark actually doing floating point stuff?
It must be. We still do lazy FP saves.
Quoted 2 lines
We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math.
No, it's more than that. There's a per-processor "used_math" flag to determine if we need to _initialize_ the FPU, but on context switches we always assume the program we're switching to will _not_ use FP, and we just set the "fault on FP" flag and do not normally restore FP state.
It seems volanomark will always use FP, if this is the hot path.
We'll only save the FP context if the thread has used the FP in _that_ particular time-slice (TS_USEDFPU).
As to why volanomark also uses FP, I don't know. I wouldn't be surprised if the benchmark was designed by somebody to not benefit from the x87 state save optimization.
On the other hand, I also wouldn't be surprised if glibc (or similar system libraries) is over-eagerly using things like SSE instructions for memcopies etc, not realizing that they can have serious downsides. I don't see why volanomark would use FP, but hey, it's billed as a java VM and thread torture test for "chatrooms". Whatever.
Linus
Quoted 8 lines
Quoted 3 lines
We do have the `used_math' optimisation in there which attempts to avoid doing the FP save/restore if the app isn't actually using math.
No, it's more than that. There's a per-processor "
used_math" flag to determine if we need to _initialize_ the FPU, but on context switches we always assume the program we're switching to will _not_ use FP, and we just set the "fault on FP" flag and do not normally restore FP state.
This shows room for optimization; if an app is consistently faulting to use FP after a context switch, in principle the kernel could start to assume that it will in the next timeslice as well.
Quoted 1 line
On the other hand, I also wouldn't be surprised if glibc (or similar
I doubt glibc is normally, at least most distros don't ship an SSE enabled glibc, only an "i686" one.
Quoted 19 lines · Chuck Ebbert
This patch makes
restore_fpu()an inline. When L1/L2 cache are saturated
it makes a measurable difference.Results from profiling Volanomark follow. Sample rate was 2000
samples/sec
(HZ = 250, profile multiplier = 8) on a dual-processor Pentium II Xeon.Before:
10680
restore_fpu333.7500
8351device_not_available203.6829
3823math_state_restore59.7344
-----
22854After:
Quoted 8 lines
12534
math_state_restore130.5625
8354device_not_available203.7561
-----
20888Patch is "obviously correct" and cuts 9% of the overhead. Please apply.
...
If this patch makes a difference, could you do me a favour and check whether replacing the current cpu_has_fxsr #define in include/asm-i386/cpufeature.h with
#define cpu_has_fxsr 1
on top of your patch brings an additional improvement?
Quoted 1 line
Chuck
TIA
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 8 lines · Adrian Bunk
If this patch makes a difference, could you do me a favour and check whether replacing the current
cpu_has_fxsr#define ininclude/asm-i386/cpufeature.hwith#define cpu_has_fxsr 1on top of your patch brings an additional improvement?
It would be really sad if it made a difference. There might be a branch mispredict, but the real expense of the fnsave/fxsave will be that instruction itself, and any cache misses associated with it. The 9% performace difference would almost have to be due to a memory bank conflict or something (likely some unnecessary I$ prefetching that interacts badly with the writeback needed for the _big_ memory write forced by the fxsave).
I can't see any way that a single branch mispredict could make that big of a difference, but I _can_ see how bad memory access patterns could do it.
Btw, the switch from fnsave to fxsave (and thus the change from a 112-byte save area to a 512-byte one, or whatever the exact details are) caused _huge_ performance degradation for various context switching benchmarks. I really hated that, but obviously the need to support SSE2 made it non-optional. The point being that the real overhead is that big memory read/write in fxrestor/fxsave.
What _could_ make a bigger difference is not doing the lazy FPU at all. That lazy FPU is a huge optimization on 99.9% of all loads, but it sounds like java/volanomark are broken and always use the FPU, and then we take a big hit on doing the FP restore exception (an exception is a lot more expensive than a mispredict).
Something like the following (totally untested) should make it be non-lazy. It's going to slow down normal task switches, but might speed up the "restoring FP context all the time" case.
Chuck? This should work fine with or without your inline thing. Does it make any difference?
Linus
-----
arch/i386/kernel/process.c +16 -8
@@ -678,8 +678,16 @@ struct task_struct fastcall * __switch_tstruct tss_struct *tss = &per_cpu (init_tss, cpu);/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */ - -__unlazy_fpu (prev_p); +if (prev_p->thread_info->status &TS_USEDFPU ) { +__save_init_fpu (prev_p); +if (tsk_used_math (next_p)) +init_fpu (next_p); +restore_fpu (next_p); + next_p->thread_info->status |=TS_USEDFPU ; + }else { +stts (); + next_p->thread_info->status &= ~TS_USEDFPU ; + }/* * Reload esp0, LDT and the page table pointer:
@@ -710,13 +718,13 @@ struct task_struct fastcall * __switch_t* Now maybe reload the debug registers */ if (unlikely (next->debugreg[7 ])) { -set_debugreg (current->thread.debugreg[0 ],0 ); -set_debugreg (current->thread.debugreg[1 ],1 ); -set_debugreg (current->thread.debugreg[2 ],2 ); -set_debugreg (current->thread.debugreg[3 ],3 ); +set_debugreg (next->debugreg[0 ],0 ); +set_debugreg (next->debugreg[1 ],1 ); +set_debugreg (next->debugreg[2 ],2 ); +set_debugreg (next->debugreg[3 ],3 );/* no 4 and 5 */ -set_debugreg (current->thread.debugreg[6 ],6 ); -set_debugreg (current->thread.debugreg[7 ],7 ); +set_debugreg (next->debugreg[6 ],6 ); +set_debugreg (next->debugreg[7 ],7 ); }if (unlikely (prev->io_bitmap_ptr || next->io_bitmap_ptr))
Quoted 27 lines · Linus Torvalds
Quoted 6 lines · Adrian Bunk
If this patch makes a difference, could you do me a favour and check whether replacing the current
cpu_has_fxsr#define ininclude/asm-i386/cpufeature.hwith#define
cpu_has_fxsr1on top of your patch brings an additional improvement?
It would be really sad if it made a difference. There might be a branch mispredict, but the real expense of the fnsave/fxsave will be that instruction itself, and any cache misses associated with it. The 9% performace difference would almost have to be due to a memory bank conflict or something (likely some unnecessary I$ prefetching that interacts badly with the writeback needed for the _big_ memory write forced by the fxsave).
I can't see any way that a single branch mispredict could make that big of a difference, but I _can_ see how bad memory access patterns could do it.
Btw, the switch from fnsave to fxsave (and thus the change from a 112-byte
save area to a 512-byte one, or whatever the exact details are) caused
_huge_ performance degradation for various context switching benchmarks. I
really hated that, but obviously the need to support SSE2 made it
non-optional. The point being that the real overhead is that big memory read/write in fxrestor/fxsave.
Quoted 4 lines
What _could_ make a bigger difference is not doing the lazy FPU at all. That lazy FPU is a huge optimization on 99.9% of all loads, but it sounds like java/volanomark are broken and always use the FPU, and then we take a
big hit on doing the FP restore exception (an exception is a lot more
expensive than a mispredict).
It seems expensive to do the save/restore when it isn't needed, that's why the code got lazy. Would it be useful to have a small flag or count field and start by assuming that FPU is not used, and if the exception takes place set the count to unconditionally save the FP state for some number of context switches and then try reverting to lazy save?
That 99.9% may be a guess, but I suspect that there are a lot of applications which alternate between using FPU and not, even if they do use FPU for some parts of the application. That way the performance of lazy save would be realized for the common applications, and the overhead of exception would be greatly reduced for both the ill-behaved and legitimately FPU intensive application.
Quoted 5 lines
Something like the following (totally untested) should make it be
non-lazy. It's going to slow down normal task switches, but might speed up the "restoring FP context all the time" case.Chuck? This should work fine with or without your inline thing. Does it make any difference?
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 3 lines · Chuck Ebbert
This patch makes
restore_fpu()an inline. When L1/L2 cache are saturated
it makes a measurable difference.
I've now pushed out an alternative fix for this, which imho is much cleaner.
We've long had infrastructure for "alternative asm instructions" that are conditional on CPU features, and I just made restore_fpu() use that instead:
/* * The "nop" is needed to make the instructions the same * length. */ #define restore_fpu(tsk) \ alternative_input( \ "nop ; frstor %1", \ "fxrstor %1", \ X86_FEATURE_FXSR, \ "m" ((tsk)->thread.i387.fsave))
which not only makes it inline, but makes it a single instruction instead of the mess it was before.
Now, we should do the same for "fnsave ; fwait" vs "fxsave ; fnclex" too, but I was lazy. If somebody wants to try that, it would need an "alternative_output()" define but should otherwise be trivial too.
Linus