x86,fpu: various small FPU cleanups and optimizations
cover x86,fpu: various small FPU cleanups and optimizations
This includes the three patches by Oleg that are not in -tip yet, and five more by myself.
I believe the changes to my patches address all the comments by reviewers on the previous version.
PATCH 1/8 x86, fpu: unlazy_fpu: don't reset thread.fpu_counter
From: Oleg Nesterov <[email protected]>
It is not clear why the "else" branch clears ->fpu_counter, this makes no sense.
If use_eager_fpu() then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if __thread_has_fpu().
--- arch/x86/kernel/i387.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kernel/i387.c +1 -2
@@ -122,8 +122,7 @@ void unlazy_fpu(struct task_struct *tsk)if (__thread_has_fpu (tsk)) {__save_init_fpu (tsk);__thread_fpu_end (tsk); - }else - tsk->thread.fpu_counter =0 ; + }preempt_enable (); }EXPORT_SYMBOL (unlazy_fpu);
Signature · 1 line
1.9.3
Borislav Petkov
maintainerRe: [PATCH 1/8] x86, fpu: unlazy_fpu: don't reset thread.fpu_counter
Quoted 4 lines · On Fri, Feb 06, 2015 at 03:01:58PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
It is not clear why the "else" branch clears ->fpu_counter, this makes no sense.
See below for why.
Quoted 3 lines
If
use_eager_fpu()then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if__thread_has_fpu().
and AFAICT, switch_fpu_prepare() already does that so I'd go ahead and venture a guess that this has survived all the cleanups (I had to go surf down memory lane to find Arjan's patch) over the years and is still there for no apparent reason.
Quoted 11 lines
Signed-off-by Oleg Nesterov <[email protected]>Signed-off-by Rik van Riel <[email protected]>--- arch/x86/kernel/i387.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/kernel/i387.c +0
-0
arch/x86/kernel/i387.c void unlazy_fpu(struct task_struct *tsk) · line 125
Quoted 6 lines
arch/x86/kernel/i387.c +1
-2@@ -122,8 +122,7 @@ void unlazy_fpu(struct task_struct *tsk)if (__thread_has_fpu (tsk)) {__save_init_fpu (tsk);__thread_fpu_end (tsk); - }else - tsk->thread.fpu_counter =0 ; + }
... and so by looking at the unlazy_fpu() call sites, I think this makes sense.
So how's that for a commit message instead:
---
x86, fpu, unlazy_fpu: Don't reset thread.fpu_counter
The "else" branch clears ->fpu_counter as a remnant of the lazy FPU usage counting:
e07e23e1fd30 ("[PATCH] non lazy "sleazy" fpu implementation")
However, switch_fpu_prepare() does this now so that else branch is superfluous.
If we do use_eager_fpu(), then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if __thread_has_fpu().
---
?
Thanks.
---e07e23e1fd300:
From: Arjan van de Ven <[email protected]>
Date: Tue, 26 Sep 2006 10:52:36 +0200
Subject: [PATCH] [PATCH] non lazy "sleazy" fpu implementation
Right now the kernel on x86-64 has a 100% lazy fpu behavior: after *every* context switch a trap is taken for the first FPU use to restore the FPU context lazily. This is of course great for applications that have very sporadic or no FPU use (since then you avoid doing the expensive save/restore all the time). However for very frequent FPU users... you take an extra trap every context switch.
The patch below adds a simple heuristic to this code: After 5 consecutive context switches of FPU use, the lazy behavior is disabled and the context gets restored every context switch. If the app indeed uses the FPU, the trap is avoided. (the chance of the 6th time slice using FPU after the previous 5 having done so are quite high obviously).
After 256 switches, this is reset and lazy behavior is returned (until there are 5 consecutive ones again). The reason for this is to give apps that do longer bursts of FPU use still the lazy behavior back after some time.
--- arch/x86_64/kernel/process.c | 10 ++++++++++ arch/x86_64/kernel/traps.c | 1 + include/asm-x86_64/i387.h | 5 ++++- include/linux/sched.h | 9 +++++++++ 4 files changed, 24 insertions(+), 1 deletion(-)
arch/x86_64/kernel/process.c +10 -0
@@ -552,6 +552,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)int cpu =smp_processor_id ();struct tss_struct *tss = &per_cpu (init_tss, cpu); +/* we're going to use this soon, after a few expensive things */ +if (next_p->fpu_counter>5 ) +prefetch (&next->i387.fxsave); +/* * Reload esp0, LDT and the page table pointer: */
@@ -629,6 +633,12 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) || test_tsk_thread_flag(prev_p, TIF_IO_BITMAP))__switch_to_xtra (prev_p, next_p, tss); +/* If the task has used fpu the last 5 timeslices, just do a full +* restore of the math state immediately to avoid the trap; the +* chances of needing FPU soon are obviously high now +*/ +if (next_p->fpu_counter>5 ) +math_state_restore ();return prev_p; }
arch/x86_64/kernel/traps.c +1 -0
@@ -1136,6 +1136,7 @@ asmlinkage void math_state_restore(void)init_fpu (me);restore_fpu_checking (&me->thread.i387.fxsave);task_thread_info (me)->status |=TS_USEDFPU ; + me->fpu_counter++; }void __init trap_init (void )
include/asm-x86_64/i387.h +4 -1
@@ -24,6 +24,7 @@ extern unsigned int mxcsr_feature_mask;extern void mxcsr_feature_mask_init (void );extern void init_fpu (struct task_struct *child);extern int save_i387 (struct _fpstate __user *buf); +extern asmlinkage void math_state_restore(void);/* * FPU lazy state save handling...
@@ -31,7 +32,9 @@ extern int save_i387(struct _fpstate __user *buf);#define unlazy_fpu (tsk) do { \ if (task_thread_info(tsk)->status & TS_USEDFPU) \ - save_init_fpu(tsk); \ + save_init_fpu(tsk); \ + else \ + tsk->fpu_counter = 0; \ } while (0)/* Ignore delayed exceptions from user space */
include/linux/sched.h +9 -0
@@ -865,6 +865,15 @@ struct task_struct {struct key *thread_keyring;/* keyring private to this thread */ unsigned char jit_keyring;/* default keyring to attach requested keys to */ #endif +/* +* fpu_counter contains the number of consecutive context switches +* that the FPU is used. If this is over a threshold, the lazy fpu +* saving becomes unlazy to save the trap. This is an unsigned char +* so that after 256 times the counter wraps and the behavior turns +* lazy again; this to deal with bursty apps that only use FPU for +* a short time +*/ +unsigned char fpu_counter;int oomkilladj;/* OOM kill score adjustment (bit shift). */ char comm[TASK_COMM_LEN ];/* executable name excluding path - access with [gs]et_task_comm (which lock
Signature · 8 lines
2.2.0.33.gc18b867
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH 1/8] x86, fpu: unlazy_fpu: don't reset thread.fpu_counter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 15 lines · Borislav Petkov
... and so by looking at the
unlazy_fpu()call sites, I think this makes sense.So how's that for a commit message instead:
--- x86, fpu,
unlazy_fpu: Don't reset thread.fpu_counterThe "else" branch clears ->fpu_counter as a remnant of the lazy FPU usage counting:
e07e23e1fd30("[PATCH] non lazy "sleazy" fpu implementation")However,
switch_fpu_prepare()does this now so that else branch is superfluous.
Quoted 6 lines
If we do
use_eager_fpu(), then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if__thread_has_fpu(). ---?
Good detective work.
Your changelog makes sense to me.
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU4i/NAAoJEM553pKExN6DWn4H/ArcSaAwA5ClOmxNACOTS644 m+8d/+3XZd+nWKBxLY2iyigLQ3FSek2dPSilDMSxtN97oFsG9mFw14gvP1/oR9OU 6vVhfuBWPRQ+Ay7CAL40v1acvzbYSTQJ6i9IJq20EFhuAS1CFwksOr9gQrhRHed1 FxFUpSnss0+p6Jp+RXY+UG2gNtSUOwgLzZFb1cmgeD5APC+RShGqHgDPTlNUESGK f9g3QzOD3KAz8fVNzTwumN1nJzPNx6yBXuazu3U8HAxzcMskVvWSOEyEBlo48CdR XxjLpI95qgzwUUQGFj6yax8ciq6nQGk6TsgqbdqHlpHSQ2zWU1F8XefGAS/r0vE= =CXAX
-----END PGP SIGNATURE-----
Re: [PATCH 1/8] x86, fpu: unlazy_fpu: don't reset thread.fpu_counter
Quoted 26 lines · Rik van Riel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 20 lines · Borislav Petkov
... and so by looking at the
unlazy_fpu()call sites, I think this makes sense.So how's that for a commit message instead:
--- x86, fpu,
unlazy_fpu: Don't reset thread.fpu_counterThe "else" branch clears ->fpu_counter as a remnant of the lazy FPU usage counting:
e07e23e1fd30("[PATCH] non lazy "sleazy" fpu implementation")However,
switch_fpu_prepare()does this now so that else branch is superfluous.If we do
use_eager_fpu(), then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if__thread_has_fpu(). ---
Quoted 5 lines
Quoted 1 line
?
Good detective work.
Your changelog makes sense to me.
And to me, thanks.
Should I resend?
Oleg.
Borislav Petkov
maintainerRe: [PATCH 1/8] x86, fpu: unlazy_fpu: don't reset thread.fpu_counter
Quoted 1 line · Oleg Nesterov
Should I resend?
No need, I've changed it before applying. :-)
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Oleg Nesterov
merge bot[tip:x86/fpu] x86/fpu: Don't reset thread.fpu_counter
In reply to [email protected]x86/fpu: Don't reset thread.fpu_counter
Commit-ID: a9241ea5fd709fc935dade130f4e3b2612bbe9e3 Gitweb: http://git.kernel.org/tip/a9241ea5fd709fc935dade130f4e3b2612bbe9e3 Author: Oleg Nesterov <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:01:58 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:12:40 +0100
x86/fpu: Don't reset thread.fpu_counter
The "else" branch clears ->fpu_counter as a remnant of the lazy FPU usage counting:
e07e23e1fd30 ("[PATCH] non lazy "sleazy" fpu implementation")
However, switch_fpu_prepare() does this now so that else branch is superfluous.
If we do use_eager_fpu(), then this has no effect. Otherwise, if we actually wanted to prevent fpu preload after the context switch we would need to reset it unconditionally, even if __thread_has_fpu().
--- arch/x86/kernel/i387.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kernel/i387.c +1 -2
@@ -108,8 +108,7 @@ void unlazy_fpu(struct task_struct *tsk)if (__thread_has_fpu (tsk)) {__save_init_fpu (tsk);__thread_fpu_end (tsk); - }else - tsk->thread.fpu_counter =0 ; + }preempt_enable (); }EXPORT_SYMBOL (unlazy_fpu);
PATCH 2/8 x86, fpu: unlazy_fpu: don't do __thread_fpu_end() if use_eager_fpu()
In reply to [email protected]From: Oleg Nesterov <[email protected]>
unlazy_fpu()->__thread_fpu_end() doesn't look right if use_eager_fpu(). Unconditional __thread_fpu_end() is only correct if we know that this thread can't return to user-mode and use FPU.
Fortunately it has only 2 callers. fpu_copy() checks use_eager_fpu(), and init_fpu(current) can be only called by the coredumping thread via regset->get(). But it is exported to modules, and imo this should be fixed anyway.
And if we check use_eager_fpu() we can use __save_fpu() like fpu_copy() and save_init_fpu() do.
- It seems that even !use_eager_fpu() case doesn't need the unconditional __thread_fpu_end(), we only need it if __save_init_fpu() returns 0.
- It is still not clear to me if __save_init_fpu() can safely nest with another save + restore from __kernel_fpu_begin(). If not, we can use kernel_fpu_disable() to fix the race.
--- arch/x86/kernel/i387.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/kernel/i387.c +6 -2
@@ -120,8 +120,12 @@ void unlazy_fpu(struct task_struct *tsk) {preempt_disable ();if (__thread_has_fpu (tsk)) { -__save_init_fpu (tsk); -__thread_fpu_end (tsk); +if (use_eager_fpu ()) { +__save_fpu (tsk); + }else { +__save_init_fpu (tsk); +__thread_fpu_end (tsk); + } }preempt_enable (); }
Signature · 1 line
1.9.3
Borislav Petkov
maintainerRe: [PATCH 2/8] x86, fpu: unlazy_fpu: don't do __thread_fpu_end() if use_eager_fpu()
Quoted 11 lines · On Fri, Feb 06, 2015 at 03:01:59PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
unlazy_fpu()->__thread_fpu_end() doesn't look right if
use_eager_fpu(). Unconditional__thread_fpu_end()is only correct if we know that this thread can't return to user-mode and use FPU.Fortunately it has only 2 callers.
fpu_copy()checksuse_eager_fpu(), andinit_fpu(current)can be only called by the coredumping thread viaregset->get(). But it is exported to modules, and imo this should be fixed anyway.
Quoted 5 lines
And if we check
use_eager_fpu()we can use__save_fpu()likefpu_copy()andsave_init_fpu()do.- It seems that even !use_eager_fpu() case doesn't need the unconditional
__thread_fpu_end(), we only need it if__save_init_fpu()returns 0.
I can follow so far.
Quoted 3 lines
- It is still not clear to me if
__save_init_fpu()can safely nest with another save + restore from__kernel_fpu_begin(). If not, we can usekernel_fpu_disable()to fix the race.
Well, my primitive understanding would say no, not safely, for the simple reason that we have only one XSAVE state area per thread. However, __kernel_fpu_begin() is called with preemption disabled so ... I guess I'm still not seeing the race.
Btw, what is kernel_fpu_disable()? Can't find it here.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH 2/8] x86, fpu: unlazy_fpu: don't do __thread_fpu_end() if use_eager_fpu()
Quoted 21 lines · Borislav Petkov
Quoted 11 lines · On Fri, Feb 06, 2015 at 03:01:59PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
unlazy_fpu()->__thread_fpu_end() doesn't look right if
use_eager_fpu(). Unconditional__thread_fpu_end()is only correct if we know that this thread can't return to user-mode and use FPU.Fortunately it has only 2 callers.
fpu_copy()checksuse_eager_fpu(), andinit_fpu(current)can be only called by the coredumping thread viaregset->get(). But it is exported to modules, and imo this should be fixed anyway.Quoted 5 lines
And if we check
use_eager_fpu()we can use__save_fpu()likefpu_copy()andsave_init_fpu()do.- It seems that even !use_eager_fpu() case doesn't need the unconditional
__thread_fpu_end(), we only need it if__save_init_fpu()returns 0.I can follow so far.
Quoted 8 lines
Quoted 3 lines
- It is still not clear to me if
__save_init_fpu()can safely nest with another save + restore from__kernel_fpu_begin(). If not, we can usekernel_fpu_disable()to fix the race.Well, my primitive understanding would say no, not safely, for the simple reason that we have only one XSAVE state area per thread. However,
__kernel_fpu_begin()is called with preemption disabled so ... I guess I'm still not seeing the race.
This is not about preemption. But let me first say that I do not know how the FPU hardware actually works, and I do not understand the FPU asm code at all.
Let's look at this code
if (__thread_has_fpu(tsk)) {
__save_init_fpu(tsk); // interrupt -> kernel_fpu_begin()
__thread_fpu_end(tsk);
}Suppose that kernel_fpu_begin() from interrupt races with __save_init_fpu() in progress. Is this safe? I do not know.
My concern is that (I think) __save_init_fpu() can save the FPU state _and_ modify it (say, it can reset some register to default value). This means that the nested __save_init_fpu() from __kernel_fpu_begin() can save the modified register again to current->thread.fpu.
If my understanding is wrong, then why switch_fpu_prepare() clears .last_cpu if __save_init_fpu() returns 0 (which iiuc means that CPU's state does not match the saved state) ?
Plus I have other (more vague) concerns...
Quoted 1 line
Btw, what is
kernel_fpu_disable()? Can't find it here.
It's already in Linus's tree, see
14e153ef75eecae8fd0738ffb42120f4962a00cd x86, fpu: Introduce per-cpu in_kernel_fpu state 33a3ebdc077fd85f1bf4d4586eea579b297461ae x86, fpu: Don't abuse has_fpu in __kernel_fpu_begin/end() 7575637ab293861a799f3bbafe0d8c597389f4e9 x86, fpu: Fix math_state_restore() race with kernel_fpu_begin()
And, Borislav, thanks for looking at this!
Oleg.
Borislav Petkov
maintainerRe: [PATCH 2/8] x86, fpu: unlazy_fpu: don't do __thread_fpu_end() if use_eager_fpu()
Quoted 26 lines · Oleg Nesterov
Quoted 25 lines · Borislav Petkov
Quoted 11 lines · On Fri, Feb 06, 2015 at 03:01:59PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
unlazy_fpu()->__thread_fpu_end() doesn't look right if
use_eager_fpu(). Unconditional__thread_fpu_end()is only correct if we know that this thread can't return to user-mode and use FPU.Fortunately it has only 2 callers.
fpu_copy()checksuse_eager_fpu(), andinit_fpu(current)can be only called by the coredumping thread viaregset->get(). But it is exported to modules, and imo this should be fixed anyway.Quoted 5 lines
And if we check
use_eager_fpu()we can use__save_fpu()likefpu_copy()andsave_init_fpu()do.- It seems that even !use_eager_fpu() case doesn't need the unconditional
__thread_fpu_end(), we only need it if__save_init_fpu()returns 0.I can follow so far.
Quoted 3 lines
- It is still not clear to me if
__save_init_fpu()can safely nest with another save + restore from__kernel_fpu_begin(). If not, we can usekernel_fpu_disable()to fix the race.
Quoted 8 lines
Quoted 4 lines
Well, my primitive understanding would say no, not safely, for the simple reason that we have only one XSAVE state area per thread. However,
__kernel_fpu_begin()is called with preemption disabled so ... I guess I'm still not seeing the race.This is not about preemption. But let me first say that I do not know how the FPU hardware actually works, and I do not understand the FPU asm code at all.
Same here :-P
Quoted 9 lines
Let's look at this code
if (__thread_has_fpu(tsk)) { __save_init_fpu(tsk); // interrupt -> kernel_fpu_begin() __thread_fpu_end(tsk); }Suppose that
kernel_fpu_begin()from interrupt races with__save_init_fpu()in progress. Is this safe? I do not know.
I guess this was the reason for
WARN_ON_ONCE(!irq_fpu_usable());
in kernel_fpu_begin(). And even that solution is not sufficient. Check out this thread:
https://lkml.kernel.org/r/[email protected]
for similar issues with the current handling of FPU state.
Quoted 4 lines
My concern is that (I think)
__save_init_fpu()can save the FPU state _and_ modify it (say, it can reset some register to default value). This means that the nested__save_init_fpu()from__kernel_fpu_begin()can save the modified register again tocurrent->thread.fpu.
Oh ok, I see what you mean.
Quoted 3 lines
If my understanding is wrong, then why
switch_fpu_prepare()clears .last_cpu if__save_init_fpu()returns 0 (which iiuc means that CPU's state does not match the saved state) ?
Right, this should be the reason why fpu_save_init() signals intact FPU state with its retval.
Although I don't know what that XSTATE_FP bit checking is supposed to mean as manuals say XCR0[0] is always 1. And nothing says that XSAVE would ever clear it...
But the FNSAVE case, for example and IMHO, says that after FWAIT which would raise unmasked FPU exceptions and thus clear FPU control register bits...
Long story short, yeah, we better prevent concurrent __save_init_fpu() from happening...
Quoted 1 line
Plus I have other (more vague) concerns...
Same here. :-\
Our FPU code is nuts and misses a lot of comments.
Quoted 7 lines
Quoted 1 line
Btw, what is
kernel_fpu_disable()? Can't find it here.It's already in Linus's tree, see
14e153ef75eecae8fd0738ffb42120f4962a00cdx86, fpu: Introduce per-cpuin_kernel_fpustate33a3ebdc077fd85f1bf4d4586eea579b297461aex86, fpu: Don't abusehas_fpuin __kernel_fpu_begin/end()7575637ab293861a799f3bbafe0d8c597389f4e9x86, fpu: Fixmath_state_restore()race withkernel_fpu_begin()
Ah, and I was working on a plain 3.19, that's why I didn't see it. I'll rebase your patches then.
Quoted 1 line
And, Borislav, thanks for looking at this!
Sure. Thanks for the patience and explaining stuff to me! :)
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Oleg Nesterov
merge bot[tip:x86/fpu] x86/fpu: Don't do __thread_fpu_end() if use_eager_fpu()
In reply to [email protected]x86/fpu: Don't do __thread_fpu_end() if use_eager_fpu()
Commit-ID: 1a2a7f4ec8e3a7ac582dac4d01fcc7e8acd3bb30 Gitweb: http://git.kernel.org/tip/1a2a7f4ec8e3a7ac582dac4d01fcc7e8acd3bb30 Author: Oleg Nesterov <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:01:59 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:12:46 +0100
x86/fpu: Don't do __thread_fpu_end() if use_eager_fpu()
unlazy_fpu()->__thread_fpu_end() doesn't look right if use_eager_fpu(). Unconditional __thread_fpu_end() is only correct if we know that this thread can't return to user-mode and use FPU.
Fortunately it has only 2 callers. fpu_copy() checks use_eager_fpu(), and init_fpu(current) can be only called by the coredumping thread via regset->get(). But it is exported to modules, and imo this should be fixed anyway.
And if we check use_eager_fpu() we can use __save_fpu() like fpu_copy() and save_init_fpu() do.
- It seems that even !use_eager_fpu() case doesn't need the unconditional __thread_fpu_end(), we only need it if __save_init_fpu() returns 0.
- It is still not clear to me if __save_init_fpu() can safely nest with another save + restore from __kernel_fpu_begin(). If not, we can use kernel_fpu_disable() to fix the race.
--- arch/x86/kernel/i387.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/kernel/i387.c +6 -2
@@ -106,8 +106,12 @@ void unlazy_fpu(struct task_struct *tsk) {preempt_disable ();if (__thread_has_fpu (tsk)) { -__save_init_fpu (tsk); -__thread_fpu_end (tsk); +if (use_eager_fpu ()) { +__save_fpu (tsk); + }else { +__save_init_fpu (tsk); +__thread_fpu_end (tsk); + } }preempt_enable (); }
PATCH 3/8 x86, fpu: kill save_init_fpu(), change math_error() to use unlazy_fpu()
In reply to [email protected]From: Oleg Nesterov <[email protected]>
math_error() calls save_init_fpu() after conditional_sti(), this means that the caller can be preempted. If !use_eager_fpu() we can hit the WARN_ON_ONCE(!__thread_has_fpu(tsk)) and/or save the wrong FPU state.
Change math_error() to use unlazy_fpu() and kill save_init_fpu().
--- arch/x86/include/asm/fpu-internal.h | 18 ------------------ arch/x86/kernel/traps.c | 2 +- 2 files changed, 1 insertion(+), 19 deletions(-)
arch/x86/include/asm/fpu-internal.h +0 -18
@@ -520,24 +520,6 @@ static inline void __save_fpu(struct task_struct *tsk) }/* -* These disable preemption on their own and are safe -*/ -static inline void save_init_fpu (struct task_struct *tsk) -{ -WARN_ON_ONCE (!__thread_has_fpu (tsk)); - -if (use_eager_fpu ()) { -__save_fpu (tsk); -return ; - } - -preempt_disable (); -__save_init_fpu (tsk); -__thread_fpu_end (tsk); -preempt_enable (); -} - -/* * i387 state interaction */ static inline unsigned short get_fpu_cwd (struct task_struct *tsk)
arch/x86/kernel/traps.c +1 -1
@@ -663,7 +663,7 @@ static void math_error(struct pt_regs *regs, int error_code, int trapnr)/* * Save the info for the exception handler and clear the error. */ -save_init_fpu (task); +unlazy_fpu (task); task->thread.trap_nr = trapnr; task->thread.error_code = error_code; info.si_signo =SIGFPE ;
Signature · 1 line
1.9.3
Borislav Petkov
maintainerRe: [PATCH 3/8] x86, fpu: kill save_init_fpu(), change math_error() to use unlazy_fpu()
Quoted 49 lines · On Fri, Feb 06, 2015 at 03:02:00PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
math_error()callssave_init_fpu()afterconditional_sti(), this means that the caller can be preempted. If !use_eager_fpu() we can hit the WARN_ON_ONCE(!__thread_has_fpu(tsk)) and/or save the wrong FPU state.Change
math_error()to useunlazy_fpu()and killsave_init_fpu().Signed-off-by Oleg Nesterov <[email protected]>Signed-off-by Rik van Riel <[email protected]>--- arch/x86/include/asm/fpu-internal.h | 18 ------------------ arch/x86/kernel/traps.c | 2 +- 2 files changed, 1 insertion(+), 19 deletions(-)arch/x86/include/asm/fpu-internal.h +0
-18@@ -520,24 +520,6 @@ static inline void __save_fpu(struct task_struct *tsk) }/* -* These disable preemption on their own and are safe -*/ -static inline void save_init_fpu (struct task_struct *tsk) -{ -WARN_ON_ONCE (!__thread_has_fpu (tsk)); - -if (use_eager_fpu ()) { -__save_fpu (tsk); -return ; - } - -preempt_disable (); -__save_init_fpu (tsk); -__thread_fpu_end (tsk); -preempt_enable (); -} - -/* * i387 state interaction */ static inline unsigned short get_fpu_cwd (struct task_struct *tsk)arch/x86/kernel/traps.c +0
-0
arch/x86/kernel/traps.c static void math_error(struct pt_regs *regs, int error_code, int trapnr) · line 666
Quoted 5 lines
arch/x86/kernel/traps.c +1
-1@@ -663,7 +663,7 @@ static void math_error(struct pt_regs *regs, int error_code, int trapnr)/* * Save the info for the exception handler and clear the error. */ -save_init_fpu (task); +unlazy_fpu (task);
Do I see it correctly that even with this there's a not-so-small hole *after* conditional_sti() and *before* unlazy_fpu() where caller can still get preempted?
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH 3/8] x86, fpu: kill save_init_fpu(), change math_error() to use unlazy_fpu()
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 14 lines · Borislav Petkov
Quoted 9 lines · On Fri, Feb 06, 2015 at 03:02:00PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
arch/x86/kernel/traps.c +0
-0
a/arch/x86/kernel/traps.c+++b/arch/x86/kernel/traps.c@@ -663,7 +663,7 @@ static voidmath_error(struct pt_regs *regs, int error_code, int trapnr)/* * Save the info for the exception
handler and clear the error. */ -save_init_fpu(task); +unlazy_fpu(task);Do I see it correctly that even with this there's a not-so-small hole *after*
conditional_sti()and *before*unlazy_fpu()where caller can still get preempted?
That's ok, the context switch will save the register contents to memory in that case. At that point unlazy_fpu will potentially do nothing, and the task will process the FPU context that was saved to memory previously.
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU4mGKAAoJEM553pKExN6DHoAH/1R3Pak5bZrvnCEBd3lp9/uD bqtms5VjiWNNhlG0Hpob0eS71p8fO9La8+hPwFQJdK9yZ2lwvrDEXMCMGOG1P8by HHbtxS74tUIm49qlIIUNxoiI+HMDnOQNhYv9dA4VdTMcY//2Asr6g4LM3wFH+f1q tZJ5CU5/f0g2SCtTEzXxZDQdlqgpGXpvcGVymENe45uzn5ZYgcbCAiix+Dy5SiCw +GjURoRW07iRSvK48w8vTVstTwl2WKgHpWEGVGhL1RyJY6aMIft3b4FH1jUrhVVJ w3ncQAptA2ry/anU/6A2aTrZ646ebJuYejhF99gwTucolJ3netdX7xohOaFEpSc= =/rU9
-----END PGP SIGNATURE-----
Re: [PATCH 3/8] x86, fpu: kill save_init_fpu(), change math_error() to use unlazy_fpu()
Quoted 20 lines · Rik van Riel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 14 lines · Borislav Petkov
Quoted 9 lines · On Fri, Feb 06, 2015 at 03:02:00PM -0500, [email protected] wrote:
From: Oleg Nesterov <[email protected]>
arch/x86/kernel/traps.c +0
-0
a/arch/x86/kernel/traps.c+++b/arch/x86/kernel/traps.c@@ -663,7 +663,7 @@ static voidmath_error(struct pt_regs *regs, int error_code, int trapnr)/* * Save the info for the exception
handler and clear the error. */ -save_init_fpu(task); +unlazy_fpu(task);Do I see it correctly that even with this there's a not-so-small hole *after*
conditional_sti()and *before*unlazy_fpu()where caller can still get preempted?
Quoted 2 lines
That's ok, the context switch will save the register contents to memory in that case.
Yes, thanks.
Quoted 3 lines
At that point
unlazy_fpuwill potentially
do nothing, and the task will process the FPU context that was saved to memory previously.
Or, if __thread_has_fpu() will be true again, it will save the registers again.
And this equally applies to any other user of unlazy_fpu().
Oleg.
tip-bot for Oleg Nesterov
merge bot[tip:x86/fpu] x86/fpu: Change math_error() to use unlazy_fpu(), kill (now) unused save_init_fpu()
In reply to [email protected]x86/fpu: Change math_error() to use unlazy_fpu(), kill (now) unused save_init_fpu()
Commit-ID: 08a744c6bfded3d5fa66f94263f81773226113d1 Gitweb: http://git.kernel.org/tip/08a744c6bfded3d5fa66f94263f81773226113d1 Author: Oleg Nesterov <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:00 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:03 +0100
x86/fpu: Change math_error() to use unlazy_fpu(), kill (now) unused save_init_fpu()
math_error() calls save_init_fpu() after conditional_sti(), this means that the caller can be preempted. If !use_eager_fpu() we can hit the WARN_ON_ONCE(!__thread_has_fpu(tsk)) and/or save the wrong FPU state.
Change math_error() to use unlazy_fpu() and kill save_init_fpu().
--- arch/x86/include/asm/fpu-internal.h | 18 ------------------ arch/x86/kernel/traps.c | 2 +- 2 files changed, 1 insertion(+), 19 deletions(-)
arch/x86/include/asm/fpu-internal.h +0 -18
@@ -518,24 +518,6 @@ static inline void __save_fpu(struct task_struct *tsk) }/* -* These disable preemption on their own and are safe -*/ -static inline void save_init_fpu (struct task_struct *tsk) -{ -WARN_ON_ONCE (!__thread_has_fpu (tsk)); - -if (use_eager_fpu ()) { -__save_fpu (tsk); -return ; - } - -preempt_disable (); -__save_init_fpu (tsk); -__thread_fpu_end (tsk); -preempt_enable (); -} - -/* * i387 state interaction */ static inline unsigned short get_fpu_cwd (struct task_struct *tsk)
arch/x86/kernel/traps.c +1 -1
@@ -663,7 +663,7 @@ static void math_error(struct pt_regs *regs, int error_code, int trapnr)/* * Save the info for the exception handler and clear the error. */ -save_init_fpu (task); +unlazy_fpu (task); task->thread.trap_nr = trapnr; task->thread.error_code = error_code; info.si_signo =SIGFPE ;
PATCH 4/8 x86,fpu: move lazy restore functions up a few lines
In reply to [email protected]From: Rik van Riel <[email protected]>
We need another lazy restore related function, that will be called from a function that is above where the lazy restore functions are now. It would be nice to keep all three functions grouped together.
--- arch/x86/include/asm/fpu-internal.h | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
arch/x86/include/asm/fpu-internal.h +18 -18
@@ -67,6 +67,24 @@ extern void finit_soft_fpu(struct i387_soft_struct *soft);static inline void finit_soft_fpu (struct i387_soft_struct *soft) {}#endif +/* +* Must be run with preemption disabled: this clears the fpu_owner_task, +* on this CPU. +* +* This will disable any lazy FPU state restore of the current FPU state, +* but if the current thread owns the FPU, it will still be saved by. +*/ +static inline void __cpu_disable_lazy_restore (unsigned int cpu) +{ +per_cpu (fpu_owner_task, cpu) =NULL ; +} + +static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) +{ +return new ==this_cpu_read_stable (fpu_owner_task) && + cpu == new->thread.fpu.last_cpu; +} +static inline int is_ia32_compat_frame (void ) { return config_enabled(CONFIG_IA32_EMULATION) &&
@@ -400,24 +418,6 @@ static inline void drop_init_fpu(struct task_struct *tsk)*/ typedef struct {int preload; }fpu_switch_t ; -/* -* Must be run with preemption disabled: this clears the fpu_owner_task, -* on this CPU. -* -* This will disable any lazy FPU state restore of the current FPU state, -* but if the current thread owns the FPU, it will still be saved by. -*/ -static inline void __cpu_disable_lazy_restore (unsigned int cpu) -{ -per_cpu (fpu_owner_task, cpu) =NULL ; -} - -static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) -{ -return new ==this_cpu_read_stable (fpu_owner_task) && - cpu == new->thread.fpu.last_cpu; -} -static inline fpu_switch_t switch_fpu_prepare (struct task_struct *old,struct task_struct *new,int cpu) {fpu_switch_t fpu;
Signature · 1 line
1.9.3
tip-bot for Rik van Riel
merge bot[tip:x86/fpu] x86/fpu: Move lazy restore functions up a few lines
x86/fpu: Move lazy restore functions up a few lines
Commit-ID: 1c927eea4cad83c439cb51e9c96ad19cb005157d Gitweb: http://git.kernel.org/tip/1c927eea4cad83c439cb51e9c96ad19cb005157d Author: Rik van Riel <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:01 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:53 +0100
x86/fpu: Move lazy restore functions up a few lines
We need another lazy restore related function, that will be called from a function that is above where the lazy restore functions are now. It would be nice to keep all three functions grouped together.
--- arch/x86/include/asm/fpu-internal.h | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-)
arch/x86/include/asm/fpu-internal.h +18 -18
@@ -67,6 +67,24 @@ extern void finit_soft_fpu(struct i387_soft_struct *soft);static inline void finit_soft_fpu (struct i387_soft_struct *soft) {}#endif +/* +* Must be run with preemption disabled: this clears the fpu_owner_task, +* on this CPU. +* +* This will disable any lazy FPU state restore of the current FPU state, +* but if the current thread owns the FPU, it will still be saved by. +*/ +static inline void __cpu_disable_lazy_restore (unsigned int cpu) +{ +per_cpu (fpu_owner_task, cpu) =NULL ; +} + +static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) +{ +return new ==this_cpu_read_stable (fpu_owner_task) && + cpu == new->thread.fpu.last_cpu; +} +static inline int is_ia32_compat_frame (void ) { return config_enabled(CONFIG_IA32_EMULATION) &&
@@ -398,24 +416,6 @@ static inline void drop_init_fpu(struct task_struct *tsk)*/ typedef struct {int preload; }fpu_switch_t ; -/* -* Must be run with preemption disabled: this clears the fpu_owner_task, -* on this CPU. -* -* This will disable any lazy FPU state restore of the current FPU state, -* but if the current thread owns the FPU, it will still be saved by. -*/ -static inline void __cpu_disable_lazy_restore (unsigned int cpu) -{ -per_cpu (fpu_owner_task, cpu) =NULL ; -} - -static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) -{ -return new ==this_cpu_read_stable (fpu_owner_task) && - cpu == new->thread.fpu.last_cpu; -} -static inline fpu_switch_t switch_fpu_prepare (struct task_struct *old,struct task_struct *new,int cpu) {fpu_switch_t fpu;
PATCH 5/8 x86,fpu: introduce task_disable_lazy_fpu_restore helper
In reply to [email protected]From: Rik van Riel <[email protected]>
Currently there are a few magic assignments sprinkled through the code that disable lazy FPU state restoring, some more effective than others, and all equally mystifying.
It would be easier to have a helper to explicitly disable lazy FPU state restoring for a task.
--- arch/x86/include/asm/fpu-internal.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/include/asm/fpu-internal.h +10 -0
@@ -79,6 +79,16 @@ static inline void __cpu_disable_lazy_restore(unsigned int cpu)per_cpu (fpu_owner_task, cpu) =NULL ; } +/* +* Used to indicate that the FPU state in memory is newer than the FPU +* state in registers, and the FPU state should be reloaded next time the +* task is run. Only safe on the current task, or non-running tasks. +*/ +static inline void task_disable_lazy_fpu_restore (struct task_struct *tsk) +{ + tsk->thread.fpu.last_cpu = ~0 ; +} +static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) { return new == this_cpu_read_stable(fpu_owner_task) &&
Signature · 1 line
1.9.3
tip-bot for Rik van Riel
merge bot[tip:x86/fpu] x86/fpu: Introduce task_disable_lazy_fpu_restore() helper
x86/fpu: Introduce task_disable_lazy_fpu_restore() helper
Commit-ID: 33e03dedd759cc9396252d9641b25d01909a26bb Gitweb: http://git.kernel.org/tip/33e03dedd759cc9396252d9641b25d01909a26bb Author: Rik van Riel <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:02 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:53 +0100
x86/fpu: Introduce task_disable_lazy_fpu_restore() helper
Currently there are a few magic assignments sprinkled through the code that disable lazy FPU state restoring, some more effective than others, and all equally mystifying.
It would be easier to have a helper to explicitly disable lazy FPU state restoring for a task.
--- arch/x86/include/asm/fpu-internal.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/include/asm/fpu-internal.h +10 -0
@@ -79,6 +79,16 @@ static inline void __cpu_disable_lazy_restore(unsigned int cpu)per_cpu (fpu_owner_task, cpu) =NULL ; } +/* +* Used to indicate that the FPU state in memory is newer than the FPU +* state in registers, and the FPU state should be reloaded next time the +* task is run. Only safe on the current task, or non-running tasks. +*/ +static inline void task_disable_lazy_fpu_restore (struct task_struct *tsk) +{ + tsk->thread.fpu.last_cpu = ~0 ; +} +static inline int fpu_lazy_restore (struct task_struct *new,unsigned int cpu) { return new == this_cpu_read_stable(fpu_owner_task) &&
PATCH 6/8 x86,fpu: use an explicit if/else in switch_fpu_prepare
In reply to [email protected]From: Rik van Riel <[email protected]>
Use an explicit if/else branch after __save_init_fpu(old) in switch_fpu_prepare. This makes substituting the assignment with a call to task_disable_lazy_fpu() in the next patch easier to review.
--- arch/x86/include/asm/fpu-internal.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/include/asm/fpu-internal.h +3 -2
@@ -440,8 +440,9 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct ta new->thread.fpu_counter > 5);if (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - cpu = ~0 ; - old->thread.fpu.last_cpu = cpu; + old->thread.fpu.last_cpu = ~0 ; +else + old->thread.fpu.last_cpu = cpu; old->thread.fpu.has_fpu =0 ;/* But leave fpu_owner_task! */ /* Don't change CR0.TS if we just switch! */
Signature · 1 line
1.9.3
Borislav Petkov
maintainerRe: [PATCH 6/8] x86,fpu: use an explicit if/else in switch_fpu_prepare
Quoted 8 lines · On Fri, Feb 06, 2015 at 03:02:03PM -0500, [email protected] wrote:
From: Rik van Riel <[email protected]>
Use an explicit if/else branch after
__save_init_fpu(old)inswitch_fpu_prepare. This makes substituting the assignment with a call totask_disable_lazy_fpu()in the next patch easier to review.Signed-off-by Rik van Riel <[email protected]>
I did space-out stuff a bit ontop, it looks much more readable to me this way and that fpu.preload assignment is actually parseable at a quick glance now :-)
---
From: Rik van Riel <[email protected]>
Subject: [PATCH] x86, fpu: Use an explicit if/else in switch_fpu_prepare
Use an explicit if/else branch after __save_init_fpu(old) in switch_fpu_prepare. This makes substituting the assignment with a call in task_disable_lazy_fpu_restore() in the next patch easier to review.
--- arch/x86/include/asm/fpu-internal.h | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu-internal.h +9 -5
@@ -434,13 +434,17 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct ta* If the task has used the math, pre-load the FPU on xsave processors * or if the past 5 consecutive context-switches used math. */ - fpu.preload =tsk_used_math (new) && (use_eager_fpu () || - new->thread.fpu_counter >5 ); + fpu.preload =tsk_used_math (new) && + (use_eager_fpu () || new->thread.fpu_counter >5 ); +if (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - cpu = ~0 ; - old->thread.fpu.last_cpu = cpu; - old->thread.fpu.has_fpu =0 ;/* But leave fpu_owner_task! */ + old->thread.fpu.last_cpu = ~0 ; +else + old->thread.fpu.last_cpu = cpu; + +/* But leave fpu_owner_task! */ + old->thread.fpu.has_fpu =0 ;/* Don't change CR0.TS if we just switch! */ if (fpu.preload) {
Signature · 8 lines
2.2.0.33.gc18b867
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Rik van Riel
merge bot[tip:x86/fpu] x86/fpu: Use an explicit if/ else in switch_fpu_prepare()
In reply to [email protected]x86/fpu: Use an explicit if/else in switch_fpu_prepare()
Commit-ID: 1361ef29c7e49ae7cf37220c25fac1904b77f71a Gitweb: http://git.kernel.org/tip/1361ef29c7e49ae7cf37220c25fac1904b77f71a Author: Rik van Riel <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:03 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:54 +0100
x86/fpu: Use an explicit if/else in switch_fpu_prepare()
Use an explicit if/else branch after __save_init_fpu(old) in switch_fpu_prepare(). This makes substituting the assignment with a call in task_disable_lazy_fpu_restore() in the next patch easier to review.
--- arch/x86/include/asm/fpu-internal.h | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu-internal.h +9 -5
@@ -434,13 +434,17 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct ta* If the task has used the math, pre-load the FPU on xsave processors * or if the past 5 consecutive context-switches used math. */ - fpu.preload =tsk_used_math (new) && (use_eager_fpu () || - new->thread.fpu_counter >5 ); + fpu.preload =tsk_used_math (new) && + (use_eager_fpu () || new->thread.fpu_counter >5 ); +if (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - cpu = ~0 ; - old->thread.fpu.last_cpu = cpu; - old->thread.fpu.has_fpu =0 ;/* But leave fpu_owner_task! */ + old->thread.fpu.last_cpu = ~0 ; +else + old->thread.fpu.last_cpu = cpu; + +/* But leave fpu_owner_task! */ + old->thread.fpu.has_fpu =0 ;/* Don't change CR0.TS if we just switch! */ if (fpu.preload) {
PATCH 7/8 x86,fpu: use disable_task_lazy_fpu_restore helper
In reply to [email protected]From: Rik van Riel <[email protected]>
Replace magic assignments of fpu.last_cpu = ~0 with more explicit disable_task_lazy_fpu_restore calls.
This also fixes the lazy FPU restore disabling in drop_fpu, which only really works when !use_eager_fpu(). This is fine for now, because fpu_lazy_restore() is only used when !use_eager_fpu() currently, but we may want to expand that.
--- arch/x86/include/asm/fpu-internal.h | 4 ++-- arch/x86/kernel/i387.c | 2 +- arch/x86/kernel/process.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu-internal.h +2 -2
@@ -440,7 +440,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct ta new->thread.fpu_counter > 5);if (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);else old->thread.fpu.last_cpu = cpu; old->thread.fpu.has_fpu =0 ;/* But leave fpu_owner_task! */
@@ -454,7 +454,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct tastts (); }else { old->thread.fpu_counter =0 ; - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);if (fpu.preload) { new->thread.fpu_counter++;if (!use_eager_fpu () &&fpu_lazy_restore (new, cpu))
arch/x86/kernel/i387.c +1 -1
@@ -250,7 +250,7 @@ int init_fpu(struct task_struct *tsk)if (tsk_used_math (tsk)) {if (cpu_has_fpu && tsk == current)unlazy_fpu (tsk); - tsk->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (tsk);return 0 ; }
arch/x86/kernel/process.c +1 -1
@@ -68,8 +68,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.fpu_counter =0 ; dst->thread.fpu.has_fpu =0 ; - dst->thread.fpu.last_cpu = ~0 ; dst->thread.fpu.state =NULL ; +task_disable_lazy_fpu_restore (dst);if (tsk_used_math (src)) {int err =fpu_alloc (&dst->thread.fpu);if (err)
Signature · 1 line
1.9.3
Borislav Petkov
maintainerRe: [PATCH 7/8] x86,fpu: use disable_task_lazy_fpu_restore helper
Quoted 9 lines · On Fri, Feb 06, 2015 at 03:02:04PM -0500, [email protected] wrote:
From: Rik van Riel <[email protected]>
Replace magic assignments of fpu.last_cpu = ~0 with more explicit
disable_task_lazy_fpu_restorecalls.This also fixes the lazy FPU restore disabling in
drop_fpu, which only really works when !use_eager_fpu(). This is fine for now, becausefpu_lazy_restore()is only used when !use_eager_fpu() currently, but we may want to expand that.
Sorry, I can't follow here. The only non-trivial change below is in arch_dup_task_struct(). Please clarify.
[ leaving in the rest for context ]
Quoted 66 lines
Signed-off-by Rik van Riel <[email protected]>--- arch/x86/include/asm/fpu-internal.h | 4 ++-- arch/x86/kernel/i387.c | 2 +- arch/x86/kernel/process.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)arch/x86/include/asm/fpu-internal.h +2
-2@@ -440,7 +440,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct ta new->thread.fpu_counter > 5);if (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);else old->thread.fpu.last_cpu = cpu; old->thread.fpu.has_fpu =0 ;/* But leave fpu_owner_task! */ @@ -454,7 +454,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct tastts (); }else { old->thread.fpu_counter =0 ; - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);if (fpu.preload) { new->thread.fpu_counter++;if (!use_eager_fpu () &&fpu_lazy_restore (new, cpu))arch/x86/kernel/i387.c +1
-1@@ -250,7 +250,7 @@ int init_fpu(struct task_struct *tsk)if (tsk_used_math (tsk)) {if (cpu_has_fpu && tsk == current)unlazy_fpu (tsk); - tsk->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (tsk);return 0 ; }arch/x86/kernel/process.c +1
-1@@ -68,8 +68,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.fpu_counter =0 ; dst->thread.fpu.has_fpu =0 ; - dst->thread.fpu.last_cpu = ~0 ; dst->thread.fpu.state =NULL ; +task_disable_lazy_fpu_restore (dst);if (tsk_used_math (src)) {int err =fpu_alloc (&dst->thread.fpu);if (err)Signature · 7 lines
1.9.3 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--Re: [PATCH 7/8] x86,fpu: use disable_task_lazy_fpu_restore helper
Quoted 7 lines · Borislav Petkov
Quoted 5 lines · On Fri, Feb 06, 2015 at 03:02:04PM -0500, [email protected] wrote:
From: Rik van Riel <[email protected]>
Replace magic assignments of fpu.last_cpu = ~0 with more explicit
disable_task_lazy_fpu_restorecalls.
Quoted 7 lines
Quoted 4 lines
This also fixes the lazy FPU restore disabling in
drop_fpu, which only really works when !use_eager_fpu(). This is fine for now, becausefpu_lazy_restore()is only used when !use_eager_fpu() currently, but we may want to expand that.Sorry, I can't follow here. The only non-trivial change below is in
arch_dup_task_struct(). Please clarify.
I guess Rik forgot to update the changelog ;)
Initial version of this patch played with drop_fpu() as well, this part was removed after discussion. So I guess the last sentence in the changelog should go away too.
Oleg.
Borislav Petkov
maintainerRe: [PATCH 7/8] x86,fpu: use disable_task_lazy_fpu_restore helper
Quoted 3 lines · Oleg Nesterov
Initial version of this patch played with
drop_fpu()as well, this part was removed after discussion. So I guess the last sentence in the changelog should go away too.
Ok, removed.
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--tip-bot for Rik van Riel
merge bot[tip:x86/fpu] x86/fpu: Use task_disable_lazy_fpu_restore() helper
In reply to [email protected]x86/fpu: Use task_disable_lazy_fpu_restore() helper
Commit-ID: 6a5fe8952bd676baf382d14df21e7b32b5d8943e Gitweb: http://git.kernel.org/tip/6a5fe8952bd676baf382d14df21e7b32b5d8943e Author: Rik van Riel <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:04 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:55 +0100
x86/fpu: Use task_disable_lazy_fpu_restore() helper
Replace magic assignments of fpu.last_cpu = ~0 with more explicit task_disable_lazy_fpu_restore() calls.
--- arch/x86/include/asm/fpu-internal.h | 4 ++-- arch/x86/kernel/i387.c | 2 +- arch/x86/kernel/process.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu-internal.h +2 -2
@@ -439,7 +439,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct taif (__thread_has_fpu (old)) {if (!__save_init_fpu (old)) - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);else old->thread.fpu.last_cpu = cpu;
@@ -455,7 +455,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct tastts (); }else { old->thread.fpu_counter =0 ; - old->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (old);if (fpu.preload) { new->thread.fpu_counter++;if (!use_eager_fpu () &&fpu_lazy_restore (new, cpu))
arch/x86/kernel/i387.c +1 -1
@@ -236,7 +236,7 @@ int init_fpu(struct task_struct *tsk)if (tsk_used_math (tsk)) {if (cpu_has_fpu && tsk == current)unlazy_fpu (tsk); - tsk->thread.fpu.last_cpu = ~0 ; +task_disable_lazy_fpu_restore (tsk);return 0 ; }
arch/x86/kernel/process.c +1 -1
@@ -68,8 +68,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.fpu_counter =0 ; dst->thread.fpu.has_fpu =0 ; - dst->thread.fpu.last_cpu = ~0 ; dst->thread.fpu.state =NULL ; +task_disable_lazy_fpu_restore (dst);if (tsk_used_math (src)) {int err =fpu_alloc (&dst->thread.fpu);if (err)
PATCH 8/8 x86,fpu: also check fpu_lazy_restore when use_eager_fpu
In reply to [email protected]From: Rik van Riel <[email protected]>
With Oleg's patch "x86, fpu: don't abuse FPU in kernel threads if use_eager_fpu()", kernel threads no longer have an FPU state even on systems with use_eager_fpu()
That in turn means that a task may still have its FPU state loaded in the FPU registers, if the task only got interrupted by kernel threads from when it went to sleep, to when it woke up again.
In that case, there is no need to restore the FPU state for this task, since it is still in the registers.
The kernel can simply use the same logic to determine this as is used for !use_eager_fpu() systems.
--- arch/x86/include/asm/fpu-internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/fpu-internal.h +1 -1
@@ -457,7 +457,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct tatask_disable_lazy_fpu_restore (old);if (fpu.preload) { new->thread.fpu_counter++; -if (!use_eager_fpu () &&fpu_lazy_restore (new, cpu)) +if (fpu_lazy_restore (new, cpu)) fpu.preload =0 ;else prefetch (new->thread.fpu.state);
Signature · 1 line
1.9.3
tip-bot for Rik van Riel
merge bot[tip:x86/fpu] x86/fpu: Also check fpu_lazy_restore() when use_eager_fpu()
x86/fpu: Also check fpu_lazy_restore() when use_eager_fpu()
Commit-ID: 728e53fef429a0f3c9dda3587c3ccc57ad268b70 Gitweb: http://git.kernel.org/tip/728e53fef429a0f3c9dda3587c3ccc57ad268b70 Author: Rik van Riel <[email protected]> AuthorDate: Fri, 6 Feb 2015 15:02:05 -0500 Committer: Borislav Petkov <[email protected]> CommitDate: Thu, 19 Feb 2015 11:15:55 +0100
x86/fpu: Also check fpu_lazy_restore() when use_eager_fpu()
With Oleg's patch:
33a3ebdc077f ("x86, fpu: Don't abuse has_fpu in __kernel_fpu_begin/end()")
kernel threads no longer have an FPU state even on systems with use_eager_fpu().
That in turn means that a task may still have its FPU state loaded in the FPU registers, if the task only got interrupted by kernel threads from when it went to sleep, to when it woke up again.
In that case, there is no need to restore the FPU state for this task, since it is still in the registers.
The kernel can simply use the same logic to determine this as is used for !use_eager_fpu() systems.
--- arch/x86/include/asm/fpu-internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/fpu-internal.h +1 -1
@@ -458,7 +458,7 @@ static inline fpu_switch_t switch_fpu_prepare(struct task_struct *old, struct tatask_disable_lazy_fpu_restore (old);if (fpu.preload) { new->thread.fpu_counter++; -if (!use_eager_fpu () &&fpu_lazy_restore (new, cpu)) +if (fpu_lazy_restore (new, cpu)) fpu.preload =0 ;else prefetch (new->thread.fpu.state);
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 5 lines · On 02/06/2015 03:01 PM, [email protected] wrote:
This includes the three patches by Oleg that are not in -tip yet, and five more by myself.
I believe the changes to my patches address all the comments by reviewers on the previous version.
Ingo, Borislav, Peter,
what can I do to get these patches merged?
kind regards,
Rik
- --
All rights reversed
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU4gwbAAoJEM553pKExN6DeccH/1W8AqviPWFRd0GTwU2BuswK YFoF6F/Hn/TGtAVUEkdB+7m3jJGDwPVj0UNC4osaP7mHtb8KY2snlzrUmPmreuw1 Y6lxCcx7y3CkD6UQJ4v7CJSAqdRwaQq+YB2llx/fprQkRmZIMuQQHTGGbYgg0uoG pOxEYn6LvuFQzZlnvbpJNf0xKjLQzGNUukzXTiSpvp1q2HG3QCJj3IbULhzxEB4g r+y9ePxej1ijiuqvaK/3rV3M7clUV5axZH+V0PH3Fk4mPgWk/8Zy5x4l1q8Al6YK Tk/hIkWySCHpAgQZvZATtVwc6ilZ79rzmZlenGUJ5sY7XjUz0cWeaCUxgh0jgrU= =yljP
-----END PGP SIGNATURE-----
Borislav Petkov
maintainerQuoted 3 lines · Rik van Riel
Ingo, Borislav, Peter,
what can I do to get these patches merged?
Let me take a look, as far as I can. I'll collect them if all is fine and unless someone else chimes in with a desire to collect them :-)
Thanks.
Signature · 5 lines
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
--