sched: Make task_struct::thread constant size, x86/fpu: Remove thread::fpu
cover sched: Make task_struct::thread constant size, x86/fpu: Remove thread::fpu
This series is one of the dependencies of the fast-headers work, which aims to reduce header complexity by removing <asm/processor.h> from the <linux/sched.h> dependency chain, which headers are headers are fat enough already even if we do not combine them.
To achieve that decoupling, one of the key steps is to not embedd any C types from <asm/processor.h> into task_struct.
The only architecture that relies on that in a serious fashion is x86, via the 'struct thread::fpu' variable size structure. The series below attempts to resolve it by using a calculated fpu context area address value via the x86_task_fpu() helper. The allocation layout of task_struct + fpu-save-area doesn't change.
The -v5 version is a refresh of the -v4 series to v6.15-rc1.
The Git tree of these commits can also be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git WIP.x86/fpu
Thanks,
Ingo
===============>
Ingo Molnar (8):
x86/fpu: Introduce the x86_task_fpu() helper method
x86/fpu: Convert task_struct::thread.fpu accesses to use x86_task_fpu()
x86/fpu: Make task_struct::thread constant size
x86/fpu: Remove the thread::fpu pointer
x86/fpu: Push 'fpu' pointer calculation into the fpu__drop() call
x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit x86/fpu: Remove init_task FPU state dependencies, add debugging warning for PF_KTHREAD tasks x86/fpu: Use 'fpstate' variable names consistently
arch/x86/include/asm/fpu/api.h | 2 +- arch/x86/include/asm/fpu/sched.h | 4 +- arch/x86/include/asm/processor.h | 23 ++++++------ arch/x86/kernel/fpu/context.h | 4 +- arch/x86/kernel/fpu/core.c | 80 +++++++++++++++++++++++----------------- arch/x86/kernel/fpu/init.c | 21 ++++++----- arch/x86/kernel/fpu/regset.c | 22 +++++------ arch/x86/kernel/fpu/signal.c | 18 ++++----- arch/x86/kernel/fpu/xstate.c | 27 ++++++-------- arch/x86/kernel/fpu/xstate.h | 6 +-- arch/x86/kernel/process.c | 9 ++--- arch/x86/kernel/signal.c | 6 +-- arch/x86/kernel/traps.c | 2 +- arch/x86/math-emu/fpu_aux.c | 2 +- arch/x86/math-emu/fpu_entry.c | 4 +- arch/x86/math-emu/fpu_system.h | 2 +- arch/x86/mm/extable.c | 2 +- include/linux/sched.h | 15 ++------ 18 files changed, 126 insertions(+), 123 deletions(-)
Signature · 1 line
2.45.2
PATCH 1/8 x86/fpu: Introduce the x86_task_fpu() helper method
The per-task FPU context/save area is allocated right next to task_struct, currently in a variable-size array via task_struct::thread.fpu[], but we plan to fully hide it from the C type scope.
Introduce the x86_task_fpu() accessor that gets to the FPU context pointer explicitly from the task pointer.
Right now this is a simple (task)->thread.fpu wrapper.
--- arch/x86/include/asm/processor.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/include/asm/processor.h +2 -0
@@ -523,6 +523,8 @@ struct thread_struct {*/ }; +#define x86_task_fpu (task) (&(task)->thread.fpu) +extern void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size); static inline void arch_thread_struct_whitelist(unsigned long *offset,
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Introduce the x86_task_fpu() helper method
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: 77fbccede633a5565cae084348b5459f6849086d Gitweb: https://git.kernel.org/tip/77fbccede633a5565cae084348b5459f6849086d Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:20 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Introduce the x86_task_fpu() helper method
The per-task FPU context/save area is allocated right next to task_struct, currently in a variable-size array via task_struct::thread.fpu[], but we plan to fully hide it from the C type scope.
Introduce the x86_task_fpu() accessor that gets to the FPU context pointer explicitly from the task pointer.
Right now this is a simple (task)->thread.fpu wrapper.
--- arch/x86/include/asm/processor.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/include/asm/processor.h +2 -0
@@ -523,6 +523,8 @@ struct thread_struct {*/ }; +#define x86_task_fpu (task) (&(task)->thread.fpu) +extern void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size); static inline void arch_thread_struct_whitelist(unsigned long *offset,
PATCH 2/8 x86/fpu: Convert task_struct::thread.fpu accesses to use x86_task_fpu()
In reply to Ingo MolnarThis will make the removal of the task_struct::thread.fpu array easier.
No change in functionality - code generated before and after this commit is identical on x86-defconfig:
kepler:~/tip> diff -up vmlinux.before.asm vmlinux.after.asm kepler:~/tip>
--- arch/x86/include/asm/fpu/sched.h | 2 +- arch/x86/kernel/fpu/context.h | 4 ++-- arch/x86/kernel/fpu/core.c | 30 +++++++++++++++--------------- arch/x86/kernel/fpu/init.c | 8 ++++---- arch/x86/kernel/fpu/regset.c | 22 +++++++++++----------- arch/x86/kernel/fpu/signal.c | 18 +++++++++--------- arch/x86/kernel/fpu/xstate.c | 22 +++++++++++----------- arch/x86/kernel/fpu/xstate.h | 6 +++--- arch/x86/kernel/process.c | 6 +++--- arch/x86/kernel/signal.c | 6 +++--- arch/x86/kernel/traps.c | 2 +- arch/x86/math-emu/fpu_aux.c | 2 +- arch/x86/math-emu/fpu_entry.c | 4 ++-- arch/x86/math-emu/fpu_system.h | 2 +- arch/x86/mm/extable.c | 2 +- 15 files changed, 68 insertions(+), 68 deletions(-)
arch/x86/include/asm/fpu/sched.h +1 -1
@@ -41,7 +41,7 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu) {if (cpu_feature_enabled (X86_FEATURE_FPU ) && !(old->flags & (PF_KTHREAD |PF_USER_WORKER ))) { -struct fpu *old_fpu = &old->thread.fpu; +struct fpu *old_fpu =x86_task_fpu (old);save_fpregs_to_fpstate (old_fpu);/*
arch/x86/kernel/fpu/context.h +2 -2
@@ -53,7 +53,7 @@ static inline void fpregs_activate(struct fpu *fpu)/* Internal helper for switch_fpu_return() and signal frame setup */ static inline void fpregs_restore_userregs (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);int cpu =smp_processor_id ();if (WARN_ON_ONCE (current->flags & (PF_KTHREAD |PF_USER_WORKER )))
@@ -67,7 +67,7 @@ static inline void fpregs_restore_userregs(void)* If PKRU is enabled, then the PKRU value is already * correct because it was either set in switch_to() or in * flush_thread(). So it is excluded because it might be -* not up to date in current->thread.fpu.xsave state. +* not up to date in current->thread.fpu->xsave state. * * XFD state is handled in restore_fpregs_from_fpstate(). */
arch/x86/kernel/fpu/core.c +15 -15
@@ -211,7 +211,7 @@ static void fpu_init_guest_permissions(struct fpu_guest *gfpu)return ;spin_lock_irq (¤t->sighand->siglock); - fpuperm = ¤t->group_leader->thread.fpu.guest_perm; + fpuperm = &x86_task_fpu (current->group_leader)->guest_perm; perm = fpuperm->__state_perm;/* First fpstate allocation locks down permissions. */
@@ -323,7 +323,7 @@ EXPORT_SYMBOL_GPL(fpu_update_guest_xfd);*/ void fpu_sync_guest_vmexit_xfd_state (void ) { -struct fpstate *fps = current->thread.fpu.fpstate; +struct fpstate *fps =x86_task_fpu (current)->fpstate;lockdep_assert_irqs_disabled ();if (fpu_state_size_dynamic ()) {
@@ -337,7 +337,7 @@ EXPORT_SYMBOL_GPL(fpu_sync_guest_vmexit_xfd_state);int fpu_swap_kvm_fpstate (struct fpu_guest *guest_fpu,bool enter_guest) {struct fpstate *guest_fps = guest_fpu->fpstate; -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);struct fpstate *cur_fps = fpu->fpstate;fpregs_lock ();
@@ -438,7 +438,7 @@ void kernel_fpu_begin_mask(unsigned int kfpu_mask)if (!(current->flags & (PF_KTHREAD |PF_USER_WORKER )) && !test_thread_flag (TIF_NEED_FPU_LOAD )) {set_thread_flag (TIF_NEED_FPU_LOAD ); -save_fpregs_to_fpstate (¤t->thread.fpu); +save_fpregs_to_fpstate (x86_task_fpu (current)); }__cpu_invalidate_fpregs_state ();
@@ -467,7 +467,7 @@ EXPORT_SYMBOL_GPL(kernel_fpu_end);*/ void fpu_sync_fpstate (struct fpu *fpu) { -WARN_ON_FPU (fpu != ¤t->thread.fpu); +WARN_ON_FPU (fpu !=x86_task_fpu (current));fpregs_lock ();trace_x86_fpu_before_save (fpu);
@@ -552,7 +552,7 @@ void fpstate_reset(struct fpu *fpu)static inline void fpu_inherit_perms (struct fpu *dst_fpu) {if (fpu_state_size_dynamic ()) { -struct fpu *src_fpu = ¤t->group_leader->thread.fpu; +struct fpu *src_fpu =x86_task_fpu (current->group_leader);spin_lock_irq (¤t->sighand->siglock);/* Fork also inherits the permissions of the parent */
@@ -572,7 +572,7 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)if (!ssp)return 0 ; - xstate =get_xsave_addr (&dst->thread.fpu.fpstate->regs.xsave, + xstate =get_xsave_addr (&x86_task_fpu (dst)->fpstate->regs.xsave,XFEATURE_CET_USER );/*
@@ -593,8 +593,8 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long ssp) { -struct fpu *src_fpu = ¤t->thread.fpu; -struct fpu *dst_fpu = &dst->thread.fpu; +struct fpu *src_fpu =x86_task_fpu (current); +struct fpu *dst_fpu =x86_task_fpu (dst);/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -686,7 +686,7 @@ void fpu__drop(struct fpu *fpu) {preempt_disable (); -if (fpu == ¤t->thread.fpu) { +if (fpu ==x86_task_fpu (current)) {/* Ignore delayed exceptions from user space */ asmvolatile ("1: fwait\n" "2:\n"
@@ -720,7 +720,7 @@ static inline void restore_fpregs_from_init_fpstate(u64 features_mask)*/ static void fpu_reset_fpregs (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);fpregs_lock ();__fpu_invalidate_fpregs_state (fpu);
@@ -749,7 +749,7 @@ static void fpu_reset_fpregs(void)*/ void fpu__clear_user_states (struct fpu *fpu) { -WARN_ON_FPU (fpu != ¤t->thread.fpu); +WARN_ON_FPU (fpu !=x86_task_fpu (current));fpregs_lock ();if (!cpu_feature_enabled (X86_FEATURE_FPU )) {
@@ -782,7 +782,7 @@ void fpu__clear_user_states(struct fpu *fpu)void fpu_flush_thread (void ) { -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));fpu_reset_fpregs (); }/*
@@ -823,7 +823,7 @@ void fpregs_lock_and_load(void)*/ void fpregs_assert_state_consistent (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);if (test_thread_flag (TIF_NEED_FPU_LOAD ))return ;
@@ -835,7 +835,7 @@ EXPORT_SYMBOL_GPL(fpregs_assert_state_consistent);void fpregs_mark_activate (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);fpregs_activate (fpu); fpu->last_cpu =smp_processor_id ();
arch/x86/kernel/fpu/init.c +4 -4
@@ -38,7 +38,7 @@ static void fpu__init_cpu_generic(void)/* Flush out any pending x87 state: */ #ifdef CONFIG_MATH_EMULATION if (!boot_cpu_has (X86_FEATURE_FPU )) -fpstate_init_soft (¤t->thread.fpu.fpstate->regs.soft); +fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); else#endif asmvolatile ("fninit" );
@@ -154,7 +154,7 @@ static void __init fpu__init_task_struct_size(void)* Subtract off the static size of the register state. * It potentially has a bunch of padding. */ - task_size -=sizeof (current->thread.fpu.__fpstate.regs); + task_size -=sizeof (union fpregs_state );/* * Add back the dynamically-calculated register state
@@ -204,7 +204,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_kernel_cfg.default_size = size; fpu_user_cfg.max_size = size; fpu_user_cfg.default_size = size; -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current)); }/*
@@ -213,7 +213,7 @@ static void __init fpu__init_system_xstate_size_legacy(void)*/ void __init fpu__init_system (void ) { -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));fpu__init_system_early_generic ();/*
arch/x86/kernel/fpu/regset.c +11 -11
@@ -45,7 +45,7 @@ int regset_xregset_fpregs_active(struct task_struct *target, const struct user_r*/ static void sync_fpstate (struct fpu *fpu) { -if (fpu == ¤t->thread.fpu) +if (fpu ==x86_task_fpu (current))fpu_sync_fpstate (fpu); }
@@ -63,7 +63,7 @@ static void fpu_force_restore(struct fpu *fpu)* Only stopped child tasks can be used to modify the FPU * state in the fpstate buffer: */ -WARN_ON_FPU (fpu == ¤t->thread.fpu); +WARN_ON_FPU (fpu ==x86_task_fpu (current));__fpu_invalidate_fpregs_state (fpu); }
@@ -71,7 +71,7 @@ static void fpu_force_restore(struct fpu *fpu)int xfpregs_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);if (!cpu_feature_enabled (X86_FEATURE_FXSR ))return -ENODEV ;
@@ -91,7 +91,7 @@ int xfpregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct fxregs_state newstate;int ret;
@@ -133,7 +133,7 @@ int xstateregs_get(struct task_struct *target, const struct user_regset *regset,if (!cpu_feature_enabled (X86_FEATURE_XSAVE ))return -ENODEV ; -sync_fpstate (&target->thread.fpu); +sync_fpstate (x86_task_fpu (target));copy_xstate_to_uabi_buf (to, target,XSTATE_COPY_XSAVE );return 0 ;
@@ -143,7 +143,7 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct xregs_state *tmpbuf =NULL ;int ret;
@@ -187,7 +187,7 @@ int ssp_active(struct task_struct *target, const struct user_regset *regset)int ssp_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);struct cet_user_state *cetregs; if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) ||
@@ -214,7 +214,7 @@ int ssp_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct xregs_state *xsave = &fpu->fpstate->regs.xsave;struct cet_user_state *cetregs;unsigned long user_ssp;
@@ -368,7 +368,7 @@ static void __convert_from_fxsr(struct user_i387_ia32_struct *env,void convert_from_fxsr (struct user_i387_ia32_struct *env,struct task_struct *tsk) { -__convert_from_fxsr (env, tsk, &tsk->thread.fpu.fpstate->regs.fxsave); +__convert_from_fxsr (env, tsk, &x86_task_fpu (tsk)->fpstate->regs.fxsave); } void convert_to_fxsr(struct fxregs_state *fxsave,
@@ -401,7 +401,7 @@ void convert_to_fxsr(struct fxregs_state *fxsave,int fpregs_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);struct user_i387_ia32_struct env;struct fxregs_state fxsave, *fx;
@@ -433,7 +433,7 @@ int fpregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct user_i387_ia32_struct env;int ret;
arch/x86/kernel/fpu/signal.c +9 -9
@@ -43,13 +43,13 @@ static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,* fpstate layout with out copying the extended state information * in the memory layout. */ -if (__get_user (magic2, (__u32 __user *)(fpstate + current->thread.fpu.fpstate->user_size))) +if (__get_user (magic2, (__u32 __user *)(fpstate +x86_task_fpu (current)->fpstate->user_size)))return false;if (likely (magic2 ==FP_XSTATE_MAGIC2 ))return true; setfx: -trace_x86_fpu_xstate_check_failed (¤t->thread.fpu); +trace_x86_fpu_xstate_check_failed (x86_task_fpu (current));/* Set the parameters for fx only state */ fx_sw->magic1 =0 ;
@@ -64,13 +64,13 @@ static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,static inline bool save_fsave_header (struct task_struct *tsk,void __user *buf) {if (use_fxsr ()) { -struct xregs_state *xsave = &tsk->thread.fpu.fpstate->regs.xsave; +struct xregs_state *xsave = &x86_task_fpu (tsk)->fpstate->regs.xsave;struct user_i387_ia32_struct env;struct _fpstate_32 __user *fp = buf;fpregs_lock ();if (!test_thread_flag (TIF_NEED_FPU_LOAD )) -fxsave (&tsk->thread.fpu.fpstate->regs.fxsave); +fxsave (&x86_task_fpu (tsk)->fpstate->regs.fxsave);fpregs_unlock ();convert_from_fxsr (&env, tsk);
@@ -184,7 +184,7 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf, u32 pkbool copy_fpstate_to_sigframe (void __user *buf,void __user *buf_fx,int size,u32 pkru) {struct task_struct *tsk = current; -struct fpstate *fpstate = tsk->thread.fpu.fpstate; +struct fpstate *fpstate =x86_task_fpu (tsk)->fpstate;bool ia32_fxstate = (buf != buf_fx);int ret;
@@ -272,7 +272,7 @@ static int __restore_fpregs_from_user(void __user *buf, u64 ufeatures,*/ static bool restore_fpregs_from_user (void __user *buf,u64 xrestore,bool fx_only) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);int ret;/* Restore enabled features only. */
@@ -332,7 +332,7 @@ static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, bool ia32_fxstate) {struct task_struct *tsk = current; -struct fpu *fpu = &tsk->thread.fpu; +struct fpu *fpu =x86_task_fpu (tsk);struct user_i387_ia32_struct env;bool success, fx_only = false;union fpregs_state *fpregs;
@@ -452,7 +452,7 @@ static inline unsigned int xstate_sigframe_size(struct fpstate *fpstate)*/ bool fpu__restore_sig (void __user *buf,int ia32_frame) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);void __user *buf_fx = buf;bool ia32_fxstate = false;bool success = false;
@@ -499,7 +499,7 @@ unsigned long fpu__alloc_mathframe(unsigned long sp, int ia32_frame, unsigned long *buf_fx, unsigned long *size) { - unsigned long frame_size = xstate_sigframe_size(current->thread.fpu.fpstate); + unsigned long frame_size = xstate_sigframe_size(x86_task_fpu(current)->fpstate); *buf_fx = sp =round_down (sp - frame_size,64 );if (ia32_frame &&use_fxsr ()) {
arch/x86/kernel/fpu/xstate.c +11 -11
@@ -727,7 +727,7 @@ static void __init fpu__init_disable_system_xstate(unsigned int legacy_size)*/ init_fpstate.xfd =0 ; -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current)); }/*
@@ -835,7 +835,7 @@ void __init fpu__init_system_xstate(unsigned int legacy_size) goto out_disable;/* Reset the state for the current task */ -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));/* * Update info used for ptrace frames; use standard-format size and no
@@ -909,7 +909,7 @@ void fpu__resume_cpu(void) }if (fpu_state_size_dynamic ()) -wrmsrl (MSR_IA32_XFD , current->thread.fpu.fpstate->xfd); +wrmsrl (MSR_IA32_XFD ,x86_task_fpu (current)->fpstate->xfd); }/*
@@ -1191,8 +1191,8 @@ void __copy_xstate_to_uabi_buf(struct membuf to, struct fpstate *fpstate,void copy_xstate_to_uabi_buf (struct membuf to,struct task_struct *tsk,enum xstate_copy_mode copy_mode) { -__copy_xstate_to_uabi_buf (to, tsk->thread.fpu.fpstate, - tsk->thread.fpu.fpstate->user_xfeatures, +__copy_xstate_to_uabi_buf (to,x86_task_fpu (tsk)->fpstate, +x86_task_fpu (tsk)->fpstate->user_xfeatures, tsk->thread.pkru, copy_mode); }
@@ -1332,7 +1332,7 @@ int copy_uabi_from_kernel_to_xstate(struct fpstate *fpstate, const void *kbuf, uint copy_sigframe_from_user_to_xstate (struct task_struct *tsk,const void __user *ubuf) { -return copy_uabi_to_xstate (tsk->thread.fpu.fpstate,NULL , ubuf, &tsk->thread.pkru); +return copy_uabi_to_xstate (x86_task_fpu (tsk)->fpstate,NULL , ubuf, &tsk->thread.pkru); }static bool validate_independent_components (u64 mask)
@@ -1426,7 +1426,7 @@ static bool xstate_op_valid(struct fpstate *fpstate, u64 mask, bool rstor)* The XFD MSR does not match fpstate->xfd. That's invalid when * the passed in fpstate is current's fpstate. */ -if (fpstate->xfd == current->thread.fpu.fpstate->xfd) +if (fpstate->xfd ==x86_task_fpu (current)->fpstate->xfd)return false;/*
@@ -1503,7 +1503,7 @@ void fpstate_free(struct fpu *fpu)static int fpstate_realloc (u64 xfeatures,unsigned int ksize,unsigned int usize,struct fpu_guest *guest_fpu) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);struct fpstate *curfps, *newfps =NULL ;unsigned int fpsize;bool in_use;
@@ -1596,7 +1596,7 @@ static int __xstate_request_perm(u64 permitted, u64 requested, bool guest)* AVX512. */ bool compacted =cpu_feature_enabled (X86_FEATURE_XCOMPACTED ); -struct fpu *fpu = ¤t->group_leader->thread.fpu; +struct fpu *fpu =x86_task_fpu (current->group_leader);struct fpu_state_perm *perm;unsigned int ksize, usize;u64 mask;
@@ -1699,7 +1699,7 @@ int __xfd_enable_feature(u64 xfd_err, struct fpu_guest *guest_fpu)return -EPERM ; } - fpu = ¤t->group_leader->thread.fpu; + fpu =x86_task_fpu (current->group_leader); perm = guest_fpu ? &fpu->guest_perm : &fpu->perm; ksize = perm->__state_size; usize = perm->__user_state_size;
@@ -1804,7 +1804,7 @@ long fpu_xstate_prctl(int option, unsigned long arg2)*/ static void avx512_status (struct seq_file *m,struct task_struct *task) { -unsigned long timestamp =READ_ONCE (task->thread.fpu.avx512_timestamp); +unsigned long timestamp =READ_ONCE (x86_task_fpu (task)->avx512_timestamp);long delta;if (!timestamp) {
arch/x86/kernel/fpu/xstate.h +3 -3
@@ -22,7 +22,7 @@ static inline void xstate_init_xcomp_bv(struct xregs_state *xsave, u64 mask)static inline u64 xstate_get_group_perm (bool guest) { -struct fpu *fpu = ¤t->group_leader->thread.fpu; +struct fpu *fpu =x86_task_fpu (current->group_leader);struct fpu_state_perm *perm;/* Pairs with WRITE_ONCE() in xstate_request_perm() */
@@ -288,7 +288,7 @@ static inline int xsave_to_user_sigframe(struct xregs_state __user *buf, u32 pkr* internally, e.g. PKRU. That's user space ABI and also required * to allow the signal handler to modify PKRU. */ -struct fpstate *fpstate = current->thread.fpu.fpstate; +struct fpstate *fpstate =x86_task_fpu (current)->fpstate;u64 mask = fpstate->user_xfeatures;u32 lmask;u32 hmask;
@@ -322,7 +322,7 @@ static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64u32 hmask = mask >>32 ;int err; -xfd_validate_state (current->thread.fpu.fpstate, mask, true); +xfd_validate_state (x86_task_fpu (current)->fpstate, mask, true);stac ();XSTATE_OP (XRSTOR , xstate, lmask, hmask, err);
arch/x86/kernel/process.c +3 -3
@@ -103,7 +103,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.vm86 =NULL ;#endif /* Drop the copied pointer to current's fpstate */ - dst->thread.fpu.fpstate =NULL ; +x86_task_fpu (dst)->fpstate =NULL ;return 0 ; }
@@ -112,7 +112,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)void arch_release_task_struct (struct task_struct *tsk) {if (fpu_state_size_dynamic ()) -fpstate_free (&tsk->thread.fpu); +fpstate_free (x86_task_fpu (tsk)); }#endif
@@ -122,7 +122,7 @@ void arch_release_task_struct(struct task_struct *tsk)void exit_thread (struct task_struct *tsk) {struct thread_struct *t = &tsk->thread; -struct fpu *fpu = &t->fpu; +struct fpu *fpu =x86_task_fpu (tsk);if (test_thread_flag (TIF_IO_BITMAP ))io_bitmap_exit (tsk);
arch/x86/kernel/signal.c +3 -3
@@ -255,7 +255,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { bool stepping, failed; -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);if (v8086_mode (regs))save_v86_state ((struct kernel_vm86_regs *) regs,VM86_SIGNAL );
@@ -423,14 +423,14 @@ bool sigaltstack_size_valid(size_t ss_size)if (!fpu_state_size_dynamic () && !strict_sigaltstack_size)return true; - fsize += current->group_leader->thread.fpu.perm.__user_state_size; + fsize +=x86_task_fpu (current->group_leader)->perm.__user_state_size;if (likely (ss_size > fsize))return true;if (strict_sigaltstack_size)return ss_size > fsize; - mask = current->group_leader->thread.fpu.perm.__state_perm; + mask =x86_task_fpu (current->group_leader)->perm.__state_perm;if (mask &XFEATURE_MASK_USER_DYNAMIC )return ss_size > fsize;
arch/x86/kernel/traps.c +1 -1
@@ -1295,7 +1295,7 @@ DEFINE_IDTENTRY_RAW(exc_debug)static void math_error (struct pt_regs *regs,int trapnr) {struct task_struct *task = current; -struct fpu *fpu = &task->thread.fpu; +struct fpu *fpu =x86_task_fpu (task);int si_code;char *str = (trapnr ==X86_TRAP_MF ) ?"fpu exception" :"simd exception" ;
arch/x86/math-emu/fpu_aux.c +1 -1
@@ -53,7 +53,7 @@ void fpstate_init_soft(struct swregs_state *soft)void finit (void ) { -fpstate_init_soft (¤t->thread.fpu.fpstate->regs.soft); +fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); }/*
arch/x86/math-emu/fpu_entry.c +2 -2
@@ -641,7 +641,7 @@ int fpregs_soft_set(struct task_struct *target, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft; + struct swregs_state *s387 = &x86_task_fpu(target)->fpstate->regs.soft;void *space = s387->st_space;int ret;int offset, other, i, tags, regnr, tag, newtop;
@@ -692,7 +692,7 @@ int fpregs_soft_get(struct task_struct *target, const struct user_regset *regset, struct membuf to) { -struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft; +struct swregs_state *s387 = &x86_task_fpu (target)->fpstate->regs.soft;const void *space = s387->st_space;int offset = (S387 ->ftop &7 ) *10 , other =80 - offset;
arch/x86/math-emu/fpu_system.h +1 -1
@@ -73,7 +73,7 @@ static inline bool seg_writable(struct desc_struct *d)return (d->type &SEG_TYPE_EXECUTE_MASK ) ==SEG_TYPE_WRITABLE ; } -#define I387 (¤t->thread.fpu.fpstate->regs) +#define I387 (&x86_task_fpu(current)->fpstate->regs)#define FPU_info (I387->soft.info)#define FPU_CS (*(unsigned short *) &(FPU_info->regs->cs))
arch/x86/mm/extable.c +1 -1
@@ -111,7 +111,7 @@ static bool ex_handler_sgx(const struct exception_table_entry *fixup,/* * Handler for when we fail to restore a task's FPU state. We should never get -* here because the FPU state of a task using the FPU (task->thread.fpu.state) +* here because the FPU state of a task using the FPU (struct fpu::fpstate) * should always be valid. However, past bugs have allowed userspace to set * reserved bits in the XSAVE area using PTRACE_SETREGSET or sys_rt_sigreturn(). * These caused XRSTOR to fail when switching to the task, leaking the FPU
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Convert task_struct::thread.fpu accesses to use x86_task_fpu()
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: e3bfa3859936da3edd1e16d0b74fdaaa19bb5087 Gitweb: https://git.kernel.org/tip/e3bfa3859936da3edd1e16d0b74fdaaa19bb5087 Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:21 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Convert task_struct::thread.fpu accesses to use x86_task_fpu()
This will make the removal of the task_struct::thread.fpu array easier.
No change in functionality - code generated before and after this commit is identical on x86-defconfig:
kepler:~/tip> diff -up vmlinux.before.asm vmlinux.after.asm kepler:~/tip>
--- arch/x86/include/asm/fpu/sched.h | 2 +- arch/x86/kernel/fpu/context.h | 4 ++-- arch/x86/kernel/fpu/core.c | 30 +++++++++++++++--------------- arch/x86/kernel/fpu/init.c | 8 ++++---- arch/x86/kernel/fpu/regset.c | 22 +++++++++++----------- arch/x86/kernel/fpu/signal.c | 18 +++++++++--------- arch/x86/kernel/fpu/xstate.c | 22 +++++++++++----------- arch/x86/kernel/fpu/xstate.h | 6 +++--- arch/x86/kernel/process.c | 6 +++--- arch/x86/kernel/signal.c | 6 +++--- arch/x86/kernel/traps.c | 2 +- arch/x86/math-emu/fpu_aux.c | 2 +- arch/x86/math-emu/fpu_entry.c | 4 ++-- arch/x86/math-emu/fpu_system.h | 2 +- arch/x86/mm/extable.c | 2 +- 15 files changed, 68 insertions(+), 68 deletions(-)
arch/x86/include/asm/fpu/sched.h +1 -1
@@ -41,7 +41,7 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu) {if (cpu_feature_enabled (X86_FEATURE_FPU ) && !(old->flags & (PF_KTHREAD |PF_USER_WORKER ))) { -struct fpu *old_fpu = &old->thread.fpu; +struct fpu *old_fpu =x86_task_fpu (old);save_fpregs_to_fpstate (old_fpu);/*
arch/x86/kernel/fpu/context.h +2 -2
@@ -53,7 +53,7 @@ static inline void fpregs_activate(struct fpu *fpu)/* Internal helper for switch_fpu_return() and signal frame setup */ static inline void fpregs_restore_userregs (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);int cpu =smp_processor_id ();if (WARN_ON_ONCE (current->flags & (PF_KTHREAD |PF_USER_WORKER )))
@@ -67,7 +67,7 @@ static inline void fpregs_restore_userregs(void)* If PKRU is enabled, then the PKRU value is already * correct because it was either set in switch_to() or in * flush_thread(). So it is excluded because it might be -* not up to date in current->thread.fpu.xsave state. +* not up to date in current->thread.fpu->xsave state. * * XFD state is handled in restore_fpregs_from_fpstate(). */
arch/x86/kernel/fpu/core.c +15 -15
@@ -211,7 +211,7 @@ static void fpu_init_guest_permissions(struct fpu_guest *gfpu)return ;spin_lock_irq (¤t->sighand->siglock); - fpuperm = ¤t->group_leader->thread.fpu.guest_perm; + fpuperm = &x86_task_fpu (current->group_leader)->guest_perm; perm = fpuperm->__state_perm;/* First fpstate allocation locks down permissions. */
@@ -323,7 +323,7 @@ EXPORT_SYMBOL_GPL(fpu_update_guest_xfd);*/ void fpu_sync_guest_vmexit_xfd_state (void ) { -struct fpstate *fps = current->thread.fpu.fpstate; +struct fpstate *fps =x86_task_fpu (current)->fpstate;lockdep_assert_irqs_disabled ();if (fpu_state_size_dynamic ()) {
@@ -337,7 +337,7 @@ EXPORT_SYMBOL_GPL(fpu_sync_guest_vmexit_xfd_state);int fpu_swap_kvm_fpstate (struct fpu_guest *guest_fpu,bool enter_guest) {struct fpstate *guest_fps = guest_fpu->fpstate; -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);struct fpstate *cur_fps = fpu->fpstate;fpregs_lock ();
@@ -438,7 +438,7 @@ void kernel_fpu_begin_mask(unsigned int kfpu_mask)if (!(current->flags & (PF_KTHREAD |PF_USER_WORKER )) && !test_thread_flag (TIF_NEED_FPU_LOAD )) {set_thread_flag (TIF_NEED_FPU_LOAD ); -save_fpregs_to_fpstate (¤t->thread.fpu); +save_fpregs_to_fpstate (x86_task_fpu (current)); }__cpu_invalidate_fpregs_state ();
@@ -467,7 +467,7 @@ EXPORT_SYMBOL_GPL(kernel_fpu_end);*/ void fpu_sync_fpstate (struct fpu *fpu) { -WARN_ON_FPU (fpu != ¤t->thread.fpu); +WARN_ON_FPU (fpu !=x86_task_fpu (current));fpregs_lock ();trace_x86_fpu_before_save (fpu);
@@ -552,7 +552,7 @@ void fpstate_reset(struct fpu *fpu)static inline void fpu_inherit_perms (struct fpu *dst_fpu) {if (fpu_state_size_dynamic ()) { -struct fpu *src_fpu = ¤t->group_leader->thread.fpu; +struct fpu *src_fpu =x86_task_fpu (current->group_leader);spin_lock_irq (¤t->sighand->siglock);/* Fork also inherits the permissions of the parent */
@@ -572,7 +572,7 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)if (!ssp)return 0 ; - xstate =get_xsave_addr (&dst->thread.fpu.fpstate->regs.xsave, + xstate =get_xsave_addr (&x86_task_fpu (dst)->fpstate->regs.xsave,XFEATURE_CET_USER );/*
@@ -593,8 +593,8 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long ssp) { -struct fpu *src_fpu = ¤t->thread.fpu; -struct fpu *dst_fpu = &dst->thread.fpu; +struct fpu *src_fpu =x86_task_fpu (current); +struct fpu *dst_fpu =x86_task_fpu (dst);/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -686,7 +686,7 @@ void fpu__drop(struct fpu *fpu) {preempt_disable (); -if (fpu == ¤t->thread.fpu) { +if (fpu ==x86_task_fpu (current)) {/* Ignore delayed exceptions from user space */ asmvolatile ("1: fwait\n" "2:\n"
@@ -720,7 +720,7 @@ static inline void restore_fpregs_from_init_fpstate(u64 features_mask)*/ static void fpu_reset_fpregs (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);fpregs_lock ();__fpu_invalidate_fpregs_state (fpu);
@@ -749,7 +749,7 @@ static void fpu_reset_fpregs(void)*/ void fpu__clear_user_states (struct fpu *fpu) { -WARN_ON_FPU (fpu != ¤t->thread.fpu); +WARN_ON_FPU (fpu !=x86_task_fpu (current));fpregs_lock ();if (!cpu_feature_enabled (X86_FEATURE_FPU )) {
@@ -782,7 +782,7 @@ void fpu__clear_user_states(struct fpu *fpu)void fpu_flush_thread (void ) { -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));fpu_reset_fpregs (); }/*
@@ -823,7 +823,7 @@ void fpregs_lock_and_load(void)*/ void fpregs_assert_state_consistent (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);if (test_thread_flag (TIF_NEED_FPU_LOAD ))return ;
@@ -835,7 +835,7 @@ EXPORT_SYMBOL_GPL(fpregs_assert_state_consistent);void fpregs_mark_activate (void ) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);fpregs_activate (fpu); fpu->last_cpu =smp_processor_id ();
arch/x86/kernel/fpu/init.c +4 -4
@@ -38,7 +38,7 @@ static void fpu__init_cpu_generic(void)/* Flush out any pending x87 state: */ #ifdef CONFIG_MATH_EMULATION if (!boot_cpu_has (X86_FEATURE_FPU )) -fpstate_init_soft (¤t->thread.fpu.fpstate->regs.soft); +fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); else#endif asmvolatile ("fninit" );
@@ -154,7 +154,7 @@ static void __init fpu__init_task_struct_size(void)* Subtract off the static size of the register state. * It potentially has a bunch of padding. */ - task_size -=sizeof (current->thread.fpu.__fpstate.regs); + task_size -=sizeof (union fpregs_state );/* * Add back the dynamically-calculated register state
@@ -204,7 +204,7 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_kernel_cfg.default_size = size; fpu_user_cfg.max_size = size; fpu_user_cfg.default_size = size; -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current)); }/*
@@ -213,7 +213,7 @@ static void __init fpu__init_system_xstate_size_legacy(void)*/ void __init fpu__init_system (void ) { -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));fpu__init_system_early_generic ();/*
arch/x86/kernel/fpu/regset.c +11 -11
@@ -45,7 +45,7 @@ int regset_xregset_fpregs_active(struct task_struct *target, const struct user_r*/ static void sync_fpstate (struct fpu *fpu) { -if (fpu == ¤t->thread.fpu) +if (fpu ==x86_task_fpu (current))fpu_sync_fpstate (fpu); }
@@ -63,7 +63,7 @@ static void fpu_force_restore(struct fpu *fpu)* Only stopped child tasks can be used to modify the FPU * state in the fpstate buffer: */ -WARN_ON_FPU (fpu == ¤t->thread.fpu); +WARN_ON_FPU (fpu ==x86_task_fpu (current));__fpu_invalidate_fpregs_state (fpu); }
@@ -71,7 +71,7 @@ static void fpu_force_restore(struct fpu *fpu)int xfpregs_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);if (!cpu_feature_enabled (X86_FEATURE_FXSR ))return -ENODEV ;
@@ -91,7 +91,7 @@ int xfpregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct fxregs_state newstate;int ret;
@@ -133,7 +133,7 @@ int xstateregs_get(struct task_struct *target, const struct user_regset *regset,if (!cpu_feature_enabled (X86_FEATURE_XSAVE ))return -ENODEV ; -sync_fpstate (&target->thread.fpu); +sync_fpstate (x86_task_fpu (target));copy_xstate_to_uabi_buf (to, target,XSTATE_COPY_XSAVE );return 0 ;
@@ -143,7 +143,7 @@ int xstateregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct xregs_state *tmpbuf =NULL ;int ret;
@@ -187,7 +187,7 @@ int ssp_active(struct task_struct *target, const struct user_regset *regset)int ssp_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);struct cet_user_state *cetregs; if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK) ||
@@ -214,7 +214,7 @@ int ssp_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct xregs_state *xsave = &fpu->fpstate->regs.xsave;struct cet_user_state *cetregs;unsigned long user_ssp;
@@ -368,7 +368,7 @@ static void __convert_from_fxsr(struct user_i387_ia32_struct *env,void convert_from_fxsr (struct user_i387_ia32_struct *env,struct task_struct *tsk) { -__convert_from_fxsr (env, tsk, &tsk->thread.fpu.fpstate->regs.fxsave); +__convert_from_fxsr (env, tsk, &x86_task_fpu (tsk)->fpstate->regs.fxsave); } void convert_to_fxsr(struct fxregs_state *fxsave,
@@ -401,7 +401,7 @@ void convert_to_fxsr(struct fxregs_state *fxsave,int fpregs_get (struct task_struct *target,const struct user_regset *regset,struct membuf to) { -struct fpu *fpu = &target->thread.fpu; +struct fpu *fpu =x86_task_fpu (target);struct user_i387_ia32_struct env;struct fxregs_state fxsave, *fx;
@@ -433,7 +433,7 @@ int fpregs_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct fpu *fpu = &target->thread.fpu; + struct fpu *fpu = x86_task_fpu(target);struct user_i387_ia32_struct env;int ret;
arch/x86/kernel/fpu/signal.c +9 -9
@@ -43,13 +43,13 @@ static inline bool check_xstate_in_sigframe(struct fxregs_state __user *fxbuf,* fpstate layout with out copying the extended state information * in the memory layout. */ -if (__get_user (magic2, (__u32 __user *)(fpstate + current->thread.fpu.fpstate->user_size))) +if (__get_user (magic2, (__u32 __user *)(fpstate +x86_task_fpu (current)->fpstate->user_size)))return false;if (likely (magic2 ==FP_XSTATE_MAGIC2 ))return true; setfx: -trace_x86_fpu_xstate_check_failed (¤t->thread.fpu); +trace_x86_fpu_xstate_check_failed (x86_task_fpu (current));/* Set the parameters for fx only state */ fx_sw->magic1 =0 ;
@@ -64,13 +64,13 @@ setfx:static inline bool save_fsave_header (struct task_struct *tsk,void __user *buf) {if (use_fxsr ()) { -struct xregs_state *xsave = &tsk->thread.fpu.fpstate->regs.xsave; +struct xregs_state *xsave = &x86_task_fpu (tsk)->fpstate->regs.xsave;struct user_i387_ia32_struct env;struct _fpstate_32 __user *fp = buf;fpregs_lock ();if (!test_thread_flag (TIF_NEED_FPU_LOAD )) -fxsave (&tsk->thread.fpu.fpstate->regs.fxsave); +fxsave (&x86_task_fpu (tsk)->fpstate->regs.fxsave);fpregs_unlock ();convert_from_fxsr (&env, tsk);
@@ -184,7 +184,7 @@ static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf, u32 pkbool copy_fpstate_to_sigframe (void __user *buf,void __user *buf_fx,int size,u32 pkru) {struct task_struct *tsk = current; -struct fpstate *fpstate = tsk->thread.fpu.fpstate; +struct fpstate *fpstate =x86_task_fpu (tsk)->fpstate;bool ia32_fxstate = (buf != buf_fx);int ret;
@@ -272,7 +272,7 @@ static int __restore_fpregs_from_user(void __user *buf, u64 ufeatures,*/ static bool restore_fpregs_from_user (void __user *buf,u64 xrestore,bool fx_only) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);int ret;/* Restore enabled features only. */
@@ -332,7 +332,7 @@ static bool __fpu_restore_sig(void __user *buf, void __user *buf_fx, bool ia32_fxstate) {struct task_struct *tsk = current; -struct fpu *fpu = &tsk->thread.fpu; +struct fpu *fpu =x86_task_fpu (tsk);struct user_i387_ia32_struct env;bool success, fx_only = false;union fpregs_state *fpregs;
@@ -452,7 +452,7 @@ static inline unsigned int xstate_sigframe_size(struct fpstate *fpstate)*/ bool fpu__restore_sig (void __user *buf,int ia32_frame) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);void __user *buf_fx = buf;bool ia32_fxstate = false;bool success = false;
@@ -499,7 +499,7 @@ unsigned long fpu__alloc_mathframe(unsigned long sp, int ia32_frame, unsigned long *buf_fx, unsigned long *size) { - unsigned long frame_size = xstate_sigframe_size(current->thread.fpu.fpstate); + unsigned long frame_size = xstate_sigframe_size(x86_task_fpu(current)->fpstate); *buf_fx = sp =round_down (sp - frame_size,64 );if (ia32_frame &&use_fxsr ()) {
arch/x86/kernel/fpu/xstate.c +11 -11
@@ -763,7 +763,7 @@ static void __init fpu__init_disable_system_xstate(unsigned int legacy_size)*/ init_fpstate.xfd =0 ; -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current)); }/*
@@ -871,7 +871,7 @@ void __init fpu__init_system_xstate(unsigned int legacy_size) goto out_disable;/* Reset the state for the current task */ -fpstate_reset (¤t->thread.fpu); +fpstate_reset (x86_task_fpu (current));/* * Update info used for ptrace frames; use standard-format size and no
@@ -945,7 +945,7 @@ void fpu__resume_cpu(void) }if (fpu_state_size_dynamic ()) -wrmsrl (MSR_IA32_XFD , current->thread.fpu.fpstate->xfd); +wrmsrl (MSR_IA32_XFD ,x86_task_fpu (current)->fpstate->xfd); }/*
@@ -1227,8 +1227,8 @@ out:void copy_xstate_to_uabi_buf (struct membuf to,struct task_struct *tsk,enum xstate_copy_mode copy_mode) { -__copy_xstate_to_uabi_buf (to, tsk->thread.fpu.fpstate, - tsk->thread.fpu.fpstate->user_xfeatures, +__copy_xstate_to_uabi_buf (to,x86_task_fpu (tsk)->fpstate, +x86_task_fpu (tsk)->fpstate->user_xfeatures, tsk->thread.pkru, copy_mode); }
@@ -1368,7 +1368,7 @@ int copy_uabi_from_kernel_to_xstate(struct fpstate *fpstate, const void *kbuf, uint copy_sigframe_from_user_to_xstate (struct task_struct *tsk,const void __user *ubuf) { -return copy_uabi_to_xstate (tsk->thread.fpu.fpstate,NULL , ubuf, &tsk->thread.pkru); +return copy_uabi_to_xstate (x86_task_fpu (tsk)->fpstate,NULL , ubuf, &tsk->thread.pkru); }static bool validate_independent_components (u64 mask)
@@ -1462,7 +1462,7 @@ static bool xstate_op_valid(struct fpstate *fpstate, u64 mask, bool rstor)* The XFD MSR does not match fpstate->xfd. That's invalid when * the passed in fpstate is current's fpstate. */ -if (fpstate->xfd == current->thread.fpu.fpstate->xfd) +if (fpstate->xfd ==x86_task_fpu (current)->fpstate->xfd)return false;/*
@@ -1539,7 +1539,7 @@ void fpstate_free(struct fpu *fpu)static int fpstate_realloc (u64 xfeatures,unsigned int ksize,unsigned int usize,struct fpu_guest *guest_fpu) { -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);struct fpstate *curfps, *newfps =NULL ;unsigned int fpsize;bool in_use;
@@ -1632,7 +1632,7 @@ static int __xstate_request_perm(u64 permitted, u64 requested, bool guest)* AVX512. */ bool compacted =cpu_feature_enabled (X86_FEATURE_XCOMPACTED ); -struct fpu *fpu = ¤t->group_leader->thread.fpu; +struct fpu *fpu =x86_task_fpu (current->group_leader);struct fpu_state_perm *perm;unsigned int ksize, usize;u64 mask;
@@ -1735,7 +1735,7 @@ int __xfd_enable_feature(u64 xfd_err, struct fpu_guest *guest_fpu)return -EPERM ; } - fpu = ¤t->group_leader->thread.fpu; + fpu =x86_task_fpu (current->group_leader); perm = guest_fpu ? &fpu->guest_perm : &fpu->perm; ksize = perm->__state_size; usize = perm->__user_state_size;
@@ -1840,7 +1840,7 @@ long fpu_xstate_prctl(int option, unsigned long arg2)*/ static void avx512_status (struct seq_file *m,struct task_struct *task) { -unsigned long timestamp =READ_ONCE (task->thread.fpu.avx512_timestamp); +unsigned long timestamp =READ_ONCE (x86_task_fpu (task)->avx512_timestamp);long delta;if (!timestamp) {
arch/x86/kernel/fpu/xstate.h +3 -3
@@ -22,7 +22,7 @@ static inline void xstate_init_xcomp_bv(struct xregs_state *xsave, u64 mask)static inline u64 xstate_get_group_perm (bool guest) { -struct fpu *fpu = ¤t->group_leader->thread.fpu; +struct fpu *fpu =x86_task_fpu (current->group_leader);struct fpu_state_perm *perm;/* Pairs with WRITE_ONCE() in xstate_request_perm() */
@@ -288,7 +288,7 @@ static inline int xsave_to_user_sigframe(struct xregs_state __user *buf, u32 pkr* internally, e.g. PKRU. That's user space ABI and also required * to allow the signal handler to modify PKRU. */ -struct fpstate *fpstate = current->thread.fpu.fpstate; +struct fpstate *fpstate =x86_task_fpu (current)->fpstate;u64 mask = fpstate->user_xfeatures;u32 lmask;u32 hmask;
@@ -322,7 +322,7 @@ static inline int xrstor_from_user_sigframe(struct xregs_state __user *buf, u64u32 hmask = mask >>32 ;int err; -xfd_validate_state (current->thread.fpu.fpstate, mask, true); +xfd_validate_state (x86_task_fpu (current)->fpstate, mask, true);stac ();XSTATE_OP (XRSTOR , xstate, lmask, hmask, err);
arch/x86/kernel/process.c +3 -3
@@ -103,7 +103,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.vm86 =NULL ;#endif /* Drop the copied pointer to current's fpstate */ - dst->thread.fpu.fpstate =NULL ; +x86_task_fpu (dst)->fpstate =NULL ;return 0 ; }
@@ -112,7 +112,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)void arch_release_task_struct (struct task_struct *tsk) {if (fpu_state_size_dynamic ()) -fpstate_free (&tsk->thread.fpu); +fpstate_free (x86_task_fpu (tsk)); }#endif
@@ -122,7 +122,7 @@ void arch_release_task_struct(struct task_struct *tsk)void exit_thread (struct task_struct *tsk) {struct thread_struct *t = &tsk->thread; -struct fpu *fpu = &t->fpu; +struct fpu *fpu =x86_task_fpu (tsk);if (test_thread_flag (TIF_IO_BITMAP ))io_bitmap_exit (tsk);
arch/x86/kernel/signal.c +3 -3
@@ -255,7 +255,7 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs) { bool stepping, failed; -struct fpu *fpu = ¤t->thread.fpu; +struct fpu *fpu =x86_task_fpu (current);if (v8086_mode (regs))save_v86_state ((struct kernel_vm86_regs *) regs,VM86_SIGNAL );
@@ -423,14 +423,14 @@ bool sigaltstack_size_valid(size_t ss_size)if (!fpu_state_size_dynamic () && !strict_sigaltstack_size)return true; - fsize += current->group_leader->thread.fpu.perm.__user_state_size; + fsize +=x86_task_fpu (current->group_leader)->perm.__user_state_size;if (likely (ss_size > fsize))return true;if (strict_sigaltstack_size)return ss_size > fsize; - mask = current->group_leader->thread.fpu.perm.__state_perm; + mask =x86_task_fpu (current->group_leader)->perm.__state_perm;if (mask &XFEATURE_MASK_USER_DYNAMIC )return ss_size > fsize;
arch/x86/kernel/traps.c +1 -1
@@ -1295,7 +1295,7 @@ DEFINE_IDTENTRY_RAW(exc_debug)static void math_error (struct pt_regs *regs,int trapnr) {struct task_struct *task = current; -struct fpu *fpu = &task->thread.fpu; +struct fpu *fpu =x86_task_fpu (task);int si_code;char *str = (trapnr ==X86_TRAP_MF ) ?"fpu exception" :"simd exception" ;
arch/x86/math-emu/fpu_aux.c +1 -1
@@ -53,7 +53,7 @@ void fpstate_init_soft(struct swregs_state *soft)void finit (void ) { -fpstate_init_soft (¤t->thread.fpu.fpstate->regs.soft); +fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); }/*
arch/x86/math-emu/fpu_entry.c +2 -2
@@ -641,7 +641,7 @@ int fpregs_soft_set(struct task_struct *target, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) { - struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft; + struct swregs_state *s387 = &x86_task_fpu(target)->fpstate->regs.soft;void *space = s387->st_space;int ret;int offset, other, i, tags, regnr, tag, newtop;
@@ -692,7 +692,7 @@ int fpregs_soft_get(struct task_struct *target, const struct user_regset *regset, struct membuf to) { -struct swregs_state *s387 = &target->thread.fpu.fpstate->regs.soft; +struct swregs_state *s387 = &x86_task_fpu (target)->fpstate->regs.soft;const void *space = s387->st_space;int offset = (S387 ->ftop &7 ) *10 , other =80 - offset;
arch/x86/math-emu/fpu_system.h +1 -1
@@ -73,7 +73,7 @@ static inline bool seg_writable(struct desc_struct *d)return (d->type &SEG_TYPE_EXECUTE_MASK ) ==SEG_TYPE_WRITABLE ; } -#define I387 (¤t->thread.fpu.fpstate->regs) +#define I387 (&x86_task_fpu(current)->fpstate->regs)#define FPU_info (I387->soft.info)#define FPU_CS (*(unsigned short *) &(FPU_info->regs->cs))
arch/x86/mm/extable.c +1 -1
@@ -111,7 +111,7 @@ static bool ex_handler_sgx(const struct exception_table_entry *fixup,/* * Handler for when we fail to restore a task's FPU state. We should never get -* here because the FPU state of a task using the FPU (task->thread.fpu.state) +* here because the FPU state of a task using the FPU (struct fpu::fpstate) * should always be valid. However, past bugs have allowed userspace to set * reserved bits in the XSAVE area using PTRACE_SETREGSET or sys_rt_sigreturn(). * These caused XRSTOR to fail when switching to the task, leaking the FPU
PATCH 3/8 x86/fpu: Make task_struct::thread constant size
In reply to Ingo MolnarTurn thread.fpu into a pointer. Since most FPU code internals work by passing around the FPU pointer already, the code generation impact is small.
This allows us to remove the old kludge of task_struct being variable size:
struct task_struct { ...
/*
* New fields for task_struct should be added above here, so that
* they are included in the randomized portion of task_struct.
*/
randomized_struct_fields_end
/* CPU-specific state of this task: */
struct thread_struct thread;
/*
* WARNING: on x86, 'thread_struct' contains a variable-sized
* structure. It *MUST* be at the end of 'task_struct'.
*
* Do not put anything below here!
*/};
... which creates a number of problems, such as requiring thread_struct to be the last member of the struct - not allowing it to be struct-randomized, etc.
But the primary motivation is to allow the decoupling of task_struct from hardware details (<asm/processor.h> in particular), and to eventually allow the per-task infrastructure:
DECLARE_PER_TASK(type, name); ... per_task(current, name) = val;
... which requires task_struct to be a constant size struct.
The fpu_thread_struct_whitelist() quirk to hardened usercopy can be removed, now that the FPU structure is not embedded in the task struct anymore, which reduces text footprint a bit.
--- arch/x86/include/asm/processor.h | 20 +++++++++----------- arch/x86/kernel/fpu/core.c | 23 ++++++++++++----------- arch/x86/kernel/fpu/init.c | 17 ++++++++++------- arch/x86/kernel/process.c | 2 +- include/linux/sched.h | 15 ++++----------- 5 files changed, 36 insertions(+), 41 deletions(-)
arch/x86/include/asm/processor.h +9 -11
@@ -516,21 +516,19 @@ struct thread_struct {#endif /* Floating point and extended processor state */ -struct fpu fpu; -/* -* WARNING: 'fpu' is dynamically-sized. It *MUST* be at -* the end. -*/ +struct fpu *fpu; }; -#define x86_task_fpu (task) (&(task)->thread.fpu) - -extern void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size); +#define x86_task_fpu (task) ((task)->thread.fpu) -static inline void arch_thread_struct_whitelist (unsigned long *offset, -unsigned long *size) +/* +* X86 doesn't need any embedded-FPU-struct quirks: +*/ +static inline void +arch_thread_struct_whitelist (unsigned long *offset,unsigned long *size) { -fpu_thread_struct_whitelist (offset, size); + *offset =0 ; + *size =0 ; } static inline void
arch/x86/kernel/fpu/core.c +12 -11
@@ -593,8 +593,19 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long ssp) { +/* +* We allocate the new FPU structure right after the end of the task struct. +* task allocation size already took this into account. +* +* This is safe because task_struct size is a multiple of cacheline size. +*/ struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu =x86_task_fpu (dst); +struct fpu *dst_fpu = (void *)dst +sizeof (*dst); + +BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 ); +BUG_ON (!src_fpu); + + dst->thread.fpu = dst_fpu;/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -663,16 +674,6 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,return 0 ; } -/* -* Whitelist the FPU register state embedded into task_struct for hardened -* usercopy. -*/ -void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size) -{ - *offset = offsetof(struct thread_struct, fpu.__fpstate.regs); - *size = fpu_kernel_cfg.default_size; -} -/* * Drops current FPU state: deactivates the fpregs and * the fpstate. NOTE: it still leaves previous contents
arch/x86/kernel/fpu/init.c +10 -7
@@ -71,8 +71,15 @@ static bool __init fpu__probe_without_cpuid(void)return fsw ==0 && (fcw &0x103f ) ==0x003f ; } +static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly; +static void __init fpu__init_system_early_generic (void ) { +fpstate_reset (&x86_init_fpu); + current->thread.fpu = &x86_init_fpu; +set_thread_flag (TIF_NEED_FPU_LOAD ); + x86_init_fpu.last_cpu =-1 ; +if (!boot_cpu_has (X86_FEATURE_CPUID ) && !test_bit (X86_FEATURE_FPU , (unsigned long *)cpu_caps_cleared)) {if (fpu__probe_without_cpuid ())
@@ -150,6 +157,8 @@ static void __init fpu__init_task_struct_size(void) {int task_size =sizeof (struct task_struct ); + task_size +=sizeof (struct fpu ); +/* * Subtract off the static size of the register state. * It potentially has a bunch of padding.
@@ -164,14 +173,9 @@ static void __init fpu__init_task_struct_size(void)/* * We dynamically size 'struct fpu', so we require that -* it be at the end of 'thread_struct' and that -* 'thread_struct' be at the end of 'task_struct'. If -* you hit a compile error here, check the structure to -* see if something got added to the end. +* 'state' be at the end of 'it: */ CHECK_MEMBER_AT_END_OF(struct fpu, __fpstate); - CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu); - CHECK_MEMBER_AT_END_OF(struct task_struct, thread); arch_task_struct_size = task_size; }
@@ -213,7 +217,6 @@ static void __init fpu__init_system_xstate_size_legacy(void)*/ void __init fpu__init_system (void ) { -fpstate_reset (x86_task_fpu (current));fpu__init_system_early_generic ();/*
arch/x86/kernel/process.c +1 -1
@@ -103,7 +103,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.vm86 =NULL ;#endif /* Drop the copied pointer to current's fpstate */ -x86_task_fpu (dst)->fpstate =NULL ; + dst->thread.fpu =NULL ;return 0 ; }
include/linux/sched.h +4 -11
@@ -1646,22 +1646,15 @@ struct task_struct {struct user_event_mm *user_event_mm;#endif -/* -* New fields for task_struct should be added above here, so that -* they are included in the randomized portion of task_struct. -*/ -randomized_struct_fields_end -/* CPU-specific state of this task: */ struct thread_struct thread;/* -* WARNING: on x86, 'thread_struct' contains a variable-sized -* structure. It *MUST* be at the end of 'task_struct'. -* -* Do not put anything below here! +* New fields for task_struct should be added above here, so that +* they are included in the randomized portion of task_struct. */ -}; +randomized_struct_fields_end +} __attribute__ ((aligned (64)));#define TASK_REPORT_IDLE (TASK_REPORT + 1)#define TASK_REPORT_MAX (TASK_REPORT_IDLE << 1)
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Make task_struct::thread constant size
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: cb7ca40a3882360ce87191793449d48df0b29184 Gitweb: https://git.kernel.org/tip/cb7ca40a3882360ce87191793449d48df0b29184 Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:22 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Make task_struct::thread constant size
Turn thread.fpu into a pointer. Since most FPU code internals work by passing around the FPU pointer already, the code generation impact is small.
This allows us to remove the old kludge of task_struct being variable size:
struct task_struct { ...
/*
* New fields for task_struct should be added above here, so that
* they are included in the randomized portion of task_struct.
*/
randomized_struct_fields_end
/* CPU-specific state of this task: */
struct thread_struct thread;
/*
* WARNING: on x86, 'thread_struct' contains a variable-sized
* structure. It *MUST* be at the end of 'task_struct'.
*
* Do not put anything below here!
*/};
... which creates a number of problems, such as requiring thread_struct to be the last member of the struct - not allowing it to be struct-randomized, etc.
But the primary motivation is to allow the decoupling of task_struct from hardware details (<asm/processor.h> in particular), and to eventually allow the per-task infrastructure:
DECLARE_PER_TASK(type, name); ... per_task(current, name) = val;
... which requires task_struct to be a constant size struct.
The fpu_thread_struct_whitelist() quirk to hardened usercopy can be removed, now that the FPU structure is not embedded in the task struct anymore, which reduces text footprint a bit.
--- arch/x86/include/asm/processor.h | 20 +++++++++----------- arch/x86/kernel/fpu/core.c | 23 ++++++++++++----------- arch/x86/kernel/fpu/init.c | 17 ++++++++++------- arch/x86/kernel/process.c | 2 +- include/linux/sched.h | 15 ++++----------- 5 files changed, 36 insertions(+), 41 deletions(-)
arch/x86/include/asm/processor.h +9 -11
@@ -516,21 +516,19 @@ struct thread_struct {#endif /* Floating point and extended processor state */ -struct fpu fpu; -/* -* WARNING: 'fpu' is dynamically-sized. It *MUST* be at -* the end. -*/ +struct fpu *fpu; }; -#define x86_task_fpu (task) (&(task)->thread.fpu) - -extern void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size); +#define x86_task_fpu (task) ((task)->thread.fpu) -static inline void arch_thread_struct_whitelist (unsigned long *offset, -unsigned long *size) +/* +* X86 doesn't need any embedded-FPU-struct quirks: +*/ +static inline void +arch_thread_struct_whitelist (unsigned long *offset,unsigned long *size) { -fpu_thread_struct_whitelist (offset, size); + *offset =0 ; + *size =0 ; } static inline void
arch/x86/kernel/fpu/core.c +12 -11
@@ -593,8 +593,19 @@ static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long ssp) { +/* +* We allocate the new FPU structure right after the end of the task struct. +* task allocation size already took this into account. +* +* This is safe because task_struct size is a multiple of cacheline size. +*/ struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu =x86_task_fpu (dst); +struct fpu *dst_fpu = (void *)dst +sizeof (*dst); + +BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 ); +BUG_ON (!src_fpu); + + dst->thread.fpu = dst_fpu;/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -664,16 +675,6 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal, }/* -* Whitelist the FPU register state embedded into task_struct for hardened -* usercopy. -*/ -void fpu_thread_struct_whitelist (unsigned long *offset,unsigned long *size) -{ - *offset = offsetof(struct thread_struct, fpu.__fpstate.regs); - *size = fpu_kernel_cfg.default_size; -} - -/* * Drops current FPU state: deactivates the fpregs and * the fpstate. NOTE: it still leaves previous contents * in the fpregs in the eager-FPU case.
arch/x86/kernel/fpu/init.c +10 -7
@@ -71,8 +71,15 @@ static bool __init fpu__probe_without_cpuid(void)return fsw ==0 && (fcw &0x103f ) ==0x003f ; } +static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly; +static void __init fpu__init_system_early_generic (void ) { +fpstate_reset (&x86_init_fpu); + current->thread.fpu = &x86_init_fpu; +set_thread_flag (TIF_NEED_FPU_LOAD ); + x86_init_fpu.last_cpu =-1 ; +if (!boot_cpu_has (X86_FEATURE_CPUID ) && !test_bit (X86_FEATURE_FPU , (unsigned long *)cpu_caps_cleared)) {if (fpu__probe_without_cpuid ())
@@ -150,6 +157,8 @@ static void __init fpu__init_task_struct_size(void) {int task_size =sizeof (struct task_struct ); + task_size +=sizeof (struct fpu ); +/* * Subtract off the static size of the register state. * It potentially has a bunch of padding.
@@ -164,14 +173,9 @@ static void __init fpu__init_task_struct_size(void)/* * We dynamically size 'struct fpu', so we require that -* it be at the end of 'thread_struct' and that -* 'thread_struct' be at the end of 'task_struct'. If -* you hit a compile error here, check the structure to -* see if something got added to the end. +* 'state' be at the end of 'it: */ CHECK_MEMBER_AT_END_OF(struct fpu, __fpstate); - CHECK_MEMBER_AT_END_OF(struct thread_struct, fpu); - CHECK_MEMBER_AT_END_OF(struct task_struct, thread); arch_task_struct_size = task_size; }
@@ -213,7 +217,6 @@ static void __init fpu__init_system_xstate_size_legacy(void)*/ void __init fpu__init_system (void ) { -fpstate_reset (x86_task_fpu (current));fpu__init_system_early_generic ();/*
arch/x86/kernel/process.c +1 -1
@@ -103,7 +103,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.vm86 =NULL ;#endif /* Drop the copied pointer to current's fpstate */ -x86_task_fpu (dst)->fpstate =NULL ; + dst->thread.fpu =NULL ;return 0 ; }
include/linux/sched.h +4 -11
@@ -1646,22 +1646,15 @@ struct task_struct {struct user_event_mm *user_event_mm;#endif -/* -* New fields for task_struct should be added above here, so that -* they are included in the randomized portion of task_struct. -*/ -randomized_struct_fields_end -/* CPU-specific state of this task: */ struct thread_struct thread;/* -* WARNING: on x86, 'thread_struct' contains a variable-sized -* structure. It *MUST* be at the end of 'task_struct'. -* -* Do not put anything below here! +* New fields for task_struct should be added above here, so that +* they are included in the randomized portion of task_struct. */ -}; +randomized_struct_fields_end +} __attribute__ ((aligned (64)));#define TASK_REPORT_IDLE (TASK_REPORT + 1)#define TASK_REPORT_MAX (TASK_REPORT_IDLE << 1)
PATCH 4/8 x86/fpu: Remove the thread::fpu pointer
In reply to Ingo MolnarAs suggested by Oleg, remove the thread::fpu pointer, as we can calculate it via x86_task_fpu() at compile-time.
This improves code generation a bit:
kepler:~/tip> size vmlinux.before vmlinux.after
text data bss dec hex filename 26475405 10435342 1740804 38651551 24dc69f vmlinux.before 26475339 10959630 1216516 38651485 24dc65d vmlinux.after
--- arch/x86/include/asm/processor.h | 5 +---- arch/x86/kernel/fpu/core.c | 4 +--- arch/x86/kernel/fpu/init.c | 1 - arch/x86/kernel/process.c | 2 -- arch/x86/kernel/vmlinux.lds.S | 4 ++++ 5 files changed, 6 insertions(+), 10 deletions(-)
arch/x86/include/asm/processor.h +1 -4
@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; }; -#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))/* * X86 doesn't need any embedded-FPU-struct quirks:
arch/x86/kernel/fpu/core.c +1 -3
@@ -600,13 +600,11 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* This is safe because task_struct size is a multiple of cacheline size. */ struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu = (void *)dst +sizeof (*dst); +struct fpu *dst_fpu =x86_task_fpu (dst);BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 );BUG_ON (!src_fpu); - dst->thread.fpu = dst_fpu; -/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
arch/x86/kernel/fpu/init.c +0 -1
@@ -76,7 +76,6 @@ static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly;static void __init fpu__init_system_early_generic (void ) {fpstate_reset (&x86_init_fpu); - current->thread.fpu = &x86_init_fpu;set_thread_flag (TIF_NEED_FPU_LOAD ); x86_init_fpu.last_cpu =-1 ;
arch/x86/kernel/process.c +0 -2
@@ -102,8 +102,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)#ifdef CONFIG_VM86 dst->thread.vm86 =NULL ;#endif -/* Drop the copied pointer to current's fpstate */ - dst->thread.fpu =NULL ;return 0 ; }
arch/x86/kernel/vmlinux.lds.S +4 -0
@@ -181,6 +181,10 @@ SECTIONS /* equivalent to task_pt_regs(&init_task) */ __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; + __x86_init_fpu_begin = .; + . = __x86_init_fpu_begin + 128*PAGE_SIZE; + __x86_init_fpu_end = .; + #ifdef CONFIG_X86_32 /* 32 bit has nosave before _edata */ NOSAVE_DATA
Signature · 1 line
2.45.2
Peter Zijlstra
maintainerRe: [PATCH 4/8] x86/fpu: Remove the thread::fpu pointer
Quoted 5 lines · Ingo Molnar
arch/x86/include/asm/processor.h +0
-0
arch/x86/include/asm/processor.h struct thread_struct { · line 519
Quoted 10 lines
arch/x86/include/asm/processor.h +1
-4@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; }; -#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))
Doesn't our FPU state need to be cacheline aligned? Looking at struct fpu, it has a bunch of members that have __aligned__(64) on them, and as such the whole of struct fpu will inherit this alignment requirement.
This means that both task and sizeof(*task) need to be cacheline aligned for this to work.
Would it make sense to add something like:
static_assert(sizeof(struct task_struct) % 64 == 0);
And I didn't check, but isn't task a page pointer and as such always page aligned?
Re: [PATCH 4/8] x86/fpu: Remove the thread::fpu pointer
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Ingo Molnar
arch/x86/include/asm/processor.h +0
-3@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; };
Quoted 4 lines
arch/x86/include/asm/processor.h struct thread_struct { · line 519
Quoted 2 lines
arch/x86/include/asm/processor.h +1
-1-#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))Doesn't our FPU state need to be cacheline aligned?
Yeah, and we do have a check for that:
+ BUILD_BUG_ON(sizeof(*dst) % SMP_CACHE_BYTES != 0);
And task_struct is allocated cache-aligned, which means when we do this in fpu_clone():
+ struct fpu *dst_fpu = (void *)dst + sizeof(*dst);
the FPU pointer is guaranteed to be cacheline aligned as well.
'dst' in that context is the new task_struct.
BTW., Oleg suggested in a previous discussion for us to replace the task->thread.fpu pointer with a build-time calculation - but I'm still not sure it's a good idea.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 4/8] x86/fpu: Remove the thread::fpu pointer
Quoted 18 lines · Ingo Molnar
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Ingo Molnar
arch/x86/include/asm/processor.h +0
-3@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; };
Quoted 8 lines
Quoted 4 lines
arch/x86/include/asm/processor.h struct thread_struct { · line 519
Quoted 2 lines
arch/x86/include/asm/processor.h +1
-1-#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))Doesn't our FPU state need to be cacheline aligned?
Yeah, and we do have a check for that:
+
BUILD_BUG_ON(sizeof(*dst) % SMP_CACHE_BYTES != 0);
Ah, missed that. Clearly I need to improve my reading skillz :-)
PATCH 1/1 x86/fpu: Clarify FPU context cacheline alignment
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Ingo Molnar
Quoted 19 lines · Peter Zijlstra
Quoted 5 lines · Ingo Molnar
arch/x86/include/asm/processor.h +0
-0arch/x86/include/asm/processor.h struct thread_struct { · line 519
Quoted 10 lines
arch/x86/include/asm/processor.h +1
-4@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; }; -#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))Doesn't our FPU state need to be cacheline aligned?
Quoted 5 lines
Quoted 3 lines
Yeah, and we do have a check for that:
+
BUILD_BUG_ON(sizeof(*dst) % SMP_CACHE_BYTES != 0);Ah, missed that. Clearly I need to improve my reading skillz :-)
Admittedly it's written a bit obtusely - how about the patch below?
Thanks,
Ingo
============================>
From: Ingo Molnar <[email protected]>
Date: Thu, 10 Apr 2025 12:52:16 +0200
Subject: [PATCH] x86/fpu: Clarify FPU context cacheline alignment
--- arch/x86/kernel/fpu/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/fpu/core.c +2 -1
@@ -607,7 +607,8 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* We allocate the new FPU structure right after the end of the task struct. * task allocation size already took this into account. * -* This is safe because task_struct size is a multiple of cacheline size. +* This is safe because task_struct size is a multiple of cacheline size, +* thus x86_task_fpu() will always be cacheline aligned as well. */ struct fpu *dst_fpu = (void *)dst +sizeof (*dst);
[tip: x86/merge] x86/fpu: Clarify FPU context cacheline alignment
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: e3a52b67f54aa36ab21265eeea016460b5fe1c46 Gitweb: https://git.kernel.org/tip/e3a52b67f54aa36ab21265eeea016460b5fe1c46 Author: Ingo Molnar <[email protected]> AuthorDate: Thu, 10 Apr 2025 12:52:16 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Clarify FPU context cacheline alignment
--- arch/x86/kernel/fpu/core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/fpu/core.c +2 -1
@@ -607,7 +607,8 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* We allocate the new FPU structure right after the end of the task struct. * task allocation size already took this into account. * -* This is safe because task_struct size is a multiple of cacheline size. +* This is safe because task_struct size is a multiple of cacheline size, +* thus x86_task_fpu() will always be cacheline aligned as well. */ struct fpu *dst_fpu = (void *)dst +sizeof (*dst);
Re: [PATCH 4/8] x86/fpu: Remove the thread::fpu pointer
In reply to Ingo MolnarQuoted 3 lines · Ingo Molnar
BTW., Oleg suggested in a previous discussion for us to replace the
task->thread.fpupointer with a build-time calculation - but I'm still not sure it's a good idea.
Actually, I've implemented that already:
arch/x86/include/asm/processor.h:# define x86_task_fpu(task) ((struct fpu *)((void *)(task) + sizeof(*(task))))
... then promptly forgot about it. So never mind me. :-)
Thanks,
Ingo
Re: [PATCH 4/8] x86/fpu: Remove the thread::fpu pointer
In reply to Ingo MolnarQuoted 19 lines · Ingo Molnar
Quoted 6 lines · Peter Zijlstra
arch/x86/include/asm/processor.h struct thread_struct { · line 519
Quoted 2 lines · Ingo Molnar
arch/x86/include/asm/processor.h +1
-1-#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))Doesn't our FPU state need to be cacheline aligned?
Yeah, and we do have a check for that:
+
BUILD_BUG_ON(sizeof(*dst) % SMP_CACHE_BYTES != 0);And
task_structis allocated cache-aligned, which means when we do this infpu_clone():+ struct fpu
*dst_fpu= (void *)dst +sizeof(*dst);
Quoted 7 lines
the FPU pointer is guaranteed to be cacheline aligned as well.
'dst' in that context is the new
task_struct.BTW., Oleg suggested in a previous discussion for us to replace the
task->thread.fpupointer with a build-time calculation - but I'm still not sure it's a good idea.
To be honest, I forgot everything we discussed before ;)
But I have found this email,
https://lore.kernel.org/all/[email protected]/
Perhaps
#define X86_TASK_SIZE \ ALIGN(sizeof(struct task_struct), __alignof__(union fpregs_state))
and
#define x86_task_fpu(task) \ ((struct fpu *)((void *)(task) + X86_TASK_SIZE))
plus a bit more similar changes make more sense than __attribute__ ((aligned (64))) for struct task_struct?
OK, even if yes, we can do this later on top of this series.
I'll try to read it tomorrow.
Oleg.
[tip: x86/merge] x86/fpu: Remove the thread::fpu pointer
In reply to Ingo MolnarThe following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: 55bc30f2e34dcc17a370d1f6c1c992be107c4502 Gitweb: https://git.kernel.org/tip/55bc30f2e34dcc17a370d1f6c1c992be107c4502 Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:23 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Remove the thread::fpu pointer
As suggested by Oleg, remove the thread::fpu pointer, as we can calculate it via x86_task_fpu() at compile-time.
This improves code generation a bit:
kepler:~/tip> size vmlinux.before vmlinux.after
text data bss dec hex filename 26475405 10435342 1740804 38651551 24dc69f vmlinux.before 26475339 10959630 1216516 38651485 24dc65d vmlinux.after
--- arch/x86/include/asm/processor.h | 5 +---- arch/x86/kernel/fpu/core.c | 4 +--- arch/x86/kernel/fpu/init.c | 1 - arch/x86/kernel/process.c | 2 -- arch/x86/kernel/vmlinux.lds.S | 4 ++++ 5 files changed, 6 insertions(+), 10 deletions(-)
arch/x86/include/asm/processor.h +1 -4
@@ -514,12 +514,9 @@ struct thread_struct {struct thread_shstk shstk;#endif - -/* Floating point and extended processor state */ -struct fpu *fpu; }; -#define x86_task_fpu (task) ((task)->thread.fpu) +#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task))))/* * X86 doesn't need any embedded-FPU-struct quirks:
arch/x86/kernel/fpu/core.c +1 -3
@@ -600,13 +600,11 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* This is safe because task_struct size is a multiple of cacheline size. */ struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu = (void *)dst +sizeof (*dst); +struct fpu *dst_fpu =x86_task_fpu (dst);BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 );BUG_ON (!src_fpu); - dst->thread.fpu = dst_fpu; -/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
arch/x86/kernel/fpu/init.c +0 -1
@@ -76,7 +76,6 @@ static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly;static void __init fpu__init_system_early_generic (void ) {fpstate_reset (&x86_init_fpu); - current->thread.fpu = &x86_init_fpu;set_thread_flag (TIF_NEED_FPU_LOAD ); x86_init_fpu.last_cpu =-1 ;
arch/x86/kernel/process.c +0 -2
@@ -102,8 +102,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)#ifdef CONFIG_VM86 dst->thread.vm86 =NULL ;#endif -/* Drop the copied pointer to current's fpstate */ - dst->thread.fpu =NULL ;return 0 ; }
arch/x86/kernel/vmlinux.lds.S +4 -0
@@ -181,6 +181,10 @@ SECTIONS /* equivalent to task_pt_regs(&init_task) */ __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; + __x86_init_fpu_begin = .; + . = __x86_init_fpu_begin + 128*PAGE_SIZE; + __x86_init_fpu_end = .; + #ifdef CONFIG_X86_32 /* 32 bit has nosave before _edata */ NOSAVE_DATA
PATCH 5/8 x86/fpu: Push 'fpu' pointer calculation into the fpu__drop() call
In reply to Ingo MolnarThis encapsulates the fpu__drop() functionality better, and it will also enable other changes that want to check a task for PF_KTHREAD before calling x86_task_fpu().
--- arch/x86/include/asm/fpu/sched.h | 2 +- arch/x86/kernel/fpu/core.c | 4 +++- arch/x86/kernel/process.c | 3 +-- 3 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu/sched.h +1 -1
@@ -10,7 +10,7 @@#include <asm/trace/fpu.h> extern void save_fpregs_to_fpstate (struct fpu *fpu); -extern void fpu__drop (struct fpu *fpu); +extern void fpu__drop (struct task_struct *tsk);extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long shstk_addr);extern void fpu_flush_thread (void );
arch/x86/kernel/fpu/core.c +3 -1
@@ -681,8 +681,10 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* a state-restore is coming: either an explicit one, * or a reschedule. */ -void fpu__drop (struct fpu *fpu) +void fpu__drop (struct task_struct *tsk) { +struct fpu *fpu =x86_task_fpu (tsk); +preempt_disable ();if (fpu ==x86_task_fpu (current)) {
arch/x86/kernel/process.c +1 -2
@@ -120,7 +120,6 @@ void arch_release_task_struct(struct task_struct *tsk)void exit_thread (struct task_struct *tsk) {struct thread_struct *t = &tsk->thread; -struct fpu *fpu =x86_task_fpu (tsk);if (test_thread_flag (TIF_IO_BITMAP ))io_bitmap_exit (tsk);
@@ -128,7 +127,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t);shstk_free (tsk); -fpu__drop (fpu); +fpu__drop (tsk); }static int set_new_tls (struct task_struct *p,unsigned long tls)
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Push 'fpu' pointer calculation into the fpu__drop() call
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: ec2227e03a46a162f2721917262adc553b212e2d Gitweb: https://git.kernel.org/tip/ec2227e03a46a162f2721917262adc553b212e2d Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:24 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Push 'fpu' pointer calculation into the fpu__drop() call
This encapsulates the fpu__drop() functionality better, and it will also enable other changes that want to check a task for PF_KTHREAD before calling x86_task_fpu().
--- arch/x86/include/asm/fpu/sched.h | 2 +- arch/x86/kernel/fpu/core.c | 4 +++- arch/x86/kernel/process.c | 3 +-- 3 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu/sched.h +1 -1
@@ -10,7 +10,7 @@#include <asm/trace/fpu.h> extern void save_fpregs_to_fpstate (struct fpu *fpu); -extern void fpu__drop (struct fpu *fpu); +extern void fpu__drop (struct task_struct *tsk);extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal,unsigned long shstk_addr);extern void fpu_flush_thread (void );
arch/x86/kernel/fpu/core.c +3 -1
@@ -681,8 +681,10 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* a state-restore is coming: either an explicit one, * or a reschedule. */ -void fpu__drop (struct fpu *fpu) +void fpu__drop (struct task_struct *tsk) { +struct fpu *fpu =x86_task_fpu (tsk); +preempt_disable ();if (fpu ==x86_task_fpu (current)) {
arch/x86/kernel/process.c +1 -2
@@ -120,7 +120,6 @@ void arch_release_task_struct(struct task_struct *tsk)void exit_thread (struct task_struct *tsk) {struct thread_struct *t = &tsk->thread; -struct fpu *fpu =x86_task_fpu (tsk);if (test_thread_flag (TIF_IO_BITMAP ))io_bitmap_exit (tsk);
@@ -128,7 +127,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t);shstk_free (tsk); -fpu__drop (fpu); +fpu__drop (tsk); }static int set_new_tls (struct task_struct *p,unsigned long tls)
PATCH 6/8 x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit
In reply to Ingo Molnarfpu__drop() and arch_release_task_struct() calls x86_task_fpu() unconditionally, while the FPU context area will not be present if it's the init task, and should not be in use when it's some other type of kthread.
Return early for PF_KTHREAD or PF_USER_WORKER tasks. The debug warning in x86_task_fpu() will catch any kthreads attempting to use the FPU save area.
--- arch/x86/kernel/fpu/core.c | 8 +++++++- arch/x86/kernel/process.c | 2 +- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/fpu/core.c +7 -1
@@ -683,7 +683,13 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,*/ void fpu__drop (struct task_struct *tsk) { -struct fpu *fpu =x86_task_fpu (tsk); +struct fpu *fpu; + +/* PF_KTHREAD tasks do not use the FPU context area: */ +if (tsk->flags & (PF_KTHREAD |PF_USER_WORKER )) +return ; + + fpu =x86_task_fpu (tsk);preempt_disable ();
arch/x86/kernel/process.c +1 -1
@@ -109,7 +109,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)#ifdef CONFIG_X86_64 void arch_release_task_struct (struct task_struct *tsk) { -if (fpu_state_size_dynamic ()) +if (fpu_state_size_dynamic () && !(current->flags & (PF_KTHREAD |PF_USER_WORKER )))fpstate_free (x86_task_fpu (tsk)); }#endif
Signature · 1 line
2.45.2
Re: [PATCH 6/8] x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit
arch/x86/kernel/process.c int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) · line 114
Quoted 7 lines · Ingo Molnar
void arch_release_task_struct(struct task_struct *tsk) {arch/x86/kernel/process.c +1
-1-if (fpu_state_size_dynamic ()) +if (fpu_state_size_dynamic () && !(current->flags & (PF_KTHREAD |PF_USER_WORKER )))fpstate_free (x86_task_fpu (tsk)); }
Looks like the AMX system couldn't boot properly, and I found that referencing tsk->flags here resolves the issue.
Thanks,
Chang
Re: [PATCH 6/8] x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit
Quoted 10 lines · Chang S. Bae
Review
Quoted 6 lines · Ingo Molnar
void arch_release_task_struct(struct task_struct *tsk) {(quoted diff) +1
-1- if (fpu_state_size_dynamic()) + if (fpu_state_size_dynamic() && !(current->flags & (PF_KTHREAD | PF_USER_WORKER))) fpstate_free(x86_task_fpu(tsk)); }Looks like the AMX system couldn't boot properly, and I found that referencing
tsk->flagshere resolves the issue.
Indeed, fixed!
I've also added your Fixed-by tag to the patch:
Thanks,
Ingo
==========>
arch/x86/kernel/process.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/process.c +1 -1
@@ -109,7 +109,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)#ifdef CONFIG_X86_64 void arch_release_task_struct (struct task_struct *tsk) { -if (fpu_state_size_dynamic () && !(current->flags & (PF_KTHREAD |PF_USER_WORKER ))) +if (fpu_state_size_dynamic () && !(tsk->flags & (PF_KTHREAD |PF_USER_WORKER )))fpstate_free (x86_task_fpu (tsk)); }#endif
[tip: x86/merge] x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit
In reply to Ingo MolnarThe following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: c360bdc593b8a8b6e94166026728764085919cff Gitweb: https://git.kernel.org/tip/c360bdc593b8a8b6e94166026728764085919cff Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:25 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Make sure x86_task_fpu() doesn't get called for PF_KTHREAD|PF_USER_WORKER tasks during exit
fpu__drop() and arch_release_task_struct() calls x86_task_fpu() unconditionally, while the FPU context area will not be present if it's the init task, and should not be in use when it's some other type of kthread.
Return early for PF_KTHREAD or PF_USER_WORKER tasks. The debug warning in x86_task_fpu() will catch any kthreads attempting to use the FPU save area.
--- arch/x86/kernel/fpu/core.c | 8 +++++++- arch/x86/kernel/process.c | 2 +- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/fpu/core.c +7 -1
@@ -683,7 +683,13 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,*/ void fpu__drop (struct task_struct *tsk) { -struct fpu *fpu =x86_task_fpu (tsk); +struct fpu *fpu; + +/* PF_KTHREAD tasks do not use the FPU context area: */ +if (tsk->flags & (PF_KTHREAD |PF_USER_WORKER )) +return ; + + fpu =x86_task_fpu (tsk);preempt_disable ();
arch/x86/kernel/process.c +1 -1
@@ -109,7 +109,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)#ifdef CONFIG_X86_64 void arch_release_task_struct (struct task_struct *tsk) { -if (fpu_state_size_dynamic ()) +if (fpu_state_size_dynamic () && !(tsk->flags & (PF_KTHREAD |PF_USER_WORKER )))fpstate_free (x86_task_fpu (tsk)); }#endif
PATCH 7/8 x86/fpu: Remove init_task FPU state dependencies, add debugging warning for PF_KTHREAD tasks
In reply to Ingo Molnarinit_task's FPU state initialization was a bit of a hack:
__x86_init_fpu_begin = .; . = __x86_init_fpu_begin + 128*PAGE_SIZE; __x86_init_fpu_end = .;
But the init task isn't supposed to be using the FPU context in any case, so remove the hack and add in some debug warnings.
As Linus noted in the discussion, the init task (and other PF_KTHREAD tasks) *can* use the FPU via kernel_fpu_begin()/_end(), but they don't need the context area because their FPU use is not preemptible or reentrant, and they don't return to user-space.
--- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/fpu/core.c | 15 +++++++++++---- arch/x86/kernel/fpu/init.c | 3 +-- arch/x86/kernel/fpu/xstate.c | 3 --- arch/x86/kernel/vmlinux.lds.S | 4 ---- 5 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/include/asm/processor.h +5 -1
@@ -516,7 +516,11 @@ struct thread_struct {#endif }; -#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task)))) +#ifdef CONFIG_X86_DEBUG_FPU +extern struct fpu *x86_task_fpu (struct task_struct *task); +#else +# define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task)))) +#endif /* * X86 doesn't need any embedded-FPU-struct quirks:
arch/x86/kernel/fpu/core.c +11 -4
@@ -51,6 +51,16 @@ static DEFINE_PER_CPU(bool, in_kernel_fpu);*/ DEFINE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx); +#ifdef CONFIG_X86_DEBUG_FPU +struct fpu *x86_task_fpu (struct task_struct *task) +{ +if (WARN_ON_ONCE (task->flags &PF_KTHREAD )) +return NULL ; + +return (void *)task +sizeof (*task); +} +#endif +/* * Can we use the FPU in kernel mode with the * whole "kernel_fpu_begin/end()" sequence?
@@ -599,11 +609,9 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* * This is safe because task_struct size is a multiple of cacheline size. */ -struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu =x86_task_fpu (dst); +struct fpu *dst_fpu = (void *)dst +sizeof (*dst);BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 ); -BUG_ON (!src_fpu);/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -666,7 +674,6 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,if (update_fpu_shstk (dst, ssp))return 1 ; -trace_x86_fpu_copy_src (src_fpu);trace_x86_fpu_copy_dst (dst_fpu);return 0 ;
arch/x86/kernel/fpu/init.c +1 -2
@@ -38,7 +38,7 @@ static void fpu__init_cpu_generic(void)/* Flush out any pending x87 state: */ #ifdef CONFIG_MATH_EMULATION if (!boot_cpu_has (X86_FEATURE_FPU )) -fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); + ; else#endif asmvolatile ("fninit" );
@@ -207,7 +207,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_kernel_cfg.default_size = size; fpu_user_cfg.max_size = size; fpu_user_cfg.default_size = size; -fpstate_reset (x86_task_fpu (current)); }/*
arch/x86/kernel/fpu/xstate.c +0 -3
@@ -834,9 +834,6 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)if (err) goto out_disable; -/* Reset the state for the current task */ -fpstate_reset (x86_task_fpu (current)); -/* * Update info used for ptrace frames; use standard-format size and no * supervisor xstates:
arch/x86/kernel/vmlinux.lds.S +0 -4
@@ -181,10 +181,6 @@ SECTIONS /* equivalent to task_pt_regs(&init_task) */ __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; - __x86_init_fpu_begin = .; - . = __x86_init_fpu_begin + 128*PAGE_SIZE; - __x86_init_fpu_end = .; - #ifdef CONFIG_X86_32 /* 32 bit has nosave before _edata */ NOSAVE_DATA
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Remove init_task FPU state dependencies, add debugging warning for PF_KTHREAD tasks
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: 22aafe3bcb67472effdea1ccf0df20280192bbaf Gitweb: https://git.kernel.org/tip/22aafe3bcb67472effdea1ccf0df20280192bbaf Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:26 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Remove init_task FPU state dependencies, add debugging warning for PF_KTHREAD tasks
init_task's FPU state initialization was a bit of a hack:
__x86_init_fpu_begin = .; . = __x86_init_fpu_begin + 128*PAGE_SIZE; __x86_init_fpu_end = .;
But the init task isn't supposed to be using the FPU context in any case, so remove the hack and add in some debug warnings.
As Linus noted in the discussion, the init task (and other PF_KTHREAD tasks) *can* use the FPU via kernel_fpu_begin()/_end(), but they don't need the context area because their FPU use is not preemptible or reentrant, and they don't return to user-space.
--- arch/x86/include/asm/processor.h | 6 +++++- arch/x86/kernel/fpu/core.c | 15 +++++++++++---- arch/x86/kernel/fpu/init.c | 3 +-- arch/x86/kernel/fpu/xstate.c | 3 --- arch/x86/kernel/vmlinux.lds.S | 4 ---- 5 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/include/asm/processor.h +5 -1
@@ -516,7 +516,11 @@ struct thread_struct {#endif }; -#define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task)))) +#ifdef CONFIG_X86_DEBUG_FPU +extern struct fpu *x86_task_fpu (struct task_struct *task); +#else +# define x86_task_fpu (task) ((struct fpu *)((void *)(task) + sizeof(*(task)))) +#endif /* * X86 doesn't need any embedded-FPU-struct quirks:
arch/x86/kernel/fpu/core.c +11 -4
@@ -51,6 +51,16 @@ static DEFINE_PER_CPU(bool, in_kernel_fpu);*/ DEFINE_PER_CPU(struct fpu *, fpu_fpregs_owner_ctx); +#ifdef CONFIG_X86_DEBUG_FPU +struct fpu *x86_task_fpu (struct task_struct *task) +{ +if (WARN_ON_ONCE (task->flags &PF_KTHREAD )) +return NULL ; + +return (void *)task +sizeof (*task); +} +#endif +/* * Can we use the FPU in kernel mode with the * whole "kernel_fpu_begin/end()" sequence?
@@ -599,11 +609,9 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,* * This is safe because task_struct size is a multiple of cacheline size. */ -struct fpu *src_fpu =x86_task_fpu (current); -struct fpu *dst_fpu =x86_task_fpu (dst); +struct fpu *dst_fpu = (void *)dst +sizeof (*dst);BUILD_BUG_ON (sizeof (*dst) %SMP_CACHE_BYTES !=0 ); -BUG_ON (!src_fpu);/* The new task's FPU state cannot be valid in the hardware. */ dst_fpu->last_cpu =-1 ;
@@ -666,7 +674,6 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,if (update_fpu_shstk (dst, ssp))return 1 ; -trace_x86_fpu_copy_src (src_fpu);trace_x86_fpu_copy_dst (dst_fpu);return 0 ;
arch/x86/kernel/fpu/init.c +1 -2
@@ -38,7 +38,7 @@ static void fpu__init_cpu_generic(void)/* Flush out any pending x87 state: */ #ifdef CONFIG_MATH_EMULATION if (!boot_cpu_has (X86_FEATURE_FPU )) -fpstate_init_soft (&x86_task_fpu (current)->fpstate->regs.soft); + ; else#endif asmvolatile ("fninit" );
@@ -207,7 +207,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_kernel_cfg.default_size = size; fpu_user_cfg.max_size = size; fpu_user_cfg.default_size = size; -fpstate_reset (x86_task_fpu (current)); }/*
arch/x86/kernel/fpu/xstate.c +0 -3
@@ -870,9 +870,6 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)if (err) goto out_disable; -/* Reset the state for the current task */ -fpstate_reset (x86_task_fpu (current)); -/* * Update info used for ptrace frames; use standard-format size and no * supervisor xstates:
arch/x86/kernel/vmlinux.lds.S +0 -4
@@ -181,10 +181,6 @@ SECTIONS /* equivalent to task_pt_regs(&init_task) */ __top_init_kernel_stack = __end_init_stack - TOP_OF_KERNEL_STACK_PADDING - PTREGS_SIZE; - __x86_init_fpu_begin = .; - . = __x86_init_fpu_begin + 128*PAGE_SIZE; - __x86_init_fpu_end = .; - #ifdef CONFIG_X86_32 /* 32 bit has nosave before _edata */ NOSAVE_DATA
PATCH 8/8 x86/fpu: Use 'fpstate' variable names consistently
In reply to Ingo MolnarA few uses of 'fps' snuck in, which is rather confusing (to me) as it suggests frames-per-second. ;-)
Rename them to the canonical 'fpstate' name.
No change in functionality.
--- arch/x86/include/asm/fpu/api.h | 2 +- arch/x86/kernel/fpu/core.c | 14 +++++++------- arch/x86/kernel/fpu/xstate.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-)
arch/x86/include/asm/fpu/api.h +1 -1
@@ -136,7 +136,7 @@ static inline void fpstate_free(struct fpu *fpu) { }#endif /* fpstate-related functions which are exported to KVM */ -extern void fpstate_clear_xstate_component (struct fpstate *fps,unsigned int xfeature); +extern void fpstate_clear_xstate_component (struct fpstate *fpstate,unsigned int xfeature);extern u64 xstate_get_guest_group_perm (void );
arch/x86/kernel/fpu/core.c +7 -7
@@ -273,16 +273,16 @@ EXPORT_SYMBOL_GPL(fpu_alloc_guest_fpstate);void fpu_free_guest_fpstate (struct fpu_guest *gfpu) { -struct fpstate *fps = gfpu->fpstate; +struct fpstate *fpstate = gfpu->fpstate; -if (!fps) +if (!fpstate)return ; -if (WARN_ON_ONCE (!fps->is_valloc || !fps->is_guest || fps->in_use)) +if (WARN_ON_ONCE (!fpstate->is_valloc || !fpstate->is_guest || fpstate->in_use))return ; gfpu->fpstate =NULL ; -vfree (fps); +vfree (fpstate); }EXPORT_SYMBOL_GPL (fpu_free_guest_fpstate);
@@ -333,12 +333,12 @@ EXPORT_SYMBOL_GPL(fpu_update_guest_xfd);*/ void fpu_sync_guest_vmexit_xfd_state (void ) { -struct fpstate *fps =x86_task_fpu (current)->fpstate; +struct fpstate *fpstate =x86_task_fpu (current)->fpstate;lockdep_assert_irqs_disabled ();if (fpu_state_size_dynamic ()) { -rdmsrl (MSR_IA32_XFD , fps->xfd); -__this_cpu_write (xfd_state, fps->xfd); +rdmsrl (MSR_IA32_XFD , fpstate->xfd); +__this_cpu_write (xfd_state, fpstate->xfd); } }EXPORT_SYMBOL_GPL (fpu_sync_guest_vmexit_xfd_state);
arch/x86/kernel/fpu/xstate.c +2 -2
@@ -1395,9 +1395,9 @@ void xrstors(struct xregs_state *xstate, u64 mask) }#if IS_ENABLED (CONFIG_KVM ) -void fpstate_clear_xstate_component (struct fpstate *fps,unsigned int xfeature) +void fpstate_clear_xstate_component (struct fpstate *fpstate,unsigned int xfeature) { -void *addr =get_xsave_addr (&fps->regs.xsave, xfeature); +void *addr =get_xsave_addr (&fpstate->regs.xsave, xfeature);if (addr)memset (addr,0 , xstate_sizes[xfeature]);
Signature · 1 line
2.45.2
[tip: x86/merge] x86/fpu: Use 'fpstate' variable names consistently
The following commit has been merged into the x86/merge branch of tip:
The following commit has been merged into the x86/merge branch of tip:
Commit-ID: 8b2a7a7294b34fa00adbecbd352ef19eb780261b Gitweb: https://git.kernel.org/tip/8b2a7a7294b34fa00adbecbd352ef19eb780261b Author: Ingo Molnar <[email protected]> AuthorDate: Wed, 09 Apr 2025 23:11:27 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Apr 2025 08:18:29 +02:00
x86/fpu: Use 'fpstate' variable names consistently
A few uses of 'fps' snuck in, which is rather confusing (to me) as it suggests frames-per-second. ;-)
Rename them to the canonical 'fpstate' name.
No change in functionality.
--- arch/x86/include/asm/fpu/api.h | 2 +- arch/x86/kernel/fpu/core.c | 14 +++++++------- arch/x86/kernel/fpu/xstate.c | 4 ++-- 3 files changed, 10 insertions(+), 10 deletions(-)
arch/x86/include/asm/fpu/api.h +1 -1
@@ -136,7 +136,7 @@ static inline void fpstate_free(struct fpu *fpu) { }#endif /* fpstate-related functions which are exported to KVM */ -extern void fpstate_clear_xstate_component (struct fpstate *fps,unsigned int xfeature); +extern void fpstate_clear_xstate_component (struct fpstate *fpstate,unsigned int xfeature);extern u64 xstate_get_guest_group_perm (void );
arch/x86/kernel/fpu/core.c +7 -7
@@ -273,16 +273,16 @@ EXPORT_SYMBOL_GPL(fpu_alloc_guest_fpstate);void fpu_free_guest_fpstate (struct fpu_guest *gfpu) { -struct fpstate *fps = gfpu->fpstate; +struct fpstate *fpstate = gfpu->fpstate; -if (!fps) +if (!fpstate)return ; -if (WARN_ON_ONCE (!fps->is_valloc || !fps->is_guest || fps->in_use)) +if (WARN_ON_ONCE (!fpstate->is_valloc || !fpstate->is_guest || fpstate->in_use))return ; gfpu->fpstate =NULL ; -vfree (fps); +vfree (fpstate); }EXPORT_SYMBOL_GPL (fpu_free_guest_fpstate);
@@ -333,12 +333,12 @@ EXPORT_SYMBOL_GPL(fpu_update_guest_xfd);*/ void fpu_sync_guest_vmexit_xfd_state (void ) { -struct fpstate *fps =x86_task_fpu (current)->fpstate; +struct fpstate *fpstate =x86_task_fpu (current)->fpstate;lockdep_assert_irqs_disabled ();if (fpu_state_size_dynamic ()) { -rdmsrl (MSR_IA32_XFD , fps->xfd); -__this_cpu_write (xfd_state, fps->xfd); +rdmsrl (MSR_IA32_XFD , fpstate->xfd); +__this_cpu_write (xfd_state, fpstate->xfd); } }EXPORT_SYMBOL_GPL (fpu_sync_guest_vmexit_xfd_state);
arch/x86/kernel/fpu/xstate.c +2 -2
@@ -1431,9 +1431,9 @@ void xrstors(struct xregs_state *xstate, u64 mask) }#if IS_ENABLED (CONFIG_KVM ) -void fpstate_clear_xstate_component (struct fpstate *fps,unsigned int xfeature) +void fpstate_clear_xstate_component (struct fpstate *fpstate,unsigned int xfeature) { -void *addr =get_xsave_addr (&fps->regs.xsave, xfeature); +void *addr =get_xsave_addr (&fpstate->regs.xsave, xfeature);if (addr)memset (addr,0 , xstate_sizes[xfeature]);
Ingo, sorry for delay.
So just in case, the whole series looks good to me. I am going to send a couple of minor cleanups on top of it, but let me ask first if I missed something or not.
- x86_init_fpu is not really used after 4/8, it can be killed
- DEFINE_EVENT(x86_fpu, x86_fpu_copy_src) can be killed after 7/8
- arch_dup_task_struct() still does
/* init_task is not dynamically sized (incomplete FPU state) */ if (unlikely(src == &init_task)) memcpy_and_pad(dst, arch_task_struct_size, src, sizeof(init_task), 0); else memcpy(dst, src, arch_task_struct_size);
and I don't understand why do we need to check src == &init_task. It seems that we can always do
memcpy_and_pad(dst, arch_task_struct_size, src, sizeof(struct task_struct), 0);
or even just
memcpy(dst, src, sizeof(struct task_struct));
fpu_clone() will initialize the "dst_fpu" memory correctly.
- fpu__drop() does
/* PF_KTHREAD tasks do not use the FPU context area: */ if (tsk->flags & (PF_KTHREAD | PF_USER_WORKER)) return;
and this is correct. But perhaps
if (test_tsk_thread_flag(tsk, TIF_NEED_FPU_LOAD)) return;
makes more sense? PF_KTHREAD's should never clear TIF_NEED_FPU_LOAD, and this way we can avoid the unnecessary "fwait" if, say, the exiting task does context_switch() at least once on its way to exit_thread().
- Finally, with or without these changes, it seems that the switch_fpu_prepare() + switch_fpu_finish() logic can be simplified, I'll write another email.
Quoted 4 lines · Oleg Nesterov
Ingo, sorry for delay.
So just in case, the whole series looks good to me. I am going to send a couple of minor cleanups on top of it,
Great, please do!
Quoted 1 line
[...] but let me ask first if I missed something or not.
You probably didn't. :-)
Quoted 1 line
-
x86_init_fpuis not really used after 4/8, it can be killed
Indeed!
Quoted 2 lines
-
DEFINE_EVENT(x86_fpu, x86_fpu_copy_src)can be killed after 7/8
Agreed.
Quoted 11 lines
-
arch_dup_task_struct()still does/* init_task is not dynamically sized (incomplete FPU state) */ if (unlikely(src == &init_task)) memcpy_and_pad(dst, arch_task_struct_size, src, sizeof(init_task), 0); else memcpy(dst, src, arch_task_struct_size);and I don't understand why do we need to check src ==
&init_task. It seems that we can always do
Quoted 7 lines
memcpy_and_pad(dst, arch_task_struct_size, src, sizeof(struct task_struct), 0);or even just
memcpy(dst, src, sizeof(struct task_struct));
fpu_clone()will initialize the "dst_fpu" memory correctly.
Unecessary paranoia on my part, please send a patch to simplify this.
Quoted 6 lines
-
fpu__drop()does/* PF_KTHREAD tasks do not use the FPU context area: */ if (tsk->flags & (PF_KTHREAD | PF_USER_WORKER)) return;
Quoted 8 lines
and this is correct. But perhaps
if (test_tsk_thread_flag(tsk, TIF_NEED_FPU_LOAD)) return;makes more sense? PF_KTHREAD's should never clear
TIF_NEED_FPU_LOAD, and this way we can avoid the unnecessary "fwait" if, say, the exiting task doescontext_switch()at least once on its way toexit_thread().
I think you are right here as well.
Quoted 3 lines
- Finally, with or without these changes, it seems that the
switch_fpu_prepare()+switch_fpu_finish()logic can be simplified, I'll write another email.
Thank you!
Ingo
question about switch_fpu_prepare/switch_fpu_finish
In reply to Ingo MolnarI must have missed something, but I can't understand this logic, it seems unnecessarily complicated today.
1. Now that switch_fpu_finish() doesn't load the FPU state, I think it can be folded into switch_fpu_prepare().
2. But the main question is that I fail to understand why __switch_to() -> switch_fpu_finish() uses the "next" task to set TIF_NEED_FPU_LOAD.
I think that set_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD) makes more sense.
Just in case, note that fpu_clone() sets TIF_NEED_FPU_LOAD, so we should not worry about the 1st __switch_to(next_p).
IOW, can you explain why the (untested) patch below could be wrong?
We can even remove the PF_KTHREAD check in switch_fpu_prepare(), kthreads should never clear TIF_NEED_FPU_LOAD...
Oleg.
---
arch/x86/include/asm/fpu/sched.h +2 -10
@@ -54,18 +54,10 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu)*/ old_fpu->last_cpu = cpu; +set_tsk_thread_flag (old,TIF_NEED_FPU_LOAD ); + trace_x86_fpu_regs_deactivated(old_fpu); } } -/* -* Delay loading of the complete FPU state until the return to userland. -* PKRU is handled separately. -*/ -static inline void switch_fpu_finish (struct task_struct *new) -{ -if (cpu_feature_enabled (X86_FEATURE_FPU )) -set_tsk_thread_flag (new,TIF_NEED_FPU_LOAD ); -} -#endif /* _ASM_X86_FPU_SCHED_H */
arch/x86/kernel/process_32.c +0 -2
@@ -208,8 +208,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p); -switch_fpu_finish (next_p); -/* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in (next_p);
arch/x86/kernel/process_64.c +0 -2
@@ -671,8 +671,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p);raw_cpu_write (cpu_current_top_of_stack,task_top_of_stack (next_p)); -switch_fpu_finish (next_p); -/* Reload sp0. */ update_task_stack (next_p);
Re: question about switch_fpu_prepare/switch_fpu_finish
Quoted 5 lines · Oleg Nesterov
I must have missed something, but I can't understand this logic, it seems unnecessarily complicated today.
1. Now that
switch_fpu_finish()doesn't load the FPU state, I think it can be folded intoswitch_fpu_prepare().
Agreed.
Quoted 11 lines
2. But the main question is that I fail to understand why
__switch_to()->switch_fpu_finish()uses the "next" task to setTIF_NEED_FPU_LOAD.I think that
set_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD)makes more sense.Just in case, note that
fpu_clone()setsTIF_NEED_FPU_LOAD, so we should not worry about the 1st__switch_to(next_p).IOW, can you explain why the (untested) patch below could be wrong?
I think your patch should work.
Quoted 2 lines
We can even remove the
PF_KTHREADcheck inswitch_fpu_prepare(), kthreads should never clearTIF_NEED_FPU_LOAD...
Agreed.
Thanks,
Ingo
PATCH 1/6 x86/fpu: simplify the switch_fpu_prepare() + switch_fpu_finish() logic
In reply to Ingo MolnarNow that switch_fpu_finish() doesn't load the FPU state, it makes more sense to fold it into switch_fpu_prepare() renamed to switch_fpu(), and more importantly, use the "prev_p" task as a target for TIF_NEED_FPU_LOAD. It doesn't make any sense to delay set_tsk_thread_flag(TIF_NEED_FPU_LOAD) until "prev_p" is scheduled again.
There is no worry about the very first context switch, fpu_clone() must always set TIF_NEED_FPU_LOAD.
Also, shift the test_tsk_thread_flag(TIF_NEED_FPU_LOAD) from the callers to switch_fpu().
Note that the "PF_KTHREAD | PF_USER_WORKER" check can be removed but this deserves a separate patch which can change more functions, say, kernel_fpu_begin_mask().
--- arch/x86/include/asm/fpu/sched.h | 34 +++++++++----------------------- arch/x86/kernel/process_32.c | 5 +---- arch/x86/kernel/process_64.c | 5 +---- 3 files changed, 11 insertions(+), 33 deletions(-)
arch/x86/include/asm/fpu/sched.h +9 -25
@@ -18,31 +18,25 @@ extern void fpu_flush_thread(void);/* * FPU state switching for scheduling. * -* This is a two-stage process: +* switch_fpu() saves the old state and sets TIF_NEED_FPU_LOAD if +* TIF_NEED_FPU_LOAD is not set. This is done within the context +* of the old process. * -* - switch_fpu_prepare() saves the old state. -* This is done within the context of the old process. -* -* - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state -* will get loaded on return to userspace, or when the kernel needs it. -* -* If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers -* are saved in the current thread's FPU register state. -* -* If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not -* hold current()'s FPU registers. It is required to load the +* Once TIF_NEED_FPU_LOAD is set, it is required to load the * registers before returning to userland or using the content * otherwise. * * The FPU context is only stored/restored for a user task and * PF_KTHREAD is used to distinguish between kernel and user threads. */ -static inline void switch_fpu_prepare (struct task_struct *old,int cpu) +static inline void switch_fpu (struct task_struct *old,int cpu) { -if (cpu_feature_enabled (X86_FEATURE_FPU ) && +if (!test_tsk_thread_flag (old,TIF_NEED_FPU_LOAD ) && +cpu_feature_enabled (X86_FEATURE_FPU ) && !(old->flags & (PF_KTHREAD |PF_USER_WORKER ))) {struct fpu *old_fpu =x86_task_fpu (old); +set_tsk_thread_flag (old,TIF_NEED_FPU_LOAD );save_fpregs_to_fpstate (old_fpu);/* * The save operation preserved register state, so the
@@ -50,7 +44,7 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu)* current CPU number in @old_fpu, so the next return * to user space can avoid the FPU register restore * when is returns on the same CPU and still owns the -* context. +* context. See fpregs_restore_userregs(). */ old_fpu->last_cpu = cpu;
@@ -58,14 +52,4 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu) } } -/* -* Delay loading of the complete FPU state until the return to userland. -* PKRU is handled separately. -*/ -static inline void switch_fpu_finish (struct task_struct *new) -{ -if (cpu_feature_enabled (X86_FEATURE_FPU )) -set_tsk_thread_flag (new,TIF_NEED_FPU_LOAD ); -} -#endif /* _ASM_X86_FPU_SCHED_H */
arch/x86/kernel/process_32.c +1 -4
@@ -160,8 +160,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */ -if (!test_tsk_thread_flag (prev_p,TIF_NEED_FPU_LOAD )) -switch_fpu_prepare (prev_p, cpu); +switch_fpu (prev_p, cpu);/* * Save away %gs. No need to save %fs, as it was saved on the
@@ -208,8 +207,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p); -switch_fpu_finish (next_p); -/* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in (next_p);
arch/x86/kernel/process_64.c +1 -4
@@ -616,8 +616,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ENTRY ) &&this_cpu_read (hardirq_stack_inuse)); -if (!test_tsk_thread_flag (prev_p,TIF_NEED_FPU_LOAD )) -switch_fpu_prepare (prev_p, cpu); +switch_fpu (prev_p, cpu);/* We must save %fs and %gs before load_TLS() because * %fs and %gs may be cleared by load_TLS().
@@ -671,8 +670,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p);raw_cpu_write (cpu_current_top_of_stack,task_top_of_stack (next_p)); -switch_fpu_finish (next_p); -/* Reload sp0. */ update_task_stack (next_p);
Signature · 1 line
2.25.1.362.g51ebf55
[tip: x86/fpu] x86/fpu: Simplify the switch_fpu_prepare() + switch_fpu_finish() logic
The following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 730faa15a069f4025a0f8c2a5244c3067da7ecbe Gitweb: https://git.kernel.org/tip/730faa15a069f4025a0f8c2a5244c3067da7ecbe Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:38:30 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:24 +02:00
x86/fpu: Simplify the switch_fpu_prepare() + switch_fpu_finish() logic
Now that switch_fpu_finish() doesn't load the FPU state, it makes more sense to fold it into switch_fpu_prepare() renamed to switch_fpu(), and more importantly, use the "prev_p" task as a target for TIF_NEED_FPU_LOAD. It doesn't make any sense to delay set_tsk_thread_flag(TIF_NEED_FPU_LOAD) until "prev_p" is scheduled again.
There is no worry about the very first context switch, fpu_clone() must always set TIF_NEED_FPU_LOAD.
Also, shift the test_tsk_thread_flag(TIF_NEED_FPU_LOAD) from the callers to switch_fpu().
Note that the "PF_KTHREAD | PF_USER_WORKER" check can be removed but this deserves a separate patch which can change more functions, say, kernel_fpu_begin_mask().
--- arch/x86/include/asm/fpu/sched.h | 34 ++++++++----------------------- arch/x86/kernel/process_32.c | 5 +---- arch/x86/kernel/process_64.c | 5 +---- 3 files changed, 11 insertions(+), 33 deletions(-)
arch/x86/include/asm/fpu/sched.h +9 -25
@@ -18,31 +18,25 @@ extern void fpu_flush_thread(void);/* * FPU state switching for scheduling. * -* This is a two-stage process: +* switch_fpu() saves the old state and sets TIF_NEED_FPU_LOAD if +* TIF_NEED_FPU_LOAD is not set. This is done within the context +* of the old process. * -* - switch_fpu_prepare() saves the old state. -* This is done within the context of the old process. -* -* - switch_fpu_finish() sets TIF_NEED_FPU_LOAD; the floating point state -* will get loaded on return to userspace, or when the kernel needs it. -* -* If TIF_NEED_FPU_LOAD is cleared then the CPU's FPU registers -* are saved in the current thread's FPU register state. -* -* If TIF_NEED_FPU_LOAD is set then CPU's FPU registers may not -* hold current()'s FPU registers. It is required to load the +* Once TIF_NEED_FPU_LOAD is set, it is required to load the * registers before returning to userland or using the content * otherwise. * * The FPU context is only stored/restored for a user task and * PF_KTHREAD is used to distinguish between kernel and user threads. */ -static inline void switch_fpu_prepare (struct task_struct *old,int cpu) +static inline void switch_fpu (struct task_struct *old,int cpu) { -if (cpu_feature_enabled (X86_FEATURE_FPU ) && +if (!test_tsk_thread_flag (old,TIF_NEED_FPU_LOAD ) && +cpu_feature_enabled (X86_FEATURE_FPU ) && !(old->flags & (PF_KTHREAD |PF_USER_WORKER ))) {struct fpu *old_fpu =x86_task_fpu (old); +set_tsk_thread_flag (old,TIF_NEED_FPU_LOAD );save_fpregs_to_fpstate (old_fpu);/* * The save operation preserved register state, so the
@@ -50,7 +44,7 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu)* current CPU number in @old_fpu, so the next return * to user space can avoid the FPU register restore * when is returns on the same CPU and still owns the -* context. +* context. See fpregs_restore_userregs(). */ old_fpu->last_cpu = cpu;
@@ -58,14 +52,4 @@ static inline void switch_fpu_prepare(struct task_struct *old, int cpu) } } -/* -* Delay loading of the complete FPU state until the return to userland. -* PKRU is handled separately. -*/ -static inline void switch_fpu_finish (struct task_struct *new) -{ -if (cpu_feature_enabled (X86_FEATURE_FPU )) -set_tsk_thread_flag (new,TIF_NEED_FPU_LOAD ); -} -#endif /* _ASM_X86_FPU_SCHED_H */
arch/x86/kernel/process_32.c +1 -4
@@ -160,8 +160,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)/* never put a printk in __switch_to... printk() calls wake_up*() indirectly */ -if (!test_tsk_thread_flag (prev_p,TIF_NEED_FPU_LOAD )) -switch_fpu_prepare (prev_p, cpu); +switch_fpu (prev_p, cpu);/* * Save away %gs. No need to save %fs, as it was saved on the
@@ -208,8 +207,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p); -switch_fpu_finish (next_p); -/* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in (next_p);
arch/x86/kernel/process_64.c +1 -4
@@ -616,8 +616,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)WARN_ON_ONCE (IS_ENABLED (CONFIG_DEBUG_ENTRY ) &&this_cpu_read (hardirq_stack_inuse)); -if (!test_tsk_thread_flag (prev_p,TIF_NEED_FPU_LOAD )) -switch_fpu_prepare (prev_p, cpu); +switch_fpu (prev_p, cpu);/* We must save %fs and %gs before load_TLS() because * %fs and %gs may be cleared by load_TLS().
@@ -671,8 +670,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)raw_cpu_write (current_task, next_p);raw_cpu_write (cpu_current_top_of_stack,task_top_of_stack (next_p)); -switch_fpu_finish (next_p); -/* Reload sp0. */ update_task_stack (next_p);
PATCH 2/6 x86/fpu: kill x86_init_fpu
In reply to Ingo MolnarIt is not actually used after the commit 55bc30f2e34d ("x86/fpu: Remove the thread::fpu pointer").
--- arch/x86/kernel/fpu/init.c | 4 ---- 1 file changed, 4 deletions(-)
arch/x86/kernel/fpu/init.c +0 -4
@@ -71,13 +71,9 @@ static bool __init fpu__probe_without_cpuid(void)return fsw ==0 && (fcw &0x103f ) ==0x003f ; } -static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly; -static void __init fpu__init_system_early_generic (void ) { -fpstate_reset (&x86_init_fpu);set_thread_flag (TIF_NEED_FPU_LOAD ); - x86_init_fpu.last_cpu =-1 ;if (!boot_cpu_has (X86_FEATURE_CPUID ) && !test_bit (X86_FEATURE_FPU , (unsigned long *)cpu_caps_cleared)) {
Signature · 1 line
2.25.1.362.g51ebf55
[tip: x86/fpu] x86/fpu: Remove x86_init_fpu
The following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 392bbe11c7cf90e65cba32e90af3b969a981c4fe Gitweb: https://git.kernel.org/tip/392bbe11c7cf90e65cba32e90af3b969a981c4fe Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:38:37 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:24 +02:00
x86/fpu: Remove x86_init_fpu
It is not actually used after:
55bc30f2e34d ("x86/fpu: Remove the thread::fpu pointer")
--- arch/x86/kernel/fpu/init.c | 4 ---- 1 file changed, 4 deletions(-)
arch/x86/kernel/fpu/init.c +0 -4
@@ -71,13 +71,9 @@ static bool __init fpu__probe_without_cpuid(void)return fsw ==0 && (fcw &0x103f ) ==0x003f ; } -static struct fpu x86_init_fpu __attribute__ ((aligned (64))) __read_mostly; -static void __init fpu__init_system_early_generic (void ) { -fpstate_reset (&x86_init_fpu);set_thread_flag (TIF_NEED_FPU_LOAD ); - x86_init_fpu.last_cpu =-1 ;if (!boot_cpu_has (X86_FEATURE_CPUID ) && !test_bit (X86_FEATURE_FPU , (unsigned long *)cpu_caps_cleared)) {
PATCH 3/6 x86/fpu: kill DEFINE_EVENT(x86_fpu, x86_fpu_copy_src)
In reply to Ingo Molnartrace_x86_fpu_copy_src() has no users after the commit 22aafe3bcb67 ("x86/fpu: Remove init_task FPU state dependencies, add debugging
warning for PF_KTHREAD tasks").
--- arch/x86/include/asm/trace/fpu.h | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/include/asm/trace/fpu.h +0 -5
@@ -74,11 +74,6 @@ DEFINE_EVENT(x86_fpu, x86_fpu_dropped,TP_ARGS (fpu) ); -DEFINE_EVENT (x86_fpu, x86_fpu_copy_src, - TP_PROTO(struct fpu *fpu), -TP_ARGS (fpu) -); -DEFINE_EVENT (x86_fpu, x86_fpu_copy_dst, TP_PROTO(struct fpu *fpu), TP_ARGS(fpu)
Signature · 1 line
2.25.1.362.g51ebf55
[tip: x86/fpu] x86/fpu: Remove DEFINE_EVENT(x86_fpu, x86_fpu_copy_src)
The following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 8e269c030ecafbfebf4f55e24fb336fd7b489708 Gitweb: https://git.kernel.org/tip/8e269c030ecafbfebf4f55e24fb336fd7b489708 Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:38:43 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:25 +02:00
x86/fpu: Remove DEFINE_EVENT(x86_fpu, x86_fpu_copy_src)
trace_x86_fpu_copy_src() has no users after:
22aafe3bcb67 ("x86/fpu: Remove init_task FPU state dependencies, add debugging warning for PF_KTHREAD tasks")
Remove the event.
--- arch/x86/include/asm/trace/fpu.h | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/include/asm/trace/fpu.h +0 -5
@@ -74,11 +74,6 @@ DEFINE_EVENT(x86_fpu, x86_fpu_dropped,TP_ARGS (fpu) ); -DEFINE_EVENT (x86_fpu, x86_fpu_copy_src, - TP_PROTO(struct fpu *fpu), -TP_ARGS (fpu) -); -DEFINE_EVENT (x86_fpu, x86_fpu_copy_dst, TP_PROTO(struct fpu *fpu), TP_ARGS(fpu)
PATCH 4/6 x86/fpu: arch_dup_task_struct: always use memcpy_and_pad()
In reply to Ingo MolnarIt makes no sense to copy the bytes after sizeof(struct task_struct), FPU state will be initialized in fpu_clone().
A plain memcpy(dst, src, sizeof(struct task_struct)) should work too, but "_and_pad" looks more safe.
--- arch/x86/kernel/process.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/kernel/process.c +3 -5
@@ -93,11 +93,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(__tss_limit_invalid);*/ int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src) { -/* init_task is not dynamically sized (incomplete FPU state) */ -if (unlikely (src == &init_task)) -memcpy_and_pad (dst, arch_task_struct_size, src,sizeof (init_task),0 ); -else -memcpy (dst, src, arch_task_struct_size); +/* fpu_clone() will initialize the "dst_fpu" memory */ +memcpy_and_pad (dst, arch_task_struct_size, src, +sizeof (struct task_struct ),0 );#ifdef CONFIG_VM86 dst->thread.vm86 =NULL ;
Signature · 1 line
2.25.1.362.g51ebf55
[tip: x86/fpu] x86/fpu: Always use memcpy_and_pad() in arch_dup_task_struct()
The following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 2d299e3d773d519ee93e5aaa3ffddd4a6276b005 Gitweb: https://git.kernel.org/tip/2d299e3d773d519ee93e5aaa3ffddd4a6276b005 Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:38:50 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:25 +02:00
x86/fpu: Always use memcpy_and_pad() in arch_dup_task_struct()
It makes no sense to copy the bytes after sizeof(struct task_struct), FPU state will be initialized in fpu_clone().
A plain memcpy(dst, src, sizeof(struct task_struct)) should work too, but "_and_pad" looks safer.
[ mingo: Simplify it a bit more. ]
--- arch/x86/kernel/process.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
arch/x86/kernel/process.c +2 -5
@@ -93,11 +93,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(__tss_limit_invalid);*/ int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src) { -/* init_task is not dynamically sized (incomplete FPU state) */ -if (unlikely (src == &init_task)) -memcpy_and_pad (dst, arch_task_struct_size, src,sizeof (init_task),0 ); -else -memcpy (dst, src, arch_task_struct_size); +/* fpu_clone() will initialize the "dst_fpu" memory */ +memcpy_and_pad (dst, arch_task_struct_size, src,sizeof (*dst),0 );#ifdef CONFIG_VM86 dst->thread.vm86 =NULL ;
PATCH 5/6 x86/fpu: fpu__drop: check TIF_NEED_FPU_LOAD instead of PF_KTHREAD|PF_USER_WORKER
In reply to Ingo MolnarPF_KTHREAD|PF_USER_WORKER tasks should never clear TIF_NEED_FPU_LOAD, so the TIF_NEED_FPU_LOAD check should equally filter them out.
And this way an exiting userspace task can avoid the unnecessary "fwait" if it does context_switch() at least once on its way to exit_thread().
--- arch/x86/kernel/fpu/core.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kernel/fpu/core.c +1 -2
@@ -693,8 +693,7 @@ void fpu__drop(struct task_struct *tsk) {struct fpu *fpu; -/* PF_KTHREAD tasks do not use the FPU context area: */ -if (tsk->flags & (PF_KTHREAD |PF_USER_WORKER )) +if (test_tsk_thread_flag (tsk,TIF_NEED_FPU_LOAD ))return ; fpu =x86_task_fpu (tsk);
Signature · 1 line
2.25.1.362.g51ebf55
[tip: x86/fpu] x86/fpu: Check TIF_NEED_FPU_LOAD instead of PF_KTHREAD|PF_USER_WORKER in fpu__drop()
The following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 016a2e6f8ae5ed544ba8fb2b6d78f64ddfd9d01b Gitweb: https://git.kernel.org/tip/016a2e6f8ae5ed544ba8fb2b6d78f64ddfd9d01b Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:38:56 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:25 +02:00
x86/fpu: Check TIF_NEED_FPU_LOAD instead of PF_KTHREAD|PF_USER_WORKER in fpu__drop()
PF_KTHREAD|PF_USER_WORKER tasks should never clear TIF_NEED_FPU_LOAD, so the TIF_NEED_FPU_LOAD check should equally filter them out.
And this way an exiting userspace task can avoid the unnecessary "fwait" if it does context_switch() at least once on its way to exit_thread().
--- arch/x86/kernel/fpu/core.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/kernel/fpu/core.c +1 -2
@@ -693,8 +693,7 @@ void fpu__drop(struct task_struct *tsk) {struct fpu *fpu; -/* PF_KTHREAD tasks do not use the FPU context area: */ -if (tsk->flags & (PF_KTHREAD |PF_USER_WORKER )) +if (test_tsk_thread_flag (tsk,TIF_NEED_FPU_LOAD ))return ; fpu =x86_task_fpu (tsk);
PATCH 6/6 x86/fpu: shift fpregs_assert_state_consistent() from arch_exit_work() to its caller
In reply to Ingo MolnarIf CONFIG_X86_DEBUG_FPU=Y, arch_exit_to_user_mode_prepare() calls arch_exit_work() even if ti_work == 0. There only reason is that we want to call fpregs_assert_state_consistent() if TIF_NEED_FPU_LOAD is not set.
This looks confusing. arch_exit_to_user_mode_prepare() can just call fpregs_assert_state_consistent() unconditionally, it depends on CONFIG_X86_DEBUG_FPU and checks TIF_NEED_FPU_LOAD itself.
--- arch/x86/include/asm/entry-common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/include/asm/entry-common.h +3 -2
@@ -53,7 +53,6 @@ static inline void arch_exit_work(unsigned long ti_work)if (unlikely (ti_work & _TIF_IO_BITMAP))tss_update_io_bitmap (); -fpregs_assert_state_consistent ();if (unlikely (ti_work & _TIF_NEED_FPU_LOAD))switch_fpu_return (); }
@@ -61,7 +60,9 @@ static inline void arch_exit_work(unsigned long ti_work)static inline void arch_exit_to_user_mode_prepare (struct pt_regs *regs,unsigned long ti_work) { -if (IS_ENABLED (CONFIG_X86_DEBUG_FPU ) ||unlikely (ti_work)) +fpregs_assert_state_consistent (); + +if (unlikely (ti_work))arch_exit_work (ti_work);fred_update_rsp0 ();
Signature · 1 line
2.25.1.362.g51ebf55
Re: [PATCH tip/x86/fpu 6/6] x86/fpu: shift fpregs_assert_state_consistent() from arch_exit_work() to its caller
Quoted 5 lines · Oleg Nesterov
If
CONFIG_X86_DEBUG_FPU=Y,arch_exit_to_user_mode_prepare()callsarch_exit_work()even ifti_work== 0. There only reason is that we want to callfpregs_assert_state_consistent()ifTIF_NEED_FPU_LOADis not set.
Quoted 8 lines
This looks confusing.
arch_exit_to_user_mode_prepare()can just callfpregs_assert_state_consistent()unconditionally, it depends onCONFIG_X86_DEBUG_FPUand checksTIF_NEED_FPU_LOADitself.Signed-off-by Oleg Nesterov <[email protected]>--- arch/x86/include/asm/entry-common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Thanks Oleg! I've applied these improvements to tip:x86/fpu.
Note that there's still a 32-bit hardened-usercopy regression/crash that Boris reported against tip:x86/fpu:
https://lore.kernel.org/r/20250503120712.GJaBYG8A-D77MllFZ3@fat_crate.local
Which may result in the subsequent rebasing of your commits - but otherwise your series looks good to me.
Thanks,
Ingo
[tip: x86/fpu] x86/fpu: Shift fpregs_assert_state_consistent() from arch_exit_work() to its caller
In reply to Oleg NesterovThe following commit has been merged into the x86/fpu branch of tip:
The following commit has been merged into the x86/fpu branch of tip:
Commit-ID: 46c158e3ad0fc633007802c338c409c188ec0a12 Gitweb: https://git.kernel.org/tip/46c158e3ad0fc633007802c338c409c188ec0a12 Author: Oleg Nesterov <[email protected]> AuthorDate: Sat, 03 May 2025 16:39:02 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 10:29:25 +02:00
x86/fpu: Shift fpregs_assert_state_consistent() from arch_exit_work() to its caller
If CONFIG_X86_DEBUG_FPU=Y, arch_exit_to_user_mode_prepare() calls arch_exit_work() even if ti_work == 0. There only reason is that we want to call fpregs_assert_state_consistent() if TIF_NEED_FPU_LOAD is not set.
This looks confusing. arch_exit_to_user_mode_prepare() can just call fpregs_assert_state_consistent() unconditionally, it depends on CONFIG_X86_DEBUG_FPU and checks TIF_NEED_FPU_LOAD itself.
--- arch/x86/include/asm/entry-common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/x86/include/asm/entry-common.h +3 -2
@@ -53,7 +53,6 @@ static inline void arch_exit_work(unsigned long ti_work)if (unlikely (ti_work & _TIF_IO_BITMAP))tss_update_io_bitmap (); -fpregs_assert_state_consistent ();if (unlikely (ti_work & _TIF_NEED_FPU_LOAD))switch_fpu_return (); }
@@ -61,7 +60,9 @@ static inline void arch_exit_work(unsigned long ti_work)static inline void arch_exit_to_user_mode_prepare (struct pt_regs *regs,unsigned long ti_work) { -if (IS_ENABLED (CONFIG_X86_DEBUG_FPU ) ||unlikely (ti_work)) +fpregs_assert_state_consistent (); + +if (unlikely (ti_work))arch_exit_work (ti_work);fred_update_rsp0 ();