x86/fpu: Spring cleaning and PKRU sanitizing
Subsystems (4)
cover x86/fpu: Spring cleaning and PKRU sanitizing
This is a follow up to these patch series:
- [patch V2 00/14] x86/fpu: Mop up XSAVES and related damage
https://lore.kernel.org/r/[email protected]
- [PATCH 00/18] x86/pkeys: stop managing PKRU with XSAVE
https://lore.kernel.org/r/[email protected]
- [PATCH 0/2] x86/fpu: Clean up "dynamic" APIs
https://lore.kernel.org/r/[email protected]
The analysis of the subtle bugs in the FPU code triggered a larger discussion about the general state of this code. The above patch series are all related to this and the following series combines them into one because the already started consolidation work and the PKRU rework collided all over the place.
The main parts of this series are:
- Simplification and removal/replacement of redundant and/or overengineered code.
- Name space cleanup as the existing names were just a permanent source of confusion.
- Clear seperation of user ABI and kernel internal state handling.
- Removal of PKRU from being XSTATE managed in the kernel because PKRU has to be eagerly restored on context switch and keeping it in sync in the xstate buffer is just pointless overhead and fragile.
The kernel still XSAVEs PKRU on context switch but the value in the buffer is not longer used and never restored from the buffer.
This still needs to be cleaned up, but the series is already 40+ patches large and the cleanup of this is not a functional problem.
The functional issues of PKRU management are fully addressed with the series as is.
It applies on top of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git master
and is also available via git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git x86/fpu
Thanks,
tglx
--- arch/x86/events/intel/lbr.c | 6 arch/x86/include/asm/fpu/internal.h | 155 ++++------- arch/x86/include/asm/fpu/xstate.h | 61 +++- arch/x86/include/asm/pgtable.h | 57 ---- arch/x86/include/asm/pkeys.h | 9 arch/x86/include/asm/processor.h | 9 arch/x86/include/asm/special_insns.h | 14 - arch/x86/kernel/cpu/common.c | 29 -- arch/x86/kernel/fpu/core.c | 243 ++++++++++++------ arch/x86/kernel/fpu/init.c | 4 arch/x86/kernel/fpu/regset.c | 116 ++++---- arch/x86/kernel/fpu/signal.c | 59 ++-- arch/x86/kernel/fpu/xstate.c | 456 +++++++++++------------------------ arch/x86/kernel/process.c | 19 + arch/x86/kernel/process_64.c | 28 ++ arch/x86/kvm/svm/sev.c | 1 arch/x86/kvm/x86.c | 52 ++- arch/x86/mm/extable.c | 2 arch/x86/mm/fault.c | 2 arch/x86/mm/pkeys.c | 22 - b/arch/x86/include/asm/pkru.h | 62 ++++ include/linux/pkeys.h | 4 22 files changed, 671 insertions(+), 739 deletions(-)
PATCH 1/41 Revert a5eff7259790 ("x86/pkeys: Add PKRU value to init_fpstate")
This cannot work and it's unclear how that ever made a difference.
init_fpstate.xsave.header.xfeatures is always 0 so get_xsave_addr() will always return a NULL pointer, which will prevent storing the default PKRU value in initfp_state.
--- arch/x86/kernel/cpu/common.c | 5 ----- arch/x86/mm/pkeys.c | 6 ------ 2 files changed, 11 deletions(-)
arch/x86/kernel/cpu/common.c +0 -5
@@ -466,8 +466,6 @@ static bool pku_disabled; static __always_inline void setup_pku(struct cpuinfo_x86 *c) { -struct pkru_state *pk; -/* check the boot processor, plus compile options for PKU: */ if (!cpu_feature_enabled (X86_FEATURE_PKU ))return ;
@@ -478,9 +476,6 @@ static __always_inline void setup_pku(streturn ;cr4_set_bits (X86_CR4_PKE ); - pk =get_xsave_addr (&init_fpstate.xsave,XFEATURE_PKRU ); -if (pk) - pk->pkru = init_pkru_value;/* * Setting X86_CR4_PKE will cause the X86_FEATURE_OSPKE * cpuid bit to be set. We need to ensure that we
arch/x86/mm/pkeys.c +0 -6
@@ -10,7 +10,6 @@#include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/mmu_context.h> /* vma_pkey() */ -#include <asm/fpu/internal.h> /* init_fpstate */ int __execute_only_pkey (struct mm_struct *mm) {
@@ -154,7 +153,6 @@ static ssize_t init_pkru_read_file(strucstatic ssize_t init_pkru_write_file (struct file *file,const char __user *user_buf,size_t count,loff_t *ppos) { -struct pkru_state *pk;char buf[32 ];ssize_t len;u32 new_init_pkru;
@@ -177,10 +175,6 @@ static ssize_t init_pkru_write_file(strureturn -EINVAL ;WRITE_ONCE (init_pkru_value, new_init_pkru); - pk =get_xsave_addr (&init_fpstate.xsave,XFEATURE_PKRU ); -if (!pk) -return -EINVAL ; - pk->pkru = new_init_pkru;return count; }
Re: [patch 01/41] Revert a5eff7259790 ("x86/pkeys: Add PKRU value to init_fpstate")
Quoted 1 line · Thomas Gleixner
This cannot work and it's unclear how that ever made a difference.
Subject: Re: [patch 01/41] Revert a5eff7259790 ("x86/pkeys: Add PKRU value to init_fpstate")
Add subject prefix pls:
x86/pkeys: Revert...
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergPATCH 2/41 x86/fpu: Mark various FPU states __ro_after_init
In reply to Thomas GleixnerNothing modifies these after booting.
--- arch/x86/kernel/fpu/init.c | 4 ++-- arch/x86/kernel/fpu/xstate.c | 16 ++++++++++------ 2 files changed, 12 insertions(+), 8 deletions(-)
arch/x86/kernel/fpu/init.c +2 -2
@@ -89,7 +89,7 @@ static void fpu__init_system_early_gener/* * Boot time FPU feature detection code: */ -unsigned int mxcsr_feature_mask__read_mostly =0xffffffffu ; +unsigned int mxcsr_feature_mask__ro_after_init =0xffffffffu ;EXPORT_SYMBOL_GPL (mxcsr_feature_mask);static void __init fpu__init_system_mxcsr (void )
@@ -135,7 +135,7 @@ static void __init fpu__init_system_gene* This is inherent to the XSAVE architecture which puts all state * components into a single, continuous memory block: */ -unsigned int fpu_kernel_xstate_size; +unsigned int fpu_kernel_xstate_size__ro_after_init ;EXPORT_SYMBOL_GPL (fpu_kernel_xstate_size);/* Get alignment of the TYPE. */
arch/x86/kernel/fpu/xstate.c +10 -6
@@ -59,19 +59,23 @@ static short xsave_cpuid_features[] __in* This represents the full set of bits that should ever be set in a kernel * XSAVE buffer, both supervisor and user xstates. */ -u64 xfeatures_mask_all__read_mostly ; +u64 xfeatures_mask_all__ro_after_init ; -static unsigned int xstate_offsets[XFEATURE_MAX ] = { [0 ...XFEATURE_MAX -1 ] =-1 }; -static unsigned int xstate_sizes[XFEATURE_MAX ] = { [0 ...XFEATURE_MAX -1 ] =-1 }; -static unsigned int xstate_comp_offsets[XFEATURE_MAX ] = { [0 ...XFEATURE_MAX -1 ] =-1 }; -static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX ] = { [0 ...XFEATURE_MAX -1 ] =-1 }; +static unsigned int xstate_offsets[XFEATURE_MAX ]__ro_after_init = + { [0 ...XFEATURE_MAX -1 ] =-1 }; +static unsigned int xstate_sizes[XFEATURE_MAX ]__ro_after_init = + { [0 ...XFEATURE_MAX -1 ] =-1 }; +static unsigned int xstate_comp_offsets[XFEATURE_MAX ]__ro_after_init = + { [0 ...XFEATURE_MAX -1 ] =-1 }; +static unsigned int xstate_supervisor_only_offsets[XFEATURE_MAX ]__ro_after_init = + { [0 ...XFEATURE_MAX -1 ] =-1 };/* * The XSAVE area of kernel can be in standard or compacted format; * it is always in standard format for user mode. This is the user * mode standard format size used for signal and ptrace frames. */ -unsigned int fpu_user_xstate_size; +unsigned int fpu_user_xstate_size__ro_after_init ;/* * Return whether the system supports a given xfeature.
Re: [patch 02/41] x86/fpu: Mark various FPU states __ro_after_init
Quoted 7 lines · Thomas Gleixner
Nothing modifies these after booting.
Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/kernel/fpu/init.c | 4 ++-- arch/x86/kernel/fpu/xstate.c | 16 ++++++++++------ 2 files changed, 12 insertions(+), 8 deletions(-)
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergRe: [patch 02/41] x86/fpu: Mark various FPU states __ro_after_init
In reply to Thomas GleixnerQuoted 2 lines · Thomas Gleixner
Nothing modifies these after booting.
PATCH 3/41 x86/fpu: Remove unused get_xsave_field_ptr()
In reply to Thomas Gleixner---
V4: New patch
--- arch/x86/include/asm/fpu/xstate.h | 1 - arch/x86/kernel/fpu/xstate.c | 30 ------------------------------ 2 files changed, 31 deletions(-)
arch/x86/include/asm/fpu/xstate.h +0 -1
@@ -101,7 +101,6 @@ extern void __init update_regset_xstate_ u64 xstate_mask);void *get_xsave_addr (struct xregs_state *xsave,int xfeature_nr); -const void *get_xsave_field_ptr (int xfeature_nr);int using_compacted_format (void );int xfeature_size (int xfeature_nr);struct membuf ;
arch/x86/kernel/fpu/xstate.c +0 -30
@@ -955,36 +955,6 @@ void *get_xsave_addr(struct xregs_state }EXPORT_SYMBOL_GPL (get_xsave_addr); -/* -* This wraps up the common operations that need to occur when retrieving -* data from xsave state. It first ensures that the current task was -* using the FPU and retrieves the data in to a buffer. It then calculates -* the offset of the requested field in the buffer. -* -* This function is safe to call whether the FPU is in use or not. -* -* Note that this only works on the current task. -* -* Inputs: -* @xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, -* XFEATURE_SSE, etc...) -* Output: -* address of the state in the xsave area or NULL if the state -* is not present or is in its 'init state'. -*/ -const void *get_xsave_field_ptr (int xfeature_nr) -{ -struct fpu *fpu = ¤t->thread.fpu; - -/* -* fpu__save() takes the CPU's xstate registers -* and saves them off to the 'fpu memory buffer. -*/ -fpu__save (fpu); - -return get_xsave_addr (&fpu->state.xsave, xfeature_nr); -} -#ifdef CONFIG_ARCH_HAS_PKEYS /*
Re: [patch 03/41] x86/fpu: Remove unused get_xsave_field_ptr()
On 6/11/21 9:15 AM, Thomas Gleixner wrote:
PATCH 4/41 x86/fpu: Move inlines where they belong
In reply to Thomas GleixnerThey are only used in fpstate_init() and there is no point to have them in a header just to make reading the code harder.
---
V2: New patch
--- arch/x86/include/asm/fpu/internal.h | 14 -------------- arch/x86/kernel/fpu/core.c | 15 +++++++++++++++ 2 files changed, 15 insertions(+), 14 deletions(-)
arch/x86/include/asm/fpu/internal.h +0 -14
@@ -87,20 +87,6 @@ extern void fpstate_init_soft(struct swrstatic inline void fpstate_init_soft (struct swregs_state *soft) {}#endif -static inline void fpstate_init_xstate (struct xregs_state *xsave) -{ -/* -* XRSTORS requires these bits set in xcomp_bv, or it will -* trigger #GP: -*/ - xsave->header.xcomp_bv =XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; -} - -static inline void fpstate_init_fxstate (struct fxregs_state *fx) -{ - fx->cwd =0x37f ; - fx->mxcsr =MXCSR_DEFAULT ; -}extern void fpstate_sanitize_xstate (struct fpu *fpu); #define user_insn(insn, output, input...) \
arch/x86/kernel/fpu/core.c +15 -0
@@ -181,6 +181,21 @@ void fpu__save(struct fpu *fpu)fpregs_unlock (); } +static inline void fpstate_init_xstate (struct xregs_state *xsave) +{ +/* +* XRSTORS requires these bits set in xcomp_bv, or it will +* trigger #GP: +*/ + xsave->header.xcomp_bv =XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask_all; +} + +static inline void fpstate_init_fxstate (struct fxregs_state *fx) +{ + fx->cwd =0x37f ; + fx->mxcsr =MXCSR_DEFAULT ; +} +/* * Legacy x87 fpstate state init: */
PATCH 5/41 x86/fpu: Limit xstate copy size in xstateregs_set()
In reply to Thomas GleixnerIf the count argument is larger than the xstate size, this will happily copy beyond the end of xstate.
--- arch/x86/kernel/fpu/regset.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/fpu/regset.c +1 -1
@@ -117,7 +117,7 @@ int xstateregs_set(struct task_struct *t/* * A whole standard-format XSAVE buffer is needed: */ -if ((pos !=0 ) || (count < fpu_user_xstate_size)) +if (pos !=0 || count != fpu_user_xstate_size)return -EFAULT ; xsave = &fpu->state.xsave;
Re: [patch 05/41] x86/fpu: Limit xstate copy size in xstateregs_set()
Quoted 12 lines · Thomas Gleixner
If the count argument is larger than the xstate size, this will happily copy beyond the end of xstate.
Fixes 91c3dba7dbc1 ("x86/fpu/xstate: Fix PTRACE frames for XSAVES")Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/kernel/fpu/regset.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/fpu/regset.c int xstateregs_set(struct task_struct *t · line 123
Quoted 8 lines
arch/x86/kernel/fpu/regset.c +1
-1@@ -117,7 +117,7 @@ int xstateregs_set(struct task_struct *t/* * A whole standard-format XSAVE buffer is needed: */ -if ((pos !=0 ) || (count < fpu_user_xstate_size)) +if (pos !=0 || count != fpu_user_xstate_size)return -EFAULT ; xsave = &fpu->state.xsave;
Patches 3-5:
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergRe: [patch 05/41] x86/fpu: Limit xstate copy size in xstateregs_set()
In reply to Thomas GleixnerQuoted 2 lines · Thomas Gleixner
If the count argument is larger than the xstate size, this will happily copy beyond the end of xstate.
This interface is horrible. Oh well.
Re: [patch 05/41] x86/fpu: Limit xstate copy size in xstateregs_set()
Quoted 7 lines · Andy Lutomirski
Quoted 2 lines · Thomas Gleixner
If the count argument is larger than the xstate size, this will happily copy beyond the end of xstate.
Reviewed-by Andy Lutomirski <[email protected]>This interface is horrible. Oh well.
It's ptrace is must be horrible :)
PATCH 6/41 x86/fpu: Sanitize xstateregs_set()
In reply to Thomas Gleixnerxstateregs_set() operates on a stopped task and tries to copy the provided buffer into the tasks fpu.state.xsave buffer.
Any error while copying or invalid state detected after copying results in wiping the target tasks FPU state completely including supervisor states.
That's just wrong. The caller supplied invalid data or has a problem with unmapped memory, so there is absolutely no justification to corrupt the target state.
Fix this with the following modifications:
1) If data has to be copied from userspace, allocate a buffer and copy from user first.
2) Use copy_kernel_to_xstate() unconditionally so that header checking works correctly.
3) Return on error without corrupting the target state.
This prevents corrupting states and lets the caller deal with the problem it caused in the first place.
---
V2: Use copy_from_user() - Dave, Yu
Rename xbuf to tmpbuf - Borislav
--- arch/x86/include/asm/fpu/xstate.h | 4 --- arch/x86/kernel/fpu/regset.c | 41 +++++++++++++++----------------------- arch/x86/kernel/fpu/xstate.c | 14 +++++++----- 3 files changed, 25 insertions(+), 34 deletions(-)
arch/x86/include/asm/fpu/xstate.h +0 -4
@@ -111,8 +111,4 @@ void copy_supervisor_to_kernel(struct xrvoid copy_dynamic_supervisor_to_kernel (struct xregs_state *xstate,u64 mask);void copy_kernel_to_dynamic_supervisor (struct xregs_state *xstate,u64 mask); - -/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ -int validate_user_xstate_header (const struct xstate_header *hdr); -#endif
arch/x86/kernel/fpu/regset.c +17 -24
64 lines not on this page · raw message
arch/x86/kernel/fpu/xstate.c +8 -6
@@ -519,7 +519,7 @@ int using_compacted_format(void) }/* Validate an xstate header supplied by userspace (ptrace or sigreturn) */ -int validate_user_xstate_header (const struct xstate_header *hdr) +static int validate_user_xstate_header (const struct xstate_header *hdr) {/* No unknown or supervisor features may be set */ if (hdr->xfeatures & ~xfeatures_mask_user ())
@@ -1099,7 +1099,7 @@ void copy_xstate_to_kernel(struct membuf }/* -* Convert from a ptrace standard-format kernel buffer to kernel XSAVES format +* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S] format * and copy to the target thread. This is called from xstateregs_set(). */ int copy_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf)
@@ -1146,14 +1146,16 @@ int copy_kernel_to_xstate(struct xregs_s*/ xsave->header.xfeatures |= hdr.xfeatures; +/* mxcsr reserved bits must be masked to zero for security reasons. */ + xsave->i387.mxcsr &= mxcsr_feature_mask; +return 0 ; }/* -* Convert from a ptrace or sigreturn standard-format user-space buffer to -* kernel XSAVES format and copy to the target thread. This is called from -* xstateregs_set(), as well as potentially from the sigreturn() and -* rt_sigreturn() system calls. +* Convert from a sigreturn standard-format user-space buffer to kernel +* XSAVE[S] format and copy to the target thread. This is called from the +* sigreturn() and rt_sigreturn() system calls. */ int copy_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf) {
Andy Lutomirski
maintainerRe: [patch 06/41] x86/fpu: Sanitize xstateregs_set()
Quoted 11 lines · Thomas Gleixner
xstateregs_set()operates on a stopped task and tries to copy the provided buffer into the tasks fpu.state.xsave buffer.Any error while copying or invalid state detected after copying results in wiping the target tasks FPU state completely including supervisor states.
That's just wrong. The caller supplied invalid data or has a problem with unmapped memory, so there is absolutely no justification to corrupt the target state.
arch/x86/kernel/fpu/xstate.c int copy_kernel_to_xstate(struct xregs_s · line 1150
Quoted 5 lines
arch/x86/kernel/fpu/xstate.c +2
-0@@ -1146,14 +1146,16 @@ int copy_kernel_to_xstate(struct xregs_s*/ xsave->header.xfeatures |= hdr.xfeatures; +/* mxcsr reserved bits must be masked to zero for security reasons. */ + xsave->i387.mxcsr &= mxcsr_feature_mask;
This comment is vague. At least it should say:
A subsequent XRSTOR(S) will fail if MXCSR has bits set that are not accepted by the current CPU. Mask out unsupported bits.
But a much nicer fix IMO would be to just return an error.
--Andy
Re: [patch 06/41] x86/fpu: Sanitize xstateregs_set()
Quoted 20 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
xstateregs_set()operates on a stopped task and tries to copy the provided buffer into the tasks fpu.state.xsave buffer.Any error while copying or invalid state detected after copying results in wiping the target tasks FPU state completely including supervisor states.
That's just wrong. The caller supplied invalid data or has a problem with unmapped memory, so there is absolutely no justification to corrupt the target state.
arch/x86/kernel/fpu/xstate.c int copy_kernel_to_xstate(struct xregs_s · line 1150
Quoted 5 lines
arch/x86/kernel/fpu/xstate.c +2
-0@@ -1146,14 +1146,16 @@ int copy_kernel_to_xstate(struct xregs_s*/ xsave->header.xfeatures |= hdr.xfeatures; +/* mxcsr reserved bits must be masked to zero for security reasons. */ + xsave->i387.mxcsr &= mxcsr_feature_mask;This comment is vague. At least it should say:
Quoted 4 lines
A subsequent XRSTOR(S) will fail if MXCSR has bits set that are not accepted by the current CPU. Mask out unsupported bits.
But a much nicer fix IMO would be to just return an error.
That was the "fix" for CVE-2003-0248, but I forgot the details of this issue and why the masking was done instead of returning an error. The problem was not that it caused a #GP, there was some other implication which I can't remember.
The reason why this masking was done was that the SDM was pretty vague about the reserved bits and only newer processor generations had an issue while older ones just ignored them and of course some existing user space was sloppy and got away with random values in reserved bits up to the point where newer CPUs got more sensitive about this.
Yes, the comment could get some love and we probably can just error out today. Those ancient binaries should be in the bit bucket by now.
Thanks,
tglx
PATCH 7/41 x86/fpu: Simplify PTRACE_GETREGS code
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
ptrace() has interfaces that let a ptracer inspect a ptracee's register state. This includes XSAVE state. The ptrace() ABI includes a hardware-format XSAVE buffer for both the SETREGS and GETREGS interfaces.
In the old days, the kernel buffer and the ptrace() ABI buffer were the same boring non-compacted format. But, since the advent of supervisor states and the compacted format, the kernel buffer has diverged from the format presented in the ABI.
This leads to two paths in the kernel:
1. Effectively a verbatim copy_to_user() which just copies the kernel buffer out to userspace. This is used when the kernel buffer is kept in the non-compacted form which means that it shares a format with the ptrace ABI.
2. A one-state-at-a-time path: copy_xstate_to_kernel(). This is theoretically slower since it does a bunch of piecemeal copies.
Remove the verbatim copy case. Speed probably does not matter in this path, and the vast majority of new hardware will use the one-state-at-a-time path anyway. This ensures greater testing for the "slow" path.
This also makes enabling PKRU in this interface easier since a single path can be patched instead of two.
---
V4: Picked up from Dave's PKRU series
--- arch/x86/kernel/fpu/regset.c | 22 ++-------------------- arch/x86/kernel/fpu/xstate.c | 6 +++--- 2 files changed, 5 insertions(+), 23 deletions(-)
arch/x86/kernel/fpu/regset.c +2 -20
@@ -79,32 +79,14 @@ int xstateregs_get(struct task_struct *t struct membuf to) {struct fpu *fpu = &target->thread.fpu; -struct xregs_state *xsave;if (!boot_cpu_has (X86_FEATURE_XSAVE ))return -ENODEV ; - xsave = &fpu->state.xsave; -fpu__prepare_read (fpu); -if (using_compacted_format ()) { -copy_xstate_to_kernel (to, xsave); -return 0 ; - }else { -fpstate_sanitize_xstate (fpu); -/* -* Copy the 48 bytes defined by the software into the xsave -* area in the thread struct, so that we can copy the whole -* area to user using one user_regset_copyout(). -*/ -memcpy (&xsave->i387.sw_reserved, xstate_fx_sw_bytes,sizeof (xstate_fx_sw_bytes)); - -/* -* Copy the xstate memory layout. -*/ -return membuf_write (&to, xsave, fpu_user_xstate_size); - } +copy_xstate_to_kernel (to, &fpu->state.xsave); +return 0 ; } int xstateregs_set(struct task_struct *target, const struct user_regset *regset,
arch/x86/kernel/fpu/xstate.c +3 -3
@@ -1040,11 +1040,11 @@ static void copy_part(struct membuf *to, }/* -* Convert from kernel XSAVES compacted format to standard format and copy -* to a kernel-space ptrace buffer. +* Convert from kernel XSAVE or XSAVES compacted format to UABI +* non-compacted format and copy to a kernel-space ptrace buffer. * * It supports partial copy but pos always starts from zero. This is called -* from xstateregs_get() and there we check the CPU has XSAVES. +* from xstateregs_get() and there we check the CPU has XSAVE. */ void copy_xstate_to_kernel (struct membuf to,struct xregs_state *xsave) {
Re: [patch 07/41] x86/fpu: Simplify PTRACE_GETREGS code
Quoted 5 lines · Thomas Gleixner
From: Dave Hansen <[email protected]>
ptrace()has interfaces that let a ptracer inspect a ptracee's register state. This includes XSAVE state. Theptrace()ABI includes a hardware-format XSAVE buffer for both the SETREGS and GETREGS interfaces.
Re: [patch 07/41] x86/fpu: Simplify PTRACE_GETREGS code
In reply to Thomas GleixnerQuoted 26 lines · Thomas Gleixner
From: Dave Hansen <[email protected]>
ptrace()has interfaces that let a ptracer inspect a ptracee's register state. This includes XSAVE state. Theptrace()ABI includes a hardware-format XSAVE buffer for both the SETREGS and GETREGS interfaces.In the old days, the kernel buffer and the
ptrace()ABI buffer were the same boring non-compacted format. But, since the advent of supervisor states and the compacted format, the kernel buffer has diverged from the format presented in the ABI.This leads to two paths in the kernel:
1. Effectively a verbatimcopy_to_user()which just copies the kernel buffer out to userspace. This is used when the kernel buffer is kept in the non-compacted form which means that it shares a format with the ptrace ABI.
2. A one-state-at-a-time path:copy_xstate_to_kernel(). This is theoretically slower since it does a bunch of piecemeal copies.Remove the verbatim copy case. Speed probably does not matter in this path, and the vast majority of new hardware will use the one-state-at-a-time path anyway. This ensures greater testing for the "slow" path.
This also makes enabling PKRU in this interface easier since a single path can be patched instead of two.
Quoted 8 lines
Signed-off-by Dave Hansen <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>---
V4: Picked up from Dave's PKRU series--- arch/x86/kernel/fpu/regset.c | 22 ++-------------------- arch/x86/kernel/fpu/xstate.c | 6 +++--- 2 files changed, 5 insertions(+), 23 deletions(-)
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergPATCH 8/41 x86/fpu: Restrict fpstate sanitizing to legacy components
In reply to Thomas Gleixnerxstateregs_get() does not longer use fpstate_sanitize_xstate() and the only callers are the regset functions for the legacy FP/SSE components.
Move the function to the callsites and remove the extended features part.
--- arch/x86/include/asm/fpu/internal.h | 2 arch/x86/kernel/fpu/regset.c | 41 ++++++++++++++++-- arch/x86/kernel/fpu/xstate.c | 79 ------------------------------------ 3 files changed, 37 insertions(+), 85 deletions(-)
arch/x86/include/asm/fpu/internal.h +0 -2
@@ -87,8 +87,6 @@ extern void fpstate_init_soft(struct swrstatic inline void fpstate_init_soft (struct swregs_state *soft) {}#endif -extern void fpstate_sanitize_xstate (struct fpu *fpu); - #define user_insn(insn, output, input...) \ ({ \ int err; \
arch/x86/kernel/fpu/regset.c +37 -4
71 lines not on this page · raw message
arch/x86/kernel/fpu/xstate.c +0 -79
85 lines not on this page · raw message
Andy Lutomirski
maintainerRe: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
Quoted 1 line · Thomas Gleixner
xstateregs_get()does not longer usefpstate_sanitize_xstate()and the only
s/does not longer use/no longer uses/
\
arch/x86/kernel/fpu/regset.c line 15
Quoted 8 lines
arch/x86/kernel/fpu/regset.c +2
-0@@ -11,6 +11,39 @@#include <linux/sched/task_stack.h> +/* +* When executing XSAVEOPT (or other optimized XSAVE instructions), if
The kernel doesn't use XSAVEOPT any more. How about:
When executing XSAVES (or other optimized XSAVE instructions)
arch/x86/kernel/fpu/regset.c line 19
Quoted 4 lines
arch/x86/kernel/fpu/regset.c +4
-0+* a processor implementation detects that an FPU state component is still +* (or is again) in its initialized state, it may clear the corresponding +* bit in the header.xfeatures field, and can skip the writeout of registers +* to the corresponding memory layout.
Additionally, copy_xxx_to_xstate() may result in an xsave buffer with a bit clear in xfeatures but the corresponding state region not containing the state's init value.
arch/x86/kernel/fpu/regset.c line 22
Quoted 3 lines
arch/x86/kernel/fpu/regset.c +3
-0+* +* This means that when the bit is zero, the state component might still +* contain some previous - non-initialized register state.
Maybe say what the function does, e.g.:
This function fills in the init values for the X87 and SSE states if the corresponding xfeatures bits are clear.
arch/x86/kernel/fpu/regset.c line 32
Quoted 10 lines
arch/x86/kernel/fpu/regset.c +10
-0+* +* This is required for the legacy regset functions. +*/ +static void fpstate_sanitize_legacy (struct fpu *fpu) +{ +struct fxregs_state *fx = &fpu->state.fxsave; +u64 xfeatures; + +if (!use_xsaveopt ()) +return ;
This is confusing, since we never use xsaveopt. It's also wrong -- see above. How about just removing it?
Quoted 5 lines
arch/x86/kernel/fpu/regset.c +5
-0+ + xfeatures = fpu->state.xsave.header.xfeatures; + +/* If FP is in init state, reinitialize it */ +if (!(xfeatures &XFEATURE_MASK_FP )) {
arch/x86/kernel/fpu/regset.c line 47
Quoted 10 lines
arch/x86/kernel/fpu/regset.c +9
-0+memset (fx,0 ,sizeof (*fx)); + fx->cwd =0x37f ; + } + +/* If SSE is in init state, clear the storage */ +if (!(xfeatures &XFEATURE_MASK_SSE )) +memset (fx->xmm_space,0 ,sizeof (fx->xmm_space)); +} +
Does this result in the mxcsr_mask and mxcsr fields being correct? There is a silly number of special cases there.
Andy Lutomirski
maintainerRe: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
Quoted 61 lines · Andy Lutomirski
Quoted 1 line · Thomas Gleixner
xstateregs_get()does not longer usefpstate_sanitize_xstate()and the onlys/does not longer use/no longer uses/
\arch/x86/kernel/fpu/regset.c line 15
Quoted 8 lines
arch/x86/kernel/fpu/regset.c +2
-0@@ -11,6 +11,39 @@#include <linux/sched/task_stack.h> +/* +* When executing XSAVEOPT (or other optimized XSAVE instructions), if The kernel doesn't use XSAVEOPT any more. How about:
When executing XSAVES (or other optimized XSAVE instructions)
arch/x86/kernel/fpu/regset.c line 19
Quoted 4 lines
arch/x86/kernel/fpu/regset.c +4
-0+* a processor implementation detects that an FPU state component is still +* (or is again) in its initialized state, it may clear the corresponding +* bit in the header.xfeatures field, and can skip the writeout of registers +* to the corresponding memory layout. Additionally,
copy_xxx_to_xstate()may result in an xsave buffer with a bit clear in xfeatures but the corresponding state region not containing the state's init value.arch/x86/kernel/fpu/regset.c line 22
Quoted 3 lines
arch/x86/kernel/fpu/regset.c +3
-0+* +* This means that when the bit is zero, the state component might still +* contain some previous - non-initialized register state. Maybe say what the function does, e.g.:
This function fills in the init values for the X87 and SSE states if the corresponding xfeatures bits are clear.
arch/x86/kernel/fpu/regset.c line 32
Quoted 10 lines
arch/x86/kernel/fpu/regset.c +10
-0+* +* This is required for the legacy regset functions. +*/ +static void fpstate_sanitize_legacy (struct fpu *fpu) +{ +struct fxregs_state *fx = &fpu->state.fxsave; +u64 xfeatures; + +if (!use_xsaveopt ()) +return ;This is confusing, since we never use xsaveopt. It's also wrong -- see above. How about just removing it?
Quoted 10 lines
arch/x86/kernel/fpu/regset.c +10
-0+ + xfeatures = fpu->state.xsave.header.xfeatures; + +/* If FP is in init state, reinitialize it */ +if (!(xfeatures &XFEATURE_MASK_FP )) { +memset (fx,0 ,sizeof (*fx)); + fx->cwd =0x37f ; + } + +/* If SSE is in init state, clear the storage */
Quoted 9 lines
arch/x86/kernel/fpu/regset.c line 47
Quoted 5 lines
arch/x86/kernel/fpu/regset.c +4
-0+if (!(xfeatures &XFEATURE_MASK_SSE )) +memset (fx->xmm_space,0 ,sizeof (fx->xmm_space)); +} +Does this result in the
mxcsr_maskand mxcsr fields being correct? There is a silly number of special cases there.
The SDM says, in a footnote in XRSTOR:
There is an exception if RFBM[1] = 0 and RFBM[2] = 1. In this case, the standard form of XRSTOR will load MXCSR from memory, even though MXCSR is part of state component 1 — SSE. The compacted form of XRSTOR does not make this exception.
which makes me think that this code has a bug. Also, the code is manifest nonsense for a different reason:
if (!FP)
memset(the whole damn thing, 0);
if (!SSE)
memset(just part of it, 0);
which means that, if the FP bit is clear but the SSE bit is set, this thing will clobber the SSE state.
This code is garbage. So is the architecture that gave rise to this code.
--Andy
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
Quoted 4 lines · Andy Lutomirski
Quoted 3 lines · Andy Lutomirski
On 6/11/21 9:15 AM, Thomas Gleixner wrote:
Does this result in themxcsr_maskand mxcsr fields being correct? There is a silly number of special cases there.
Let me stare at that.
Quoted 13 lines
The SDM says, in a footnote in XRSTOR:
There is an exception if RFBM[1] = 0 and RFBM[2] = 1. In this case, the standard form of XRSTOR will load MXCSR from memory, even though MXCSR is part of state component 1 — SSE. The compacted form of XRSTOR does not make this exception.
which makes me think that this code has a bug. Also, the code is manifest nonsense for a different reason:
if (!FP)
memset(the whole damn thing, 0);
Quoted 7 lines
if (!SSE)
memset(just part of it, 0);which means that, if the FP bit is clear but the SSE bit is set, this thing will clobber the SSE state.
This code is garbage. So is the architecture that gave rise to this code.
No argument about that :)
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
Quoted 20 lines · Thomas Gleixner
Quoted 4 lines · Andy Lutomirski
Quoted 3 lines · Andy Lutomirski
On 6/11/21 9:15 AM, Thomas Gleixner wrote:
Does this result in themxcsr_maskand mxcsr fields being correct? There is a silly number of special cases there.Let me stare at that.
Quoted 12 lines
The SDM says, in a footnote in XRSTOR:
There is an exception if RFBM[1] = 0 and RFBM[2] = 1. In this case, the standard form of XRSTOR will load MXCSR from memory, even though MXCSR is part of state component 1 — SSE. The compacted form of XRSTOR does not make this exception.
which makes me think that this code has a bug. Also, the code is manifest nonsense for a different reason:
if (!FP)
memset(the whole damn thing, 0);
That's actually my bug. The code I replaced was different. Lemme stare at it and fix it.
Quoted 9 lines
Quoted 7 lines
if (!SSE)
memset(just part of it, 0);which means that, if the FP bit is clear but the SSE bit is set, this thing will clobber the SSE state.
This code is garbage. So is the architecture that gave rise to this code.
No argument about that :)
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
In reply to Andy LutomirskiQuoted 6 lines · Andy Lutomirski
Quoted 3 lines · Thomas Gleixner
xstateregs_get()does not longer usefpstate_sanitize_xstate()and the only +/*
+ * When executing XSAVEOPT (or other optimized XSAVE instructions), ifThe kernel doesn't use XSAVEOPT any more. How about:
You sure?
#define XSTATE_XSAVE(st, lmask, hmask, err) \ asm volatile(ALTERNATIVE_2(XSAVE, \ XSAVEOPT, X86_FEATURE_XSAVEOPT, \ XSAVES, X86_FEATURE_XSAVES) \
Thanks,
tglx
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
In reply to Andy LutomirskiQuoted 14 lines · Andy Lutomirski
arch/x86/kernel/fpu/regset.c line 32
Quoted 10 lines · Thomas Gleixner
+ *
+ * This is required for the legacy regset functions.
+ */arch/x86/kernel/fpu/regset.c +7
-0+static void fpstate_sanitize_legacy (struct fpu *fpu) +{ +struct fxregs_state *fx = &fpu->state.fxsave; +u64 xfeatures; + +if (!use_xsaveopt ()) +return ;This is confusing, since we never use xsaveopt. It's also wrong -- see above. How about just removing it?
We do and this code is buggy because xsaves does not clear the component storage either. Neither does xsavec which we in fact do not use in the kernel.
So here is how the different opcodes behave on a buffer filled with 0xff when looking the first four 64bit words of the buffer after doing a xrstor with all feature bits cleared
Intel SKLX
XSAVE 000000000000037f 0000000000000000 0000000000000000 0000ffff00001f80
XSAVEOPT ffffffffffffffff ffffffffffffffff ffffffffffffffff 0000ffff00001f80
XSAVEC ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
XSAVES ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
AMD ZEN2
XSAVE 000000000000037f 0000000000000000 0000000000000000 0002ffff00001f80
XSAVEOPT ffffffffffffffff ffffffffffffffff ffffffffffffffff 0002ffff00001f80
XSAVEC ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
XSAVES ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
I verified that all saved buffers have xstate.header.xstate_bv == 0
So nothing about any of this is consistent and correct. But it magically works for unknown reasons.
Thanks,
tglx
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
Quoted 16 lines · Thomas Gleixner
Quoted 14 lines · Andy Lutomirski
arch/x86/kernel/fpu/regset.c line 32
Quoted 10 lines · Thomas Gleixner
+ *
+ * This is required for the legacy regset functions.
+ */arch/x86/kernel/fpu/regset.c +7
-0+static void fpstate_sanitize_legacy (struct fpu *fpu) +{ +struct fxregs_state *fx = &fpu->state.fxsave; +u64 xfeatures; + +if (!use_xsaveopt ()) +return ;This is confusing, since we never use xsaveopt. It's also wrong -- see above. How about just removing it?
Quoted 3 lines
We do and this code is buggy because xsaves does not clear the component storage either. Neither does xsavec which we in fact do not use in the kernel.
So that kinda works because the CPUs which have XSAVES have XSAVEOPT too. But XSAVESOPT can be disabled on the command line, which then makes that condition false...
Oh well.
Re: [patch 08/41] x86/fpu: Restrict fpstate sanitizing to legacy components
In reply to Thomas GleixnerQuoted 38 lines · Thomas Gleixner
Quoted 14 lines · Andy Lutomirski
arch/x86/kernel/fpu/regset.c line 32
Quoted 10 lines · Thomas Gleixner
+ *
+ * This is required for the legacy regset functions.
+ */arch/x86/kernel/fpu/regset.c +7
-0+static void fpstate_sanitize_legacy (struct fpu *fpu) +{ +struct fxregs_state *fx = &fpu->state.fxsave; +u64 xfeatures; + +if (!use_xsaveopt ()) +return ;This is confusing, since we never use xsaveopt. It's also wrong -- see above. How about just removing it?
We do and this code is buggy because xsaves does not clear the component storage either. Neither does xsavec which we in fact do not use in the kernel.
So here is how the different opcodes behave on a buffer filled with
0xffwhen looking the first four 64bit words of the buffer after doing a xrstor with all feature bits clearedIntel SKLX
XSAVE
000000000000037f0000000000000000 00000000000000000000ffff00001f80
XSAVEOPT ffffffffffffffff ffffffffffffffff ffffffffffffffff0000ffff00001f80
XSAVEC ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
XSAVES ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffffAMD ZEN2
XSAVE
000000000000037f0000000000000000 00000000000000000002ffff00001f80
XSAVEOPT ffffffffffffffff ffffffffffffffff ffffffffffffffff0002ffff00001f80
XSAVEC ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
XSAVES ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
Quoted 4 lines
I verified that all saved buffers have xstate.header.xstate_bv == 0
So nothing about any of this is consistent and correct. But it magically works for unknown reasons.
What's even worse is the following. setup_init_fpu_buf() does:
copy_kernel_to_xregs_booting(&init_fpstate.xsave); /* * Dump the init state again. This is to identify the init state * of any feature which is not represented by all zero's. */ copy_xregs_to_kernel_booting(&init_fpstate.xsave);
That comment is blatantly wrong with XSAVES/XRSTORS. init_fpstate is initially all zeros and it stays that way with XRSTORS. Oh well.
And as the intialization values at least for mxcsr_mask differ on AMD and INTEL making them hardcoded is just wrong. Sigh...
So we could save the state with XSAVE into a different buffer and copy the components to the correct place in the compacted init_fpstate, but the only interesting part of all this with any variant of XSAVE is the legacy part. Everything else is just zeros (except for HDC and HWP states which we do not use and they can be accessed via MSRs if the need ever arises). We could just save the legacy part with fxsave and be done with it.
Aside of that this whole _booting() stuff is complete nonsense. It's completely sufficient to XRSTOR from an all zeroes buffer which brings every components into init state and then do all the other muck _after_ alternatives have been patched. Absolutely nothing uses FPU muck before that point.
While staring at all this I figured out why that sanitizing does not and _cannot_ touch MXCSR and MXCSR_MASK.
MXCSR and MXCSR_MASK are located in the FP component storage and used by SSE. But MXCSR is also XSAVEd when AVX is in use.
So the sanitizing cannot touch it without checking whether AVX is in use. This is really all well thought out in hardware _AND_ software.
Let me try to beat some more sense into this trainwreck.
Thanks,
tglx
PATCH 9/41 x86/kvm: Avoid looking up PKRU in XSAVE buffer
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
PKRU is being removed from the kernel XSAVE/FPU buffers. This removal will probably include warnings for code that look up PKRU in those buffers.
KVM currently looks up the location of PKRU but doesn't even use the pointer that it gets back. Rework the code to avoid calling get_xsave_addr() except in cases where its result is actually used.
This makes the code more clear and also avoids the inevitable PKRU warnings.
This is probably a good cleanup and could go upstream idependently of any PKRU rework.
---
arch/x86/kvm/x86.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-)
arch/x86/kvm/x86.c +22 -19
@@ -4589,20 +4589,21 @@ static void fill_xsave(u8 *dest, struct*/ valid = xstate_bv & ~XFEATURE_MASK_FPSSE ;while (valid) { +u32 size, offset, ecx, edx;u64 xfeature_mask = valid & -valid;int xfeature_nr =fls64 (xfeature_mask) -1 ; -void *src =get_xsave_addr (xsave, xfeature_nr); +void *src; -if (src) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (dest + offset, &vcpu->arch.pkru, -sizeof (vcpu->arch.pkru)); -else -memcpy (dest + offset, src, size); +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx); +if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (dest + offset, &vcpu->arch.pkru, +sizeof (vcpu->arch.pkru)); + }else { + src =get_xsave_addr (xsave, xfeature_nr); +if (src) +memcpy (dest + offset, src, size); } valid -= xfeature_mask;
@@ -4632,18 +4633,20 @@ static void load_xsave(struct kvm_vcpu **/ valid = xstate_bv & ~XFEATURE_MASK_FPSSE ;while (valid) { +u32 size, offset, ecx, edx;u64 xfeature_mask = valid & -valid;int xfeature_nr =fls64 (xfeature_mask) -1 ; -void *dest =get_xsave_addr (xsave, xfeature_nr); -if (dest) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (&vcpu->arch.pkru, src + offset, -sizeof (vcpu->arch.pkru)); -else +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx); + +if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (&vcpu->arch.pkru, src + offset, +sizeof (vcpu->arch.pkru)); + }else { +void *dest =get_xsave_addr (xsave, xfeature_nr); + +if (dest)memcpy (dest, src + offset, size); }
Re: [patch 09/41] x86/kvm: Avoid looking up PKRU in XSAVE buffer
Quoted 12 lines · Thomas Gleixner
arch/x86/kvm/x86.c +2
-9-if (src) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (dest + offset, &vcpu->arch.pkru, -sizeof (vcpu->arch.pkru)); -else -memcpy (dest + offset, src, size); +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx);
arch/x86/kvm/x86.c static void fill_xsave(u8 *dest, struct · line 4609
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0+if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (dest + offset, &vcpu->arch.pkru, +sizeof (vcpu->arch.pkru)); + }else { + src =get_xsave_addr (xsave, xfeature_nr); +if (src) +memcpy (dest + offset, src, size); } valid -= xfeature_mask;
How about pulling up that PKRU case even further (pasting the whole changed loop instead of an unreadable diff) and keeping the CPUID access and the xsave address handling separate?
valid = xstate_bv & ~XFEATURE_MASK_FPSSE;
while (valid) {
u32 size, offset, ecx, edx;
u64 xfeature_mask = valid & -valid;
int xfeature_nr = fls64(xfeature_mask) - 1;
void *src;
if (xfeature_nr == XFEATURE_PKRU) {
memcpy(dest + offset, &vcpu->arch.pkru, sizeof(vcpu->arch.pkru));
continue;
}
cpuid_count(XSTATE_CPUID, xfeature_nr, &size, &offset, &ecx, &edx);
src = get_xsave_addr(xsave, xfeature_nr);
if (src)
memcpy(dest + offset, src, size);
valid -= xfeature_mask;
}Btw, I'm wondering if that CPUID read in a loop can be replaced with adding accessors for xstate_{offsets,sizes,..} etc and providing them to kvm...
Quoted 16 lines
arch/x86/kvm/x86.c +1
-8@@ -4632,18 +4633,20 @@ static void load_xsave(struct kvm_vcpu **/ valid = xstate_bv & ~XFEATURE_MASK_FPSSE ;while (valid) { +u32 size, offset, ecx, edx;u64 xfeature_mask = valid & -valid;int xfeature_nr =fls64 (xfeature_mask) -1 ; -void *dest =get_xsave_addr (xsave, xfeature_nr); -if (dest) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (&vcpu->arch.pkru, src + offset, -sizeof (vcpu->arch.pkru));
arch/x86/kvm/x86.c static void load_xsave(struct kvm_vcpu * · line 4648
Quoted 10 lines
arch/x86/kvm/x86.c +9
-1-else +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx); + +if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (&vcpu->arch.pkru, src + offset, +sizeof (vcpu->arch.pkru)); + }else { +void *dest =get_xsave_addr (xsave, xfeature_nr); +
^ Superfluous newline.
Quoted 5 lines
arch/x86/kvm/x86.c +1
-0+if (dest)memcpy (dest, src + offset, size); }
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergRe: [patch 09/41] x86/kvm: Avoid looking up PKRU in XSAVE buffer
Quoted 40 lines · Borislav Petkov
Quoted 12 lines · Thomas Gleixner
(quoted diff) +2
-9- if (src) { - u32 size, offset, ecx, edx; - cpuid_count(XSTATE_CPUID, xfeature_nr, - &size, &offset, &ecx, &edx); - if (xfeature_nr == XFEATURE_PKRU) - memcpy(dest + offset, &vcpu->arch.pkru, - sizeof(vcpu->arch.pkru)); - else - memcpy(dest + offset, src, size); + cpuid_count(XSTATE_CPUID, xfeature_nr, + &size, &offset, &ecx, &edx);Review
Quoted 10 lines
(quoted diff) +7
-0+ if (xfeature_nr == XFEATURE_PKRU) { + memcpy(dest + offset, &vcpu->arch.pkru, + sizeof(vcpu->arch.pkru)); + } else { + src = get_xsave_addr(xsave, xfeature_nr); + if (src) + memcpy(dest + offset, src, size); } valid -= xfeature_mask;How about pulling up that PKRU case even further (pasting the whole changed loop instead of an unreadable diff) and keeping the CPUID access and the xsave address handling separate?
valid = xstate_bv & ~XFEATURE_MASK_FPSSE; while (valid) { u32 size, offset, ecx, edx; u64 xfeature_mask = valid & -valid; int xfeature_nr = fls64(xfeature_mask) - 1; void *src; if (xfeature_nr == XFEATURE_PKRU) { memcpy(dest + offset, &vcpu->arch.pkru, sizeof(vcpu->arch.pkru)); continue; }
Quoted 8 lines
cpuid_count(XSTATE_CPUID, xfeature_nr, &size, &offset, &ecx, &edx);
src = get_xsave_addr(xsave, xfeature_nr);
if (src)
memcpy(dest + offset, src, size);
valid -= xfeature_mask;
}I gave that a shot. Two wrinkles: The PKRU memcpy() needs 'offset' from cpuid_count() and the PKRU case also needs the 'valid -=' manipulation.
The result is attached, and while it makes the diff look better, I
don't think the resulting code is an improvement.
Quoted 3 lines
Btw, I'm wondering if that CPUID read in a loop can be replaced with adding accessors for xstate_{offsets,sizes,..} etc and providing them to kvm...
I *think* these are already stored in xfeature_uncompacted_offset[]. It would be a pretty simple matter to export it. I just assumed that this is a slow enough path that the KVM folks don't care.
Quoted 21 lines
Quoted 21 lines
(quoted diff) +5
-9@@ -4632,18 +4633,20 @@ static void load_xsave(struct kvm_vcpu * */ valid = xstate_bv & ~XFEATURE_MASK_FPSSE; while (valid) { + u32 size, offset, ecx, edx; u64 xfeature_mask = valid & -valid; int xfeature_nr = fls64(xfeature_mask) - 1; - void *dest = get_xsave_addr(xsave, xfeature_nr); - if (dest) { - u32 size, offset, ecx, edx; - cpuid_count(XSTATE_CPUID, xfeature_nr, - &size, &offset, &ecx, &edx); - if (xfeature_nr == XFEATURE_PKRU) - memcpy(&vcpu->arch.pkru, src + offset, - sizeof(vcpu->arch.pkru)); - else + cpuid_count(XSTATE_CPUID, xfeature_nr, + &size, &offset, &ecx, &edx); + + if (xfeature_nr == XFEATURE_PKRU) {
Quoted 8 lines
Review static void load_xsave(struct kvm_vcpu * · line 4648
Quoted 5 lines
(quoted diff) +5
-0+ memcpy(&vcpu->arch.pkru, src + offset, + sizeof(vcpu->arch.pkru)); + } else { + void *dest = get_xsave_addr(xsave, xfeature_nr); +^ Superfluous newline.
I'm happy to change it, but I usually like to separate declarations from pure code. Although, I guess that's a bit inconsistent in that file.
commit a761b0a48fb62429bd98c9a1275ef3ce33f9925a
Author: Dave Hansen <[email protected]> Date: Thu Jun 3 16:08:12 2021 -0700
x86/kvm: Avoid looking up PKRU in XSAVE buffer
PKRU is being removed from the kernel XSAVE/FPU buffers. This removal will probably include warnings for code that look up PKRU in those buffers.
KVM currently looks up the location of PKRU but doesn't even use the pointer that it gets back. Rework the code to avoid calling get_xsave_addr() except in cases where its result is actually used.
This makes the code more clear and also avoids the inevitable PKRU warnings.
This is probably a good cleanup and could go upstream idependently of any PKRU rework.
--
arch/x86/kvm/x86.c +24 -21
@@ -4589,20 +4589,21 @@ static void fill_xsave(u8 *dest, struct kvm_vcpu *vcpu)*/ valid = xstate_bv & ~XFEATURE_MASK_FPSSE ;while (valid) { +u32 size, offset, ecx, edx;u64 xfeature_mask = valid & -valid;int xfeature_nr =fls64 (xfeature_mask) -1 ; -void *src =get_xsave_addr (xsave, xfeature_nr); - -if (src) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (dest + offset, &vcpu->arch.pkru, -sizeof (vcpu->arch.pkru)); -else -memcpy (dest + offset, src, size); +void *src; + +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx); +if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (dest + offset, &vcpu->arch.pkru, +sizeof (vcpu->arch.pkru)); + }else { + src =get_xsave_addr (xsave, xfeature_nr); +if (src) +memcpy (dest + offset, src, size); } valid -= xfeature_mask;
@@ -4632,18 +4633,20 @@ static void load_xsave(struct kvm_vcpu *vcpu, u8 *src)*/ valid = xstate_bv & ~XFEATURE_MASK_FPSSE ;while (valid) { +u32 size, offset, ecx, edx;u64 xfeature_mask = valid & -valid;int xfeature_nr =fls64 (xfeature_mask) -1 ; -void *dest =get_xsave_addr (xsave, xfeature_nr); - -if (dest) { -u32 size, offset, ecx, edx; -cpuid_count (XSTATE_CPUID , xfeature_nr, - &size, &offset, &ecx, &edx); -if (xfeature_nr ==XFEATURE_PKRU ) -memcpy (&vcpu->arch.pkru, src + offset, -sizeof (vcpu->arch.pkru)); -else + +cpuid_count (XSTATE_CPUID , xfeature_nr, + &size, &offset, &ecx, &edx); + +if (xfeature_nr ==XFEATURE_PKRU ) { +memcpy (&vcpu->arch.pkru, src + offset, +sizeof (vcpu->arch.pkru)); + }else { +void *dest =get_xsave_addr (xsave, xfeature_nr); + +if (dest)memcpy (dest, src + offset, size); }
Re: [patch 09/41] x86/kvm: Avoid looking up PKRU in XSAVE buffer
Quoted 4 lines · Dave Hansen
I gave that a shot. Two wrinkles: The PKRU
memcpy()needs 'offset' fromcpuid_count()and the PKRU case also needs the 'valid -=' manipulation.
The result is attached, and while it makes the diff look better, I
don't think the resulting code is an improvement.
Bah, that was too much wishful and faulty thinking on my part, forget what I said.
Quoted 3 lines
I *think* these are already stored in xfeature_uncompacted_offset[]. It would be a pretty simple matter to export it. I just assumed that this is a slow enough path that the KVM folks don't care.
I guess. Yeah, let's cleanup the FPU mess first and then see what makes sense or not.
Quoted 2 lines
I'm happy to change it, but I usually like to separate declarations from pure code. Although, I guess that's a bit inconsistent in that file.
No, this is what I mean:
+ src = get_xsave_addr(xsave, xfeature_nr);
+ if (src)
+ memcpy(dest + offset, src, size);
vs
+ void *dest = get_xsave_addr(xsave, xfeature_nr);
+
+ if (dest)
memcpy(dest, src + offset, size);
both in your patch.
It is a lot easier when reading the code to have the error handling glued together with the previous function call.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
SUSE Software Solutions Germany GmbH, GF: Felix Imendörffer, HRB 36809, AG NürnbergPATCH 10/41 x86/fpu: Cleanup arch_set_user_pkey_access()
In reply to Thomas GleixnerThe function is having a sanity check with a WARN_ON_ONCE() but happily proceeds when the pkey argument is out of range.
Clean it up.
--- arch/x86/kernel/fpu/xstate.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
arch/x86/kernel/fpu/xstate.c +6 -5
@@ -887,11 +887,10 @@ EXPORT_SYMBOL_GPL(get_xsave_addr);* rights for @pkey to @init_val. */ int arch_set_user_pkey_access (struct task_struct *tsk,int pkey, -unsigned long init_val) +unsigned long init_val) { -u32 old_pkru; -int pkey_shift = (pkey *PKRU_BITS_PER_PKEY ); -u32 new_pkru_bits =0 ; +u32 old_pkru, new_pkru_bits =0 ; +int pkey_shift;/* * This check implies XSAVE support. OSPKE only gets
@@ -905,7 +904,8 @@ int arch_set_user_pkey_access(struct tas* values originating from in-kernel users. Complain * if a bad value is observed. */ -WARN_ON_ONCE (pkey >=arch_max_pkey ()); +if (WARN_ON_ONCE (pkey >=arch_max_pkey ())) +return -EINVAL ;/* Set the bits we need in PKRU: */ if (init_val &PKEY_DISABLE_ACCESS )
@@ -914,6 +914,7 @@ int arch_set_user_pkey_access(struct tas new_pkru_bits |=PKRU_WD_BIT ;/* Shift the bits in to the correct place in PKRU for pkey: */ + pkey_shift = pkey *PKRU_BITS_PER_PKEY ; new_pkru_bits <<= pkey_shift;/* Get old PKRU and mask off any old bits in place: */
PATCH 11/41 x86/fpu: Get rid of copy_supervisor_to_kernel()
In reply to Thomas GleixnerIf the fast path of restoring the FPU state on sigreturn fails or is not taken and the current task's FPU is active then the FPU has to be deactivated for the slow path to allow a safe update of the tasks FPU memory state.
With supervisor states enabled, this requires to save the supervisor state in the memory state first. Supervisor states require XSAVES so saving only the supervisor state requires to reshuffle the memory buffer because XSAVES uses the compacted format and therefore stores the supervisor states at the beginning of the memory state. That's just an overengineered optimization.
Get rid of it and save the full state for this case.
---
V4: New patch
--- arch/x86/include/asm/fpu/xstate.h | 1 arch/x86/kernel/fpu/signal.c | 13 +++++--- arch/x86/kernel/fpu/xstate.c | 55 -------------------------------------- 3 files changed, 8 insertions(+), 61 deletions(-)
arch/x86/include/asm/fpu/xstate.h +0 -1
@@ -107,7 +107,6 @@ struct membuf;void copy_xstate_to_kernel (struct membuf to,struct xregs_state *xsave);int copy_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf);int copy_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf); -void copy_supervisor_to_kernel (struct xregs_state *xsave);void copy_dynamic_supervisor_to_kernel (struct xregs_state *xstate,u64 mask);void copy_kernel_to_dynamic_supervisor (struct xregs_state *xstate,u64 mask);
arch/x86/kernel/fpu/signal.c +8 -5
@@ -411,15 +411,18 @@ static int __fpu__restore_sig(void __use* the optimisation). */ fpregs_lock (); -if (!test_thread_flag (TIF_NEED_FPU_LOAD )) { -/* -* Supervisor states are not modified by user space input. Save -* current supervisor states first and invalidate the FPU regs. +* If supervisor states are available then save the +* hardware state in current's fpstate so that the +* supervisor state is preserved. Save the full state for +* simplicity. There is no point in optimizing this by only +* saving the supervisor states and then shuffle them to +* the right place in memory. This is the slow path and the +* above XRSTOR failed or ia32_fxstate is true. Shrug. */ if (xfeatures_mask_supervisor ()) -copy_supervisor_to_kernel (&fpu->state.xsave); +copy_xregs_to_kernel (&fpu->state.xsave);set_thread_flag (TIF_NEED_FPU_LOAD ); }__fpu_invalidate_fpregs_state (fpu);
arch/x86/kernel/fpu/xstate.c +0 -55
61 lines not on this page · raw message
Re: [patch 11/41] x86/fpu: Get rid of copy_supervisor_to_kernel()
Quoted 5 lines · Thomas Gleixner
If the fast path of restoring the FPU state on sigreturn fails or is not taken and the current task's FPU is active then the FPU has to be deactivated for the slow path to allow a safe update of the tasks FPU memory state.
Quoted 7 lines
With supervisor states enabled, this requires to save the supervisor state in the memory state first. Supervisor states require XSAVES so saving only the supervisor state requires to reshuffle the memory buffer because XSAVES uses the compacted format and therefore stores the supervisor states at the beginning of the memory state. That's just an overengineered optimization.
Get rid of it and save the full state for this case.
Hallelujah.
PATCH 12/41 x86/fpu: Rename copy_xregs_to_kernel() and copy_kernel_to_xregs()
In reply to Thomas GleixnerThe function names for xsave[s]/xrstor[s] operations are horribly named and a permanent source of confusion.
Rename:
copy_xregs_to_kernel() to xsave_to_kernel() copy_kernel_to_xregs() to xrstor_from_kernel()
so it's entirely clear what this is about.
No functional change.
--- arch/x86/include/asm/fpu/internal.h | 12 ++++++------ arch/x86/kernel/fpu/core.c | 6 +++--- arch/x86/kernel/fpu/signal.c | 14 +++++++------- arch/x86/kernel/fpu/xstate.c | 4 ++-- 4 files changed, 18 insertions(+), 18 deletions(-)
arch/x86/include/asm/fpu/internal.h +6 -6
@@ -256,7 +256,7 @@ static inline void copy_fxregs_to_kernel* This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet. */ -static inline void copy_xregs_to_kernel_booting (struct xregs_state *xstate) +static inline void xsave_to_kernel_booting (struct xregs_state *xstate) {u64 mask = xfeatures_mask_all;u32 lmask = mask;
@@ -278,7 +278,7 @@ static inline void copy_xregs_to_kernel_* This function is called only during boot time when x86 caps are not set * up and alternative can not be used yet. */ -static inline void copy_kernel_to_xregs_booting (struct xregs_state *xstate) +static inline void xrstor_from_kernel_booting (struct xregs_state *xstate) {u64 mask =-1 ;u32 lmask = mask;
@@ -302,7 +302,7 @@ static inline void copy_kernel_to_xregs_/* * Save processor xstate to xsave area. */ -static inline void copy_xregs_to_kernel (struct xregs_state *xstate) +static inline void xsave_to_kernel (struct xregs_state *xstate) {u64 mask = xfeatures_mask_all;u32 lmask = mask;
@@ -320,7 +320,7 @@ static inline void copy_xregs_to_kernel(/* * Restore processor xstate from xsave area. */ -static inline void copy_kernel_to_xregs (struct xregs_state *xstate,u64 mask) +static inline void xrstor_from_kernel (struct xregs_state *xstate,u64 mask) {u32 lmask = mask;u32 hmask = mask >>32 ;
@@ -381,7 +381,7 @@ static inline int copy_user_to_xregs(str* Restore xstate from kernel space xsave area, return an error code instead of * an exception. */ -static inline int copy_kernel_to_xregs_err (struct xregs_state *xstate,u64 mask) +static inline int xrstor_from_kernel_err (struct xregs_state *xstate,u64 mask) {u32 lmask = mask;u32 hmask = mask >>32 ;
@@ -400,7 +400,7 @@ extern int copy_fpregs_to_fpstate(structstatic inline void __copy_kernel_to_fpregs (union fpregs_state *fpstate,u64 mask) {if (use_xsave ()) { -copy_kernel_to_xregs (&fpstate->xsave, mask); +xrstor_from_kernel (&fpstate->xsave, mask); }else {if (use_fxsr ())copy_kernel_to_fxregs (&fpstate->fxsave);
arch/x86/kernel/fpu/core.c +3 -3
@@ -95,7 +95,7 @@ EXPORT_SYMBOL(irq_fpu_usable);int copy_fpregs_to_fpstate (struct fpu *fpu) {if (likely (use_xsave ())) { -copy_xregs_to_kernel (&fpu->state.xsave); +xsave_to_kernel (&fpu->state.xsave);/* * AVX512 state is tracked here because its use is
@@ -358,7 +358,7 @@ void fpu__drop(struct fpu *fpu)static inline void copy_init_fpstate_to_fpregs (u64 features_mask) {if (use_xsave ()) -copy_kernel_to_xregs (&init_fpstate.xsave, features_mask); +xrstor_from_kernel (&init_fpstate.xsave, features_mask);else if (static_cpu_has (X86_FEATURE_FXSR ))copy_kernel_to_fxregs (&init_fpstate.fxsave);else
@@ -389,7 +389,7 @@ static void fpu__clear(struct fpu *fpu,if (user_only) {if (!fpregs_state_valid (fpu,smp_processor_id ()) &&xfeatures_mask_supervisor ()) -copy_kernel_to_xregs (&fpu->state.xsave, +xrstor_from_kernel (&fpu->state.xsave,xfeatures_mask_supervisor ());copy_init_fpstate_to_fpregs (xfeatures_mask_user ()); }else {
arch/x86/kernel/fpu/signal.c +7 -7
@@ -271,14 +271,14 @@ static int copy_user_to_fpregs_zeroing(v r =copy_user_to_fxregs (buf);if (!r) -copy_kernel_to_xregs (&init_fpstate.xsave, init_bv); +xrstor_from_kernel (&init_fpstate.xsave, init_bv);return r; }else { init_bv =xfeatures_mask_user () & ~xbv; r =copy_user_to_xregs (buf, xbv);if (!r &&unlikely (init_bv)) -copy_kernel_to_xregs (&init_fpstate.xsave, init_bv); +xrstor_from_kernel (&init_fpstate.xsave, init_bv);return r; } }else if (use_fxsr ()) {
@@ -367,7 +367,7 @@ static int __fpu__restore_sig(void __use*/ if (test_thread_flag (TIF_NEED_FPU_LOAD ) &&xfeatures_mask_supervisor ()) -copy_kernel_to_xregs (&fpu->state.xsave, +xrstor_from_kernel (&fpu->state.xsave,xfeatures_mask_supervisor ());fpregs_mark_activate ();fpregs_unlock ();
@@ -422,7 +422,7 @@ static int __fpu__restore_sig(void __use* above XRSTOR failed or ia32_fxstate is true. Shrug. */ if (xfeatures_mask_supervisor ()) -copy_xregs_to_kernel (&fpu->state.xsave); +xsave_to_kernel (&fpu->state.xsave);set_thread_flag (TIF_NEED_FPU_LOAD ); }__fpu_invalidate_fpregs_state (fpu);
@@ -440,13 +440,13 @@ static int __fpu__restore_sig(void __usefpregs_lock ();if (unlikely (init_bv)) -copy_kernel_to_xregs (&init_fpstate.xsave, init_bv); +xrstor_from_kernel (&init_fpstate.xsave, init_bv);/* * Restore previously saved supervisor xstates along with * copied-in user xstates. */ - ret =copy_kernel_to_xregs_err (&fpu->state.xsave, + ret =xrstor_from_kernel_err (&fpu->state.xsave, user_xfeatures |xfeatures_mask_supervisor ()); }else if (use_fxsr ()) {
@@ -464,7 +464,7 @@ static int __fpu__restore_sig(void __useu64 init_bv; init_bv =xfeatures_mask_user () & ~XFEATURE_MASK_FPSSE ; -copy_kernel_to_xregs (&init_fpstate.xsave, init_bv); +xrstor_from_kernel (&init_fpstate.xsave, init_bv); } ret =copy_kernel_to_fxregs_err (&fpu->state.fxsave);
arch/x86/kernel/fpu/xstate.c +2 -2
@@ -388,13 +388,13 @@ static void __init setup_init_fpu_buf(vo/* * Init all the features state with header.xfeatures being 0x0 */ -copy_kernel_to_xregs_booting (&init_fpstate.xsave); +xrstor_from_kernel_booting (&init_fpstate.xsave);/* * Dump the init state again. This is to identify the init state * of any feature which is not represented by all zero's. */ -copy_xregs_to_kernel_booting (&init_fpstate.xsave); +xsave_to_kernel_booting (&init_fpstate.xsave); }static int xfeature_uncompacted_offset (int xfeature_nr)
PATCH 13/41 x86/fpu: Rename copy_user_to_xregs() and copy_xregs_to_user()
In reply to Thomas GleixnerThe function names for xsave[s]/xrstor[s] operations are horribly named and a permanent source of confusion.
Rename:
copy_xregs_to_user() to xsave_to_user_sigframe() copy_user_to_xregs() to xrstor_from_user_sigframe()
so it's entirely clear what this is about. This is also a clear indicator of the potentially different storage format because this is user ABI and cannot use compacted format.
No functional change.
--- arch/x86/include/asm/fpu/internal.h | 4 ++-- arch/x86/kernel/fpu/signal.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu/internal.h +2 -2
@@ -338,7 +338,7 @@ static inline void xrstor_from_kernel(st* backward compatibility for old applications which don't understand * compacted format of xsave area. */ -static inline int copy_xregs_to_user (struct xregs_state __user *buf) +static inline int xsave_to_user_sigframe (struct xregs_state __user *buf) {u64 mask =xfeatures_mask_user ();u32 lmask = mask;
@@ -363,7 +363,7 @@ static inline int copy_xregs_to_user(str/* * Restore xstate from user space xsave area. */ -static inline int copy_user_to_xregs (struct xregs_state __user *buf,u64 mask) +static inline int xrstor_from_user_sigframe (struct xregs_state __user *buf,u64 mask) {struct xregs_state *xstate = ((__force struct xregs_state *)buf);u32 lmask = mask;
arch/x86/kernel/fpu/signal.c +2 -2
@@ -129,7 +129,7 @@ static inline int copy_fpregs_to_sigframint err;if (use_xsave ()) - err =copy_xregs_to_user (buf); + err =xsave_to_user_sigframe (buf);else if (use_fxsr ()) err =copy_fxregs_to_user ((struct fxregs_state __user *) buf);else
@@ -276,7 +276,7 @@ static int copy_user_to_fpregs_zeroing(v }else { init_bv =xfeatures_mask_user () & ~xbv; - r =copy_user_to_xregs (buf, xbv); + r =xrstor_from_user_sigframe (buf, xbv);if (!r &&unlikely (init_bv))xrstor_from_kernel (&init_fpstate.xsave, init_bv);return r;
PATCH 14/41 x86/fpu: Rename fxregs related copy functions
In reply to Thomas GleixnerThe function names for fxsave/fxrstor operations are horribly named and a permanent source of confusion.
Rename:
copy_fxregs_to_kernel() to fxsave_to_kernel() copy_kernel_to_fxregs() to fxrstor_from_kernel() copy_fxregs_to_user() to fxsave_to_user_sigframe() copy_user_to_fxregs() to fxrstor_from_user_sigframe()
so it's entirely clear what this is about.
No functional change.
--- arch/x86/include/asm/fpu/internal.h | 12 ++++++------ arch/x86/kernel/fpu/core.c | 4 ++-- arch/x86/kernel/fpu/signal.c | 10 +++++----- 3 files changed, 13 insertions(+), 13 deletions(-)
arch/x86/include/asm/fpu/internal.h +6 -6
@@ -132,7 +132,7 @@ static inline int copy_fregs_to_user(str return user_insn(fnsave %[fx]; fwait, [fx] "=m" (*fx), "m" (*fx)); } -static inline int copy_fxregs_to_user (struct fxregs_state __user *fx) +static inline int fxsave_to_user_sigframe (struct fxregs_state __user *fx) {if (IS_ENABLED (CONFIG_X86_32 )) return user_insn(fxsave %[fx], [fx] "=m" (*fx), "m" (*fx));
@@ -141,7 +141,7 @@ static inline int copy_fxregs_to_user(st } -static inline void copy_kernel_to_fxregs (struct fxregs_state *fx) +static inline void fxrstor_from_kernel (struct fxregs_state *fx) {if (IS_ENABLED (CONFIG_X86_32 )) kernel_insn(fxrstor %[fx], "=m" (*fx), [fx] "m" (*fx));
@@ -149,7 +149,7 @@ static inline void copy_kernel_to_fxregs kernel_insn(fxrstorq %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline int copy_kernel_to_fxregs_err (struct fxregs_state *fx) +static inline int fxrstor_from_kernel_err (struct fxregs_state *fx) {if (IS_ENABLED (CONFIG_X86_32 )) return kernel_insn_err(fxrstor %[fx], "=m" (*fx), [fx] "m" (*fx));
@@ -157,7 +157,7 @@ static inline int copy_kernel_to_fxregs_ return kernel_insn_err(fxrstorq %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline int copy_user_to_fxregs (struct fxregs_state __user *fx) +static inline int fxrstor_from_user_sigframe (struct fxregs_state __user *fx) {if (IS_ENABLED (CONFIG_X86_32 )) return user_insn(fxrstor %[fx], "=m" (*fx), [fx] "m" (*fx));
@@ -180,7 +180,7 @@ static inline int copy_user_to_fregs(str return user_insn(frstor %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline void copy_fxregs_to_kernel (struct fpu *fpu) +static inline void fxsave_to_kernel (struct fpu *fpu) {if (IS_ENABLED (CONFIG_X86_32 )) asmvolatile ("fxsave %[fx]" : [fx]"=m" (fpu->state.fxsave));
@@ -403,7 +403,7 @@ static inline void __copy_kernel_to_fprexrstor_from_kernel (&fpstate->xsave, mask); }else {if (use_fxsr ()) -copy_kernel_to_fxregs (&fpstate->fxsave); +fxrstor_from_kernel (&fpstate->fxsave);else copy_kernel_to_fregs (&fpstate->fsave); }
arch/x86/kernel/fpu/core.c +2 -2
@@ -107,7 +107,7 @@ int copy_fpregs_to_fpstate(struct fpu *f }if (likely (use_fxsr ())) { -copy_fxregs_to_kernel (fpu); +fxsave_to_kernel (fpu);return 1 ; }
@@ -360,7 +360,7 @@ static inline void copy_init_fpstate_to_if (use_xsave ())xrstor_from_kernel (&init_fpstate.xsave, features_mask);else if (static_cpu_has (X86_FEATURE_FXSR )) -copy_kernel_to_fxregs (&init_fpstate.fxsave); +fxrstor_from_kernel (&init_fpstate.fxsave);else copy_kernel_to_fregs (&init_fpstate.fsave);
arch/x86/kernel/fpu/signal.c +5 -5
@@ -64,7 +64,7 @@ static inline int save_fsave_header(strufpregs_lock ();if (!test_thread_flag (TIF_NEED_FPU_LOAD )) -copy_fxregs_to_kernel (&tsk->thread.fpu); +fxsave_to_kernel (&tsk->thread.fpu);fpregs_unlock ();convert_from_fxsr (&env, tsk);
@@ -131,7 +131,7 @@ static inline int copy_fpregs_to_sigframif (use_xsave ()) err =xsave_to_user_sigframe (buf);else if (use_fxsr ()) - err =copy_fxregs_to_user ((struct fxregs_state __user *) buf); + err =fxsave_to_user_sigframe ((struct fxregs_state __user *) buf);else err =copy_fregs_to_user ((struct fregs_state __user *) buf);
@@ -269,7 +269,7 @@ static int copy_user_to_fpregs_zeroing(vif (fx_only) { init_bv =xfeatures_mask_user () & ~XFEATURE_MASK_FPSSE ; - r =copy_user_to_fxregs (buf); + r =fxrstor_from_user_sigframe (buf);if (!r)xrstor_from_kernel (&init_fpstate.xsave, init_bv);return r;
@@ -282,7 +282,7 @@ static int copy_user_to_fpregs_zeroing(vreturn r; } }else if (use_fxsr ()) { -return copy_user_to_fxregs (buf); +return fxrstor_from_user_sigframe (buf); }else return copy_user_to_fregs (buf); }
@@ -467,7 +467,7 @@ static int __fpu__restore_sig(void __usexrstor_from_kernel (&init_fpstate.xsave, init_bv); } - ret =copy_kernel_to_fxregs_err (&fpu->state.fxsave); + ret =fxrstor_from_kernel_err (&fpu->state.fxsave); }else { ret =__copy_from_user (&fpu->state.fsave, buf_fx, state_size);if (ret)
PATCH 15/41 x86/fpu: Rename fregs related copy functions
In reply to Thomas GleixnerThe function names for fnsave/fnrstor operations are horribly named and a permanent source of confusion.
Rename:
copy_fregs_to_kernel() to fnsave_to_kernel() copy_kernel_to_fregs() to fnrstor_from_kernel()
copy_fregs_to_user() to fnsave_to_user_sigframe()
copy_user_to_fregs() to fnrstor_from_user_sigframe()
so it's entirely clear what this is about.
No functional change.
--- arch/x86/include/asm/fpu/internal.h | 10 +++++----- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/include/asm/fpu/internal.h +5 -5
@@ -127,7 +127,7 @@ static inline void fpstate_init_soft(str _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_fprestore) \ : output : input) -static inline int copy_fregs_to_user(struct fregs_state __user *fx) +static inline int fnsave_to_user_sigframe(struct fregs_state __user *fx) { return user_insn(fnsave %[fx]; fwait, [fx] "=m" (*fx), "m" (*fx)); }
@@ -165,17 +165,17 @@ static inline int fxrstor_from_user_sigf return user_insn(fxrstorq %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline void copy_kernel_to_fregs (struct fregs_state *fx) +static inline void frstor_from_kernel (struct fregs_state *fx) { kernel_insn(frstor %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline int copy_kernel_to_fregs_err (struct fregs_state *fx) +static inline int frstor_from_kernel_err (struct fregs_state *fx) { return kernel_insn_err(frstor %[fx], "=m" (*fx), [fx] "m" (*fx)); } -static inline int copy_user_to_fregs (struct fregs_state __user *fx) +static inline int frstor_from_user_sigframe (struct fregs_state __user *fx) { return user_insn(frstor %[fx], "=m" (*fx), [fx] "m" (*fx)); }
@@ -405,7 +405,7 @@ static inline void __copy_kernel_to_fpreif (use_fxsr ())fxrstor_from_kernel (&fpstate->fxsave);else -copy_kernel_to_fregs (&fpstate->fsave); +frstor_from_kernel (&fpstate->fsave); } }
arch/x86/kernel/fpu/core.c +1 -1
@@ -362,7 +362,7 @@ static inline void copy_init_fpstate_to_else if (static_cpu_has (X86_FEATURE_FXSR ))fxrstor_from_kernel (&init_fpstate.fxsave);else -copy_kernel_to_fregs (&init_fpstate.fsave); +frstor_from_kernel (&init_fpstate.fsave);if (boot_cpu_has (X86_FEATURE_OSPKE ))copy_init_pkru_to_fpregs ();
arch/x86/kernel/fpu/signal.c +3 -3
@@ -133,7 +133,7 @@ static inline int copy_fpregs_to_sigframelse if (use_fxsr ()) err =fxsave_to_user_sigframe ((struct fxregs_state __user *) buf);else - err =copy_fregs_to_user ((struct fregs_state __user *) buf); + err =fnsave_to_user_sigframe ((struct fregs_state __user *) buf);if (unlikely (err) &&__clear_user (buf, fpu_user_xstate_size)) err = -EFAULT ;
@@ -284,7 +284,7 @@ static int copy_user_to_fpregs_zeroing(v }else if (use_fxsr ()) {return fxrstor_from_user_sigframe (buf); }else -return copy_user_to_fregs (buf); +return frstor_from_user_sigframe (buf); }static int __fpu__restore_sig (void __user *buf,void __user *buf_fx,int size)
@@ -474,7 +474,7 @@ static int __fpu__restore_sig(void __use goto out;fpregs_lock (); - ret =copy_kernel_to_fregs_err (&fpu->state.fsave); + ret =frstor_from_kernel_err (&fpu->state.fsave); }if (!ret)fpregs_mark_activate ();
PATCH 16/41 x86/fpu: Rename xstate copy functions which are related to UABI
In reply to Thomas GleixnerRename them to reflect that these functions deal with user space format XSAVE buffers.
copy_xstate_to_kernel() -> copy_uabi_xstate_to_membuf()
copy_kernel_to_xstate() -> copy_uabi_from_kernel_to_xstate()
copy_user_to_xstate() -> copy_sigframe_from_user_to_xstate()Again a clear statement that these functions deal with user space ABI.
---
V4: Better names at least more consistent
V2: New patch
--- arch/x86/include/asm/fpu/xstate.h | 6 +++--- arch/x86/kernel/fpu/regset.c | 4 ++-- arch/x86/kernel/fpu/signal.c | 2 +- arch/x86/kernel/fpu/xstate.c | 7 ++++--- 4 files changed, 10 insertions(+), 9 deletions(-)
arch/x86/include/asm/fpu/xstate.h +3 -3
@@ -104,9 +104,9 @@ void *get_xsave_addr(struct xregs_stateint using_compacted_format (void );int xfeature_size (int xfeature_nr);struct membuf ; -void copy_xstate_to_kernel (struct membuf to,struct xregs_state *xsave); -int copy_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf); -int copy_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf); +void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave); +int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf); +int copy_sigframe_from_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf);void copy_dynamic_supervisor_to_kernel (struct xregs_state *xstate,u64 mask);void copy_kernel_to_dynamic_supervisor (struct xregs_state *xstate,u64 mask);
arch/x86/kernel/fpu/regset.c +2 -2
@@ -118,7 +118,7 @@ int xstateregs_get(struct task_struct *tfpu__prepare_read (fpu); -copy_xstate_to_kernel (to, &fpu->state.xsave); +copy_uabi_xstate_to_membuf (to, &fpu->state.xsave);return 0 ; }
@@ -150,7 +150,7 @@ int xstateregs_set(struct task_struct *t }fpu__prepare_write (fpu); - ret =copy_kernel_to_xstate (&fpu->state.xsave, kbuf ?: tmpbuf); + ret =copy_uabi_from_kernel_to_xstate (&fpu->state.xsave, kbuf ?: tmpbuf); out:vfree (tmpbuf);
arch/x86/kernel/fpu/signal.c +1 -1
@@ -431,7 +431,7 @@ static int __fpu__restore_sig(void __useif (use_xsave () && !fx_only) {u64 init_bv =xfeatures_mask_user () & ~user_xfeatures; - ret =copy_user_to_xstate (&fpu->state.xsave, buf_fx); + ret =copy_sigframe_from_user_to_xstate (&fpu->state.xsave, buf_fx);if (ret) goto out;
arch/x86/kernel/fpu/xstate.c +4 -3
@@ -968,7 +968,7 @@ static void copy_part(struct membuf *to,* It supports partial copy but pos always starts from zero. This is called * from xstateregs_get() and there we check the CPU has XSAVE. */ -void copy_xstate_to_kernel (struct membuf to,struct xregs_state *xsave) +void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave) {struct xstate_header header;const unsigned off_mxcsr = offsetof(struct fxregs_state , mxcsr);
@@ -1024,7 +1024,7 @@ void copy_xstate_to_kernel(struct membuf* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S] format * and copy to the target thread. This is called from xstateregs_set(). */ -int copy_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf) +int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf) {unsigned int offset, size;int i;
@@ -1079,7 +1079,8 @@ int copy_kernel_to_xstate(struct xregs_s* XSAVE[S] format and copy to the target thread. This is called from the * sigreturn() and rt_sigreturn() system calls. */ -int copy_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf) +int copy_sigframe_from_user_to_xstate (struct xregs_state *xsave, +const void __user *ubuf) {unsigned int offset, size;int i;
PATCH 17/41 x86/fpu: Deduplicate copy_uabi_from_user/kernel_to_xstate()
In reply to Thomas Gleixnercopy_uabi_from_user_to_xstate() and copy_uabi_from_kernel_to_xstate() are almost identical except for the copy function.
Unify them.
---
V4: Adjusted to the rename.
--- arch/x86/kernel/fpu/xstate.c | 84 +++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 54 deletions(-)
arch/x86/kernel/fpu/xstate.c +30 -54
@@ -1020,20 +1020,30 @@ void copy_uabi_xstate_to_membuf(struct mfill_gap (&to, &last, size); } -/* -* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S] format -* and copy to the target thread. This is called from xstateregs_set(). -*/ -int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf) +static int copy_from_buffer (void *dst,unsigned int offset,unsigned int size, +const void *kbuf,const void __user *ubuf) +{ +if (kbuf) { +memcpy (dst, kbuf + offset, size); + }else { +if (copy_from_user (dst, ubuf + offset, size)) +return -EFAULT ; + } +return 0 ; +} + +static int copy_uabi_to_xstate (struct xregs_state *xsave,const void *kbuf, +const void __user *ubuf) {unsigned int offset, size; -int i;struct xstate_header hdr; +int i; offset = offsetof(struct xregs_state , header); size =sizeof (hdr); -memcpy (&hdr, kbuf + offset, size); +if (copy_from_buffer (&hdr, offset, size, kbuf, ubuf)) +return -EFAULT ;if (validate_user_xstate_header (&hdr))return -EINVAL ;
@@ -1047,7 +1057,8 @@ int copy_uabi_from_kernel_to_xstate(stru offset = xstate_offsets[i]; size = xstate_sizes[i]; -memcpy (dst, kbuf + offset, size); +if (copy_from_buffer (dst, offset, size, kbuf, ubuf)) +return -EFAULT ; } }
@@ -1075,6 +1086,16 @@ int copy_uabi_from_kernel_to_xstate(stru }/* +* Convert from a ptrace standard-format kernel buffer to kernel XSAVE[S] +* format and copy to the target thread. This is called from +* xstateregs_set(). +*/ +int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf) +{ +return copy_uabi_to_xstate (xsave, kbuf,NULL ); +} + +/* * Convert from a sigreturn standard-format user-space buffer to kernel * XSAVE[S] format and copy to the target thread. This is called from the * sigreturn() and rt_sigreturn() system calls.
@@ -1082,52 +1103,7 @@ int copy_uabi_from_kernel_to_xstate(struint copy_sigframe_from_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf) { -unsigned int offset, size; -int i; -struct xstate_header hdr; - - offset = offsetof(struct xregs_state , header); - size =sizeof (hdr); - -if (copy_from_user (&hdr, ubuf + offset, size)) -return -EFAULT ; - -if (validate_user_xstate_header (&hdr)) -return -EINVAL ; - -for (i =0 ; i <XFEATURE_MAX ; i++) { -u64 mask = ((u64 )1 << i); - -if (hdr.xfeatures & mask) { -void *dst =__raw_xsave_addr (xsave, i); - - offset = xstate_offsets[i]; - size = xstate_sizes[i]; - -if (copy_from_user (dst, ubuf + offset, size)) -return -EFAULT ; - } - } - -if (xfeatures_mxcsr_quirk (hdr.xfeatures)) { - offset = offsetof(struct fxregs_state , mxcsr); - size =MXCSR_AND_FLAGS_SIZE ; -if (copy_from_user (&xsave->i387.mxcsr, ubuf + offset, size)) -return -EFAULT ; - } - -/* -* The state that came in from userspace was user-state only. -* Mask all the user states out of 'xfeatures': -*/ - xsave->header.xfeatures &=XFEATURE_MASK_SUPERVISOR_ALL ; - -/* -* Add back in the features that came in from userspace: -*/ - xsave->header.xfeatures |= hdr.xfeatures; - -return 0 ; +return copy_uabi_to_xstate (xsave,NULL , ubuf); }/**
PATCH 18/41 x86/fpu: Rename copy_fpregs_to_fpstate() to save_fpregs_to_fpstate()
In reply to Thomas GleixnerA copy is guaranteed to leave the source intact, which is not the case when FNSAVE is used as that reinitilizes the registers.
Rename it to save_fpregs_to_fpstate() which does not make such guarantees.
--- arch/x86/include/asm/fpu/internal.h | 4 ++-- arch/x86/kernel/fpu/core.c | 10 +++++----- arch/x86/kvm/x86.c | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-)
arch/x86/include/asm/fpu/internal.h +2 -2
@@ -395,7 +395,7 @@ static inline int xrstor_from_kernel_errreturn err; } -extern int copy_fpregs_to_fpstate (struct fpu *fpu); +extern int save_fpregs_to_fpstate (struct fpu *fpu);static inline void __copy_kernel_to_fpregs (union fpregs_state *fpstate,u64 mask) {
@@ -527,7 +527,7 @@ static inline void __fpregs_load_activatstatic inline void switch_fpu_prepare (struct fpu *old_fpu,int cpu) {if (static_cpu_has (X86_FEATURE_FPU ) && !(current->flags &PF_KTHREAD )) { -if (!copy_fpregs_to_fpstate (old_fpu)) +if (!save_fpregs_to_fpstate (old_fpu)) old_fpu->last_cpu =-1 ;else old_fpu->last_cpu = cpu;
arch/x86/kernel/fpu/core.c +5 -5
@@ -92,7 +92,7 @@ EXPORT_SYMBOL(irq_fpu_usable);* Modern FPU state can be kept in registers, if there are * no pending FP exceptions. */ -int copy_fpregs_to_fpstate (struct fpu *fpu) +int save_fpregs_to_fpstate (struct fpu *fpu) {if (likely (use_xsave ())) {xsave_to_kernel (&fpu->state.xsave);
@@ -119,7 +119,7 @@ int copy_fpregs_to_fpstate(struct fpu *freturn 0 ; } -EXPORT_SYMBOL (copy_fpregs_to_fpstate); +EXPORT_SYMBOL (save_fpregs_to_fpstate);void kernel_fpu_begin_mask (unsigned int kfpu_mask) {
@@ -137,7 +137,7 @@ void kernel_fpu_begin_mask(unsigned int* Ignore return value -- we don't care if reg state * is clobbered. */ -copy_fpregs_to_fpstate (¤t->thread.fpu); +save_fpregs_to_fpstate (¤t->thread.fpu); }__cpu_invalidate_fpregs_state ();
@@ -172,7 +172,7 @@ void fpu__save(struct fpu *fpu)trace_x86_fpu_before_save (fpu);if (!test_thread_flag (TIF_NEED_FPU_LOAD )) { -if (!copy_fpregs_to_fpstate (fpu)) { +if (!save_fpregs_to_fpstate (fpu)) {copy_kernel_to_fpregs (&fpu->state); } }
@@ -255,7 +255,7 @@ int fpu__copy(struct task_struct *dst, sif (test_thread_flag (TIF_NEED_FPU_LOAD ))memcpy (&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size); -else if (!copy_fpregs_to_fpstate (dst_fpu)) +else if (!save_fpregs_to_fpstate (dst_fpu))copy_kernel_to_fpregs (&dst_fpu->state);fpregs_unlock ();
arch/x86/kvm/x86.c +1 -1
@@ -9618,7 +9618,7 @@ static void kvm_save_current_fpu(structmemcpy (&fpu->state, ¤t->thread.fpu.state, fpu_kernel_xstate_size); else -copy_fpregs_to_fpstate (fpu ); +save_fpregs_to_fpstate (fpu ); }/* Swap (qemu) user FPU context for the guest FPU context. */
PATCH 19/41 x86/fpu: Rename copy_kernel_to_fpregs() to restore_fpregs_from_kernel()
In reply to Thomas GleixnerThis is not a copy functionality. It restores the register state from the supplied kernel buffer.
--- arch/x86/include/asm/fpu/internal.h | 8 ++++---- arch/x86/kernel/fpu/core.c | 4 ++-- arch/x86/kvm/x86.c | 4 ++-- arch/x86/mm/extable.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/include/asm/fpu/internal.h +4 -4
@@ -397,7 +397,7 @@ static inline int xrstor_from_kernel_errextern int save_fpregs_to_fpstate (struct fpu *fpu); -static inline void __copy_kernel_to_fpregs (union fpregs_state *fpstate,u64 mask) +static inline void __restore_fpregs_from_fpstate (union fpregs_state *fpstate,u64 mask) {if (use_xsave ()) {xrstor_from_kernel (&fpstate->xsave, mask);
@@ -409,7 +409,7 @@ static inline void __copy_kernel_to_fpre } } -static inline void copy_kernel_to_fpregs (union fpregs_state *fpstate) +static inline void restore_fpregs_from_fpstate (union fpregs_state *fpstate) {/* * AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception is
@@ -424,7 +424,7 @@ static inline void copy_kernel_to_fpregs : : [addr] "m" (fpstate)); } -__copy_kernel_to_fpregs (fpstate,-1 ); +__restore_fpregs_from_fpstate (fpstate,-1 ); }extern int copy_fpstate_to_sigframe (void __user *buf,void __user *fp,int size);
@@ -495,7 +495,7 @@ static inline void __fpregs_load_activatreturn ;if (!fpregs_state_valid (fpu, cpu)) { -copy_kernel_to_fpregs (&fpu->state); +restore_fpregs_from_fpstate (&fpu->state);fpregs_activate (fpu); fpu->last_cpu = cpu; }
arch/x86/kernel/fpu/core.c +2 -2
@@ -173,7 +173,7 @@ void fpu__save(struct fpu *fpu)if (!test_thread_flag (TIF_NEED_FPU_LOAD )) {if (!save_fpregs_to_fpstate (fpu)) { -copy_kernel_to_fpregs (&fpu->state); +restore_fpregs_from_fpstate (&fpu->state); } }
@@ -256,7 +256,7 @@ int fpu__copy(struct task_struct *dst, smemcpy (&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size); else if (!save_fpregs_to_fpstate(dst_fpu)) - copy_kernel_to_fpregs(&dst_fpu->state); + restore_fpregs_from_fpstate(&dst_fpu->state);fpregs_unlock ();
arch/x86/kvm/x86.c +2 -2
@@ -9634,7 +9634,7 @@ static void kvm_load_guest_fpu(struct kv*/ if (vcpu->arch.guest_fpu)/* PKRU is separately restored in kvm_x86_ops.run. */ -__copy_kernel_to_fpregs (&vcpu->arch.guest_fpu->state, +__restore_fpregs_from_fpstate (&vcpu->arch.guest_fpu->state, ~XFEATURE_MASK_PKRU );fpregs_mark_activate ();
@@ -9655,7 +9655,7 @@ static void kvm_put_guest_fpu(struct kvmif (vcpu->arch.guest_fpu)kvm_save_current_fpu (vcpu->arch.guest_fpu); -copy_kernel_to_fpregs (&vcpu->arch.user_fpu->state); +restore_fpregs_from_fpstate (&vcpu->arch.user_fpu->state);fpregs_mark_activate ();fpregs_unlock ();
arch/x86/mm/extable.c +1 -1
@@ -65,7 +65,7 @@ EXPORT_SYMBOL_GPL(ex_handler_fault);WARN_ONCE (1 ,"Bad FPU state detected at %pB, reinitializing FPU registers." , (void *)instruction_pointer (regs)); -__copy_kernel_to_fpregs (&init_fpstate,-1 ); +__restore_fpregs_from_fpstate (&init_fpstate,-1 );return true; }EXPORT_SYMBOL_GPL (ex_handler_fprestore);
PATCH 20/41 x86/fpu: Rename initstate copy functions
In reply to Thomas GleixnerAgain this not a copy. It's loading register state.
--- arch/x86/kernel/fpu/core.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/fpu/core.c +3 -3
@@ -355,7 +355,7 @@ void fpu__drop(struct fpu *fpu)* Clear FPU registers by setting them up from the init fpstate. * Caller must do fpregs_[un]lock() around it. */ -static inline void copy_init_fpstate_to_fpregs (u64 features_mask) +static inline void load_fpregs_from_init_fpstate (u64 features_mask) {if (use_xsave ())xrstor_from_kernel (&init_fpstate.xsave, features_mask);
@@ -391,9 +391,9 @@ static void fpu__clear(struct fpu *fpu, xfeatures_mask_supervisor())xrstor_from_kernel (&fpu->state.xsave,xfeatures_mask_supervisor ()); -copy_init_fpstate_to_fpregs (xfeatures_mask_user ()); +load_fpregs_from_init_fpstate (xfeatures_mask_user ()); }else { -copy_init_fpstate_to_fpregs (xfeatures_mask_all); +load_fpregs_from_init_fpstate (xfeatures_mask_all); }fpregs_mark_activate ();
PATCH 21/41 x86/fpu: Rename "dynamic" XSTATEs to "independent"
In reply to Thomas GleixnerFrom: Andy Lutomirski <[email protected]>
The salient feature of "dynamic" XSTATEs is that they are not part of the main task XSTATE buffer. The fact that they are dynamically allocated is irrelevant and will become quite confusing when user math XSTATEs start being dynamically allocated. Rename them to "independent" because they are independent of the main XSTATE code.
This is just a search-and-replace with some whitespace updates to keep things aligned.
--- arch/x86/events/intel/lbr.c | 6 +-- arch/x86/include/asm/fpu/xstate.h | 14 ++++---- arch/x86/kernel/fpu/xstate.c | 62 +++++++++++++++++++------------------- 3 files changed, 41 insertions(+), 41 deletions(-)
arch/x86/events/intel/lbr.c +3 -3
@@ -491,7 +491,7 @@ static void intel_pmu_arch_lbr_xrstors(v {struct x86_perf_task_context_arch_lbr_xsave *task_ctx = ctx; -copy_kernel_to_dynamic_supervisor (&task_ctx->xsave,XFEATURE_MASK_LBR ); +copy_kernel_to_independent_supervisor (&task_ctx->xsave,XFEATURE_MASK_LBR ); } static __always_inline bool lbr_is_reset_in_cstate(void *ctx)
@@ -576,7 +576,7 @@ static void intel_pmu_arch_lbr_xsaves(vo {struct x86_perf_task_context_arch_lbr_xsave *task_ctx = ctx; -copy_dynamic_supervisor_to_kernel (&task_ctx->xsave,XFEATURE_MASK_LBR ); +copy_independent_supervisor_to_kernel (&task_ctx->xsave,XFEATURE_MASK_LBR ); }static void __intel_pmu_lbr_save (void *ctx)
@@ -992,7 +992,7 @@ static void intel_pmu_arch_lbr_read_xsavintel_pmu_store_lbr (cpuc,NULL );return ; } -copy_dynamic_supervisor_to_kernel (&xsave->xsave,XFEATURE_MASK_LBR ); +copy_independent_supervisor_to_kernel (&xsave->xsave,XFEATURE_MASK_LBR );intel_pmu_store_lbr (cpuc, xsave->lbr.entries); }
arch/x86/include/asm/fpu/xstate.h +7 -7
@@ -56,7 +56,7 @@* - Don't set the bit corresponding to the dynamic supervisor feature in * IA32_XSS at run time, since it has been set at boot time. */ -#define XFEATURE_MASK_DYNAMIC (XFEATURE_MASK_LBR) +#define XFEATURE_MASK_INDEPENDENT (XFEATURE_MASK_LBR)/* * Unsupported supervisor features. When a supervisor feature in this mask is
@@ -66,7 +66,7 @@/* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \ - XFEATURE_MASK_DYNAMIC | \ + XFEATURE_MASK_INDEPENDENT | \ XFEATURE_MASK_SUPERVISOR_UNSUPPORTED)#ifdef CONFIG_X86_64
@@ -87,12 +87,12 @@ static inline u64 xfeatures_mask_user(voreturn xfeatures_mask_all &XFEATURE_MASK_USER_SUPPORTED ; } -static inline u64 xfeatures_mask_dynamic (void ) +static inline u64 xfeatures_mask_independent (void ) {if (!boot_cpu_has (X86_FEATURE_ARCH_LBR )) -return XFEATURE_MASK_DYNAMIC & ~XFEATURE_MASK_LBR ; +return XFEATURE_MASK_INDEPENDENT & ~XFEATURE_MASK_LBR ; -return XFEATURE_MASK_DYNAMIC ; +return XFEATURE_MASK_INDEPENDENT ; }extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS ];
@@ -107,7 +107,7 @@ struct membuf;void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave);int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf);int copy_sigframe_from_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf); -void copy_dynamic_supervisor_to_kernel (struct xregs_state *xstate,u64 mask); -void copy_kernel_to_dynamic_supervisor (struct xregs_state *xstate,u64 mask); +void copy_independent_supervisor_to_kernel (struct xregs_state *xstate,u64 mask); +void copy_kernel_to_independent_supervisor (struct xregs_state *xstate,u64 mask);#endif
arch/x86/kernel/fpu/xstate.c +31 -31
152 lines not on this page · raw message
PATCH 22/41 x86/fpu/xstate: Sanitize handling of independent features
In reply to Thomas GleixnerThe copy functions for the independent features are horribly named and the supervisor and independent part is just overengineered.
The point is that the supplied mask has either to be a subset of the independent feature or a subset of the task->fpu.xstate managed features.
Rewrite it so it checks check for invalid overlaps of these areas in the caller supplied feature mask. Rename it so it follows the new naming convention for these operations. Mop up the function documentation.
This allows to use that function for other purposes as well.
--- arch/x86/events/intel/lbr.c | 6 +- arch/x86/include/asm/fpu/xstate.h | 5 +- arch/x86/kernel/fpu/xstate.c | 93 +++++++++++++++++++------------------- 3 files changed, 53 insertions(+), 51 deletions(-)
arch/x86/events/intel/lbr.c +3 -3
@@ -491,7 +491,7 @@ static void intel_pmu_arch_lbr_xrstors(v {struct x86_perf_task_context_arch_lbr_xsave *task_ctx = ctx; -copy_kernel_to_independent_supervisor (&task_ctx->xsave,XFEATURE_MASK_LBR ); +xrstors_from_kernel (&task_ctx->xsave,XFEATURE_MASK_LBR ); } static __always_inline bool lbr_is_reset_in_cstate(void *ctx)
@@ -576,7 +576,7 @@ static void intel_pmu_arch_lbr_xsaves(vo {struct x86_perf_task_context_arch_lbr_xsave *task_ctx = ctx; -copy_independent_supervisor_to_kernel (&task_ctx->xsave,XFEATURE_MASK_LBR ); +xsaves_to_kernel (&task_ctx->xsave,XFEATURE_MASK_LBR ); }static void __intel_pmu_lbr_save (void *ctx)
@@ -992,7 +992,7 @@ static void intel_pmu_arch_lbr_read_xsavintel_pmu_store_lbr (cpuc,NULL );return ; } -copy_independent_supervisor_to_kernel (&xsave->xsave,XFEATURE_MASK_LBR ); +xsaves_to_kernel (&xsave->xsave,XFEATURE_MASK_LBR );intel_pmu_store_lbr (cpuc, xsave->lbr.entries); }
arch/x86/include/asm/fpu/xstate.h +3 -2
@@ -107,7 +107,8 @@ struct membuf;void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave);int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf);int copy_sigframe_from_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf); -void copy_independent_supervisor_to_kernel (struct xregs_state *xstate,u64 mask); -void copy_kernel_to_independent_supervisor (struct xregs_state *xstate,u64 mask); + +void xsaves_to_kernel (struct xregs_state *xsave,u64 mask); +void xrstors_from_kernel (struct xregs_state *xsave,u64 mask);#endif
arch/x86/kernel/fpu/xstate.c +47 -46
122 lines not on this page · raw message
PATCH 23/41 x86/pkeys: Move read_pkru() and write_pkru()
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
write_pkru() was originally used just to write to the PKRU register. It was mercifully short and sweet and was not out of place in pgtable.h with some other pkey-related code.
But, later work included a requirement to also modify the task XSAVE buffer when updating the register. This really is more related to the XSAVE architecture than to paging.
Move the read/write_pkru() to asm/pkru.h. pgtable.h won't miss them.
---
V2: Adapted it to the latest FPU changes, move them to a seperate file
along with the constants to avoid header hell later on.
---
arch/x86/include/asm/fpu/xstate.h | 1 arch/x86/include/asm/pgtable.h | 57 ----------------------------------- arch/x86/include/asm/pkru.h | 61 ++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/process_64.c | 1 arch/x86/kvm/svm/sev.c | 1 arch/x86/kvm/x86.c | 1 arch/x86/mm/pkeys.c | 1 7 files changed, 67 insertions(+), 56 deletions(-)
arch/x86/include/asm/fpu/xstate.h +1 -0
@@ -6,6 +6,7 @@#include <linux/types.h> #include <asm/processor.h> +#include <asm/fpu/api.h> #include <asm/user.h> /* Bit 63 of XCR0 is reserved for future expansion */
arch/x86/include/asm/pgtable.h +1 -56
75 lines not on this page · raw message
arch/x86/include/asm/pkru.h new file +61 -0
61 lines not on this page · raw message
arch/x86/kernel/process_64.c +1 -0
@@ -41,6 +41,7 @@#include <linux/syscalls.h> #include <asm/processor.h> +#include <asm/pkru.h> #include <asm/fpu/internal.h> #include <asm/mmu_context.h> #include <asm/prctl.h>
arch/x86/kvm/svm/sev.c +1 -0
@@ -19,6 +19,7 @@#include <linux/trace_events.h> #include <asm/fpu/internal.h> +#include <asm/pkru.h> #include <asm/trapnr.h> #include "x86.h"
arch/x86/kvm/x86.c +1 -0
@@ -65,6 +65,7 @@#include <asm/msr.h> #include <asm/desc.h> #include <asm/mce.h> +#include <asm/pkru.h> #include <linux/kernel_stat.h> #include <asm/fpu/internal.h> /* Ugh! */ #include <asm/pvclock.h>
arch/x86/mm/pkeys.c +1 -0
@@ -10,6 +10,7 @@#include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/mmu_context.h> /* vma_pkey() */ +#include <asm/pkru.h> /* read/write_pkru() */ int __execute_only_pkey (struct mm_struct *mm) {
PATCH 24/41 x86/fpu: Differentiate "copy" versus "move" of fpregs
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
There are three ways in the ISA to bulk save FPU state:
1. XSAVE, every CPU newer than 2008 does this
2. FXSAVE, from ~2000->2007
3. FNSAVE, pre-2000
XSAVE and FXSAVE are nice. They just copy FPU state to memory. FNSAVE is nasty; it destroys the FPU state when it writes it to memory. It is more of a "move".
Currently, copy_fpregs_to_fpstate() returns a number to its caller to say whether it used the nice, non-destructive XSAVE/FXSAVE or used the mean, clobbering FNSAVE. Some sites need special handling for the FNSAVE case to restore any FNSAVE-clobbered state. Others don't care, like when they are about to load new state anyway.
The nasty part about the copy_fpregs_to_fpstate() interface is that it's hard to tell if callers expect the "move" or the "copy" behavior.
Create a new, explicit "move" interface for callers that can handle clobbering register state. Make "copy" only do copies and never clobber register state.
== switch_fpu_prepare() optimization ==
switch_fpu_prepare() had a nice optimization for the FNSAVE case. It can handle either clobbering or preserving register state. For the XSAVE/FXSAVE case, it records that the fpregs state is still loaded, just in case a later "restore" operation can be elided. For the FNSAVE case, it marks the fpregs as not loaded on the CPU, since they were clobbered.
Instead of having switch_fpu_prepare() modify its behavior based on whether registers were clobbered or not, simply switch its behavior based on whether FNSAVE is in use. This makes it much more clear what is going on and what the common path is.
It would be simpler to just remove this FNSAVE optimization: Always save and restore in the FNSAVE case. This may incur the cost of the restore even in cases where the restored state is never used. But, it would only hurt painfully ancient (>20 years old) processors.
---
V4: Picked it up from Dave's series and adopted it to the other changes.
---
arch/x86/include/asm/fpu/internal.h | 14 ++++-- arch/x86/kernel/fpu/core.c | 84 +++++++++++++++++++++--------------- 2 files changed, 59 insertions(+), 39 deletions(-)
arch/x86/include/asm/fpu/internal.h +10 -4
@@ -395,7 +395,8 @@ static inline int xrstor_from_kernel_errreturn err; } -extern int save_fpregs_to_fpstate (struct fpu *fpu); +extern void save_fpregs_to_fpstate (struct fpu *fpu); +extern void copy_fpregs_to_fpstate (struct fpu *fpu);static inline void __restore_fpregs_from_fpstate (union fpregs_state *fpstate,u64 mask) {
@@ -527,10 +528,15 @@ static inline void __fpregs_load_activatstatic inline void switch_fpu_prepare (struct fpu *old_fpu,int cpu) {if (static_cpu_has (X86_FEATURE_FPU ) && !(current->flags &PF_KTHREAD )) { -if (!save_fpregs_to_fpstate (old_fpu)) - old_fpu->last_cpu =-1 ; -else +/* +* Avoid the expense of restoring fpregs with FNSAVE when it +* might be unnecssary. XSAVE and FXSAVE preserve the FPU state. +*/ +save_fpregs_to_fpstate (old_fpu); +if (likely (use_xsave () ||use_fxsr ())) old_fpu->last_cpu = cpu; +else + old_fpu->last_cpu =-1 ;/* But leave fpu_fpregs_owner_ctx! */ trace_x86_fpu_regs_deactivated (old_fpu);
arch/x86/kernel/fpu/core.c +49 -35
134 lines not on this page · raw message
PATCH 25/41 x86/cpu: Sanitize X86_FEATURE_OSPKE
In reply to Thomas GleixnerX86_FEATURE_OSPKE is enabled first on the boot CPU and the feature flag is set. Secondary CPUs have to enable CR4.PKE as well and set their per CPU feature flag. That's ineffective because all call sites have checks for boot_cpu_data.
Make it smarter and force the feature flag when PKU is enabled on the boot cpu which allows then to use cpu_feature_enabled(X86_FEATURE_OSPKE) all over the place. That either compiles the code out when PKEY support is disabled in Kconfig or uses a static_cpu_has() for the feature check which makes a significant difference in hotpathes, e.g. context switch.
---
V2: New patch
--- arch/x86/include/asm/pkeys.h | 8 ++++---- arch/x86/include/asm/pkru.h | 4 ++-- arch/x86/kernel/cpu/common.c | 24 +++++++++++------------- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/kernel/fpu/xstate.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/mm/fault.c | 2 +- 7 files changed, 21 insertions(+), 23 deletions(-)
arch/x86/include/asm/pkeys.h +4 -4
@@ -9,14 +9,14 @@* will be necessary to ensure that the types that store key * numbers and masks have sufficient capacity. */ -#define arch_max_pkey () (boot_cpu_has(X86_FEATURE_OSPKE) ? 16 : 1) +#define arch_max_pkey () (cpu_feature_enabled(X86_FEATURE_OSPKE) ? 16 : 1)extern int arch_set_user_pkey_access (struct task_struct *tsk,int pkey,unsigned long init_val);static inline bool arch_pkeys_enabled (void ) { -return boot_cpu_has (X86_FEATURE_OSPKE ); +return cpu_feature_enabled (X86_FEATURE_OSPKE ); }/*
@@ -26,7 +26,7 @@ static inline bool arch_pkeys_enabled(voextern int __execute_only_pkey (struct mm_struct *mm);static inline int execute_only_pkey (struct mm_struct *mm) { -if (!boot_cpu_has (X86_FEATURE_OSPKE )) +if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return ARCH_DEFAULT_PKEY ;return __execute_only_pkey (mm);
@@ -37,7 +37,7 @@ extern int __arch_override_mprotect_pkeystatic inline int arch_override_mprotect_pkey (struct vm_area_struct *vma,int prot,int pkey) { -if (!boot_cpu_has (X86_FEATURE_OSPKE )) +if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return 0 ;return __arch_override_mprotect_pkey (vma, prot, pkey);
arch/x86/include/asm/pkru.h +2 -2
@@ -32,7 +32,7 @@ static inline bool __pkru_allows_write(ustatic inline u32 read_pkru (void ) { -if (boot_cpu_has (X86_FEATURE_OSPKE )) +if (cpu_feature_enabled (X86_FEATURE_OSPKE ))return rdpkru ();return 0 ; }
@@ -41,7 +41,7 @@ static inline void write_pkru(u32 pkru) {struct pkru_state *pk; -if (!boot_cpu_has (X86_FEATURE_OSPKE )) +if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return ; pk =get_xsave_addr (¤t->thread.fpu.state.xsave,XFEATURE_PKRU );
arch/x86/kernel/cpu/common.c +11 -13
@@ -465,22 +465,20 @@ static bool pku_disabled; static __always_inline void setup_pku(struct cpuinfo_x86 *c) { -/* check the boot processor, plus compile options for PKU: */ -if (!cpu_feature_enabled (X86_FEATURE_PKU )) -return ; -/* checks the actual processor's cpuid bits: */ -if (!cpu_has (c,X86_FEATURE_PKU )) -return ; -if (pku_disabled) +if (c == &boot_cpu_data) { +if (pku_disabled || !cpu_feature_enabled (X86_FEATURE_PKU )) +return ; +/* +* Setting CR4.PKE will cause the X86_FEATURE_OSPKE cpuid +* bit to be set. Enforce it. +*/ +setup_force_cpu_cap (X86_FEATURE_OSPKE ); + + }else if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) {return ; + }cr4_set_bits (X86_CR4_PKE ); -/* -* Setting X86_CR4_PKE will cause the X86_FEATURE_OSPKE -* cpuid bit to be set. We need to ensure that we -* update that bit in this CPU's "cpu_info". -*/ -set_cpu_cap (c,X86_FEATURE_OSPKE ); }#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
arch/x86/kernel/fpu/core.c +1 -1
@@ -378,7 +378,7 @@ static inline void copy_init_fpstate_to_else frstor_from_kernel (&init_fpstate.fsave); -if (boot_cpu_has (X86_FEATURE_OSPKE )) +if (cpu_feature_enabled (X86_FEATURE_OSPKE ))copy_init_pkru_to_fpregs (); }
arch/x86/kernel/fpu/xstate.c +1 -1
@@ -896,7 +896,7 @@ int arch_set_user_pkey_access(struct tas* This check implies XSAVE support. OSPKE only gets * set if we enable XSAVE and we enable PKU in XCR0. */ -if (!boot_cpu_has (X86_FEATURE_OSPKE )) +if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return -EINVAL ;/*
arch/x86/kernel/process_64.c +1 -1
@@ -137,7 +137,7 @@ void __show_regs(struct pt_regs *regs, e log_lvl, d3, d6, d7); } -if (boot_cpu_has (X86_FEATURE_OSPKE )) +if (cpu_feature_enabled (X86_FEATURE_OSPKE ))printk ("%sPKRU: %08x\n" , log_lvl,read_pkru ()); }
arch/x86/mm/fault.c +1 -1
@@ -875,7 +875,7 @@ static inline bool bad_area_access_from_/* This code is always called on the current mm */ bool foreign = false; -if (!boot_cpu_has (X86_FEATURE_OSPKE )) +if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return false;if (error_code &X86_PF_PK )return true;
PATCH 26/41 x86/pkru: Provide pkru_get_init_value()
In reply to Thomas GleixnerWhen CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS is disabled then the following code fails to compile:
if (cpu_feature_enabled(X86_FEATURE_OSPKE)) {
u32 pkru = READ_ONCE(init_pkru_value);
..
}because init_pkru_value is defined as '0' which makes READ_ONCE() upset.
Provide an accessor macro to avoid #ifdeffery all over the place.
---
V2: New patch
--- arch/x86/include/asm/pkru.h | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/include/asm/pkru.h +2 -0
@@ -10,8 +10,10 @@#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS extern u32 init_pkru_value; +#define pkru_get_init_value () READ_ONCE(init_pkru_value)#else #define init_pkru_value 0 +#define pkru_get_init_value () 0#endif static inline bool __pkru_allows_read (u32 pkru,u16 pkey)
PATCH 27/41 x86/pkru: Provide pkru_write_default()
In reply to Thomas GleixnerProvide a simple and trivial helper which just writes the PKRU default value without trying to fiddle with the tasks xsave buffer.
---
V4: New patch
--- arch/x86/include/asm/pkru.h | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/include/asm/pkru.h +8 -0
@@ -60,4 +60,12 @@ static inline void write_pkru(u32 pkru) fpregs_unlock(); } +static inline void pkru_write_default (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) +return ; + +wrpkru (pkru_get_init_value ()); +} +#endif
PATCH 28/41 x86/cpu: Write the default PKRU value when enabling PKE
In reply to Thomas GleixnerIn preparation of making the PKRU management more independent from XSTATES, write the default PKRU value into the hardware right after enabling PKRU in CR4. This ensures that switch_to() and copy_thread() have the correct setting for init task and the per CPU idle threads right away.
--- arch/x86/kernel/cpu/common.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/cpu/common.c +2 -0
@@ -479,6 +479,8 @@ static __always_inline void setup_pku(st }cr4_set_bits (X86_CR4_PKE ); +/* Load the default PKRU value */ +pkru_write_default (); }#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
PATCH 29/41 x86/fpu: Use pkru_write_default() in copy_init_fpstate_to_fpregs()
In reply to Thomas GleixnerThere is no point in using copy_init_pkru_to_fpregs() which in turn calls write_pkru(). write_pkru() tries to fiddle with the task's xstate buffer for nothing because the XRSTOR[S](init_fpstate) just cleared the xfeature flag in the xstate header which makes get_xsave_addr() fail.
It's a useless exercise anyway because the reinitialization activates the FPU so before the task's xstate buffer can be used again a XRSTOR[S] must happen which in turn dumps the PKRU value.
Get rid of the now unused copy_init_pkru_to_fpregs().
---
V4: New patch
--- arch/x86/include/asm/pkeys.h | 1 - arch/x86/kernel/fpu/core.c | 3 +-- arch/x86/mm/pkeys.c | 17 ----------------- include/linux/pkeys.h | 4 ---- 4 files changed, 1 insertion(+), 24 deletions(-)
arch/x86/include/asm/pkeys.h +0 -1
@@ -124,7 +124,6 @@ extern int arch_set_user_pkey_access(str unsigned long init_val);extern int __arch_set_user_pkey_access (struct task_struct *tsk,int pkey,unsigned long init_val); -extern void copy_init_pkru_to_fpregs (void );static inline int vma_pkey (struct vm_area_struct *vma) {
arch/x86/kernel/fpu/core.c +1 -2
@@ -378,8 +378,7 @@ static inline void copy_init_fpstate_to_else frstor_from_kernel (&init_fpstate.fsave); -if (cpu_feature_enabled (X86_FEATURE_OSPKE )) -copy_init_pkru_to_fpregs (); +pkru_write_default (); }/*
arch/x86/mm/pkeys.c +0 -17
@@ -10,7 +10,6 @@#include <asm/cpufeature.h> /* boot_cpu_has, ... */ #include <asm/mmu_context.h> /* vma_pkey() */ -#include <asm/pkru.h> /* read/write_pkru() */ int __execute_only_pkey (struct mm_struct *mm) {
@@ -125,22 +124,6 @@ u32 init_pkru_value = PKRU_AD_KEY( 1) |PKRU_AD_KEY (10 ) |PKRU_AD_KEY (11 ) |PKRU_AD_KEY (12 ) |PKRU_AD_KEY (13 ) |PKRU_AD_KEY (14 ) |PKRU_AD_KEY (15 ); -/* -* Called from the FPU code when creating a fresh set of FPU -* registers. This is called from a very specific context where -* we know the FPU registers are safe for use and we can use PKRU -* directly. -*/ -void copy_init_pkru_to_fpregs (void ) -{ -u32 init_pkru_value_snapshot =READ_ONCE (init_pkru_value); -/* -* Override the PKRU state that came from 'init_fpstate' -* with the baseline from the process. -*/ -write_pkru (init_pkru_value_snapshot); -} -static ssize_t init_pkru_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos) {
include/linux/pkeys.h +0 -4
@@ -44,10 +44,6 @@ static inline bool arch_pkeys_enabled(voreturn false; } -static inline void copy_init_pkru_to_fpregs (void ) -{ -} -#endif /* ! CONFIG_ARCH_HAS_PKEYS */ #endif /* _LINUX_PKEYS_H */
PATCH 30/41 x86/fpu: Rename fpu__clear_all() to fpu_flush_thread()
In reply to Thomas GleixnerMake it clear what the function is about.
---
V2: New patch
--- arch/x86/include/asm/fpu/internal.h | 3 ++- arch/x86/kernel/fpu/core.c | 4 ++-- arch/x86/kernel/process.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-)
arch/x86/include/asm/fpu/internal.h +2 -1
@@ -33,9 +33,10 @@ extern int fpu__restore_sig(void __userextern void fpu__drop (struct fpu *fpu);extern int fpu__copy (struct task_struct *dst,struct task_struct *src);extern void fpu__clear_user_states (struct fpu *fpu); -extern void fpu__clear_all (struct fpu *fpu);extern int fpu__exception_code (struct fpu *fpu,int trap_nr); +extern void fpu_flush_thread (void ); +/* * Boot time FPU initialization functions: */
arch/x86/kernel/fpu/core.c +2 -2
@@ -407,9 +407,9 @@ void fpu__clear_user_states(struct fpu *fpu__clear (fpu, true); } -void fpu__clear_all (struct fpu *fpu) +void fpu_flush_thread (void ) { -fpu__clear (fpu, false); +fpu__clear (¤t->thread.fpu, false); }/*
arch/x86/kernel/process.c +1 -1
@@ -206,7 +206,7 @@ void flush_thread(void)flush_ptrace_hw_breakpoint (tsk);memset (tsk->thread.tls_array,0 ,sizeof (tsk->thread.tls_array)); -fpu__clear_all (&tsk->thread.fpu); +fpu_flush_thread (); }void disable_TSC (void )
PATCH 31/41 x86/fpu: Clean up the fpu__clear() variants
In reply to Thomas GleixnerFrom: Andy Lutomirski <[email protected]>
fpu__clear() currently resets both register state and kernel XSAVE buffer state. It has two modes: one for all state (supervisor and user) and another for user state only. fpu__clear_all() uses the "all state" (user_only=0) mode, while a number of signal paths use the user_only=1 mode.
Make fpu__clear() work only for user state (user_only=1) and remove the "all state" (user_only=0) code. Rename it to match so it can be used by the signal paths.
Replace the "all state" (user_only=0) fpu__clear() functionality. Use the TIF_NEED_FPU_LOAD functionality instead of making any actual hardware registers changes in this path.
Instead of invoking fpu__initialize() just memcpy() init_fpstate into the tasks FPU state because that has already the correct format and in case of PKRU also contains the default PKRU value. Move the actual PKRU write out into flush_thread() where it belongs and where it will end up anyway when PKRU and XSTATE have been distangled.
For bisectability a workaround is required which stores the PKRU value in the xstate memory until PKRU is distangled from XSTATE for context switching and return to user.
[ Dave Hansen: Polished changelog ]
[ tglx: Fixed the PKRU fallout ]
---
V2: Fixed PKRU fallout reported by Yu-Chen
V4: Minor adjustment to PKRU handling
--- arch/x86/kernel/fpu/core.c | 113 ++++++++++++++++++++++++++++++--------------- arch/x86/kernel/process.c | 10 +++ 2 files changed, 87 insertions(+), 36 deletions(-)
arch/x86/kernel/fpu/core.c +77 -36
@@ -283,19 +283,6 @@ int fpu__copy(struct task_struct *dst, s }/* -* Activate the current task's in-memory FPU context, -* if it has not been used before: -*/ -static void fpu__initialize (struct fpu *fpu) -{ -WARN_ON_FPU (fpu != ¤t->thread.fpu); - -set_thread_flag (TIF_NEED_FPU_LOAD ); -fpstate_init (&fpu->state); -trace_x86_fpu_init_state (fpu); -} - -/* * This function must be called before we read a task's fpstate. * * There's two cases where this gets called:
@@ -381,46 +368,100 @@ static inline void load_fpregs_from_initpkru_write_default (); } +static inline unsigned int init_fpstate_copy_size (void ) +{ +if (!use_xsave ()) +return fpu_kernel_xstate_size; + +/* XSAVE(S) just needs the legacy and the xstate header part */ +return sizeof (init_fpstate.xsave); +} + +/* Temporary workaround. Will be removed once PKRU and XSTATE are distangled. */ +static inline void pkru_set_default_in_xstate (struct xregs_state *xsave) +{ +struct pkru_state *pk; + +if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) +return ; +/* +* Force XFEATURE_PKRU to be set in the header otherwise +* get_xsave_addr() does not work and it also needs to be set to +* make XRSTOR(S) load it. +*/ + xsave->header.xfeatures |=XFEATURE_MASK_PKRU ; + pk =get_xsave_addr (xsave,XFEATURE_PKRU ); + pk->pkru =pkru_get_init_value (); +} +/* -* Clear the FPU state back to init state. -* -* Called by sys_execve(), by the signal handler code and by various -* error paths. +* Reset current->fpu memory state to the init values. */ -static void fpu__clear (struct fpu *fpu,bool user_only) +static void fpu_reset_fpstate (void ) +{ +struct fpu *fpu= ¤t->thread.fpu; + +fpregs_lock (); +fpu__drop (fpu); +/* +* This does not change the actual hardware registers. It just +* resets the memory image and sets TIF_NEED_FPU_LOAD so a +* subsequent return to usermode will reload the registers from the +* tasks memory image. +* +* Do not use fpstate_init() here. Just copy init_fpstate which has +* the correct content already except for PKRU. +*/ +memcpy (&fpu->state, &init_fpstate,init_fpstate_copy_size ()); +pkru_set_default_in_xstate (&fpu->state.xsave); +set_thread_flag (TIF_NEED_FPU_LOAD ); +fpregs_unlock (); +} + +/* +* Reset current's user FPU states to the init states. current's +* supervisor states, if any, are not modified by this function. The +* caller guarantees that the XSTATE header in memory is intact. +*/ +void fpu__clear_user_states (struct fpu *fpu) {WARN_ON_FPU (fpu != ¤t->thread.fpu); +fpregs_lock ();if (!static_cpu_has (X86_FEATURE_FPU )) { -fpu__drop (fpu); -fpu__initialize (fpu); +fpu_reset_fpstate (); +fpregs_unlock ();return ; } -fpregs_lock (); - -if (user_only) { -if (!fpregs_state_valid (fpu,smp_processor_id ()) && -xfeatures_mask_supervisor ()) -xrstor_from_kernel (&fpu->state.xsave, -xfeatures_mask_supervisor ()); -load_fpregs_from_init_fpstate (xfeatures_mask_user ()); - }else { -load_fpregs_from_init_fpstate (xfeatures_mask_all); +/* +* Ensure that current's supervisor states are loaded into their +* corresponding registers. +*/ +if (xfeatures_mask_supervisor () && + !fpregs_state_valid (fpu,smp_processor_id ())) { +xrstor_from_kernel (&fpu->state.xsave, +xfeatures_mask_supervisor ()); } +/* Reset user states in registers. */ +load_fpregs_from_init_fpstate (xfeatures_mask_user ()); + +/* +* Now all FPU registers have their desired values. Inform the FPU +* state machine that current's FPU registers are in the hardware +* registers. The memory image does not need to be updated because +* any operation relying on it has to save the registers first when +* currents FPU is marked active. +*/ fpregs_mark_activate (); -fpregs_unlock (); -} -void fpu__clear_user_states (struct fpu *fpu) -{ -fpu__clear (fpu, true); +fpregs_unlock (); }void fpu_flush_thread (void ) { -fpu__clear (¤t->thread.fpu, false); +fpu_reset_fpstate (); }/*
arch/x86/kernel/process.c +10 -0
@@ -199,6 +199,15 @@ int copy_thread(unsigned long clone_flagreturn ret; } +static void pkru_flush_thread (void ) +{ +/* +* If PKRU is enabled the default PKRU value has to be loaded into +* the hardware right here (similar to context switch). +*/ +pkru_write_default (); +} +void flush_thread (void ) {struct task_struct *tsk = current;
@@ -207,6 +216,7 @@ void flush_thread(void)memset (tsk->thread.tls_array,0 ,sizeof (tsk->thread.tls_array));fpu_flush_thread (); +pkru_flush_thread (); }void disable_TSC (void )
PATCH 32/41 x86/fpu: Rename __fpregs_load_activate() to fpregs_restore_userregs()
In reply to Thomas GleixnerRename it so that it becomes entirely clear what this function is about. It's purpose is to restore the FPU registers to the state which was saved in the task's FPU memory state either at context switch or by an in kernel FPU user.
---
V4: New patch
--- arch/x86/include/asm/fpu/internal.h | 6 ++---- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/kernel/fpu/signal.c | 2 +- 3 files changed, 4 insertions(+), 6 deletions(-)
arch/x86/include/asm/fpu/internal.h +2 -4
@@ -485,10 +485,8 @@ static inline void fpregs_activate(structrace_x86_fpu_regs_activated (fpu); } -/* -* Internal helper, do not use directly. Use switch_fpu_return() instead. -*/ -static inline void __fpregs_load_activate (void ) +/* Internal helper for switch_fpu_return() and signal frame setup */ +static inline void fpregs_restore_userregs (void ) {struct fpu *fpu = ¤t->thread.fpu;int cpu =smp_processor_id ();
arch/x86/kernel/fpu/core.c +1 -1
@@ -472,7 +472,7 @@ void switch_fpu_return(void)if (!static_cpu_has (X86_FEATURE_FPU ))return ; -__fpregs_load_activate (); +fpregs_restore_userregs (); }EXPORT_SYMBOL_GPL (switch_fpu_return);
arch/x86/kernel/fpu/signal.c +1 -1
@@ -188,7 +188,7 @@ int copy_fpstate_to_sigframe(void __user*/ fpregs_lock ();if (test_thread_flag (TIF_NEED_FPU_LOAD )) -__fpregs_load_activate (); +fpregs_restore_userregs ();pagefault_disable (); ret =copy_fpregs_to_sigframe (buf_fx);
PATCH 33/41 x86/fpu: Move FXSAVE_LEAK quirk info __copy_kernel_to_fpregs()
In reply to Thomas Gleixnercopy_kernel_to_fpregs() restores all xfeatures but it is also the place where the AMD FXSAVE_LEAK bug is handled.
That prevents fpregs_restore_userregs() to limit the restored features, which is required to distangle PKRU and XSTATE handling and also for the upcoming supervisor state management.
Move the FXSAVE_LEAK quirk into __copy_kernel_to_fpregs() and deinline that function which has become rather fat.
---
V4: New patch
--- arch/x86/include/asm/fpu/internal.h | 25 +------------------------ arch/x86/kernel/fpu/core.c | 26 ++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 24 deletions(-)
arch/x86/include/asm/fpu/internal.h +1 -24
@@ -399,33 +399,10 @@ static inline int xrstor_from_kernel_errextern void save_fpregs_to_fpstate (struct fpu *fpu);extern void copy_fpregs_to_fpstate (struct fpu *fpu); -static inline void __restore_fpregs_from_fpstate (union fpregs_state *fpstate,u64 mask) -{ -if (use_xsave ()) { -xrstor_from_kernel (&fpstate->xsave, mask); - }else { -if (use_fxsr ()) -fxrstor_from_kernel (&fpstate->fxsave); -else -frstor_from_kernel (&fpstate->fsave); - } -} +extern void __restore_fpregs_from_fpstate (union fpregs_state *fpstate,u64 mask);static inline void restore_fpregs_from_fpstate (union fpregs_state *fpstate) { -/* -* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception is -* pending. Clear the x87 state here by setting it to fixed values. -* "m" is a random variable that should be in L1. -*/ -if (unlikely (static_cpu_has_bug (X86_BUG_FXSAVE_LEAK ))) { - asmvolatile ( -"fnclex\n\t" -"emms\n\t" -"fildl %P[addr]" /* set F?P to defined value */ - : : [addr]"m" (fpstate)); - } -__restore_fpregs_from_fpstate (fpstate,-1 ); }
arch/x86/kernel/fpu/core.c +26 -0
@@ -145,6 +145,32 @@ void copy_fpregs_to_fpstate(struct fpu *restore_fpregs_from_fpstate (&fpu->state); } +void __restore_fpregs_from_fpstate (union fpregs_state *fpstate,u64 mask) +{ +/* +* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception is +* pending. Clear the x87 state here by setting it to fixed values. +* "m" is a random variable that should be in L1. +*/ +if (unlikely (static_cpu_has_bug (X86_BUG_FXSAVE_LEAK ))) { + asmvolatile ( +"fnclex\n\t" +"emms\n\t" +"fildl %P[addr]" /* set F?P to defined value */ + : : [addr]"m" (fpstate)); + } + +if (use_xsave ()) { +xrstor_from_kernel (&fpstate->xsave, mask); + }else { +if (use_fxsr ()) +fxrstor_from_kernel (&fpstate->fxsave); +else +frstor_from_kernel (&fpstate->fsave); + } +} +EXPORT_SYMBOL_GPL (__restore_fpregs_from_fpstate); +void kernel_fpu_begin_mask (unsigned int kfpu_mask) {preempt_disable ();
PATCH 34/41 x86/fpu: Rename xfeatures_mask_user() to xfeatures_mask_uabi()
In reply to Thomas GleixnerRename it so it's clear that this is about user ABI features which can differ from the feature set which the kernel saves and restores because the kernel handles e.g. PKRU differently. But the user ABI (ptrace, signal frame) expects it to be there.
---
V4: New patch
--- arch/x86/include/asm/fpu/internal.h | 7 ++++++- arch/x86/include/asm/fpu/xstate.h | 6 +++++- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/kernel/fpu/signal.c | 10 +++++----- arch/x86/kernel/fpu/xstate.c | 12 ++++++------ 5 files changed, 23 insertions(+), 14 deletions(-)
arch/x86/include/asm/fpu/internal.h +6 -1
@@ -341,7 +341,12 @@ static inline void xrstor_from_kernel(st*/ static inline int xsave_to_user_sigframe (struct xregs_state __user *buf) { -u64 mask =xfeatures_mask_user (); +/* +* Include the features which are not xsaved/rstored by the kernel +* internally, e.g. PKRU. That's user space ABI and also required +* to allow the signal handler to modify PKRU. +*/ +u64 mask =xfeatures_mask_uabi ();u32 lmask = mask;u32 hmask = mask >>32 ;int err;
arch/x86/include/asm/fpu/xstate.h +5 -1
@@ -83,7 +83,11 @@ static inline u64 xfeatures_mask_supervireturn xfeatures_mask_all &XFEATURE_MASK_SUPERVISOR_SUPPORTED ; } -static inline u64 xfeatures_mask_user (void ) +/* +* The xfeatures which are enabled in XCR0 and expected to be in ptrace +* buffers and signal frames. +*/ +static inline u64 xfeatures_mask_uabi (void ) {return xfeatures_mask_all &XFEATURE_MASK_USER_SUPPORTED ; }
arch/x86/kernel/fpu/core.c +1 -1
@@ -471,7 +471,7 @@ void fpu__clear_user_states(struct fpu * }/* Reset user states in registers. */ -load_fpregs_from_init_fpstate (xfeatures_mask_user ()); +load_fpregs_from_init_fpstate (xfeatures_mask_uabi ());/* * Now all FPU registers have their desired values. Inform the FPU
arch/x86/kernel/fpu/signal.c +5 -5
@@ -267,14 +267,14 @@ static int copy_user_to_fpregs_zeroing(vif (use_xsave ()) {if (fx_only) { - init_bv =xfeatures_mask_user () & ~XFEATURE_MASK_FPSSE ; + init_bv =xfeatures_mask_uabi () & ~XFEATURE_MASK_FPSSE ; r =fxrstor_from_user_sigframe (buf);if (!r)xrstor_from_kernel (&init_fpstate.xsave, init_bv);return r; }else { - init_bv =xfeatures_mask_user () & ~xbv; + init_bv =xfeatures_mask_uabi () & ~xbv; r =xrstor_from_user_sigframe (buf, xbv);if (!r &&unlikely (init_bv))
@@ -429,7 +429,7 @@ static int __fpu__restore_sig(void __usefpregs_unlock ();if (use_xsave () && !fx_only) { -u64 init_bv =xfeatures_mask_user () & ~user_xfeatures; +u64 init_bv =xfeatures_mask_uabi () & ~user_xfeatures; ret =copy_sigframe_from_user_to_xstate (&fpu->state.xsave, buf_fx);if (ret)
@@ -463,7 +463,7 @@ static int __fpu__restore_sig(void __useif (use_xsave ()) {u64 init_bv; - init_bv =xfeatures_mask_user () & ~XFEATURE_MASK_FPSSE ; + init_bv =xfeatures_mask_uabi () & ~XFEATURE_MASK_FPSSE ;xrstor_from_kernel (&init_fpstate.xsave, init_bv); }
@@ -558,7 +558,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 =FP_XSTATE_MAGIC1 ; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures =xfeatures_mask_user (); + fx_sw_reserved.xfeatures =xfeatures_mask_uabi (); fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) ||
arch/x86/kernel/fpu/xstate.c +6 -6
@@ -155,7 +155,7 @@ void fpu__init_cpu_xstate(void)* managed by XSAVE{C, OPT, S} and XRSTOR{S}. Only XSAVE user * states can be set here. */ -xsetbv (XCR_XFEATURE_ENABLED_MASK ,xfeatures_mask_user ()); +xsetbv (XCR_XFEATURE_ENABLED_MASK ,xfeatures_mask_uabi ());/* * MSR_IA32_XSS sets supervisor states managed by XSAVES.
@@ -443,7 +443,7 @@ int using_compacted_format(void)static int validate_user_xstate_header (const struct xstate_header *hdr) {/* No unknown or supervisor features may be set */ -if (hdr->xfeatures & ~xfeatures_mask_user ()) +if (hdr->xfeatures & ~xfeatures_mask_uabi ())return -EINVAL ;/* Userspace must use the uncompacted format */
@@ -745,7 +745,7 @@ void __init fpu__init_system_xstate(voidcpuid_count (XSTATE_CPUID ,1 , &eax, &ebx, &ecx, &edx); xfeatures_mask_all |= ecx + ((u64 )edx <<32 ); -if ((xfeatures_mask_user () &XFEATURE_MASK_FPSSE ) !=XFEATURE_MASK_FPSSE ) { +if ((xfeatures_mask_uabi () &XFEATURE_MASK_FPSSE ) !=XFEATURE_MASK_FPSSE ) {/* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue
@@ -776,7 +776,7 @@ void __init fpu__init_system_xstate(void* Update info used for ptrace frames; use standard-format size and no * supervisor xstates: */ -update_regset_xstate_info (fpu_user_xstate_size,xfeatures_mask_user ()); +update_regset_xstate_info (fpu_user_xstate_size,xfeatures_mask_uabi ());fpu__init_prepare_fx_sw_frame ();setup_init_fpu_buf ();
@@ -804,7 +804,7 @@ void fpu__resume_cpu(void)* Restore XCR0 on xsave capable CPUs: */ if (boot_cpu_has (X86_FEATURE_XSAVE )) -xsetbv (XCR_XFEATURE_ENABLED_MASK ,xfeatures_mask_user ()); +xsetbv (XCR_XFEATURE_ENABLED_MASK ,xfeatures_mask_uabi ());/* * Restore IA32_XSS. The same CPUID bit enumerates support
@@ -981,7 +981,7 @@ void copy_uabi_xstate_to_membuf(struct m*/ memset (&header,0 ,sizeof (header)); header.xfeatures = xsave->header.xfeatures; - header.xfeatures &=xfeatures_mask_user (); + header.xfeatures &=xfeatures_mask_uabi ();if (header.xfeatures &XFEATURE_MASK_FP )copy_part (&to, &last,0 , off_mxcsr, &xsave->i387);
PATCH 35/41 x86/fpu: Dont restore PKRU in fpregs_restore_userspace()
In reply to Thomas Gleixnerswitch_to(), flush_thread() write the task's PKRU value eagerly so the PKRU value of current is always valid in the hardware.
That means there is no point in restoring PKRU on exit to user or when reactivating the task's FPU registers in the signal frame setup path.
This allows to remove all the xstate buffer updates with PKRU values once the PKRU state is stored in thread struct while a task is scheduled out.
--- arch/x86/include/asm/fpu/internal.h | 12 +++++++++++- arch/x86/include/asm/fpu/xstate.h | 19 +++++++++++++++++++ arch/x86/kernel/fpu/core.c | 2 +- 3 files changed, 31 insertions(+), 2 deletions(-)
arch/x86/include/asm/fpu/internal.h +11 -1
@@ -477,7 +477,17 @@ static inline void fpregs_restore_userrereturn ;if (!fpregs_state_valid (fpu, cpu)) { -restore_fpregs_from_fpstate (&fpu->state); +/* +* This restores _all_ xstate which has not been +* established yet. +* +* If PKRU is enabled, then the PKRU value is already +* correct because it was either set in switch_to() or in +* flush_thread(). So it is excluded because it might be +* not up to date in current->thread.fpu.xsave state. +*/ +__restore_fpregs_from_fpstate (&fpu->state, +xfeatures_mask_restore_user ());fpregs_activate (fpu); fpu->last_cpu = cpu; }
arch/x86/include/asm/fpu/xstate.h +19 -0
@@ -35,6 +35,14 @@ XFEATURE_MASK_BNDREGS | \ XFEATURE_MASK_BNDCSR) +/* +* Features which are restored when returning to user space. +* PKRU is not restored on return to user space because PKRU +* is switched eagerly in switch_to() and flush_thread() +*/ +#define XFEATURE_MASK_USER_RESTORE \ + (XFEATURE_MASK_USER_SUPPORTED & ~XFEATURE_MASK_PKRU) +/* All currently supported supervisor features */ #define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)
@@ -92,6 +100,17 @@ static inline u64 xfeatures_mask_uabi(voreturn xfeatures_mask_all &XFEATURE_MASK_USER_SUPPORTED ; } +/* +* The xfeatures which are restored by the kernel when returning to user +* mode. This is not necessarily the same as xfeatures_mask_uabi() as the +* kernel does not manage all XCR0 enabled features via xsave/xrstor as +* some of them have to be switched eagerly on context switch and exec(). +*/ +static inline u64 xfeatures_mask_restore_user (void ) +{ +return xfeatures_mask_all &XFEATURE_MASK_USER_RESTORE ; +} +static inline u64 xfeatures_mask_independent (void ) {if (!boot_cpu_has (X86_FEATURE_ARCH_LBR ))
arch/x86/kernel/fpu/core.c +1 -1
@@ -471,7 +471,7 @@ void fpu__clear_user_states(struct fpu * }/* Reset user states in registers. */ -load_fpregs_from_init_fpstate (xfeatures_mask_uabi ()); +load_fpregs_from_init_fpstate (xfeatures_mask_restore_user ());/* * Now all FPU registers have their desired values. Inform the FPU
PATCH 36/41 x86/fpu: Add PKRU storage outside of task XSAVE buffer
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
PKRU is currently partly XSAVE-managed and partly not. It has space in the task XSAVE buffer and is context-switched by XSAVE/XRSTOR. However, it is switched more eagerly than FPU because there may be a need for PKRU to be up-to-date for things like copy_to/from_user() since PKRU affects user-permission memory accesses, not just accesses from userspace itself.
This leaves PKRU in a very odd position. XSAVE brings very little value to the table for how Linux uses PKRU except for signal related XSTATE handling.
Prepare to move PKRU away from being XSAVE-managed. Allocate space in the thread_struct for it and save/restore it in the context-switch path separately from the XSAVE-managed features. task->thread_struct.pkru is only valid when the task is scheduled out. For the current task the authoritative source is the hardware, i.e. it has to be retrieved via rdpkru().
Leave the XSAVE code in place for now to ensure bisectability.
---
V4: Picked up from Dave's PKRU series and adopted it to the other changes.
---
arch/x86/include/asm/processor.h | 9 +++++++++ arch/x86/kernel/process.c | 7 +++++++ arch/x86/kernel/process_64.c | 25 +++++++++++++++++++++++++ 3 files changed, 41 insertions(+)
arch/x86/include/asm/processor.h +9 -0
@@ -518,6 +518,15 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; +/* +* Protection Keys Register for Userspace. Loaded immediately on +* context switch. Store it in thread_struct to avoid a lookup in +* the tasks's FPU xstate buffer. This value is only valid when a +* task is scheduled out. For 'current' the authoritative source of +* PKRU is the hardware itself. +*/ +u32 pkru; +/* Floating point and extended processor state */ struct fpu fpu;/*
arch/x86/kernel/process.c +7 -0
@@ -157,11 +157,18 @@ int copy_thread(unsigned long clone_flag/* Kernel thread ? */ if (unlikely (p->flags &PF_KTHREAD )) { + p->thread.pkru =pkru_get_init_value ();memset (childregs,0 ,sizeof (struct pt_regs ));kthread_frame_init (frame, sp, arg);return 0 ; } +/* +* Clone current's PKRU value from hardware. tsk->thread.pkru +* is only valid when scheduled out. +*/ + p->thread.pkru =rdpkru (); + frame->bx =0 ; *childregs = *current_pt_regs (); childregs->ax =0 ;
arch/x86/kernel/process_64.c +25 -0
@@ -340,6 +340,29 @@ static __always_inline void load_seg_leg } } +/* +* Store prev's PKRU value and load next's PKRU value if they differ. PKRU +* is not XSTATE managed on context switch because that would require a +* lookup in the task's FPU xsave buffer and require to keep that updated +* in various places. +*/ +static __always_inline void x86_pkru_load(struct thread_struct *prev, +struct thread_struct *next) +{ +if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) +return ; + +/* Stash the prev task's value: */ + prev->pkru =rdpkru (); + +/* +* PKRU writes are slightly expensive. Avoid them when not +* strictly necessary: +*/ +if (prev->pkru != next->pkru) +wrpkru (next->pkru); +} + static __always_inline void x86_fsgsbase_load(struct thread_struct *prev,struct thread_struct *next) {
@@ -589,6 +612,8 @@ void compat_start_thread(struct pt_regsx86_fsgsbase_load (prev, next); +x86_pkru_load (prev, next); +/* * Switch the PDA and FPU contexts. */
PATCH 37/41 x86/fpu: Hook up PKRU into ptrace()
In reply to Thomas GleixnerFrom: Dave Hansen <[email protected]>
One nice thing about having PKRU be XSAVE-managed is that it gets naturally exposed into the XSAVE-using ABIs. Now that XSAVE will not be used to manage PKRU, these ABIs need to be manually enabled to deal with PKRU.
ptrace() uses copy_uabi_xstate_to_kernel() to collect the tracee's XSTATE. As PKRU is not in the task's XSTATE buffer, use task->thread.pkru for filling in up the ptrace buffer.
---
V4: Picked up from Dave's PKRU series, adopted to rename and dropped the
copy_user_to_fpregs_zeroing() part because task->thread.pkru is only valid when the task is scheduled out. For current the authoritative source is the hardware.
--- arch/x86/include/asm/fpu/xstate.h | 2 - arch/x86/kernel/fpu/regset.c | 12 ++++------- arch/x86/kernel/fpu/xstate.c | 40 ++++++++++++++++++++++++++------------ 3 files changed, 34 insertions(+), 20 deletions(-)
arch/x86/include/asm/fpu/xstate.h +1 -1
@@ -128,7 +128,7 @@ void *get_xsave_addr(struct xregs_stateint using_compacted_format (void );int xfeature_size (int xfeature_nr);struct membuf ; -void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave); +void copy_uabi_xstate_to_membuf (struct membuf to,struct task_struct *target);int copy_uabi_from_kernel_to_xstate (struct xregs_state *xsave,const void *kbuf);int copy_sigframe_from_user_to_xstate (struct xregs_state *xsave,const void __user *ubuf);
arch/x86/kernel/fpu/regset.c +5 -7
@@ -109,22 +109,20 @@ int xfpregs_set(struct task_struct *targ }int xstateregs_get (struct task_struct *target,const struct user_regset *regset, -struct membuf to) +struct membuf to) { -struct fpu *fpu = &target->thread.fpu; -if (!boot_cpu_has (X86_FEATURE_XSAVE ))return -ENODEV ; -fpu__prepare_read (fpu); +fpu__prepare_read (&target->thread.fpu); -copy_uabi_xstate_to_membuf (to, &fpu->state.xsave); +copy_uabi_xstate_to_membuf (to, target);return 0 ; }int xstateregs_set (struct task_struct *target,const struct user_regset *regset, -unsigned int pos,unsigned int count, -const void *kbuf,const void __user *ubuf) +unsigned int pos,unsigned int count, +const void *kbuf,const void __user *ubuf) {struct fpu *fpu = &target->thread.fpu;struct xregs_state *tmpbuf =NULL ;
arch/x86/kernel/fpu/xstate.c +28 -12
@@ -968,12 +968,13 @@ static void copy_part(struct membuf *to,* It supports partial copy but pos always starts from zero. This is called * from xstateregs_get() and there we check the CPU has XSAVE. */ -void copy_uabi_xstate_to_membuf (struct membuf to,struct xregs_state *xsave) +void copy_uabi_xstate_to_membuf (struct membuf to,struct task_struct *target) { -struct xstate_header header;const unsigned off_mxcsr = offsetof(struct fxregs_state , mxcsr); -unsigned size = to.left; -unsigned last =0 ; +struct xregs_state *xsave = &target->thread.fpu.state.xsave; +struct xstate_header header; +unsigned int size = to.left; +unsigned int last =0 ;int i;/*
@@ -983,6 +984,13 @@ void copy_uabi_xstate_to_membuf(struct m header.xfeatures = xsave->header.xfeatures; header.xfeatures &=xfeatures_mask_uabi (); +/* +* PKRU is not XSTATE managed. If enabled set the xfeature bit as +* user space expects it to be part of the XSTATE buffer. +*/ +if (cpu_feature_enabled (X86_FEATURE_OSPKE )) + header.xfeatures |=XFEATURE_MASK_PKRU ; +if (header.xfeatures &XFEATURE_MASK_FP )copy_part (&to, &last,0 , off_mxcsr, &xsave->i387);if (header.xfeatures & (XFEATURE_MASK_SSE |XFEATURE_MASK_YMM ))
@@ -1006,16 +1014,24 @@ void copy_uabi_xstate_to_membuf(struct m sizeof(header), &header);for (i =FIRST_EXTENDED_XFEATURE ; i <XFEATURE_MAX ; i++) { -/* -* Copy only in-use xstates: -*/ -if ((header.xfeatures >> i) &1 ) { -void *src =__raw_xsave_addr (xsave, i); +struct pkru_state pkru = {0 }; +void *src; -copy_part (&to, &last, xstate_offsets[i], - xstate_sizes[i], src); - } +/* Copy only in-use xstates: */ +if (!((header.xfeatures >> i) &1 )) +continue ; +if (i ==XFEATURE_PKRU ) { +/* +* PKRU is not kept in the thread's XSAVE buffer. +* Fill this part from the per-thread storage. +*/ + pkru.pkru = target->thread.pkru; + src = &pkru; + }else { + src =__raw_xsave_addr (xsave, i); + } +copy_part (&to, &last, xstate_offsets[i], xstate_sizes[i], src); }fill_gap (&to, &last, size); }
PATCH 38/41 x86/fpu: Mask PKRU from kernel XRSTOR[S] operations
In reply to Thomas GleixnerAs the PKRU state is managed seperately restoring it from the xstate buffer would be counterproductive as it might either restore a stale value or reinit the PKRU state to 0.
---
V4: New patch
--- arch/x86/include/asm/fpu/internal.h | 4 ++-- arch/x86/include/asm/fpu/xstate.h | 10 ++++++++++ arch/x86/kernel/fpu/xstate.c | 1 + arch/x86/mm/extable.c | 2 +- 4 files changed, 14 insertions(+), 3 deletions(-)
arch/x86/include/asm/fpu/internal.h +2 -2
@@ -281,7 +281,7 @@ static inline void xsave_to_kernel_booti*/ static inline void xrstor_from_kernel_booting (struct xregs_state *xstate) { -u64 mask =-1 ; +u64 mask =xfeatures_mask_fpstate ();u32 lmask = mask;u32 hmask = mask >>32 ;int err;
@@ -408,7 +408,7 @@ extern void __restore_fpregs_from_fpstatstatic inline void restore_fpregs_from_fpstate (union fpregs_state *fpstate) { -__restore_fpregs_from_fpstate (fpstate,-1 ); +__restore_fpregs_from_fpstate (fpstate,xfeatures_mask_fpstate ()); }extern int copy_fpstate_to_sigframe (void __user *buf,void __user *fp,int size);
arch/x86/include/asm/fpu/xstate.h +10 -0
@@ -111,6 +111,16 @@ static inline u64 xfeatures_mask_restorereturn xfeatures_mask_all &XFEATURE_MASK_USER_RESTORE ; } +/* +* Like xfeatures_mask_restore_user() but additionally restors the +* supported supervisor states. +*/ +static inline u64 xfeatures_mask_fpstate (void ) +{ +return xfeatures_mask_all & \ + (XFEATURE_MASK_USER_RESTORE |XFEATURE_MASK_SUPERVISOR_SUPPORTED ); +} +static inline u64 xfeatures_mask_independent (void ) {if (!boot_cpu_has (X86_FEATURE_ARCH_LBR ))
arch/x86/kernel/fpu/xstate.c +1 -0
@@ -60,6 +60,7 @@ static short xsave_cpuid_features[] __in* XSAVE buffer, both supervisor and user xstates. */ u64 xfeatures_mask_all__ro_after_init ; +EXPORT_SYMBOL_GPL (xfeatures_mask_all);static unsigned int xstate_offsets[XFEATURE_MAX ]__ro_after_init = { [0 ...XFEATURE_MAX -1 ] =-1 };
arch/x86/mm/extable.c +1 -1
@@ -65,7 +65,7 @@ EXPORT_SYMBOL_GPL(ex_handler_fault);WARN_ONCE (1 ,"Bad FPU state detected at %pB, reinitializing FPU registers." , (void *)instruction_pointer (regs)); -__restore_fpregs_from_fpstate (&init_fpstate,-1 ); +__restore_fpregs_from_fpstate (&init_fpstate,xfeatures_mask_fpstate ());return true; }EXPORT_SYMBOL_GPL (ex_handler_fprestore);
PATCH 39/41 x86/fpu: Remove PKRU handling from switch_fpu_finish()
In reply to Thomas GleixnerPKRU is already updated and the xstate is not longer the proper source of information.
---
V4: New patch.
--- arch/x86/include/asm/fpu/internal.h | 34 ++++------------------------------ 1 file changed, 4 insertions(+), 30 deletions(-)
arch/x86/include/asm/fpu/internal.h +4 -30
@@ -539,39 +539,13 @@ static inline void switch_fpu_prepare(st*/ /* -* Load PKRU from the FPU context if available. Delay loading of the -* complete FPU state until the return to userland. +* Delay loading of the complete FPU state until the return to userland. +* PKRU is handled seperately. */ static inline void switch_fpu_finish (struct fpu *new_fpu) { -u32 pkru_val = init_pkru_value; -struct pkru_state *pk; - -if (!static_cpu_has (X86_FEATURE_FPU )) -return ; - -set_thread_flag (TIF_NEED_FPU_LOAD ); - -if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) -return ; - -/* -* PKRU state is switched eagerly because it needs to be valid before we -* return to userland e.g. for a copy_to_user() operation. -*/ -if (!(current->flags &PF_KTHREAD )) { -/* -* If the PKRU bit in xsave.header.xfeatures is not set, -* then the PKRU component was in init state, which means -* XRSTOR will set PKRU to 0. If the bit is not set then -* get_xsave_addr() will return NULL because the PKRU value -* in memory is not valid. This means pkru_val has to be -* set to 0 and not to init_pkru_value. -*/ - pk =get_xsave_addr (&new_fpu->state.xsave,XFEATURE_PKRU ); - pkru_val = pk ? pk->pkru :0 ; - } -__write_pkru (pkru_val); +if (static_cpu_has (X86_FEATURE_FPU )) +set_thread_flag (TIF_NEED_FPU_LOAD ); }#endif /* _ASM_X86_FPU_INTERNAL_H */
PATCH 40/41 x86/fpu: Dont store PKRU in xstate in fpu_reset_fpstate()
In reply to Thomas GleixnerPKRU for a task is stored in task->thread.pkru when the task is scheduled out. For 'current' the authoritative source of PKRU is the hardware.
fpu_reset_fpstate() has two callers:
1) fpu__clear_user_states() for !FPU systems. For those PKRU is irrelevant
2) fpu_flush_thread() which is invoked from flush_thread(). flush_thread() resets the hardware to the kernel restrictive default value.
---
V4: New patch
--- arch/x86/kernel/fpu/core.c | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-)
arch/x86/kernel/fpu/core.c +4 -18
@@ -403,23 +403,6 @@ static inline unsigned int init_fpstate_return sizeof (init_fpstate.xsave); } -/* Temporary workaround. Will be removed once PKRU and XSTATE are distangled. */ -static inline void pkru_set_default_in_xstate (struct xregs_state *xsave) -{ -struct pkru_state *pk; - -if (!cpu_feature_enabled (X86_FEATURE_OSPKE )) -return ; -/* -* Force XFEATURE_PKRU to be set in the header otherwise -* get_xsave_addr() does not work and it also needs to be set to -* make XRSTOR(S) load it. -*/ - xsave->header.xfeatures |=XFEATURE_MASK_PKRU ; - pk =get_xsave_addr (xsave,XFEATURE_PKRU ); - pk->pkru =pkru_get_init_value (); -} -/* * Reset current->fpu memory state to the init values. */
@@ -437,9 +420,12 @@ static void fpu_reset_fpstate(void)* * Do not use fpstate_init() here. Just copy init_fpstate which has * the correct content already except for PKRU. +* +* PKRU handling does not rely on the xstate when restoring for +* user space as PKRU is eagerly written in switch_to() and +* flush_thread(). */ memcpy (&fpu->state, &init_fpstate,init_fpstate_copy_size ()); -pkru_set_default_in_xstate (&fpu->state.xsave);set_thread_flag (TIF_NEED_FPU_LOAD );fpregs_unlock (); }
PATCH 41/41 x86/pkru: Remove xstate fiddling from write_pkru()
In reply to Thomas GleixnerThe PKRU value of a task is stored in task->thread.pkru when the task is scheduled out. PKRU is restored on schedule in from there. So keeping the XSAVE buffer up to date is a pointless exercise.
Remove the xstate fiddling and cleanup all related functions.
---
V4: New patch
--- arch/x86/include/asm/pkru.h | 17 ++++------------- arch/x86/include/asm/special_insns.h | 14 +------------- arch/x86/kvm/x86.c | 4 ++-- 3 files changed, 7 insertions(+), 28 deletions(-)
arch/x86/include/asm/pkru.h +4 -13
@@ -41,23 +41,14 @@ static inline u32 read_pkru(void)static inline void write_pkru (u32 pkru) { -struct pkru_state *pk; -if (!cpu_feature_enabled (X86_FEATURE_OSPKE ))return ; - - pk =get_xsave_addr (¤t->thread.fpu.state.xsave,XFEATURE_PKRU ); -/* -* The PKRU value in xstate needs to be in sync with the value that is -* written to the CPU. The FPU restore on return to userland would -* otherwise load the previous value again. +* WRPKRU is relatively expensive compared to RDPKRU. +* Avoid WRPKRU when it would not change the value. */ -fpregs_lock (); -if (pk) - pk->pkru = pkru; -__write_pkru (pkru); -fpregs_unlock (); +if (pkru !=rdpkru ()) +wrpkru (pkru); }static inline void pkru_write_default (void )
arch/x86/include/asm/special_insns.h +1 -13
@@ -104,25 +104,13 @@ static inline void wrpkru(u32 pkru) : : "a" (pkru), "c"(ecx), "d"(edx)); } -static inline void __write_pkru (u32 pkru) -{ -/* -* WRPKRU is relatively expensive compared to RDPKRU. -* Avoid WRPKRU when it would not change the value. -*/ -if (pkru ==rdpkru ()) -return ; - -wrpkru (pkru); -} -#else static inline u32 rdpkru (void ) {return 0 ; } -static inline void __write_pkru (u32 pkru) +static inline void wrpkru (u32 pkru) { }#endif
arch/x86/kvm/x86.c +2 -2
@@ -943,7 +943,7 @@ void kvm_load_guest_xsave_state(struct k (kvm_read_cr4_bits(vcpu, X86_CR4_PKE) || (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU)) && vcpu->arch.pkru != vcpu->arch.host_pkru) -__write_pkru (vcpu->arch.pkru); +write_pkru (vcpu->arch.pkru); }EXPORT_SYMBOL_GPL (kvm_load_guest_xsave_state);
@@ -957,7 +957,7 @@ void kvm_load_host_xsave_state(struct kv (vcpu->arch.xcr0 & XFEATURE_MASK_PKRU))) { vcpu->arch.pkru =rdpkru ();if (vcpu->arch.pkru != vcpu->arch.host_pkru) -__write_pkru (vcpu->arch.host_pkru); +write_pkru (vcpu->arch.host_pkru); }if (kvm_read_cr4_bits (vcpu,X86_CR4_OSXSAVE )) {
Quoted 3 lines · Thomas Gleixner
- Removal of PKRU from being XSTATE managed in the kernel because PKRU has to be eagerly restored on context switch and keeping it in sync in the xstate buffer is just pointless overhead and fragile.
Just before anyone comes up with any complaints about the resulting inconsistency vs. xgetbv(1) in the case that the PKRU value is 0.
That inconsistency is simply a INTEL only hardware bug and there is no way to get this consistent ever no matter what kind of mechanism the kernel uses. This inconsistency can be demonstrated in user space w/o any kernel interaction.
The Intel SDM states in volume 1, chapter 13.6
PROCESSOR TRACKING OF XSAVE-MANAGED STATE
* PKRU state. PKRU state is in its initial configuration if the value of the PKRU is 0.
But that's just not true.
wrpkru(0)
assert(!(xgetbv(1) & XFEATURE_PKRU);fails on Intel but not on AMD AFAIK. xgetbv(1) returns the 'INUSE' bitmap of xstate managed features.
But the Intel SDM is blury about this:
XINUSE denotes the state-component bitmap corresponding to the init optimization. If XINUSE[i] = 0, state component i is known to be in its initial configuration; otherwise XINUSE[i] = 1. It is possible for XINUSE[i] to be 1 even when state component i is in its initial configuration. On a processor that does not support the init optimization, XINUSE[i] is always 1 for every value of i.
IOW there is no consistency vs. XINUSE and initial state guaranteed at all. So why should the kernel worry about this?
We just use the most optimized way to deal with this and that's what this patch series is doing by removing PKRU from xstate management in the kernel.
If anyone cares about consistency of XINUSE vs. the actual component state then please redirect the complaints to INTEL.
Either the hardware folks get their act together or software which relies on consistency (cough, cough) like rr has to cope with it.
Making the kernel to pretend that all of this is consistent under all circumstances is a futile attempt to ignore reality.
This inconsistency can only be fixed in hardware/ucode. End of story.
Thanks,
tglx
Quoted 6 lines · Thomas Gleixner
The Intel SDM states in volume 1, chapter 13.6
PROCESSOR TRACKING OF XSAVE-MANAGED STATE
* PKRU state. PKRU state is in its initial configuration if the value of the PKRU is 0.>
...
Quoted 2 lines
IOW there is no consistency vs. XINUSE and initial state guaranteed at all. So why should the kernel worry about this?
This is my feeling on it as well. I may be a bit corrupted by having talked to the hardware folks who have built the init state "trackers", but I've always had the opinion that XINUSE is very weakly defined.
There are some other bits of the SDM that you noted that clearly call out that XINUSE is not strictly tied to the *value* of the component.
Quoted 11 lines · Thomas Gleixner
The Intel SDM states in volume 1, chapter 13.6
PROCESSOR TRACKING OF XSAVE-MANAGED STATE
* PKRU state. PKRU state is in its initial configuration if the value of the PKRU is 0.
But that's just not true.
wrpkru(0) assert(!(xgetbv(1) & XFEATURE_PKRU);
Hi Thomas,
It's pretty clear that Intel's implementation was intentional. It was certainly no accident that it was implemented this way.
I'm a bit confused why you expected to see XINUSE[PKRU]=0 up in your example. The CPU is *free* to set XINUSE[PKRU]=0, but it appears that the example expects that it *must* set XINUSE[PKRU]=0.
I do wish the SDM had been excruciatingly explicit in defining:
initial configuration
vs.
initial state
All features in their "initial state" have their "initial configuration" values, but not all features with their "initial configuration" values are in their "initial state".
Quoted 11 lines
But the Intel SDM is blury about this:
XINUSE denotes the state-component bitmap corresponding to the init optimization. If XINUSE[i] = 0, state component i is known to be in its initial configuration; otherwise XINUSE[i] = 1. It is possible for XINUSE[i] to be 1 even when state component i is in its initial configuration. On a processor that does not support the init optimization, XINUSE[i] is always 1 for every value of i.
IOW there is no consistency vs. XINUSE and initial state guaranteed at all. So why should the kernel worry about this?
Exactly, there is really no consistency guarantee. This was written to give the CPU designers some flexibility so that they could opt to omit "init tracker" hardware if they chose. Or, so that they could be a bit lazy about implementing one.
Imagine what would happen if the AMD PKRU init tracking behavior (write all 0's, get XINUSE[PKRU]=0) was *required* XSAVE behavior. Every ZMM register write would potentially need to go checking ~2k of state to see if the rest of the state is all 0's.
Quoted 2 lines
If anyone cares about consistency of XINUSE vs. the actual component state then please redirect the complaints to INTEL.
I think we can take a _bit_ of the blame on the kernel side too. The kernel has very good reasons for managing PKRU with WRPKRU instead of XSAVE. *But*, it also tossed out XINUSE[PKRU] consistency in the process.
I'm not sure we should be looking to the hardware to bring that back.
Quoted 7 lines
Either the hardware folks get their act together or software which relies on consistency (cough, cough) like rr has to cope with it.
Making the kernel to pretend that all of this is consistent under all circumstances is a futile attempt to ignore reality.
This inconsistency can only be fixed in hardware/ucode. End of story.
I agree with this. If it's going to be fixed, the kernel simply doesn't have the tools to do it. We either need new ISA or new hardware/ucode.
I'm just not convinced it's worth fixing for PKRU.
Quoted 10 lines · Dave Hansen
Quoted 4 lines · Thomas Gleixner
Making the kernel to pretend that all of this is consistent under all circumstances is a futile attempt to ignore reality.
This inconsistency can only be fixed in hardware/ucode. End of story.
I agree with this. If it's going to be fixed, the kernel simply doesn't have the tools to do it. We either need new ISA or new hardware/ucode.
I'm just not convinced it's worth fixing for PKRU.
Me neither.