Control Flow Enforcement: Shadow Stack
Subsystems (19)
v3 cover Control Flow Enforcement: Shadow Stack
The previous version of CET patches can be found in the following link.
https://lkml.org/lkml/2018/7/10/1031
Summary of changes from v2:
Move Shadow Stack page fault handling logic to arch/x86. Update can_follow_write_pte/pmd; move logic to arch/x86. Fix problems in WRUSS in-line assembly.
Fix issues in ELF parser.
Split out IBT/PTRACE patches to a second set. Other small fixes.
Yu-cheng Yu (24):
x86/cpufeatures: Add CPUIDs for Control-flow Enforcement Technology
(CET)
x86/fpu/xstate: Change some names to separate XSAVES system and user
states
x86/fpu/xstate: Enable XSAVES system states x86/fpu/xstate: Add XSAVES system states for shadow stack Documentation/x86: Add CET description
x86/cet: Control protection exception handler x86/cet/shstk: Add Kconfig option for user-mode shadow stack
mm: Introduce VM_SHSTK for shadow stack memory
x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW
x86/mm: Introduce _PAGE_DIRTY_SW
drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for
_PAGE_DIRTY_SW
x86/mm: Shadow stack page fault error checking
mm: Handle shadow stack page fault
mm: Handle THP/HugeTLB shadow stack page fault
mm: Update can_follow_write_pte/pmd for shadow stack
mm: Introduce do_mmap_locked()
x86/cet/shstk: User-mode shadow stack support x86/cet/shstk: Introduce WRUSS instruction
x86/cet/shstk: Signal handling for shadow stack x86/cet/shstk: ELF header parsing of Shadow Stack x86/cet/shstk: Handle thread shadow stack
x86/cet/shstk: Add arch_prctl functions for Shadow Stack x86/cet/shstk: Add Shadow Stack instructions to opcode map
.../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 11 + Documentation/x86/intel_cet.rst | 252 +++++++++++++ arch/x86/Kconfig | 28 ++ arch/x86/Makefile | 7 + arch/x86/entry/entry_64.S | 2 +- arch/x86/ia32/ia32_signal.c | 13 + arch/x86/include/asm/cet.h | 42 +++ arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/elf.h | 5 + arch/x86/include/asm/fpu/internal.h | 6 +- arch/x86/include/asm/fpu/types.h | 22 ++ arch/x86/include/asm/fpu/xstate.h | 31 +- arch/x86/include/asm/mmu_context.h | 3 + arch/x86/include/asm/msr-index.h | 14 + arch/x86/include/asm/pgtable.h | 175 +++++++-- arch/x86/include/asm/pgtable_types.h | 31 +- arch/x86/include/asm/processor.h | 5 + arch/x86/include/asm/sighandling.h | 5 + arch/x86/include/asm/special_insns.h | 37 ++ arch/x86/include/asm/traps.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/include/uapi/asm/prctl.h | 5 + arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/include/uapi/asm/sigcontext.h | 17 + arch/x86/kernel/Makefile | 4 + arch/x86/kernel/cet.c | 285 +++++++++++++++ arch/x86/kernel/cet_prctl.c | 79 ++++ arch/x86/kernel/cpu/common.c | 24 ++ arch/x86/kernel/cpu/scattered.c | 1 + arch/x86/kernel/elf.c | 338 ++++++++++++++++++ arch/x86/kernel/fpu/core.c | 11 +- arch/x86/kernel/fpu/init.c | 10 - arch/x86/kernel/fpu/signal.c | 6 +- arch/x86/kernel/fpu/xstate.c | 152 +++++--- arch/x86/kernel/idt.c | 4 + arch/x86/kernel/process.c | 8 + arch/x86/kernel/process_64.c | 7 + arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kernel/signal.c | 96 +++++ arch/x86/kernel/traps.c | 58 +++ arch/x86/kvm/vmx.c | 2 +- arch/x86/lib/x86-opcode-map.txt | 26 +- arch/x86/mm/extable.c | 11 + arch/x86/mm/fault.c | 27 ++ arch/x86/mm/pgtable.c | 36 ++ drivers/gpu/drm/i915/gvt/gtt.c | 2 +- fs/binfmt_elf.c | 15 + fs/proc/task_mmu.c | 3 + include/asm-generic/pgtable.h | 52 +++ include/linux/mm.h | 26 ++ include/uapi/linux/elf.h | 1 + mm/gup.c | 8 +- mm/huge_memory.c | 12 +- mm/internal.h | 8 + mm/memory.c | 3 + tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +- 59 files changed, 1943 insertions(+), 150 deletions(-) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/cet.c create mode 100644 arch/x86/kernel/cet_prctl.c create mode 100644 arch/x86/kernel/elf.c
Signature · 1 line
2.17.1
RFC PATCH v3 1/24 x86/cpufeatures: Add CPUIDs for Control-flow Enforcement Technology (CET)
Add CPUIDs for Control-flow Enforcement Technology (CET).
CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect branch tracking
--- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/kernel/cpu/scattered.c | 1 + 2 files changed, 3 insertions(+)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -221,6 +221,7 @@#define X86_FEATURE_ZEN ( 7*32+28)/* "" CPU is AMD family 0x17 (Zen) */ #define X86_FEATURE_L1TF_PTEINV ( 7*32+29)/* "" L1TF workaround PTE inversion */ #define X86_FEATURE_IBRS_ENHANCED ( 7*32+30)/* Enhanced IBRS */ +#define X86_FEATURE_IBT ( 7*32+31)/* Indirect Branch Tracking */ /* Virtualization flags: Linux defined, word 8 */ #define X86_FEATURE_TPR_SHADOW ( 8*32+ 0)/* Intel TPR Shadow */
@@ -321,6 +322,7 @@#define X86_FEATURE_PKU (16*32+ 3)/* Protection Keys for Userspace */ #define X86_FEATURE_OSPKE (16*32+ 4)/* OS Protection Keys Enable */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6)/* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7)/* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8)/* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9)/* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10)/* Carry-Less Multiplication Double Quadword */
arch/x86/kernel/cpu/scattered.c +1 -0
@@ -21,6 +21,7 @@ struct cpuid_bit {static const struct cpuid_bit cpuid_bits[] = { {X86_FEATURE_APERFMPERF ,CPUID_ECX ,0 ,0x00000006 ,0 }, {X86_FEATURE_EPB ,CPUID_ECX ,3 ,0x00000006 ,0 }, + {X86_FEATURE_IBT ,CPUID_EDX ,20 ,0x00000007 ,0 }, {X86_FEATURE_CAT_L3 ,CPUID_EBX ,1 ,0x00000010 ,0 }, {X86_FEATURE_CAT_L2 ,CPUID_EBX ,2 ,0x00000010 ,0 }, {X86_FEATURE_CDP_L3 ,CPUID_ECX ,2 ,0x00000010 ,1 },
Signature · 1 line
2.17.1
RFC PATCH v3 2/24 x86/fpu/xstate: Change some names to separate XSAVES system and user states
In reply to Yu-cheng YuTo support XSAVES system states, change some names to distinguish user and system states.
Change:
supervisor to system
copy_init_fpstate_to_fpregs() to copy_init_user_fpstate_to_fpregs() xfeatures_mask to xfeatures_mask_user
XCNTXT_MASK to SUPPORTED_XFEATURES_MASK (states supported)
--- arch/x86/include/asm/fpu/internal.h | 5 +- arch/x86/include/asm/fpu/xstate.h | 24 ++++---- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/fpu/init.c | 2 +- arch/x86/kernel/fpu/signal.c | 6 +- arch/x86/kernel/fpu/xstate.c | 88 +++++++++++++++-------------- 6 files changed, 66 insertions(+), 63 deletions(-)
arch/x86/include/asm/fpu/internal.h +3 -2
@@ -93,7 +93,8 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave)* XRSTORS requires these bits set in xcomp_bv, or it will * trigger #GP: */ - xsave->header.xcomp_bv =XCOMP_BV_COMPACTED_FORMAT | xfeatures_mask; + xsave->header.xcomp_bv =XCOMP_BV_COMPACTED_FORMAT | + xfeatures_mask_user; }static inline void fpstate_init_fxstate (struct fxregs_state *fx)
@@ -233,7 +234,7 @@ static inline void copy_fxregs_to_kernel(struct fpu *fpu)/* * If XSAVES is enabled, it replaces XSAVEOPT because it supports a compact -* format and supervisor states in addition to modified optimization in +* format and system states in addition to modified optimization in * XSAVEOPT. * * Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT
arch/x86/include/asm/fpu/xstate.h +12 -12
@@ -19,19 +19,19 @@#define XSAVE_YMM_SIZE 256#define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* Supervisor features */ -#define XFEATURE_MASK_SUPERVISOR (XFEATURE_MASK_PT) +/* System features */ +#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT)/* All currently supported features */ -#define XCNTXT_MASK (XFEATURE_MASK_FP | \ - XFEATURE_MASK_SSE | \ - XFEATURE_MASK_YMM | \ - XFEATURE_MASK_OPMASK | \ - XFEATURE_MASK_ZMM_Hi256 | \ - XFEATURE_MASK_Hi16_ZMM | \ - XFEATURE_MASK_PKRU | \ - XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) +#define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ + XFEATURE_MASK_SSE | \ + XFEATURE_MASK_YMM | \ + XFEATURE_MASK_OPMASK | \ + XFEATURE_MASK_ZMM_Hi256 | \ + XFEATURE_MASK_Hi16_ZMM | \ + XFEATURE_MASK_PKRU | \ + XFEATURE_MASK_BNDREGS | \ + XFEATURE_MASK_BNDCSR)#ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, "
@@ -39,7 +39,7 @@#define REX_PREFIX #endif -extern u64 xfeatures_mask; +extern u64 xfeatures_mask_user;extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS ]; extern void __init update_regset_xstate_info(unsigned int size,
arch/x86/kernel/fpu/core.c +2 -2
@@ -363,7 +363,7 @@ void fpu__drop(struct fpu *fpu)* Clear FPU registers by setting them up from * the init fpstate: */ -static inline void copy_init_fpstate_to_fpregs (void ) +static inline void copy_init_user_fpstate_to_fpregs (void ) {if (use_xsave ())copy_kernel_to_xregs (&init_fpstate.xsave,-1 );
@@ -395,7 +395,7 @@ void fpu__clear(struct fpu *fpu)preempt_disable ();fpu__initialize (fpu);user_fpu_begin (); -copy_init_fpstate_to_fpregs (); +copy_init_user_fpstate_to_fpregs ();preempt_enable (); } }
arch/x86/kernel/fpu/init.c +1 -1
@@ -229,7 +229,7 @@ static void __init fpu__init_system_xstate_size_legacy(void)*/ u64 __init fpu__get_supported_xfeatures_mask (void ) { -return XCNTXT_MASK ; +return SUPPORTED_XFEATURES_MASK ; }/* Legacy code to initialize eager fpu mode. */
arch/x86/kernel/fpu/signal.c +3 -3
@@ -249,11 +249,11 @@ static inline int copy_user_to_fpregs_zeroing(void __user *buf, u64 xbv, int fx_ {if (use_xsave ()) {if ((unsigned long )buf %64 || fx_only) { -u64 init_bv = xfeatures_mask & ~XFEATURE_MASK_FPSSE ; +u64 init_bv = xfeatures_mask_user & ~XFEATURE_MASK_FPSSE ;copy_kernel_to_xregs (&init_fpstate.xsave, init_bv);return copy_user_to_fxregs (buf); }else { -u64 init_bv = xfeatures_mask & ~xbv; +u64 init_bv = xfeatures_mask_user & ~xbv;if (unlikely (init_bv))copy_kernel_to_xregs (&init_fpstate.xsave, init_bv);return copy_user_to_xregs (buf, xbv);
@@ -417,7 +417,7 @@ void fpu__init_prepare_fx_sw_frame(void) fx_sw_reserved.magic1 =FP_XSTATE_MAGIC1 ; fx_sw_reserved.extended_size = size; - fx_sw_reserved.xfeatures = xfeatures_mask; + fx_sw_reserved.xfeatures = xfeatures_mask_user; fx_sw_reserved.xstate_size = fpu_user_xstate_size; if (IS_ENABLED(CONFIG_IA32_EMULATION) ||
arch/x86/kernel/fpu/xstate.c +45 -43
280 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v3 3/24 x86/fpu/xstate: Enable XSAVES system states
In reply to Yu-cheng YuXSAVES saves both system and user states. The Linux kernel currently does not save/restore any system states. This patch creates the framework for supporting system states.
--- arch/x86/include/asm/fpu/internal.h | 3 +- arch/x86/include/asm/fpu/xstate.h | 9 ++- arch/x86/kernel/fpu/core.c | 7 +- arch/x86/kernel/fpu/init.c | 10 --- arch/x86/kernel/fpu/xstate.c | 112 +++++++++++++++++----------- 5 files changed, 80 insertions(+), 61 deletions(-)
arch/x86/include/asm/fpu/internal.h +1 -2
@@ -45,7 +45,6 @@ extern void fpu__init_cpu_xstate(void);extern void fpu__init_system (struct cpuinfo_x86 *c);extern void fpu__init_check_bugs (void );extern void fpu__resume_cpu (void ); -extern u64 fpu__get_supported_xfeatures_mask (void );/* * Debugging facility:
@@ -94,7 +93,7 @@ static inline void fpstate_init_xstate(struct xregs_state *xsave)* trigger #GP: */ xsave->header.xcomp_bv =XCOMP_BV_COMPACTED_FORMAT | - xfeatures_mask_user; + xfeatures_mask_all; }static inline void fpstate_init_fxstate (struct fxregs_state *fx)
arch/x86/include/asm/fpu/xstate.h +5 -4
@@ -19,10 +19,10 @@#define XSAVE_YMM_SIZE 256#define XSAVE_YMM_OFFSET (XSAVE_HDR_SIZE + XSAVE_HDR_OFFSET) -/* System features */ -#define XFEATURE_MASK_SYSTEM (XFEATURE_MASK_PT) - -/* All currently supported features */ +/* +* SUPPORTED_XFEATURES_MASK indicates all features +* implemented in and supported by the kernel. +*/ #define SUPPORTED_XFEATURES_MASK (XFEATURE_MASK_FP | \ XFEATURE_MASK_SSE | \ XFEATURE_MASK_YMM | \
@@ -40,6 +40,7 @@#endif extern u64 xfeatures_mask_user; +extern u64 xfeatures_mask_all;extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS ]; extern void __init update_regset_xstate_info(unsigned int size,
arch/x86/kernel/fpu/core.c +6 -1
@@ -365,8 +365,13 @@ void fpu__drop(struct fpu *fpu)*/ static inline void copy_init_user_fpstate_to_fpregs (void ) { +/* +* Only XSAVES user states are copied. +* System states are preserved. +*/ if (use_xsave ()) -copy_kernel_to_xregs (&init_fpstate.xsave,-1 ); +copy_kernel_to_xregs (&init_fpstate.xsave, + xfeatures_mask_user);else if (static_cpu_has (X86_FEATURE_FXSR ))copy_kernel_to_fxregs (&init_fpstate.fxsave);else
arch/x86/kernel/fpu/init.c +0 -10
@@ -222,16 +222,6 @@ static void __init fpu__init_system_xstate_size_legacy(void) fpu_user_xstate_size = fpu_kernel_xstate_size; } -/* -* Find supported xfeatures based on cpu features and command-line input. -* This must be called after fpu__init_parse_early_param() is called and -* xfeatures_mask is enumerated. -*/ -u64 __init fpu__get_supported_xfeatures_mask (void ) -{ -return SUPPORTED_XFEATURES_MASK ; -} -/* Legacy code to initialize eager fpu mode. */ static void __init fpu__init_system_ctx_switch (void ) {
arch/x86/kernel/fpu/xstate.c +68 -44
262 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v3 4/24 x86/fpu/xstate: Add XSAVES system states for shadow stack
In reply to Yu-cheng YuIntel Control-flow Enforcement Technology (CET) introduces the following MSRs into the XSAVES system states.
IA32_U_CET (user-mode CET settings),
IA32_PL3_SSP (user-mode shadow stack),
IA32_PL0_SSP (kernel-mode shadow stack),
IA32_PL1_SSP (ring-1 shadow stack),
IA32_PL2_SSP (ring-2 shadow stack).
--- arch/x86/include/asm/fpu/types.h | 22 +++++++++++++++++++++ arch/x86/include/asm/fpu/xstate.h | 4 +++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 10 ++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-)
arch/x86/include/asm/fpu/types.h +22 -0
@@ -114,6 +114,9 @@ enum xfeature { XFEATURE_Hi16_ZMM,XFEATURE_PT_UNIMPLEMENTED_SO_FAR ,XFEATURE_PKRU , +XFEATURE_RESERVED , +XFEATURE_SHSTK_USER , +XFEATURE_SHSTK_KERNEL , XFEATURE_MAX, };
@@ -128,6 +131,8 @@ enum xfeature {#define XFEATURE_MASK_Hi16_ZMM (1 << XFEATURE_Hi16_ZMM)#define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) +#define XFEATURE_MASK_SHSTK_USER (1 << XFEATURE_SHSTK_USER) +#define XFEATURE_MASK_SHSTK_KERNEL (1 << XFEATURE_SHSTK_KERNEL)#define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE)#define XFEATURE_MASK_AVX512 (XFEATURE_MASK_OPMASK \
@@ -229,6 +234,23 @@ struct pkru_state {u32 pad; } __packed; +/* +* State component 11 is Control flow Enforcement user states +*/ +struct cet_user_state { +u64 u_cet;/* user control flow settings */ +u64 user_ssp;/* user shadow stack pointer */ +} __packed; + +/* +* State component 12 is Control flow Enforcement kernel states +*/ +struct cet_kernel_state { +u64 kernel_ssp;/* kernel shadow stack */ +u64 pl1_ssp;/* ring-1 shadow stack */ +u64 pl2_ssp;/* ring-2 shadow stack */ +} __packed; +struct xstate_header {u64 xfeatures;u64 xcomp_bv;
arch/x86/include/asm/fpu/xstate.h +3 -1
@@ -31,7 +31,9 @@ XFEATURE_MASK_Hi16_ZMM | \ XFEATURE_MASK_PKRU | \ XFEATURE_MASK_BNDREGS | \ - XFEATURE_MASK_BNDCSR) + XFEATURE_MASK_BNDCSR | \ + XFEATURE_MASK_SHSTK_USER | \ + XFEATURE_MASK_SHSTK_KERNEL)#ifdef CONFIG_X86_64 #define REX_PREFIX "0x48, "
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -130,6 +130,8 @@#define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT)#define X86_CR4_PKE_BIT 22/* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23/* enable Control flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT)/* * x86-64 Task Priority Register, CR8
arch/x86/kernel/fpu/xstate.c +10 -0
@@ -35,6 +35,9 @@ static const char *xfeature_names[] ="Processor Trace (unused)" ,"Protection Keys User registers" ,"unknown xstate feature" , +"Control flow User registers" , +"Control flow Kernel registers" , + "unknown xstate feature" , }; static short xsave_cpuid_features[] __initdata = {
@@ -48,6 +51,9 @@ static short xsave_cpuid_features[] __initdata = {X86_FEATURE_AVX512F ,X86_FEATURE_INTEL_PT ,X86_FEATURE_PKU , +0 ,/* Unused */ +X86_FEATURE_SHSTK ,/* XFEATURE_SHSTK_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_SHSTK_KERNEL */ };/*
@@ -316,6 +322,8 @@ static void __init print_xstate_features(void)print_xstate_feature (XFEATURE_MASK_ZMM_Hi256);print_xstate_feature (XFEATURE_MASK_Hi16_ZMM);print_xstate_feature (XFEATURE_MASK_PKRU ); +print_xstate_feature (XFEATURE_MASK_SHSTK_USER ); +print_xstate_feature (XFEATURE_MASK_SHSTK_KERNEL ); }/*
@@ -562,6 +570,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_SHSTK_KERNEL, struct cet_kernel_state);/* * Make *SURE* to add any feature numbers in below if
Signature · 1 line
2.17.1
RFC PATCH v3 5/24 Documentation/x86: Add CET description
In reply to Yu-cheng YuExplain how CET works and the no_cet_shstk/no_cet_ibt kernel parameters.
--- .../admin-guide/kernel-parameters.txt | 6 + Documentation/index.rst | 1 + Documentation/x86/index.rst | 11 + Documentation/x86/intel_cet.rst | 252 ++++++++++++++++++ 4 files changed, 270 insertions(+) create mode 100644 Documentation/x86/index.rst create mode 100644 Documentation/x86/intel_cet.rst
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications + nosmap [X86] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor.
Documentation/index.rst +1 -0
@@ -104,6 +104,7 @@ implementation. :maxdepth: 2 sh/index + x86/index Filesystem Documentation ------------------------
Documentation/x86/index.rst new file +11 -0
@@ -0,0 +1,11 @@ +======================= +X86 Documentation +======================= + +Control Flow Enforcement +======================== + +.. toctree:: + :maxdepth: 1 + + intel_cet
Documentation/x86/intel_cet.rst new file +252 -0
252 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH v3 05/24] Documentation/x86: Add CET description
Hi!
Documentation/admin-guide/kernel-parameters.txt line 2771
Quoted 13 lines
Documentation/admin-guide/kernel-parameters.txt +5
-0@@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_cet_ibt [X86-64] Disable indirect branch tracking for user-mode + applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode + applications
Hmm, not too consistent with "nosmap" below. Would it make sense to have cet=on/off/ibt/shstk instead?
Documentation/admin-guide/kernel-parameters.txt line 9
Quoted 11 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+++ b/Documentation/x86/intel_cet.rst@@ -0,0 +1,252 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protection against +return/jump-oriented programing (ROP) attacks.
Can you add something like "It attempts to protect process from running arbitrary code even after attacker has control of its stack" -- for people that don't know what ROP is, and perhaps link to wikipedia explaining ROP or something...
Documentation/admin-guide/kernel-parameters.txt
Quoted 4 lines
It can be implemented
Documentation/admin-guide/kernel-parameters.txt +3
-0+to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented for the 64-bit kernel. +Thirty-two bit applications are supported under the compatibility
32-bit (for consistency).
Ok, so CET stops execution of malicious code before architectural effects are visible, correct? Does it prevent micro-architectural effects of the malicious code? (cache content would be one example; see Spectre).
Quoted 2 lines
+[3] Application Enabling
+========================
"Enabling CET in applications" ?
Documentation/admin-guide/kernel-parameters.txt
Quoted 7 lines
Documentation/admin-guide/kernel-parameters.txt +7
-0+Signal +------ + +The main program and its signal handlers use the same SHSTK. Because +the SHSTK stores only return addresses, we can estimate a large +enough SHSTK to cover the condition that both the program stack and +the sigaltstack run out.
English? Is it estimate or is it large enough? "a large" -- "a" should be deleted AFAICT.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Re: [RFC PATCH v3 05/24] Documentation/x86: Add CET description
Quoted 61 lines · Pavel Machek
Hi!
Quoted 18 lines
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -2764,6 +2764,12 @@noexec=on: enable non-executable mappings (default)
noexec=off: disable non-executable mappingsDocumentation/admin-guide/kernel-parameters.txt +1
-0+ no_cet_ibt [X86-64] Disable indirect branchtracking for user-mode
Documentation/admin-guide/kernel-parameters.txt +3
-0+ applications + + no_cet_shstk [X86-64] Disable shadow stack supportfor user-mode
Documentation/admin-guide/kernel-parameters.txt
Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +1
-0+ applicationsHmm, not too consistent with "nosmap" below. Would it make sense to have cet=on/off/ibt/shstk instead?
Quoted 11 lines
+++
b/Documentation/x86/intel_cet.rstDocumentation/admin-guide/kernel-parameters.txt +8
-0@@ -0,0 +1,252 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protectionQuoted 2 lines
against
+return/jump-oriented programing (ROP) attacks.Can you add something like "It attempts to protect process from running arbitrary code even after attacker has control of its stack" -- for people that don't know what ROP is, and perhaps link to wikipedia explaining ROP or something...
Quoted 6 lines
It can be implemented
Documentation/admin-guide/kernel-parameters.txt +2
-0+to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented for the 64-bitkernel.
+Thirty-two bit applications are supported under the compatibility32-bit (for consistency).
Ok, so CET stops execution of malicious code before architectural effects are visible, correct? Does it prevent micro-architectural effects of the malicious code? (cache content would be one example; see Spectre).
Quoted 3 lines
+[3] Application Enabling
+========================"Enabling CET in applications" ?
Quoted 1 line
Quoted 13 lines
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+Signal +------ + +The main program and its signal handlers use the sameSHSTK. Because
Documentation/admin-guide/kernel-parameters.txt +2
-0+the SHSTK stores only return addresses, we can estimate a large +enough SHSTK to cover the condition that both the program stackand
+the sigaltstack run out.English? Is it estimate or is it large enough? "a large" -- "a" should
be deleted AFAICT.
I will work on these, thanks!
Re: [RFC PATCH v3 05/24] Documentation/x86: Add CET description
In reply to Pavel MachekQuoted 13 lines · Pavel Machek
Hi!
Quoted 11 lines
Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -2764,6 +2764,12 @@ noexec=on: enable non-executable mappings (default)noexec=off: disable non-executable mappings
Documentation/admin-guide/kernel-parameters.txt +1
-0+ no_cet_ibt [X86-64] Disable indirect branch tracking for
Quoted 8 lines
Documentation/admin-guide/kernel-parameters.txt
Quoted 6 lines
user-mode
Documentation/admin-guide/kernel-parameters.txt +3
-0+ applications + + no_cet_shstk [X86-64] Disable shadow stack support for user-mode
Documentation/admin-guide/kernel-parameters.txt +1
-0+ applicationsHmm, not too consistent with "nosmap" below. Would it make sense to have cet=on/off/ibt/shstk instead?
We also have noxsave, noxsaveopt, noxsaves, etc. This style is more decisive? If "cet=" is preferred, we can change it later?
Yu-cheng
Randy Dunlap
reviewerRe: [RFC PATCH v3 05/24] Documentation/x86: Add CET description
In reply to Yu-cheng YuHi,
One spello:
Quoted 6 lines · Yu-cheng Yu
Documentation/x86/intel_cet.rst new file +0
-0
Documentation/x86/intel_cet.rst line 9
Quoted 9 lines
Documentation/x86/intel_cet.rst new file +9
-0@@ -0,0 +1,252 @@ +========================================= +Control Flow Enforcement Technology (CET) +========================================= + +[1] Overview +============ + +Control Flow Enforcement Technology (CET) provides protection against +return/jump-oriented programing (ROP) attacks. It can be implemented
programming
Quoted 4 lines
Documentation/x86/intel_cet.rst new file +4
-0+to protect both the kernel and applications. In the first phase, +only the user-mode protection is implemented for the 64-bit kernel. +Thirty-two bit applications are supported under the compatibility +mode.
Signature · 1 line
~Randy
RFC PATCH v3 6/24 x86/cet: Control protection exception handler
In reply to Yu-cheng YuA control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction.
The control protection exception handler works in a similar way as the
general protection fault handler.
--- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/traps.c | 58 ++++++++++++++++++++++++++++++++++++ 4 files changed, 66 insertions(+), 1 deletion(-)
arch/x86/entry/entry_64.S +1 -1
@@ -1000,7 +1000,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handling
arch/x86/include/asm/traps.h +3 -0
@@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void);
@@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long);#endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *, long); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long);
@@ -142,6 +144,7 @@ enum {X86_TRAP_AC ,/* 17, Alignment Check */ X86_TRAP_MC ,/* 18, Machine Check */ X86_TRAP_XF ,/* 19, SIMD Floating-Point Exception */ +X86_TRAP_CP =21 ,/* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ };
arch/x86/kernel/idt.c +4 -0
@@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = {#elif defined(CONFIG_X86_32 ) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32),#endif + +#ifdef CONFIG_X86_INTEL_CET + INTG(X86_TRAP_CP, control_protection), +#endif };/*
arch/x86/kernel/traps.c +58 -0
64 lines not on this page · raw message
Signature · 1 line
2.17.1
Jann Horn
reviewerRe: [RFC PATCH v3 06/24] x86/cet: Control protection exception handler
Quoted 9 lines · Yu-cheng Yu
A control protection exception is triggered when a control flow transfer attempt violated shadow stack or indirect branch tracking constraints. For example, the return address for a RET instruction differs from the safe copy on the shadow stack; or a JMP instruction arrives at a non- ENDBR instruction.
The control protection exception handler works in a similar way as the
general protection fault handler.
Is there a reason why all the code in this patch isn't #ifdef'ed away on builds that don't support CET? It looks like the CET handler is hooked up to the IDT conditionally, but the handler code is always built?
Quoted 113 lines
Signed-off-by Yu-cheng Yu <[email protected]>--- arch/x86/entry/entry_64.S | 2 +- arch/x86/include/asm/traps.h | 3 ++ arch/x86/kernel/idt.c | 4 +++ arch/x86/kernel/traps.c | 58 ++++++++++++++++++++++++++++++++++++ 4 files changed, 66 insertions(+), 1 deletion(-)arch/x86/entry/entry_64.S +1
-1@@ -1000,7 +1000,7 @@ idtentry spurious_interrupt_bug do_spurious_interrupt_bug has_error_code=0 idtentry coprocessor_error do_coprocessor_error has_error_code=0 idtentry alignment_check do_alignment_check has_error_code=1 idtentry simd_coprocessor_error do_simd_coprocessor_error has_error_code=0 - +idtentry control_protection do_control_protection has_error_code=1 /* * Reload gs selector with exception handlingarch/x86/include/asm/traps.h +3
-0@@ -26,6 +26,7 @@ asmlinkage void invalid_TSS(void); asmlinkage void segment_not_present(void); asmlinkage void stack_segment(void); asmlinkage void general_protection(void); +asmlinkage void control_protection(void); asmlinkage void page_fault(void); asmlinkage void async_page_fault(void); asmlinkage void spurious_interrupt_bug(void);@@ -77,6 +78,7 @@ dotraplinkage void do_stack_segment(struct pt_regs *, long); dotraplinkage void do_double_fault(struct pt_regs *, long);#endif dotraplinkage void do_general_protection(struct pt_regs *, long); +dotraplinkage void do_control_protection(struct pt_regs *, long); dotraplinkage void do_page_fault(struct pt_regs *, unsigned long); dotraplinkage void do_spurious_interrupt_bug(struct pt_regs *, long); dotraplinkage void do_coprocessor_error(struct pt_regs *, long);@@ -142,6 +144,7 @@ enum {X86_TRAP_AC ,/* 17, Alignment Check */ X86_TRAP_MC ,/* 18, Machine Check */ X86_TRAP_XF ,/* 19, SIMD Floating-Point Exception */ +X86_TRAP_CP =21 ,/* 21 Control Protection Fault */ X86_TRAP_IRET = 32, /* 32, IRET Exception */ };arch/x86/kernel/idt.c +4
-0@@ -104,6 +104,10 @@ static const __initconst struct idt_data def_idts[] = {#elif defined(CONFIG_X86_32 ) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32),#endif + +#ifdef CONFIG_X86_INTEL_CET + INTG(X86_TRAP_CP, control_protection), +#endif };/* arch/x86/kernel/traps.c +41
-0@@ -578,6 +578,64 @@ do_general_protection(struct pt_regs *regs, long error_code) }NOKPROBE_SYMBOL (do_general_protection); +static const char *control_protection_err[] = +{ +"unknown" , +"near-ret" , +"far-ret/iret" , +"endbranch" , +"rstorssp" , +"setssbsy" , +}; + +/* +* When a control protection exception occurs, send a signal +* to the responsible application. Currently, control +* protection is only enabled for the user mode. This +* exception should not come from the kernel mode. +*/ +dotraplinkage void +do_control_protection (struct pt_regs *regs,long error_code) +{ +struct task_struct *tsk; + +RCU_LOCKDEP_WARN (!rcu_is_watching (),"entry code didn't wake RCU" ); +if (notify_die (DIE_TRAP ,"control protection fault" , regs, + error_code,X86_TRAP_CP ,SIGSEGV ) ==NOTIFY_STOP ) +return ; +cond_local_irq_enable (regs); + +if (!user_mode (regs)) +die ("kernel control protection fault" , regs, error_code); + +if (!static_cpu_has (X86_FEATURE_SHSTK ) && + !static_cpu_has (X86_FEATURE_IBT )) +WARN_ONCE (1 ,"CET is disabled but got control " +"protection fault\n" ); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr =X86_TRAP_CP ; + +if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && +printk_ratelimit ()) {
arch/x86/kernel/traps.c do_general_protection(struct pt_regs *regs, long error_code) · line 631
Quoted 10 lines
arch/x86/kernel/traps.c +10
-0+unsigned int max_err; + + max_err =ARRAY_SIZE (control_protection_err) -1 ; +if ((error_code <0 ) || (error_code > max_err)) + error_code =0 ; +pr_info ("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)" , + tsk->comm,task_pid_nr (tsk), + regs->ip, regs->sp, error_code, + control_protection_err[error_code]); +print_vma_addr (" in " , regs->ip);
Shouldn't this be using KERN_CONT, like other callers of print_vma_addr(), to get the desired output?
Quoted 4 lines
arch/x86/kernel/traps.c +4
-0+pr_cont ("\n" ); + } + +force_sig_info (SIGSEGV ,SEND_SIG_PRIV , tsk);
Re: [RFC PATCH v3 06/24] x86/cet: Control protection exception handler
Quoted 5 lines · Jann Horn
Is there a reason why all the code in this patch isn't #ifdef'ed away
on builds that don't support CET? It looks like the CET handler is hooked up to the IDT conditionally, but the handler code is always built?
Yes, in idt.c, it should have been:
#ifdef CONFIG_X86_64 INTG(X86_TRAP_CP, control_protection),
#endif
I will fix it.
Quoted 34 lines
Quoted 34 lines
arch/x86/kernel/traps.c +6
-0+dotraplinkage void +do_control_protection (struct pt_regs *regs,long error_code) +{ + struct task_struct *tsk; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "entry code didn'twake RCU");
+ if (notify_die(DIE_TRAP, "control protection fault", regs, + error_code,X86_TRAP_CP, SIGSEGV) ==NOTIFY_STOP)
+ return;
+ cond_local_irq_enable(regs);
+
+ if (!user_mode(regs))
+ die("kernel control protection fault", regs,error_code);arch/x86/kernel/traps.c +11
-0+ + if (!static_cpu_has(X86_FEATURE_SHSTK) && + !static_cpu_has(X86_FEATURE_IBT)) + WARN_ONCE(1, "CET is disabled but got control " + "protection fault\n"); + + tsk = current; + tsk->thread.error_code = error_code; + tsk->thread.trap_nr = X86_TRAP_CP; + + if (show_unhandled_signals && unhandled_signal(tsk,SIGSEGV) &&
+ printk_ratelimit()) {
+ unsigned intmax_err;
+
+ max_err =ARRAY_SIZE(control_protection_err)- 1; + if ((error_code< 0) || (error_code>max_err)) + error_code = 0;
Quoted 8 lines
Quoted 6 lines
+ pr_info("%s[%d] control protection ip:%lx sp:%lx error:%lx(%s)",
+ tsk->comm,task_pid_nr(tsk),
+ regs->ip,regs->sp,error_code, + control_protection_err[error_code]); + print_vma_addr(" in ",regs->ip);Shouldn't this be using
KERN_CONT, like other callers ofprint_vma_addr(), to get the desired output?
I will change it.
RFC PATCH v3 7/24 x86/cet/shstk: Add Kconfig option for user-mode shadow stack
In reply to Yu-cheng YuIntroduce Kconfig option X86_INTEL_SHADOW_STACK_USER.
An application has shadow stack protection when all the following are true:
(1) The kernel has X86_INTEL_SHADOW_STACK_USER enabled,
(2) The running processor supports the shadow stack,
(3) The application is built with shadow stack enabled tools & libs and, and at runtime, all dependent shared libs can support shadow stack.
If this kernel config option is enabled, but (2) or (3) above is not true, the application runs without the shadow stack protection. Existing legacy applications will continue to work without the shadow stack protection.
The user-mode shadow stack protection is only implemented for the 64-bit kernel. Thirty-two bit applications are supported under the compatibility mode.
--- arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Makefile | 7 +++++++ 2 files changed, 31 insertions(+)
arch/x86/Kconfig +24 -0
@@ -1913,6 +1913,30 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config X86_INTEL_CET + def_bool n + +config ARCH_HAS_SHSTK + def_bool n + +config X86_INTEL_SHADOW_STACK_USER + prompt "Intel Shadow Stack for user-mode" + def_bool n + depends on CPU_SUP_INTEL && X86_64 + select X86_INTEL_CET + select ARCH_HAS_SHSTK + ---help--- + Shadow stack provides hardware protection against program stack + corruption. Only when all the following are true will an application + have the shadow stack protection: the kernel supports it (i.e. this + feature is enabled), the application is compiled and linked with + shadow stack enabled, and the processor supports this feature. + When the kernel has this configuration enabled, existing non shadow + stack applications will continue to work, but without shadow stack + protection. + + If unsure, say y. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/Makefile +7 -0
@@ -152,6 +152,13 @@ ifdef CONFIG_X86_X32 endif export CONFIG_X86_X32_ABI +# Check assembler shadow stack suppot +ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + ifeq ($(call as-instr, saveprevssp, y),) + $(error CONFIG_X86_INTEL_SHADOW_STACK_USER not supported by the assembler) + endif +endif + # # If the function graph tracer is used with mcount instead of fentry, # '-maccumulate-outgoing-args' is needed to prevent a GCC bug
Signature · 1 line
2.17.1
RFC PATCH v3 8/24 mm: Introduce VM_SHSTK for shadow stack memory
In reply to Yu-cheng YuVM_SHSTK indicates a shadow stack memory area.
The shadow stack is implemented only for the 64-bit kernel.
--- include/linux/mm.h | 8 ++++++++ mm/internal.h | 8 ++++++++ 2 files changed, 16 insertions(+)
include/linux/mm.h +8 -0
@@ -224,11 +224,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -266,6 +268,12 @@ extern unsigned int kobjsize(const void *objp);# define VM_MPX VM_NONE#endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +# define VM_SHSTK VM_HIGH_ARCH_5 +#else +# define VM_SHSTK VM_NONE +#endif +#ifndef VM_GROWSUP # define VM_GROWSUP VM_NONE#endif
mm/internal.h +8 -0
@@ -280,6 +280,14 @@ static inline bool is_data_mapping(vm_flags_t flags)return (flags & (VM_WRITE |VM_SHARED |VM_STACK )) ==VM_WRITE ; } +/* +* Shadow stack area +*/ +static inline bool is_shstk_mapping (vm_flags_t flags) +{ +return (flags &VM_SHSTK ); +} +/* mm/util.c */ void __vma_link_list (struct mm_struct *mm,struct vm_area_struct *vma,struct vm_area_struct *prev,struct rb_node *rb_parent);
Signature · 1 line
2.17.1
RFC PATCH v3 9/24 x86/mm: Change _PAGE_DIRTY to _PAGE_DIRTY_HW
In reply to Yu-cheng YuWe are going to create _PAGE_DIRTY_SW for non-hardware, memory management purposes. Rename _PAGE_DIRTY to _PAGE_DIRTY_HW and _PAGE_BIT_DIRTY to _PAGE_BIT_DIRTY_HW to make these PTE dirty bits more clear. There are no functional changes in this patch.
--- arch/x86/include/asm/pgtable.h | 6 +++--- arch/x86/include/asm/pgtable_types.h | 17 +++++++++-------- arch/x86/kernel/relocate_kernel_64.S | 2 +- arch/x86/kvm/vmx.c | 2 +- 4 files changed, 14 insertions(+), 13 deletions(-)
arch/x86/include/asm/pgtable.h +3 -3
@@ -316,7 +316,7 @@ static inline pte_t pte_mkexec(pte_t pte)static inline pte_t pte_mkdirty (pte_t pte) { -return pte_set_flags (pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +return pte_set_flags (pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); }static inline pte_t pte_mkyoung (pte_t pte)
@@ -390,7 +390,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)static inline pmd_t pmd_mkdirty (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +return pmd_set_flags (pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); }static inline pmd_t pmd_mkdevmap (pmd_t pmd)
@@ -444,7 +444,7 @@ static inline pud_t pud_wrprotect(pud_t pud)static inline pud_t pud_mkdirty (pud_t pud) { -return pud_set_flags (pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +return pud_set_flags (pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); }static inline pud_t pud_mkdevmap (pud_t pud)
arch/x86/include/asm/pgtable_types.h +9 -8
@@ -15,7 +15,7 @@#define _PAGE_BIT_PWT 3/* page write through */ #define _PAGE_BIT_PCD 4/* page cache disabled */ #define _PAGE_BIT_ACCESSED 5/* was accessed (raised by CPU) */ -#define _PAGE_BIT_DIRTY 6/* was written to (raised by CPU) */ +#define _PAGE_BIT_DIRTY_HW 6/* was written to (raised by CPU) */ #define _PAGE_BIT_PSE 7/* 4 MB (or 2MB) page */ #define _PAGE_BIT_PAT 7/* on 4KB pages */ #define _PAGE_BIT_GLOBAL 8/* Global TLB entry PPro+ */
@@ -45,7 +45,7 @@#define _PAGE_PWT (_AT(pteval_t, 1) << _PAGE_BIT_PWT)#define _PAGE_PCD (_AT(pteval_t, 1) << _PAGE_BIT_PCD)#define _PAGE_ACCESSED (_AT(pteval_t, 1) << _PAGE_BIT_ACCESSED) -#define _PAGE_DIRTY (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY) +#define _PAGE_DIRTY_HW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_HW)#define _PAGE_PSE (_AT(pteval_t, 1) << _PAGE_BIT_PSE)#define _PAGE_GLOBAL (_AT(pteval_t, 1) << _PAGE_BIT_GLOBAL)#define _PAGE_SOFTW1 (_AT(pteval_t, 1) << _PAGE_BIT_SOFTW1)
@@ -73,7 +73,7 @@ _PAGE_PKEY_BIT3)#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE ) -#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED) +#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY_HW | _PAGE_ACCESSED)#else #define _PAGE_KNL_ERRATUM_MASK 0#endif
@@ -112,9 +112,9 @@#define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)#define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW)#define _KERNPG_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | \ - _PAGE_ACCESSED | _PAGE_DIRTY) + _PAGE_ACCESSED | _PAGE_DIRTY_HW)/* * Set of bits not changed in pte_modify. The pte's
@@ -123,7 +123,7 @@* pte_modify() does modify it. */ #define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ - _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \ + _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ _PAGE_SOFT_DIRTY)#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE)
@@ -168,7 +168,8 @@ enum page_cache_mode { _PAGE_ACCESSED)#define __PAGE_KERNEL_EXEC \ - (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED | _PAGE_GLOBAL) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY_HW | _PAGE_ACCESSED | \ + _PAGE_GLOBAL)#define __PAGE_KERNEL (__PAGE_KERNEL_EXEC | _PAGE_NX)#define __PAGE_KERNEL_RO (__PAGE_KERNEL & ~_PAGE_RW)
@@ -187,7 +188,7 @@ enum page_cache_mode {#define _PAGE_ENC (_AT(pteval_t, sme_me_mask))#define _KERNPG_TABLE (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | \ - _PAGE_DIRTY | _PAGE_ENC) + _PAGE_DIRTY_HW | _PAGE_ENC)#define _PAGE_TABLE (_KERNPG_TABLE | _PAGE_USER)#define __PAGE_KERNEL_ENC (__PAGE_KERNEL | _PAGE_ENC)
arch/x86/kernel/relocate_kernel_64.S +1 -1
@@ -17,7 +17,7 @@ */ #define PTR(x) (x << 3) -#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY) +#define PAGE_ATTR (_PAGE_PRESENT | _PAGE_RW | _PAGE_ACCESSED | _PAGE_DIRTY_HW) /* * control_page + KEXEC_CONTROL_CODE_MAX_SIZE
arch/x86/kvm/vmx.c +1 -1
@@ -5848,7 +5848,7 @@ static int init_rmode_identity_map(struct kvm *kvm)/* Set up identity-mapping pagetable for EPT in real mode */ for (i =0 ; i <PT32_ENT_PER_PAGE ; i++) { tmp = (i <<22 ) + (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER | - _PAGE_ACCESSED | _PAGE_DIRTY | _PAGE_PSE); + _PAGE_ACCESSED | _PAGE_DIRTY_HW | _PAGE_PSE); r =kvm_write_guest_page (kvm, identity_map_pfn, &tmp, i *sizeof (tmp),sizeof (tmp));if (r <0 )
Signature · 1 line
2.17.1
RFC PATCH v3 10/24 x86/mm: Introduce _PAGE_DIRTY_SW
In reply to Yu-cheng YuA RO and dirty PTE exists in the following cases:
(a) A page is modified and then shared with a fork()'ed child;
(b) A R/O page that has been COW'ed;
(c) A SHSTK page.
The processor does not read the dirty bit for (a) and (b), but checks the dirty bit for (c). To prevent the use of non-SHSTK memory as SHSTK, we introduce a spare bit of the 64-bit PTE as _PAGE_BIT_DIRTY_SW and use that for (a) and (b). This results to the following possible PTE settings:
Modified PTE: (R/W + DIRTY_HW) Modified and shared PTE: (R/O + DIRTY_SW) R/O PTE COW'ed: (R/O + DIRTY_SW) SHSTK PTE: (R/O + DIRTY_HW) SHSTK PTE COW'ed: (R/O + DIRTY_HW) SHSTK PTE shared: (R/O + DIRTY_SW)
Note that _PAGE_BIT_DRITY_SW is only used in R/O PTEs but not R/W PTEs.
When this patch is applied, there are six free bits left in the 64-bit PTE. There is no more free bit in the 32-bit PTE (except for PAE) and shadow stack is not implemented for the 32-bit kernel.
--- arch/x86/include/asm/pgtable.h | 129 ++++++++++++++++++++++----- arch/x86/include/asm/pgtable_types.h | 14 ++- include/asm-generic/pgtable.h | 21 +++++ 3 files changed, 142 insertions(+), 22 deletions(-)
arch/x86/include/asm/pgtable.h +109 -20
@@ -119,9 +119,9 @@ extern pmdval_t early_pmd_flags;* The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty (pte_t pte) +static inline bool pte_dirty (pte_t pte) { -return pte_flags (pte) & _PAGE_DIRTY; +return pte_flags (pte) & _PAGE_DIRTY_BITS; }
@@ -143,9 +143,9 @@ static inline int pte_young(pte_t pte)return pte_flags (pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty (pmd_t pmd) +static inline bool pmd_dirty (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_DIRTY; +return pmd_flags (pmd) & _PAGE_DIRTY_BITS; }static inline int pmd_young (pmd_t pmd)
@@ -153,9 +153,9 @@ static inline int pmd_young(pmd_t pmd)return pmd_flags (pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty (pud_t pud) +static inline bool pud_dirty (pud_t pud) { -return pud_flags (pud) & _PAGE_DIRTY; +return pud_flags (pud) & _PAGE_DIRTY_BITS; }static inline int pud_young (pud_t pud)
@@ -294,9 +294,23 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER ) +static inline pte_t pte_move_flags (pte_t pte,pteval_t from,pteval_t to) +{ +if (pte_flags (pte) & from) + pte =pte_set_flags (pte_clear_flags (pte, from), to); +return pte; +} +#else +static inline pte_t pte_move_flags (pte_t pte,pteval_t from,pteval_t to) +{ +return pte; +} +#endif +static inline pte_t pte_mkclean (pte_t pte) { -return pte_clear_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_DIRTY_BITS); }static inline pte_t pte_mkold (pte_t pte)
@@ -306,6 +320,7 @@ static inline pte_t pte_mkold(pte_t pte)static inline pte_t pte_wrprotect (pte_t pte) { + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW);return pte_clear_flags (pte, _PAGE_RW); }
@@ -316,9 +331,24 @@ static inline pte_t pte_mkexec(pte_t pte)static inline pte_t pte_mkdirty (pte_t pte) { +pteval_t dirty = (!IS_ENABLED (CONFIG_X86_INTEL_SHSTK_USER ) || +pte_write (pte)) ? _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; +return pte_set_flags (pte, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk (pte_t pte) +{ + pte =pte_clear_flags (pte, _PAGE_DIRTY_SW);return pte_set_flags (pte, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pte_dirty_hw (pte_t pte) +{ +return pte_flags (pte) & _PAGE_DIRTY_HW; +} +#endif +static inline pte_t pte_mkyoung (pte_t pte) {return pte_set_flags (pte, _PAGE_ACCESSED);
@@ -326,6 +356,7 @@ static inline pte_t pte_mkyoung(pte_t pte)static inline pte_t pte_mkwrite (pte_t pte) { + pte =pte_move_flags (pte, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW);return pte_set_flags (pte, _PAGE_RW); }
@@ -373,6 +404,20 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER ) +static inline pmd_t pmd_move_flags (pmd_t pmd,pmdval_t from,pmdval_t to) +{ +if (pmd_flags (pmd) & from) + pmd =pmd_set_flags (pmd_clear_flags (pmd, from), to); +return pmd; +} +#else +static inline pmd_t pmd_move_flags (pmd_t pmd,pmdval_t from,pmdval_t to) +{ +return pmd; +} +#endif +static inline pmd_t pmd_mkold (pmd_t pmd) {return pmd_clear_flags (pmd, _PAGE_ACCESSED);
@@ -380,19 +425,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd)static inline pmd_t pmd_mkclean (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_DIRTY_BITS); }static inline pmd_t pmd_wrprotect (pmd_t pmd) { + pmd =pmd_move_flags (pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW);return pmd_clear_flags (pmd, _PAGE_RW); }static inline pmd_t pmd_mkdirty (pmd_t pmd) { +pmdval_t dirty = (!IS_ENABLED (CONFIG_X86_INTEL_SHSTK_USER ) || + (pmd_flags (pmd) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; +return pmd_set_flags (pmd, dirty | _PAGE_SOFT_DIRTY); +} + +#ifdef CONFIG_ARCH_HAS_SHSTK +static inline pmd_t pmd_mkdirty_shstk (pmd_t pmd) +{ + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY_SW);return pmd_set_flags (pmd, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); } +static inline bool pmd_dirty_hw (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_DIRTY_HW; +} +#endif +static inline pmd_t pmd_mkdevmap (pmd_t pmd) {return pmd_set_flags (pmd, _PAGE_DEVMAP);
@@ -410,6 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)static inline pmd_t pmd_mkwrite (pmd_t pmd) { + pmd =pmd_move_flags (pmd, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW);return pmd_set_flags (pmd, _PAGE_RW); }
@@ -427,6 +490,20 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER ) +static inline pud_t pud_move_flags (pud_t pud,pudval_t from,pudval_t to) +{ +if (pud_flags (pud) & from) + pud =pud_set_flags (pud_clear_flags (pud, from), to); +return pud; +} +#else +static inline pud_t pud_move_flags (pud_t pud,pudval_t from,pudval_t to) +{ +return pud; +} +#endif +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);
@@ -434,17 +511,22 @@ static inline pud_t pud_mkold(pud_t pud)static inline pud_t pud_mkclean (pud_t pud) { -return pud_clear_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_DIRTY_BITS); }static inline pud_t pud_wrprotect (pud_t pud) { + pud =pud_move_flags (pud, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW);return pud_clear_flags (pud, _PAGE_RW); }static inline pud_t pud_mkdirty (pud_t pud) { -return pud_set_flags (pud, _PAGE_DIRTY_HW | _PAGE_SOFT_DIRTY); +pudval_t dirty = (!IS_ENABLED (CONFIG_X86_INTEL_SHSTK_USER ) || + (pud_flags (pud) & _PAGE_RW)) ? + _PAGE_DIRTY_HW:_PAGE_DIRTY_SW; + +return pud_set_flags (pud, dirty | _PAGE_SOFT_DIRTY); }static inline pud_t pud_mkdevmap (pud_t pud)
@@ -464,6 +546,7 @@ static inline pud_t pud_mkyoung(pud_t pud)static inline pud_t pud_mkwrite (pud_t pud) { + pud =pud_move_flags (pud, _PAGE_DIRTY_SW, _PAGE_DIRTY_HW);return pud_set_flags (pud, _PAGE_RW); }
@@ -595,19 +678,12 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_PAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PTE_PFN_MASK ); +if ((pte_write (pte) && !(pgprot_val (newprot) & _PAGE_RW))) +return pte_move_flags (__pte (val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW);return __pte (val); } -static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) -{ -pmdval_t val =pmd_val (pmd), oldval = val; - - val &= _HPAGE_CHG_MASK; - val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; - val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK ); -return __pmd (val); -} -/* mprotect needs to preserve PAT bits when updating vm_page_prot */ #define pgprot_modify pgprot_modifystatic inline pgprot_t pgprot_modify (pgprot_t oldprot,pgprot_t newprot)
@@ -1159,6 +1235,19 @@ static inline int pmd_write(pmd_t pmd)return pmd_flags (pmd) & _PAGE_RW; } +static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) +{ +pmdval_t val =pmd_val (pmd), oldval = val; + + val &= _HPAGE_CHG_MASK; + val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; + val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK ); +if ((pmd_write (pmd) && !(pgprot_val (newprot) & _PAGE_RW))) +return pmd_move_flags (__pmd (val), _PAGE_DIRTY_HW, + _PAGE_DIRTY_SW); +return __pmd (val); +} +#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEARstatic inline pmd_t pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)
arch/x86/include/asm/pgtable_types.h +12 -2
@@ -23,6 +23,7 @@#define _PAGE_BIT_SOFTW2 10/* " */ #define _PAGE_BIT_SOFTW3 11/* " */ #define _PAGE_BIT_PAT_LARGE 12/* On 2MB or 1GB pages */ +#define _PAGE_BIT_SOFTW5 57/* available for programmer */ #define _PAGE_BIT_SOFTW4 58/* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59/* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60/* Protection Keys, bit 2/4 */
@@ -34,6 +35,7 @@#define _PAGE_BIT_CPA_TEST _PAGE_BIT_SOFTW1#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3/* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +#define _PAGE_BIT_DIRTY_SW _PAGE_BIT_SOFTW5/* was written to */ /* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */
@@ -109,6 +111,14 @@#define _PAGE_DEVMAP (_AT(pteval_t, 0))#endif +#if defined(CONFIG_X86_INTEL_SHADOW_STACK_USER ) +#define _PAGE_DIRTY_SW (_AT(pteval_t, 1) << _PAGE_BIT_DIRTY_SW) +#else +#define _PAGE_DIRTY_SW (_AT(pteval_t, 0)) +#endif + +#define _PAGE_DIRTY_BITS (_PAGE_DIRTY_HW | _PAGE_DIRTY_SW) +#define _PAGE_PROTNONE (_AT(pteval_t, 1) << _PAGE_BIT_PROTNONE)#define _PAGE_TABLE_NOENC (_PAGE_PRESENT | _PAGE_RW | _PAGE_USER |\
@@ -122,9 +132,9 @@* instance, and is *not* included in this mask since * pte_modify() does modify it. */ -#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ +#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \ _PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY_HW | \ - _PAGE_SOFT_DIRTY) + _PAGE_DIRTY_SW | _PAGE_SOFT_DIRTY)#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE)/*
include/asm-generic/pgtable.h +21 -0
@@ -1127,4 +1127,25 @@ static inline bool arch_has_pfn_modify_check(void)#endif #endif +#ifndef CONFIG_ARCH_HAS_SHSTK +static inline pte_t pte_mkdirty_shstk (pte_t pte) +{ +return pte; +} +static inline bool pte_dirty_hw (pte_t pte) +{ +return false; +} + +static inline pmd_t pmd_mkdirty_shstk (pmd_t pmd) +{ +return pmd; +} + +static inline bool pmd_dirty_hw (pmd_t pmd) +{ +return false; +} +#endif +#endif /* _ASM_GENERIC_PGTABLE_H */
Signature · 1 line
2.17.1
RFC PATCH v3 11/24 drm/i915/gvt: Update _PAGE_DIRTY to _PAGE_DIRTY_BITS
In reply to Yu-cheng YuUpdate _PAGE_DIRTY to _PAGE_DIRTY_BITS in split_2MB_gtt_entry().
In order to support Control Flow Enforcement (CET), _PAGE_DIRTY is now _PAGE_DIRTY_HW or _PAGE_DIRTY_SW.
--- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/gvt/gtt.c +1 -1
@@ -1170,7 +1170,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, }/* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse (se); ops->clear_ips (se);
Signature · 1 line
2.17.1
RFC PATCH v3 12/24 x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Yu-cheng YuWhen Shadow Stack is enabled, the read-only and PAGE_DIRTY_HW PTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we use PAGE_DIRTY_SW.
Update ptep_set_wrprotect() and pmdp_set_wrprotect().
--- arch/x86/include/asm/pgtable.h | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
arch/x86/include/asm/pgtable.h +42 -0
@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +pte_t pte; +clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + +/* +* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has +* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. +*/ + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pte_at (mm, addr, ptep, pte); }#define flush_tlb_fix_spurious_fault (vma, address) do { } while (0)
@@ -1266,7 +1287,28 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +pmd_t pmd; +clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); + pmd = *pmdp; + +/* +* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has +* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. +*/ + pmd =pmd_move_flags (pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pmd_at (mm, addr, pmdp, pmd); }#define pud_write pud_write
Signature · 1 line
2.17.1
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 33 lines · Yu-cheng Yu
When Shadow Stack is enabled, the read-only and
PAGE_DIRTY_HWPTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we usePAGE_DIRTY_SW.Update
ptep_set_wrprotect()andpmdp_set_wrprotect().Signed-off-by Yu-cheng Yu <[email protected]>--- arch/x86/include/asm/pgtable.h | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)arch/x86/include/asm/pgtable.h +12
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +pte_t pte; +clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + +/* +* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1228
Quoted 10 lines
arch/x86/include/asm/pgtable.h +9
-0+* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has +* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. +*/ + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pte_at (mm, addr, ptep, pte); }
I don't understand why it's okay that you first atomically clear the RW bit, then atomically switch from DIRTY_HW to DIRTY_SW. Doesn't that mean that between the two atomic writes, another core can incorrectly see a shadow stack?
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 53 lines · Jann Horn
Quoted 51 lines · Yu-cheng Yu
When Shadow Stack is enabled, the read-only and
PAGE_DIRTY_HWPTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we usePAGE_DIRTY_SW.Update
ptep_set_wrprotect()andpmdp_set_wrprotect().Signed-off-by Yu-cheng Yu <[email protected]>---
arch/x86/include/asm/pgtable.h | 42
++++++++++++++++++++++++++++++++++1 file changed, 42 insertions(+)arch/x86/include/asm/pgtable.h +0
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm,unsigned long addr,
pte_t*ptep)
{
+ pte_t pte;
+
clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep;
+
+ /*
+ * Some processors can start a write, but ending up seeing + * a read-only PTE by the time they get to the Dirty bit. + * In this case, they will set the Dirty bit, leaving a + * read-only, Dirty PTE which looks like a Shadow Stack PTE.
+ *
+ * However, this behavior has been improved and will not occur
+ * on processors supporting Shadow Stacks. Without this + * guarantee, a transition to a non-present PTE and flush the
+ * TLB would be needed.
+ *
+ * When change a writable PTE to read-only and if the PTE has
+ *_PAGE_DIRTY_HWset, we move that bit to_PAGE_DIRTY_SWso
+ * that the PTE is not a valid Shadow Stack PTE. + */
+ pte =pte_move_flags(pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW);
Quoted 8 lines
Quoted 2 lines
+ set_pte_at(mm, addr, ptep, pte);
}I don't understand why it's okay that you first atomically clear the RW bit, then atomically switch from
DIRTY_HWtoDIRTY_SW. Doesn't that
mean that between the two atomic writes, another core can incorrectly
see a shadow stack?
Yes, we had that concern earlier and checked.
On processors supporting Shadow Stacks, that will not happen.
Yu-cheng
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Jann HornQuoted 23 lines · Jann Horn
Quoted 23 lines
arch/x86/include/asm/pgtable.h +18
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +pte_t pte; +clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + +/* +* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has +* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE.
Quoted 8 lines
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1228
Quoted 4 lines
arch/x86/include/asm/pgtable.h +3
-0+*/ + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pte_at (mm, addr, ptep, pte); }I don't understand why it's okay that you first atomically clear the RW bit, then atomically switch from
DIRTY_HWtoDIRTY_SW. Doesn't that mean that between the two atomic writes, another core can incorrectly see a shadow stack?
Good point.
This could result in a spurious shadow-stack fault, or allow a shadow-stack write to the page in the transient state.
But, the shadow-stack permissions are more restrictive than what could be in the TLB at this point, so I don't think there's a real security implication here.
The only trouble is handling the spurious shadow-stack fault. The alternative is to go !Present for a bit, which we would probably just handle fine in the existing page fault code.
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 34 lines · Dave Hansen
Quoted 23 lines · Jann Horn
Quoted 23 lines
arch/x86/include/asm/pgtable.h +18
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +pte_t pte; +clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + +/* +* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has +* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. Quoted 8 lines
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1228
Quoted 4 lines
arch/x86/include/asm/pgtable.h +3
-0+*/ + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pte_at (mm, addr, ptep, pte); }I don't understand why it's okay that you first atomically clear the RW bit, then atomically switch from
DIRTY_HWtoDIRTY_SW. Doesn't that mean that between the two atomic writes, another core can incorrectly see a shadow stack?
Quoted 8 lines
Good point.
This could result in a spurious shadow-stack fault, or allow a shadow-stack write to the page in the transient state.
But, the shadow-stack permissions are more restrictive than what could be in the TLB at this point, so I don't think there's a real security implication here.
How about this:
Three threads (A, B, C) run with the same CR3.
1. a dirty+writable PTE is placed directly in front of B's shadow stack. (this can happen, right? or is there a guard page?)
2. C's TLB caches the dirty+writable PTE.
3. A performs some syscall that triggers ptep_set_wrprotect().
4. A's syscall calls clear_bit().
5. B's TLB caches the transient shadow stack.
[now C has write access to B's transiently-extended shadow stack]
6. B recurses into the transiently-extended shadow stack
7. C overwrites the transiently-extended shadow stack area.
8. B returns through the transiently-extended shadow stack, giving the attacker instruction pointer control in B.
9. A's syscall broadcasts a TLB flush.
Sure, it's not exactly an easy race and probably requires at least some black timing magic to exploit, if it's exploitable at all - but still. This seems suboptimal.
Quoted 3 lines
The only trouble is handling the spurious shadow-stack fault. The alternative is to go !Present for a bit, which we would probably just handle fine in the existing page fault code.
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 14 lines · Jann Horn
Three threads (A, B, C) run with the same CR3.
1. a dirty+writable PTE is placed directly in front of B's shadow stack. (this can happen, right? or is there a guard page?)
2. C's TLB caches the dirty+writable PTE.
3. A performs some syscall that triggersptep_set_wrprotect().
4. A's syscall callsclear_bit().
5. B's TLB caches the transient shadow stack.
[now C has write access to B's transiently-extended shadow stack]
6. B recurses into the transiently-extended shadow stack
7. C overwrites the transiently-extended shadow stack area.
8. B returns through the transiently-extended shadow stack, giving the attacker instruction pointer control in B.
9. A's syscall broadcasts a TLB flush.
Heh, that's a good point. The shadow stack permissions are *not* strictly reduced because a page getting marked as shadow-stack has *increased* permissions when being used as a shadow stack. Fun.
For general hardening, it seems like we want to ensure that there's a guard page at the bottom of the shadow stack. Yu-cheng, do we have a guard page?
But, to keep B's TLB from picking up the entry, I think we can just make it !Present for a moment. No TLB can cache it, and I believe the same "don't set Dirty on a !Writable entry" logic also holds for !Present (modulo a weird erratum or two).
If we do that, we just need to make sure that the fault handler knows it can get spurious faults from it, and might even run into the !Present PTE for a moment. It might be a bit confusing because it won't be a PROT_NONE, migration, or swap PTE, but will be !Present. We'll also have to make sure that we're doing this in a way which is friendly to the L1TF PTE handling.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 21 lines · Dave Hansen
Quoted 16 lines · Jann Horn
Three threads (A, B, C) run with the same CR3.
1. a dirty+writable PTE is placed directly in front of B's shadow stack.
(this can happen, right? or is there a guard page?)
2. C's TLB caches the dirty+writable PTE.
3. A performs some syscall that triggersptep_set_wrprotect().
4. A's syscall callsclear_bit().
5. B's TLB caches the transient shadow stack.
[now C has write access to B's transiently-extended shadow stack]
6. B recurses into the transiently-extended shadow stack
7. C overwrites the transiently-extended shadow stack area.
8. B returns through the transiently-extended shadow stack, giving the attacker instruction pointer control in B.
9. A's syscall broadcasts a TLB flush.Heh, that's a good point. The shadow stack permissions are *not* strictly reduced because a page getting marked as shadow-stack has *increased* permissions when being used as a shadow stack. Fun.
Quoted 5 lines
For general hardening, it seems like we want to ensure that there's a
guard page at the bottom of the shadow stack. Yu-cheng, do we have a
guard page?
We don't have the guard page now, but there is a shadow stack token there, which cannot be used as a return address.
Yu-cheng
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 2 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token there, which cannot be used as a return address.
The overall concern is that we could overflow into a page that we did not intend. Either another actual shadow stack or something that a page that the attacker constructed, like the transient scenario Jann described.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token
there, which cannot be used as a return address.
Quoted 7 lines
The overall concern is that we could overflow into a page that we did
not intend. Either another actual shadow stack or something that a page
that the attacker constructed, like the transient scenario Jann described.
A task could go beyond the bottom of its shadow stack by doing either 'ret' or 'incssp'. If it is the 'ret' case, the token prevents it. If it is the 'incssp' case, a guard page cannot prevent it entirely, right?
Yu-cheng
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 15 lines · Yu-cheng Yu
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token
there, which cannot be used as a return address.Quoted 7 lines
The overall concern is that we could overflow into a page that we did
not intend. Either another actual shadow stack or something that a page
that the attacker constructed, like the transient scenario Jann described.
Quoted 4 lines
A task could go beyond the bottom of its shadow stack by doing either 'ret' or 'incssp'. If it is the 'ret' case, the token prevents it.
If it is the 'incssp' case, a guard page cannot prevent it entirely,
right?
I mean the other direction, on "call".
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 21 lines · Jann Horn
Quoted 19 lines · Yu-cheng Yu
Quoted 16 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token
there, which cannot be used as a return address.The overall concern is that we could overflow into a page that we
did
not intend. Either another actual shadow stack or something that a
page
that the attacker constructed, like the transient scenario Jann described.
Quoted 8 lines
Quoted 7 lines
A task could go beyond the bottom of its shadow stack by doing either
'ret' or 'incssp'. If it is the 'ret' case, the token prevents it.
If it is the 'incssp' case, a guard page cannot prevent it entirely,
right?I mean the other direction, on "call".
In the flow you described, if C writes to the overflow page before B gets in with a 'call', the return address is still correct for B. To make an attack, C needs to write again before the TLB flush. I agree that is possible.
Assume we have a guard page, can someone in the short window do recursive calls in B, move ssp to the end of the guard page, and trigger the same again? He can simply take the incssp route.
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 32 lines · Yu-cheng Yu
Quoted 21 lines · Jann Horn
Quoted 19 lines · Yu-cheng Yu
Quoted 16 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token
there, which cannot be used as a return address.The overall concern is that we could overflow into a page that we
did
not intend. Either another actual shadow stack or something that a
page
that the attacker constructed, like the transient scenario Jann described.Quoted 8 lines
Quoted 7 lines
A task could go beyond the bottom of its shadow stack by doing either
'ret' or 'incssp'. If it is the 'ret' case, the token prevents it.
If it is the 'incssp' case, a guard page cannot prevent it
entirely,
right?I mean the other direction, on "call".
Quoted 8 lines
In the flow you described, if C writes to the overflow page before B gets in with a 'call', the return address is still correct for B. To make an attack, C needs to write again before the TLB flush. I agree that is possible.
Assume we have a guard page, can someone in the short window do recursive calls in B, move ssp to the end of the guard page, and trigger the same again? He can simply take the incssp route.
I don't understand what you're saying. If the shadow stack is between guard pages, you should never be able to move SSP past that area's guard pages without an appropriate shadow stack token (not even with INCSSP, since that has a maximum range of PAGE_SIZE/2), and therefore, it shouldn't matter whether memory outside that range is incorrectly marked as shadow stack. Am I missing something?
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 2 lines · Jann Horn
On Thu, Aug 30, 2018 at 10:25 PM Yu-cheng Yu <[email protected]> wrote:
...
Quoted 11 lines
Quoted 11 lines
In the flow you described, if C writes to the overflow page before B
gets in with a 'call', the return address is still correct for B. To
make an attack, C needs to write again before the TLB flush. I agree
that is possible.Assume we have a guard page, can someone in the short window do recursive calls in B, move ssp to the end of the guard page, and trigger the same again? He can simply take the incssp route.
Quoted 8 lines
I don't understand what you're saying. If the shadow stack is between
guard pages, you should never be able to move SSP past that area's guard pages without an appropriate shadow stack token (not even with INCSSP, since that has a maximum range of PAGE_SIZE/2), and therefore,
it shouldn't matter whether memory outside that range is incorrectly marked as shadow stack. Am I missing something?
INCSSP has a range of 256, but we can do multiple of that. But I realize the key is not to have the transient SHSTK page at all. The guard page is !pte_write() and even we have flaws in ptep_set_wrprotect(), there will not be any transient SHSTK pages. I will add guard pages to both ends.
Still thinking how to fix ptep_set_wrprotect().
Yu-cheng
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 25 lines · Yu-cheng Yu
Quoted 2 lines · Jann Horn
On Thu, Aug 30, 2018 at 10:25 PM Yu-cheng Yu <[email protected]> wrote:
...
Quoted 11 lines
Quoted 11 lines
In the flow you described, if C writes to the overflow page before B
gets in with a 'call', the return address is still correct for B. To
make an attack, C needs to write again before the TLB flush. I agree
that is possible.Assume we have a guard page, can someone in the short window do recursive calls in B, move ssp to the end of the guard page, and trigger the same again? He can simply take the incssp route.
Quoted 8 lines
I don't understand what you're saying. If the shadow stack is between
guard pages, you should never be able to move SSP past that area's guard pages without an appropriate shadow stack token (not even with INCSSP, since that has a maximum range of PAGE_SIZE/2), and therefore,
it shouldn't matter whether memory outside that range is incorrectly marked as shadow stack. Am I missing something?
Quoted 7 lines
INCSSP has a range of 256, but we can do multiple of that. But I realize the key is not to have the transient SHSTK page at all. The guard page is !pte_write() and even we have flaws in
ptep_set_wrprotect(), there will not be any transient SHSTK pages. I will add guard pages to both ends.Still thinking how to fix
ptep_set_wrprotect().
cmpxchg loop? Or is that slow?
Jann Horn
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 33 lines · Jann Horn
Quoted 31 lines · Yu-cheng Yu
Quoted 2 lines · Jann Horn
On Thu, Aug 30, 2018 at 10:25 PM Yu-cheng Yu <[email protected]> wrote:
...
Quoted 11 lines
Quoted 11 lines
In the flow you described, if C writes to the overflow page before B
gets in with a 'call', the return address is still correct for B. To
make an attack, C needs to write again before the TLB flush. I agree
that is possible.Assume we have a guard page, can someone in the short window do recursive calls in B, move ssp to the end of the guard page, and trigger the same again? He can simply take the incssp route.
Quoted 8 lines
I don't understand what you're saying. If the shadow stack is between
guard pages, you should never be able to move SSP past that area's guard pages without an appropriate shadow stack token (not even with INCSSP, since that has a maximum range of PAGE_SIZE/2), and therefore,
it shouldn't matter whether memory outside that range is incorrectly marked as shadow stack. Am I missing something?INCSSP has a range of 256, but we can do multiple of that. But I realize the key is not to have the transient SHSTK page at all. The guard page is !pte_write() and even we have flaws in
ptep_set_wrprotect(), there will not be any transient SHSTK pages. I will add guard pages to both ends.
Quoted 3 lines
Quoted 1 line
Still thinking how to fix
ptep_set_wrprotect().cmpxchg loop? Or is that slow?
Something like this:
static inline void ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr, pte_t *ptep)
{
pte_t pte = READ_ONCE(*ptep), new_pte;
/* ... your comment about not needing a TLB shootdown here ... */
do {
pte = pte_wrprotect(pte);
/* note: relies on _PAGE_DIRTY_HW < _PAGE_DIRTY_SW */
/* dirty direct bit-twiddling; you can probably write
this in a nicer way */
pte.pte |= (pte.pte & _PAGE_DIRTY_HW) >>
_PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW;
pte.pte &= ~_PAGE_DIRTY_HW;
pte = cmpxchg(ptep, pte, new_pte);
} while (pte != new_pte);
}I think this has the advantage of not generating weird spurious pagefaults. It's not compatible with Xen PV, but I'm guessing that this whole feature isn't going to support Xen PV anyway? So you could switch between two implementations of ptep_set_wrprotect using the pvop mechanism or so - one for environments that support shadow stacks, one for all other environments.
Or is there some arcane reason why cmpxchg doesn't work here the way I think it should?
Peter Zijlstra
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 10 lines · Jann Horn
do {
pte = pte_wrprotect(pte);
/* note: relies on _PAGE_DIRTY_HW < _PAGE_DIRTY_SW */
/* dirty direct bit-twiddling; you can probably write
this in a nicer way */
pte.pte |= (pte.pte & _PAGE_DIRTY_HW) >>
_PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW;
pte.pte &= ~_PAGE_DIRTY_HW;
pte = cmpxchg(ptep, pte, new_pte);
} while (pte != new_pte);Please use the form:
pte_t new_pte, pte = READ_ONCE(*ptep);
do {
new_pte = /* ... */;
} while (!try_cmpxchg(ptep, &pte, new_pte);Also, this will fail to build on i386-PAE, but I suspect this code will be under some CONFIG option specific to x86_64 anyway.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 16 lines · Peter Zijlstra
Quoted 13 lines · Jann Horn
do {
pte =pte_wrprotect(pte);
/* note: relies on_PAGE_DIRTY_HW<_PAGE_DIRTY_SW*/
/* dirty direct bit-twiddling; you can probably write
this in a nicer way */
pte.pte |= (pte.pte &_PAGE_DIRTY_HW) >>_PAGE_BIT_DIRTY_HW<<_PAGE_BIT_DIRTY_SW;
pte.pte &= ~_PAGE_DIRTY_HW;
pte =cmpxchg(ptep, pte, new_pte);
} while (pte !=new_pte);Please use the form:
Quoted 8 lines
pte_t new_pte, pte = READ_ONCE(*ptep); do { new_pte = /* ... */; } while (!try_cmpxchg(ptep, &pte, new_pte);Also, this will fail to build on i386-PAE, but I suspect this code will
be under some CONFIG option specific tox86_64anyway.
Thanks! I will work on it.
Yu-cheng
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 6 lines · Yu-cheng Yu
Please use the form:
pte_t new_pte, pte = READ_ONCE(*ptep); do { new_pte = /* ... */; } while (!try_cmpxchg(ptep, &pte, new_pte);
It's probably also worth doing some testing to see if you can detect the cost of the cmpxchg. It's definitely more than the old code.
A loop that does mprotect(PROT_READ) followed bymprotect(PROT_READ|PROT_WRITE) should do it.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 8 lines · Dave Hansen
Quoted 7 lines · Yu-cheng Yu
Please use the form:
pte_t new_pte, pte = READ_ONCE(*ptep); do { new_pte = /* ... */; } while (!try_cmpxchg(ptep, &pte, new_pte);
Quoted 6 lines
It's probably also worth doing some testing to see if you can detect the
cost of the cmpxchg. It's definitely more than the old code.A loop that does
mprotect(PROT_READ)followed bymprotect(PROT_READ|PROT_WRITE)should do it.
I created the test,
https://github.com/yyu168/cet-smoke-test/blob/quick/quick/mprotect_bench.c
then realized this won't work.
To trigger a race in ptep_set_wrprotect(), we need to fork from one of three pthread siblings.
Or do we measure only how much this affects fork? If there is no racing, the effect should be minimal.
Yu-cheng
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 5 lines · Yu-cheng Yu
To trigger a race in
ptep_set_wrprotect(), we need to fork from one of three pthread siblings.Or do we measure only how much this affects fork? If there is no racing, the effect should be minimal.
We don't need a race.
I think the cmpxchg will be slower, even without a race, than the code that was there before. The cmpxchg is a simple, straightforward solution, but we're putting it in place of a plain memory write, which is suboptimal.
But, before I nitpick the performance, I wanted to see if we could even detect a delta.
Peter Zijlstra
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 7 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
To trigger a race in
ptep_set_wrprotect(), we need to fork from one of three pthread siblings.Or do we measure only how much this affects fork? If there is no racing, the effect should be minimal.
Quoted 6 lines
We don't need a race.
I think the cmpxchg will be slower, even without a race, than the code that was there before. The cmpxchg is a simple, straightforward solution, but we're putting it in place of a plain memory write, which is suboptimal.
Note quite, the clear_bit() is LOCK prefixed.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Dave Hansen
Quoted 6 lines · Yu-cheng Yu
To trigger a race in
ptep_set_wrprotect(), we need to fork from one of three pthread siblings.Or do we measure only how much this affects fork? If there is no racing, the effect should be minimal.
Quoted 7 lines
Quoted 6 lines
We don't need a race.
I think the cmpxchg will be slower, even without a race, than the code that was there before. The cmpxchg is a simple, straightforward solution, but we're putting it in place of a plain memory write, which is suboptimal.
Note quite, the
clear_bit()is LOCK prefixed.
With the updated ptep_set_wrprotect() below, I did MADV_WILLNEED to a shadow stack of 8 MB, then 10,000 fork()'s, but could not prove it is more or less efficient than the other. So can we say this is probably fine in terms of efficiency?
Yu-cheng
arch/x86/include/asm/pgtable.h +0 -0
@@ -1203,7 +1203,36 @@ static inline pte_t ptep_get_and_clear_full(struct
mm_struct *mm,
static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) {
+#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER
+ pte_t new_pte, pte = READ_ONCE(*ptep);
+
+ /*
+ * Some processors can start a write, but end up
+ * seeing a read-only PTE by the time they get
+ * to the Dirty bit. In this case, they will
+ * set the Dirty bit, leaving a read-only, Dirty
+ * PTE which looks like a Shadow Stack PTE.
+ *
+ * However, this behavior has been improved and
+ * will not occur on processors supporting
+ * Shadow Stacks. Without this guarantee, a
+ * transition to a non-present PTE and flush the
+ * TLB would be needed.
+ *
+ * When changing a writable PTE to read-only and
+ * if the PTE has _PAGE_DIRTY_HW set, we move
+ * that bit to _PAGE_DIRTY_SW so that the PTE is
+ * not a valid Shadow Stack PTE.
+ */
+ do {
+ new_pte = pte_wrprotect(pte);
+ new_pte.pte |= (new_pte.pte & _PAGE_DIRTY_HW) >>
+ _PAGE_BIT_DIRTY_HW << _PAGE_BIT_DIRTY_SW;
+ new_pte.pte &= ~_PAGE_DIRTY_HW;
+ } while (!try_cmpxchg(ptep, &pte, new_pte));
+#else
clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +#endif
}
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 4 lines · Yu-cheng Yu
With the updated
ptep_set_wrprotect()below, I didMADV_WILLNEEDto a shadow stack of 8 MB, then 10,000 fork()'s, but could not prove it is more or less efficient than the other. So can we say this is probably fine in terms of efficiency?
Well, the first fork() will do all the hard work. I don't think subsequent fork()s will be affected.
Did you do something to ensure this code was being run?
I would guess that a loop like this:
for (i = 0; i < 10000; i++) {
mprotect(addr, len, PROT_READ);
mprotect(addr, len, PROT_READ|PROT_WRITE);
}might show it better.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 8 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
With the updated
ptep_set_wrprotect()below, I didMADV_WILLNEEDto a shadow stack of 8 MB, then 10,000 fork()'s, but could not prove it is more or less efficient than the other. So can we say this is probably fine in terms of efficiency?Well, the first
fork()will do all the hard work. I don't think subsequent fork()s will be affected.
Are you talking about a recent commit:
1b2de5d0 mm/cow: don't bother write protecting already write-protected pages
With that, subsequent fork()s will not do all the hard work. However, I have not done that for shadow stack PTEs (do we want to do that?). I think the additional benefit for shadow stack is small?
Quoted 11 lines
Did you do something to ensure this code was being run?
I would guess that a loop like this:
for (i = 0; i < 10000; i++) { mprotect(addr, len, PROT_READ); mprotect(addr, len, PROT_READ|PROT_WRITE); }might show it better.
Would mprotect() do copy_one_pte()? Otherwise it will not go through ptep_set_wrprotect()?
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 7 lines · Yu-cheng Yu
Quoted 6 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
With the updated
ptep_set_wrprotect()below, I didMADV_WILLNEEDto a shadow stack of 8 MB, then 10,000 fork()'s, but could not prove it is more or less efficient than the other. So can we say this is probably fine in terms of efficiency?
BTW, I wasn't particularly concerned about shadow stacks. Plain old memory is affected by this change too. Right?
Quoted 10 lines
Quoted 2 lines
Well, the first
fork()will do all the hard work. I don't think subsequent fork()s will be affected.Are you talking about a recent commit:
1b2de5d0mm/cow: don't bother write protecting already write-protected pagesWith that, subsequent fork()s will not do all the hard work. However, I have not done that for shadow stack PTEs (do we want to do that?). I think the additional benefit for shadow stack is small?
You're right. mprotect() doesn't use this path.
But, that reminds me, can you take a quick look at change_pte_range() and double-check that it's not affected by this issue?
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Jann HornQuoted 20 lines · Jann Horn
Quoted 17 lines · Yu-cheng Yu
Quoted 14 lines · Dave Hansen
Quoted 5 lines · Yu-cheng Yu
We don't have the guard page now, but there is a shadow stack token
there, which cannot be used as a return address.The overall concern is that we could overflow into a page that we did
not intend. Either another actual shadow stack or something that a page
that the attacker constructed, like the transient scenario Jann described.
Quoted 6 lines
Quoted 4 lines
A task could go beyond the bottom of its shadow stack by doing either 'ret' or 'incssp'. If it is the 'ret' case, the token prevents it. If it is the 'incssp' case, a guard page cannot prevent it entirely, right?
I mean the other direction, on "call".
I still think that shadow stacks should work just like mmap and that mmap should learn to add guard pages for all non-MAP_FIXED allocations.
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Dave HansenQuoted 24 lines · Dave Hansen
Quoted 16 lines · Jann Horn
Three threads (A, B, C) run with the same CR3.
1. a dirty+writable PTE is placed directly in front of B's shadow stack. (this can happen, right? or is there a guard page?)
2. C's TLB caches the dirty+writable PTE.
3. A performs some syscall that triggersptep_set_wrprotect().
4. A's syscall callsclear_bit().
5. B's TLB caches the transient shadow stack.
[now C has write access to B's transiently-extended shadow stack]
6. B recurses into the transiently-extended shadow stack
7. C overwrites the transiently-extended shadow stack area.
8. B returns through the transiently-extended shadow stack, giving the attacker instruction pointer control in B.
9. A's syscall broadcasts a TLB flush.Heh, that's a good point. The shadow stack permissions are *not* strictly reduced because a page getting marked as shadow-stack has *increased* permissions when being used as a shadow stack. Fun.
Quoted 8 lines
For general hardening, it seems like we want to ensure that there's a guard page at the bottom of the shadow stack. Yu-cheng, do we have a guard page?
But, to keep B's TLB from picking up the entry, I think we can just make it !Present for a moment. No TLB can cache it, and I believe the same "don't set Dirty on a !Writable entry" logic also holds for !Present (modulo a weird erratum or two).
Can we get documentation? Pretty please?
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 5 lines · Andy Lutomirski
Quoted 4 lines
But, to keep B's TLB from picking up the entry, I think we can just make it !Present for a moment. No TLB can cache it, and I believe the same "don't set Dirty on a !Writable entry" logic also holds for !Present (modulo a weird erratum or two).
Can we get documentation? Pretty please?
The accessed bit description in the SDM looks pretty good to me today:
Quoted 3 lines
Whenever the processor uses a paging-structure entry as part of linear-address translation, it sets the accessed flag in that entry (if it is not already set).
If it's !Present, it can't used as part of a translation so can't be set. I think that covers the thing I was unsure about.
But, Dirty is a bit, er, muddier, but mostly because it only gets set on leaf entries:
Quoted 5 lines
Whenever there is a write to a linear address, the processor sets the dirty flag (if it is not already set) in the paging- structure entry that identifies the final physical address for the linear address (either a PTE or a paging-structure entry in which the PS flag is 1).
That little hunk will definitely need to get updated with something like:
On processors enumerating support for CET, the processor will on set the dirty flag on paging structure entries in which the W flag is 1.
Andy Lutomirski
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 24 lines · Dave Hansen
Quoted 5 lines · Andy Lutomirski
Quoted 4 lines
But, to keep B's TLB from picking up the entry, I think we can just make it !Present for a moment. No TLB can cache it, and I believe the same "don't set Dirty on a !Writable entry" logic also holds for !Present (modulo a weird erratum or two).
Can we get documentation? Pretty please?
The accessed bit description in the SDM looks pretty good to me today:
Quoted 3 lines
Whenever the processor uses a paging-structure entry as part of linear-address translation, it sets the accessed flag in that entry (if it is not already set).
If it's !Present, it can't used as part of a translation so can't be set. I think that covers the thing I was unsure about.
But, Dirty is a bit, er, muddier, but mostly because it only gets set on leaf entries:
Quoted 5 lines
Whenever there is a write to a linear address, the processor sets the dirty flag (if it is not already set) in the paging- structure entry that identifies the final physical address for the linear address (either a PTE or a paging-structure entry in which the PS flag is 1).
Quoted 5 lines
That little hunk will definitely need to get updated with something like:
On processors enumerating support for CET, the processor will on set the dirty flag on paging structure entries in which the W flag is 1.
Can we get something much stronger, perhaps? Like this:
On processors enumerating support for CET, the processor will write to the accessed and/or dirty flags atomically, as if using the LOCK CMPXCHG instruction. The memory access, any cached entries in any paging-structure caches, and the values in the paging-structure entry before and after writing the A and/or D bits will all be consistent.
I'm sure this could be worded better. The point is that the CPU should, atomically, load the PTE, check if it allows the access, set A and/or D appropriately, write the new value to the TLB, and use that value for the access. This is clearly a little bit slower than what old CPUs could do when writing to an already-in-TLB writable non-dirty entry, but new CPUs are going to have to atomically check the W bit. (I assume that even old CPUs will *atomically* set the D bit as if by LOCK BTS, but this is all very vague in the SDM IIRC.)
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 7 lines · Andy Lutomirski
On Thu, Aug 30, 2018 at 11:55 AM, Dave Hansen
Quoted 5 lines
That little hunk will definitely need to get updated with something like:
On processors enumerating support for CET, the processor will on set the dirty flag on paging structure entries in which the W flag is 1.
Quoted 7 lines
Can we get something much stronger, perhaps? Like this:
On processors enumerating support for CET, the processor will write to the accessed and/or dirty flags atomically, as if using the LOCK CMPXCHG instruction. The memory access, any cached entries in any paging-structure caches, and the values in the paging-structure entry before and after writing the A and/or D bits will all be consistent.
There's some talk of this already in: 8.1.2.1 Automatic Locking:
Quoted 4 lines
When updating page-directory and page-table entries — When updating page-directory and page-table entries, the processor uses locked cycles to set the accessed and dirty flag in the page-directory and page-table entries.
As for the A/D consistency, I'll see if I can share that before it hits the SDM for real and see if it's sufficient for everybody.
Randy Dunlap
reviewerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Yu-cheng YuQuoted 16 lines · Yu-cheng Yu
When Shadow Stack is enabled, the read-only and
PAGE_DIRTY_HWPTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we usePAGE_DIRTY_SW.Update
ptep_set_wrprotect()andpmdp_set_wrprotect().Signed-off-by Yu-cheng Yu <[email protected]>--- arch/x86/include/asm/pgtable.h | 42 ++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)arch/x86/include/asm/pgtable.h +0
-0
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1212
Quoted 10 lines
arch/x86/include/asm/pgtable.h +6
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +pte_t pte; +clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); + pte = *ptep; + +/* +* Some processors can start a write, but ending up seeing
but end up seeing
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1222
Quoted 10 lines
arch/x86/include/asm/pgtable.h +10
-0+* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has
changing
Quoted 9 lines
arch/x86/include/asm/pgtable.h +5
-0+* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. +*/ + pte =pte_move_flags (pte, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pte_at (mm, addr, ptep, pte); }#define flush_tlb_fix_spurious_fault (vma, address) do { } while (0)
arch/x86/include/asm/pgtable.h static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, · line 1296
Quoted 10 lines
arch/x86/include/asm/pgtable.h +6
-0@@ -1266,7 +1287,28 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +pmd_t pmd; +clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); + pmd = *pmdp; + +/* +* Some processors can start a write, but ending up seeing
but end up seeing
arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1222
Quoted 10 lines
arch/x86/include/asm/pgtable.h +10
-0+* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed. +* +* When change a writable PTE to read-only and if the PTE has
changing
Quoted 9 lines
arch/x86/include/asm/pgtable.h +5
-0+* _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SW so +* that the PTE is not a valid Shadow Stack PTE. +*/ + pmd =pmd_move_flags (pmd, _PAGE_DIRTY_HW, _PAGE_DIRTY_SW); +set_pmd_at (mm, addr, pmdp, pmd); }#define pud_write pud_write
Signature · 1 line
~Randy
Re: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
Quoted 98 lines · Randy Dunlap
Quoted 31 lines · Yu-cheng Yu
When Shadow Stack is enabled, the read-only and
PAGE_DIRTY_HWPTE setting is reserved only for the Shadow Stack. To track dirty of non-Shadow Stack read-only PTEs, we usePAGE_DIRTY_SW.Update
ptep_set_wrprotect()andpmdp_set_wrprotect().Signed-off-by Yu-cheng Yu <[email protected]>---
arch/x86/include/asm/pgtable.h | 42
++++++++++++++++++++++++++++++++++1 file changed, 42 insertions(+)arch/x86/include/asm/pgtable.h +0
-0@@ -1203,7 +1203,28 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, static inline void ptep_set_wrprotect(struct mm_struct *mm,unsigned long addr,
pte_t*ptep)
{arch/x86/include/asm/pgtable.h +2
-0+pte_t pte; +
clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte);arch/x86/include/asm/pgtable.h +4
-0+ pte = *ptep; + +/* +* Some processors can start a write, but ending up Quoted 1 line
seeing
but end up seeing
Quoted 14 lines
arch/x86/include/asm/pgtable.h +3
-0+ * a read-only PTE by the time they get to the Dirty bit. + * In this case, they will set the Dirty bit, leaving a + * read-only, Dirty PTE which looks like a Shadow StackPTE.
arch/x86/include/asm/pgtable.h +2
-0+ * + * However, this behavior has been improved and will notoccur
arch/x86/include/asm/pgtable.h +2
-0+ * on processors supporting Shadow Stacks. Without this + * guarantee, a transition to a non-present PTE and flushthe
arch/x86/include/asm/pgtable.h +3
-0+ * TLB would be needed. + * + * When change a writable PTE to read-only and if the PTEQuoted 1 line
has
changing
Quoted 25 lines
arch/x86/include/asm/pgtable.h +1
-0+ * _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SWso
arch/x86/include/asm/pgtable.h +3
-0+* that the PTE is not a valid Shadow Stack PTE. +*/ + pte = pte_move_flags(pte, _PAGE_DIRTY_HW,
_PAGE_DIRTY_SW);arch/x86/include/asm/pgtable.h +1
-0+set_pte_at (mm, addr, ptep, pte);}
#defineflush_tlb_fix_spurious_fault(vma, address)do { } while
(0)arch/x86/include/asm/pgtable.h +0
-0@@ -1266,7 +1287,28 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm, static inline void pmdp_set_wrprotect(struct mm_struct *mm,unsigned long addr,
pmd_t*pmdp)
{arch/x86/include/asm/pgtable.h +2
-0+pmd_t pmd; +
clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp);arch/x86/include/asm/pgtable.h +4
-0+ pmd = *pmdp; + +/* +* Some processors can start a write, but ending up Quoted 1 line
seeing
but end up seeing
Quoted 14 lines
arch/x86/include/asm/pgtable.h +3
-0+ * a read-only PTE by the time they get to the Dirty bit. + * In this case, they will set the Dirty bit, leaving a + * read-only, Dirty PTE which looks like a Shadow StackPTE.
arch/x86/include/asm/pgtable.h +2
-0+ * + * However, this behavior has been improved and will notoccur
arch/x86/include/asm/pgtable.h +2
-0+ * on processors supporting Shadow Stacks. Without this + * guarantee, a transition to a non-present PTE and flushthe
arch/x86/include/asm/pgtable.h +3
-0+ * TLB would be needed. + * + * When change a writable PTE to read-only and if the PTEQuoted 1 line
has
changing
Quoted 1 line
Quoted 12 lines
Quoted 11 lines
arch/x86/include/asm/pgtable.h +1
-0+ * _PAGE_DIRTY_HW set, we move that bit to _PAGE_DIRTY_SWso
arch/x86/include/asm/pgtable.h +3
-0+* that the PTE is not a valid Shadow Stack PTE. +*/ + pmd = pmd_move_flags(pmd, _PAGE_DIRTY_HW,
_PAGE_DIRTY_SW);arch/x86/include/asm/pgtable.h +1
-0+set_pmd_at (mm, addr, pmdp, pmd);}
#definepud_writepud_write
Thanks, I will fix it!
Dave Hansen
maintainerRe: [RFC PATCH v3 12/24] x86/mm: Modify ptep_set_wrprotect and pmdp_set_wrprotect for _PAGE_DIRTY_SW
In reply to Yu-cheng Yuarch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1220
Quoted 9 lines · Yu-cheng Yu
arch/x86/include/asm/pgtable.h +9
-0+* Some processors can start a write, but ending up seeing +* a read-only PTE by the time they get to the Dirty bit. +* In this case, they will set the Dirty bit, leaving a +* read-only, Dirty PTE which looks like a Shadow Stack PTE. +* +* However, this behavior has been improved and will not occur +* on processors supporting Shadow Stacks. Without this +* guarantee, a transition to a non-present PTE and flush the +* TLB would be needed.
Did we publicly document this behavior anywhere? I can't seem to find it.
RFC PATCH v3 13/24 x86/mm: Shadow stack page fault error checking
In reply to Yu-cheng YuIf a page fault is triggered by a shadow stack access (e.g. call/ret) or shadow stack management instructions (e.g. wrussq), then bit[6] of the page fault error code is set.
In access_error(), we check if a shadow stack page fault is within a shadow stack memory area.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+)
arch/x86/include/asm/traps.h +2 -0
@@ -157,6 +157,7 @@ enum {* bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access +* bit 6 == 1: shadow stack access fault */ enum x86_pf_error_code {X86_PF_PROT =1 <<0 ,
@@ -165,5 +166,6 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , + X86_PF_SHSTK = 1 << 6, };#endif /* _ASM_X86_TRAPS_H */
arch/x86/mm/fault.c +18 -0
@@ -1162,6 +1162,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; +/* +* Verify X86_PF_SHSTK is within a shadow stack VMA. +* It is always an error if there is a shadow stack +* fault outside a shadow stack VMA. +*/ +if (error_code &X86_PF_SHSTK ) { +if (!(vma->vm_flags &VM_SHSTK )) +return 1 ; +return 0 ; + } +if (error_code &X86_PF_WRITE ) {/* write, present and write, not present: */ if (unlikely (!(vma->vm_flags &VM_WRITE )))
@@ -1300,6 +1311,13 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* +* If the fault is caused by a shadow stack access, +* i.e. CALL/RET/SAVEPREVSSP/RSTORSSP, then set +* FAULT_FLAG_WRITE to effect copy-on-write. +*/ +if (error_code &X86_PF_SHSTK ) + flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_WRITE ) flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_INSTR )
Signature · 1 line
2.17.1
RFC PATCH v3 14/24 mm: Handle shadow stack page fault
In reply to Yu-cheng YuWhen a task does fork(), its shadow stack must be duplicated for the child. However, the child may not actually use all pages of of the copied shadow stack. This patch implements a flow that is similar to copy-on-write of an anonymous page, but for shadow stack memory. A shadow stack PTE needs to be RO and dirty. We use this dirty bit requirement to effect the copying of shadow stack pages.
In copy_one_pte(), we clear the dirty bit from the shadow stack PTE. On the next shadow stack access to the PTE, a page fault occurs. At that time, we then copy/re-use the page and fix the PTE.
--- arch/x86/mm/pgtable.c | 10 ++++++++++ include/asm-generic/pgtable.h | 7 +++++++ mm/memory.c | 3 +++ 3 files changed, 20 insertions(+)
arch/x86/mm/pgtable.c +10 -0
@@ -872,3 +872,13 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +inline pte_t pte_set_vma_features (pte_t pte,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHSTK ) +return pte_mkdirty_shstk (pte); +else +return pte; +} +#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */
include/asm-generic/pgtable.h +7 -0
@@ -1146,6 +1146,13 @@ static inline bool pmd_dirty_hw(pmd_t pmd) {return false; } + +static inline pte_t pte_set_vma_features (pte_t pte,struct vm_area_struct *vma) +{ +return pte; +} +#else +pte_t pte_set_vma_features (pte_t pte,struct vm_area_struct *vma);#endif #endif /* _ASM_GENERIC_PGTABLE_H */
mm/memory.c +3 -0
@@ -2462,6 +2462,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf)flush_cache_page (vma, vmf->address,pte_pfn (vmf->orig_pte)); entry =pte_mkyoung (vmf->orig_pte); entry =maybe_mkwrite (pte_mkdirty (entry), vma); + entry =pte_set_vma_features (entry, vma);if (ptep_set_access_flags (vma, vmf->address, vmf->pte, entry,1 ))update_mmu_cache (vma, vmf->address, vmf->pte);pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -2535,6 +2536,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)flush_cache_page (vma, vmf->address,pte_pfn (vmf->orig_pte)); entry =mk_pte (new_page, vma->vm_page_prot); entry =maybe_mkwrite (pte_mkdirty (entry), vma); + entry =pte_set_vma_features (entry, vma);/* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition
@@ -3187,6 +3189,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot);if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); + entry =pte_set_vma_features (entry, vma); vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
Signature · 1 line
2.17.1
RFC PATCH v3 15/24 mm: Handle THP/HugeTLB shadow stack page fault
In reply to Yu-cheng YuThis patch implements THP shadow stack memory copying in the same way as the previous patch for regular PTE.
In copy_huge_pmd(), we clear the dirty bit from the PMD. On the next shadow stack access to the PMD, a page fault occurs. At that time, the page is copied/re-used and the PMD is fixed.
--- arch/x86/mm/pgtable.c | 8 ++++++++ include/asm-generic/pgtable.h | 6 ++++++ mm/huge_memory.c | 4 ++++ 3 files changed, 18 insertions(+)
arch/x86/mm/pgtable.c +8 -0
@@ -881,4 +881,12 @@ inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) else return pte; } + +inline pmd_t pmd_set_vma_features (pmd_t pmd,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHSTK ) +return pmd_mkdirty_shstk (pmd); +else +return pmd; +}#endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */
include/asm-generic/pgtable.h +6 -0
@@ -1151,8 +1151,14 @@ static inline pte_t pte_set_vma_features(pte_t pte, struct vm_area_struct *vma) {return pte; } + +static inline pmd_t pmd_set_vma_features (pmd_t pmd,struct vm_area_struct *vma) +{ +return pmd; +}#else pte_t pte_set_vma_features (pte_t pte,struct vm_area_struct *vma); +pmd_t pmd_set_vma_features (pmd_t pmd,struct vm_area_struct *vma);#endif #endif /* _ASM_GENERIC_PGTABLE_H */
mm/huge_memory.c +4 -0
@@ -597,6 +597,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, entry =mk_huge_pmd (page, vma->vm_page_prot); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); + entry =pmd_set_vma_features (entry, vma);page_add_new_anon_rmap (page, vma, haddr, true);mem_cgroup_commit_charge (page, memcg, false, true);lru_cache_add_active_or_unevictable (page, vma);
@@ -1194,6 +1195,7 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf,pte_t entry; entry =mk_pte (pages[i], vma->vm_page_prot); entry =maybe_mkwrite (pte_mkdirty (entry), vma); + entry =pte_set_vma_features (entry, vma); memcg = (void *)page_private (pages[i]);set_page_private (pages[i],0 );page_add_new_anon_rmap (pages[i], vmf->vma, haddr, false);
@@ -1278,6 +1280,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)pmd_t entry; entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); + entry =pmd_set_vma_features (entry, vma);if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); ret |=VM_FAULT_WRITE ;
@@ -1349,6 +1352,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)pmd_t entry; entry =mk_huge_pmd (new_page, vma->vm_page_prot); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); + entry =pmd_set_vma_features (entry, vma);pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd);page_add_new_anon_rmap (new_page, vma, haddr, true);mem_cgroup_commit_charge (new_page, memcg, false, true);
Signature · 1 line
2.17.1
RFC PATCH v3 16/24 mm: Update can_follow_write_pte/pmd for shadow stack
In reply to Yu-cheng Yucan_follow_write_pte/pmd look for the (RO & DIRTY) PTE/PMD to verify an exclusive RO page still exists after a broken COW.
A shadow stack PTE is RO & PAGE_DIRTY_SW when it is shared, otherwise RO & PAGE_DIRTY_HW.
Introduce pte_exclusive() and pmd_exclusive() to also verify a shadow stack PTE is exclusive.
Also rename can_follow_write_pte/pmd() to can_follow_write() to make their meaning clear; i.e. "Can we write to the page?", not "Is the PTE writable?"
--- arch/x86/mm/pgtable.c | 18 ++++++++++++++++++ include/asm-generic/pgtable.h | 18 ++++++++++++++++++ mm/gup.c | 8 +++++--- mm/huge_memory.c | 8 +++++--- 4 files changed, 46 insertions(+), 6 deletions(-)
arch/x86/mm/pgtable.c +18 -0
@@ -889,4 +889,22 @@ inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) else return pmd; } + +inline bool pte_exclusive (pte_t pte,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHSTK ) +return pte_dirty_hw (pte); +else +return pte_dirty (pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +inline bool pmd_exclusive (pmd_t pmd,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHSTK ) +return pmd_dirty_hw (pmd); +else +return pmd_dirty (pmd); +} +#endif #endif /* CONFIG_X86_INTEL_SHADOW_STACK_USER */
include/asm-generic/pgtable.h +18 -0
@@ -1156,9 +1156,27 @@ static inline pmd_t pmd_set_vma_features(pmd_t pmd, struct vm_area_struct *vma) {return pmd; } + +#ifdef CONFIG_MMU +static inline bool pte_exclusive (pte_t pte,struct vm_area_struct *vma) +{ +return pte_dirty (pte); +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline bool pmd_exclusive (pmd_t pmd,struct vm_area_struct *vma) +{ +return pmd_dirty (pmd); +} +#endif +#endif /* CONFIG_MMU */ #else pte_t pte_set_vma_features (pte_t pte,struct vm_area_struct *vma);pmd_t pmd_set_vma_features (pmd_t pmd,struct vm_area_struct *vma); +bool pte_exclusive (pte_t pte,struct vm_area_struct *vma); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +bool pmd_exclusive (pmd_t pmd,struct vm_area_struct *vma); +#endif #endif #endif /* _ASM_GENERIC_PGTABLE_H */
mm/gup.c +5 -3
@@ -64,10 +64,12 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,* FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte (pte_t pte,unsigned int flags) +static inline bool can_follow_write (pte_t pte,unsigned int flags, +struct vm_area_struct *vma) {return pte_write (pte) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pte_dirty (pte)); + ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) && +pte_exclusive (pte, vma)); } static struct page *follow_page_pte(struct vm_area_struct *vma,
@@ -105,7 +107,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if ((flags &FOLL_NUMA ) &&pte_protnone (pte)) goto no_page; -if ((flags &FOLL_WRITE ) && !can_follow_write_pte (pte, flags)) { +if ((flags &FOLL_WRITE ) && !can_follow_write (pte, flags, vma)) {pte_unmap_unlock (ptep, ptl);return NULL ; }
mm/huge_memory.c +5 -3
@@ -1387,10 +1387,12 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)* FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags) +static inline bool can_follow_write (pmd_t pmd,unsigned int flags, +struct vm_area_struct *vma) {return pmd_write (pmd) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)); + ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) && +pmd_exclusive (pmd, vma)); } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
@@ -1403,7 +1405,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags)) +if (flags &FOLL_WRITE && !can_follow_write (*pmd, flags, vma)) goto out;/* Avoid dumping huge zero page */
Signature · 1 line
2.17.1
RFC PATCH v3 17/24 mm: Introduce do_mmap_locked()
In reply to Yu-cheng YuThere are a few places that need do_mmap() with mm->mmap_sem held. Create an in-line function for that.
--- include/linux/mm.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
include/linux/mm.h +18 -0
@@ -2318,6 +2318,24 @@ static inline void mm_populate(unsigned long addr, unsigned long len)static inline void mm_populate (unsigned long addr,unsigned long len) {}#endif +static inline unsigned long do_mmap_locked (unsigned long addr, +unsigned long len,unsigned long prot,unsigned long flags, +vm_flags_t vm_flags) +{ +struct mm_struct *mm = current->mm; +unsigned long populate; + +down_write (&mm->mmap_sem); + addr =do_mmap (NULL , addr, len, prot, flags, vm_flags,0 , + &populate,NULL ); +up_write (&mm->mmap_sem); + +if (populate) +mm_populate (addr, populate); + +return addr; +} +/* These take the mm semaphore themselves */ extern int __must_check vm_brk (unsigned long ,unsigned long );extern int __must_check vm_brk_flags (unsigned long ,unsigned long ,unsigned long );
Signature · 1 line
2.17.1
RFC PATCH v3 18/24 x86/cet/shstk: User-mode shadow stack support
In reply to Yu-cheng YuThis patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with VM_SHSTK flag set and read-only protection. The shadow stack is allocated to a fixed size of RLIMIT_STACK.
--- arch/x86/include/asm/cet.h | 30 +++++++ arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 14 +++ arch/x86/include/asm/processor.h | 5 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 109 +++++++++++++++++++++++ arch/x86/kernel/cpu/common.c | 24 +++++ arch/x86/kernel/process.c | 2 + fs/proc/task_mmu.c | 3 + 9 files changed, 196 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/cet.c
arch/x86/include/asm/cet.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> + +struct task_struct ; +/* +* Per-thread CET status +*/ +struct cet_status { +unsigned long shstk_base; +unsigned long shstk_size; +unsigned int shstk_enabled:1 ; +}; + +#ifdef CONFIG_X86_INTEL_CET +int cet_setup_shstk (void ); +void cet_disable_shstk (void ); +void cet_disable_free_shstk (struct task_struct *p); +#else +static inline int cet_setup_shstk (void ) {return 0 ; } +static inline void cet_disable_shstk (void ) {} +static inline void cet_disable_free_shstk (struct task_struct *p) {} +#endif + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -56,6 +56,12 @@# define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31))#endif +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1<<(X86_FEATURE_SHSTK & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -75,7 +81,7 @@#define DISABLED_MASK13 0#define DISABLED_MASK14 0#define DISABLED_MASK15 0 -#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP) +#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP|DISABLE_SHSTK)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
arch/x86/include/asm/msr-index.h +14 -0
@@ -777,4 +777,18 @@#define MSR_VM_IGNNE 0xc0010115#define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x6a0/* user mode cet setting */ +#define MSR_IA32_S_CET 0x6a2/* kernel mode cet setting */ +#define MSR_IA32_PL0_SSP 0x6a4/* kernel shstk pointer */ +#define MSR_IA32_PL3_SSP 0x6a7/* user shstk pointer */ +#define MSR_IA32_INT_SSP_TAB 0x6a8/* exception shstk table */ + +/* MSR_IA32_U_CET and MSR_IA32_S_CET bits */ +#define MSR_IA32_CET_SHSTK_EN 0x0000000000000001ULL +#define MSR_IA32_CET_WRSS_EN 0x0000000000000002ULL +#define MSR_IA32_CET_ENDBR_EN 0x0000000000000004ULL +#define MSR_IA32_CET_LEG_IW_EN 0x0000000000000008ULL +#define MSR_IA32_CET_NO_TRACK_EN 0x0000000000000010ULL +#endif /* _ASM_X86_MSR_INDEX_H */
arch/x86/include/asm/processor.h +5 -0
@@ -24,6 +24,7 @@ struct vm86;#include <asm/special_insns.h> #include <asm/fpu/types.h> #include <asm/unwind_hints.h> +#include <asm/cet.h> #include <linux/personality.h> #include <linux/cache.h>
@@ -503,6 +504,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; unsigned int uaccess_err:1; /* uaccess failed */ +#ifdef CONFIG_X86_INTEL_CET +struct cet_status cet; +#endif +/* Floating point and extended processor state */ struct fpu fpu;/*
arch/x86/kernel/Makefile +2 -0
@@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/cet.c new file +109 -0
109 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +24 -0
@@ -411,6 +411,29 @@ static __init int setup_disable_pku(char *arg)__setup ("nopku" , setup_disable_pku);#endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ +if (cpu_feature_enabled (X86_FEATURE_SHSTK )) +cr4_set_bits (X86_CR4_CET ); +} + +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER +static __init int setup_disable_shstk (char *s) +{ +/* require an exact match without trailing characters */ +if (strlen (s)) +return 0 ; + +if (!boot_cpu_has (X86_FEATURE_SHSTK )) +return 1 ; + +setup_clear_cpu_cap (X86_FEATURE_SHSTK ); +pr_info ("x86: 'no_cet_shstk' specified, disabling Shadow Stack\n" ); +return 1 ; +} +__setup ("no_cet_shstk" , setup_disable_shstk); +#endif +/* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization
@@ -1375,6 +1398,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)x86_init_rdrand (c);x86_init_cache_qos (c);setup_pku (c); +setup_cet (c);/* * Clear/Set all flags overridden by options, need do it
arch/x86/kernel/process.c +2 -0
@@ -39,6 +39,7 @@#include <asm/desc.h> #include <asm/prctl.h> #include <asm/spec-ctrl.h> +#include <asm/cet.h> /* * per-CPU TSS segments. Threads are completely 'soft' on Linux,
@@ -134,6 +135,7 @@ void flush_thread(void)flush_ptrace_hw_breakpoint (tsk);memset (tsk->thread.tls_array,0 ,sizeof (tsk->thread.tls_array)); +cet_disable_shstk ();fpu__clear (&tsk->thread.fpu); }
fs/proc/task_mmu.c +3 -0
@@ -652,6 +652,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "",#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_INTEL_SHADOW_STACK_USER + [ilog2(VM_SHSTK)] = "ss" +#endif };size_t i;
Signature · 1 line
2.17.1
Jann Horn
reviewerRe: [RFC PATCH v3 18/24] x86/cet/shstk: User-mode shadow stack support
Quoted 7 lines · Yu-cheng Yu
This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with
VM_SHSTKflag set and read-only protection. The shadow stack is allocated to a fixed size ofRLIMIT_STACK.Signed-off-by Yu-cheng Yu <[email protected]>
[...]
Quoted 5 lines
arch/x86/kernel/cet.c new file +5
-0+static int set_shstk_ptr (unsigned long addr) +{ +u64 r; + +if (!cpu_feature_enabled (X86_FEATURE_SHSTK ))
arch/x86/kernel/cet.c line 36
Quoted 10 lines
arch/x86/kernel/cet.c new file +10
-0+return -1 ; + +if ((addr >=TASK_SIZE_MAX ) || (!IS_ALIGNED (addr,4 ))) +return -1 ; + +rdmsrl (MSR_IA32_U_CET , r); +wrmsrl (MSR_IA32_PL3_SSP , addr); +wrmsrl (MSR_IA32_U_CET , r |MSR_IA32_CET_SHSTK_EN ); +return 0 ; +}
Here's a really stupid question: Where is the logic for switching those MSRs on task switch? MSR_IA32_PL3_SSP contains a userspace pointer, so it has to be switched on task switch, right? I'm sure I'm missing something obvious, but grepping for places that set MSR_IA32_PL3_SSP to nonzero values through the entire patchset, I only see set_shstk_ptr(), which is called from:
- cet_setup_shstk() (called from arch_setup_features(), which is
called from load_elf_binary())
- cet_restore_signal() (called on signal handler return)
- cet_setup_signal() (called from signal handling code)
Re: [RFC PATCH v3 18/24] x86/cet/shstk: User-mode shadow stack support
Quoted 36 lines · Jann Horn
Quoted 8 lines · Yu-cheng Yu
This patch adds basic shadow stack enabling/disabling routines. A task's shadow stack is allocated from memory with
VM_SHSTKflag set and read-only protection. The shadow stack is allocated to a fixed size ofRLIMIT_STACK.Signed-off-by Yu-cheng Yu <[email protected]>[...]
Quoted 6 lines
arch/x86/kernel/cet.c new file +5
-0+static int set_shstk_ptr (unsigned long addr) +{ + u64 r; + + if (!cpu_feature_enabled(X86_FEATURE_SHSTK))arch/x86/kernel/cet.c line 36
Quoted 10 lines
arch/x86/kernel/cet.c new file +10
-0+ return -1; + + if ((addr >= TASK_SIZE_MAX) || (!IS_ALIGNED(addr, 4))) + return -1; + + rdmsrl(MSR_IA32_U_CET, r); + wrmsrl(MSR_IA32_PL3_SSP, addr); + wrmsrl(MSR_IA32_U_CET, r | MSR_IA32_CET_SHSTK_EN); + return 0; +}Here's a really stupid question: Where is the logic for switching those MSRs on task switch?
MSR_IA32_PL3_SSPcontains a userspace pointer, so it has to be switched on task switch, right? I'm sure I'm
missing something obvious, but grepping for places that setMSR_IA32_PL3_SSPto nonzero values through the entire patchset, I only
seeset_shstk_ptr(), which is called from:
Quoted 4 lines
-
cet_setup_shstk()(called fromarch_setup_features(), which is called fromload_elf_binary())
-cet_restore_signal()(called on signal handler return)
-cet_setup_signal()(called from signal handling code)
The MSR is in the XSAVES buffer and switched by XSAVES/XRSTORS.
Yu-cheng
RFC PATCH v3 19/24 x86/cet/shstk: Introduce WRUSS instruction
In reply to Yu-cheng YuWRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
--- arch/x86/include/asm/special_insns.h | 37 ++++++++++++++++++++++++++++ arch/x86/mm/extable.c | 11 +++++++++ arch/x86/mm/fault.c | 9 +++++++ 3 files changed, 57 insertions(+)
arch/x86/include/asm/special_insns.h +37 -0
@@ -237,6 +237,43 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_INTEL_CET + +#if defined(CONFIG_IA32_EMULATION ) || defined(CONFIG_X86_X32 ) +static inline int write_user_shstk_32 (unsigned long addr,unsigned int val) +{ +int err =0 ; + + asm volatile("1: wrussd %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + +return err; +} +#else +static inline int write_user_shstk_32 (unsigned long addr,unsigned int val) +{ +BUG (); +return 0 ; +} +#endif + +static inline int write_user_shstk_64 (unsigned long addr,unsigned long val) +{ +int err =0 ; + + asm volatile("1: wrussq %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + +return err; +} +#endif /* CONFIG_X86_INTEL_CET */ +#define nop () asm volatile ("nop")
arch/x86/mm/extable.c +11 -0
@@ -157,6 +157,17 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); +#ifdef CONFIG_X86_INTEL_CET +__visible bool ex_handler_wruss(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr) +{ + regs->ip =ex_fixup_addr (fixup); + regs->ax =-1 ; +return true; +} +EXPORT_SYMBOL (ex_handler_wruss); +#endif + __visible bool ex_has_fault_handler(unsigned long ip) {const struct exception_table_entry *e;
arch/x86/mm/fault.c +9 -0
@@ -1305,6 +1305,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, error_code |=X86_PF_USER ; flags |=FAULT_FLAG_USER ; }else { +/* +* WRUSS is a kernel instrcution and but writes +* to user shadow stack. When a fault occurs, +* both X86_PF_USER and X86_PF_SHSTK are set. +* Clear X86_PF_USER here. +*/ +if ((error_code & (X86_PF_USER |X86_PF_SHSTK )) == + (X86_PF_USER |X86_PF_SHSTK )) + error_code &= ~X86_PF_USER ;if (regs->flags &X86_EFLAGS_IF )local_irq_enable (); }
Signature · 1 line
2.17.1
Jann Horn
reviewerRe: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 11 lines · Yu-cheng Yu
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>
[...]
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 274
Quoted 12 lines
arch/x86/include/asm/special_insns.h +12
-0+static inline int write_user_shstk_64 (unsigned long addr,unsigned long val) +{ +int err =0 ; + + asmvolatile ("1: wrussq %1, (%0)\n" +"2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + +return err; +}
What's up with "err"? You set it to zero, and then you return it, but nothing can ever set it to non-zero, right?
arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 7 lines
arch/x86/mm/extable.c +7
-0+__visible bool ex_handler_wruss(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr) +{ + regs->ip =ex_fixup_addr (fixup); + regs->ax =-1 ; +return true; +}
And here you just write into regs->ax, but your "asm volatile" doesn't reserve that register. This looks wrong to me.
I think you probably want to add something like an explicit "+&a"(err) output to the asm statements.
arch/x86/mm/fault.c __do_page_fault(struct pt_regs *regs, unsigned long error_code, · line 1309
Quoted 6 lines
arch/x86/mm/fault.c +2
-0@@ -1305,6 +1305,15 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, error_code |=X86_PF_USER ; flags |=FAULT_FLAG_USER ; }else { +/* +* WRUSS is a kernel instrcution and but writes
Nits: typo ("instrcution"), weird grammar ("and but writes")
Re: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 40 lines · Jann Horn
Quoted 11 lines · Yu-cheng Yu
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>[...]
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 274
Quoted 12 lines
arch/x86/include/asm/special_insns.h +12
-0+static inline int write_user_shstk_64 (unsigned long addr,unsigned long val) +{ +int err =0 ; + + asmvolatile ("1: wrussq %1, (%0)\n" +"2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_wruss) + : + : "r" (addr), "r" (val)); + +return err; +}What's up with "err"? You set it to zero, and then you return it, but nothing can ever set it to non-zero, right?
arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 7 lines
arch/x86/mm/extable.c +7
-0+__visible bool ex_handler_wruss(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr) +{ + regs->ip =ex_fixup_addr (fixup); + regs->ax =-1 ; +return true; +}And here you just write into
regs->ax, but your "asm volatile" doesn't reserve that register. This looks wrong to me.
Quoted 2 lines
I think you probably want to add something like an explicit
"+&a"(err)output to the asm statements.
We require asm goto support these days. How about using that? You won't even need a special exception handler.
Also, please change the BUG to WARN in the you-did-it-wrong 32-bit case. And return -EFAULT.
--Andy
Re: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 52 lines · Andy Lutomirski
Quoted 49 lines · Jann Horn
On Thu, Aug 30, 2018 at 4:44 PM Yu-cheng Yu <[email protected]
Quoted 13 lines
wrote:
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>[...]
Quoted 12 lines
+static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)arch/x86/include/asm/special_insns.h +6
-0+{ + int err = 0; + + asm volatile("1: wrussq %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b,
ex_handler_wruss)
+ :
+ : "r" (addr), "r" (val));arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 254
Quoted 3 lines
arch/x86/include/asm/special_insns.h +3
-0+ + return err; +}What's up with "err"? You set it to zero, and then you return it, but
nothing can ever set it to non-zero, right?arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 10 lines
+__visible bool ex_handler_wruss(const struct
exception_table_entry*fixup,
+ structpt_regs*regs, int trapnr)arch/x86/mm/extable.c +5
-0+{ + regs->ip = ex_fixup_addr(fixup); + regs->ax = -1; + return true; +}And here you just write into
regs->ax, but your "asm volatile" doesn't
reserve that register. This looks wrong to me.Quoted 2 lines
I think you probably want to add something like an explicit
"+&a"(err)output to the asm statements.
Quoted 7 lines
We require asm goto support these days. How about using that? You won't even need a special exception handler.
Also, please change the BUG to WARN in the you-did-it-wrong 32-bit case. And return
-EFAULT.--Andy
I will look into that.
Yu-cheng
Re: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 59 lines · Yu-cheng Yu
Quoted 58 lines · Andy Lutomirski
Quoted 55 lines · Jann Horn
On Thu, Aug 30, 2018 at 4:44 PM Yu-cheng Yu <[email protected] om
Quoted 14 lines
wrote:
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>[...]
Quoted 13 lines
+static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)arch/x86/include/asm/special_insns.h +6
-0+{ + int err = 0; + + asm volatile("1: wrussq %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b,
ex_handler_wruss)
+ :
+ : "r" (addr), "r" (val));arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 254
Quoted 3 lines
arch/x86/include/asm/special_insns.h +3
-0+ + return err; +}What's up with "err"? You set it to zero, and then you return it,
but
nothing can ever set it to non-zero, right?arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 11 lines
+__visible bool ex_handler_wruss(const struct
exception_table_entry*fixup,
+ structpt_regs*regs, int trapnr)arch/x86/mm/extable.c +5
-0+{ + regs->ip = ex_fixup_addr(fixup); + regs->ax = -1; + return true; +}And here you just write into
regs->ax, but your "asm volatile" doesn't
reserve that register. This looks wrong to me.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I think you probably want to add something like an explicit
"+&a"(err)output to the asm statements.We require asm goto support these days. How about using that? You
won't even need a special exception handler.
Maybe something like this? It looks simple now.
static inline int write_user_shstk_64(unsigned long addr, unsigned long val)
{
asm_volatile_goto("wrussq %1, (%0)\n"
"jmp %l[ok]\n"
".section .fixup,\"ax\"n"
"jmp %l[fail]\n"
".previous\n"
:: "r" (addr), "r" (val)
:: ok, fail);
ok:
return 0;
fail:
return -1;
}Andy Lutomirski
maintainerRe: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 70 lines · Yu-cheng Yu
Quoted 59 lines · Yu-cheng Yu
Quoted 58 lines · Andy Lutomirski
Quoted 55 lines · Jann Horn
On Thu, Aug 30, 2018 at 4:44 PM Yu-cheng Yu <[email protected] om
Quoted 14 lines
wrote:
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>[...]
Quoted 11 lines
+static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)arch/x86/include/asm/special_insns.h +6
-0+{ +int err =0 ; + + asmvolatile ("1: wrussq %1, (%0)\n" +"2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b,
ex_handler_wruss)arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 254
Quoted 5 lines
arch/x86/include/asm/special_insns.h +5
-0+ : + : "r" (addr), "r" (val)); + +return err; +}What's up with "err"? You set it to zero, and then you return it,
but
nothing can ever set it to non-zero, right?arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 11 lines
+__visible bool ex_handler_wruss(const struct
exception_table_entry*fixup,
+ structpt_regs*regs, int
trapnr)arch/x86/mm/extable.c +5
-0+{ + regs->ip =ex_fixup_addr (fixup); + regs->ax =-1 ; +return true; +}And here you just write into
regs->ax, but your "asm volatile" doesn't
reserve that register. This looks wrong to me.Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I think you probably want to add something like an explicit
"+&a"(err)output to the asm statements.We require asm goto support these days. How about using that? You
won't even need a special exception handler.Maybe something like this? It looks simple now.
static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)
Quoted 14 lines
{
asm_volatile_goto("wrussq %1, (%0)\n"
"jmp %l[ok]\n"
".section .fixup,\"ax\"n"
"jmp %l[fail]\n"
".previous\n"
:: "r" (addr), "r" (val)
:: ok, fail);
ok:
return 0;
fail:
return -1;
}I think you can get rid of 'jmp %l[ok]' and the ok label and just fall through. And you don't need an explicit jmp to fail -- just set the _EX_HANDLER entry to land on the fail label.
Re: [RFC PATCH v3 19/24] x86/cet/shstk: Introduce WRUSS instruction
Quoted 91 lines · Andy Lutomirski
Quoted 74 lines · Yu-cheng Yu
Quoted 65 lines · Yu-cheng Yu
Quoted 63 lines · Andy Lutomirski
Quoted 59 lines · Jann Horn
On Thu, Aug 30, 2018 at 4:44 PM Yu-cheng Yu <[email protected] om
Quoted 15 lines
wrote:
WRUSS is a new kernel-mode instruction but writes directly to user shadow stack memory. This is used to construct a return address on the shadow stack for the signal handler.
This instruction can fault if the user shadow stack is invalid shadow stack memory. In that case, the kernel does fixup.
Signed-off-by Yu-cheng Yu <[email protected]>[...]
Quoted 14 lines
+static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)arch/x86/include/asm/special_insns.h +6
-0+{ + int err = 0; + + asm volatile("1: wrussq %1, (%0)\n" + "2:\n" + _ASM_EXTABLE_HANDLE(1b, 2b,
ex_handler_wruss)
+ :
+ : "r" (addr), "r" (val));arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 254
Quoted 3 lines
arch/x86/include/asm/special_insns.h +3
-0+ + return err; +}What's up with "err"? You set it to zero, and then you return it,
but
nothing can ever set it to non-zero, right?arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 167
Quoted 12 lines
+__visible bool ex_handler_wruss(const struct
exception_table_entry*fixup,
+ structpt_regs*regs, int trapnr)arch/x86/mm/extable.c +5
-0+{ + regs->ip = ex_fixup_addr(fixup); + regs->ax = -1; + return true; +}And here you just write into
regs->ax, but your "asm volatile" doesn't
reserve that register. This looks wrong to me.Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I think you probably want to add something like an explicit
"+&a"(err)output to the asm statements.We require asm goto support these days. How about using that? You
won't even need a special exception handler.Maybe something like this? It looks simple now.
Quoted 16 lines
static inline int
write_user_shstk_64(unsigned long addr, unsigned long val)
{
asm_volatile_goto("wrussq %1, (%0)\n"
"jmp %l[ok]\n"
".section .fixup,\"ax\"n"
"jmp %l[fail]\n"
".previous\n"
:: "r" (addr), "r" (val)
:: ok, fail);
ok:
return 0;
fail:
return -1;
}
Quoted 3 lines
I think you can get rid of 'jmp %l[ok]' and the ok label and just fall through. And you don't need an explicit jmp to fail -- just set the
_EX_HANDLERentry to land on the fail label.
Thanks! This now looks simple and much better.
Yu-cheng
+static inline int write_user_shstk_64(unsigned long addr, unsigned long val) +{
+ asm_volatile_goto("1: wrussq %1, (%0)\n"
+ _ASM_EXTABLE(1b, %l[fail])
+ :: "r" (addr), "r" (val)
+ :: fail);
+ return 0;
+fail:
+ return -1;
+}
RFC PATCH v3 20/24 x86/cet/shstk: Signal handling for shadow stack
In reply to Yu-cheng YuWhen setting up a signal, the kernel creates a shadow stack restore token at the current SHSTK address and then stores the token's address in the signal frame, right after the FPU state. Before restoring a signal, the kernel verifies and then uses the restore token to set the SHSTK pointer.
--- arch/x86/ia32/ia32_signal.c | 13 +++ arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/asm/sighandling.h | 5 ++ arch/x86/include/uapi/asm/sigcontext.h | 17 ++++ arch/x86/kernel/cet.c | 115 +++++++++++++++++++++++++ arch/x86/kernel/signal.c | 96 +++++++++++++++++++++ 6 files changed, 251 insertions(+)
arch/x86/ia32/ia32_signal.c +13 -0
@@ -34,6 +34,7 @@#include <asm/sigframe.h> #include <asm/sighandling.h> #include <asm/smap.h> +#include <asm/cet.h> /* * Do a signal return; undo the signal stack.
@@ -108,6 +109,9 @@ static int ia32_restore_sigcontext(struct pt_regs *regs, err |=fpu__restore_sig (buf,1 ); +if (!err) + err =restore_sigcontext_ext (buf); +force_iret ();return err;
@@ -234,6 +238,10 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs,if (fpu->initialized) {unsigned long fx_aligned, math_size; +/* sigcontext extension */ +if (boot_cpu_has (X86_FEATURE_SHSTK )) + sp -= (sizeof (struct sc_ext ) +8 ); + sp =fpu__alloc_mathframe (sp,1 , &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; if (copy_fpstate_to_sigframe(*fpstate, (void __user *)fx_aligned,
@@ -277,6 +285,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig,if (ia32_setup_sigcontext (&frame->sc, fpstate, regs, set->sig[0 ]))return -EFAULT ; +if (setup_sigcontext_ext (ksig, fpstate)) +return -EFAULT ;if (_COMPAT_NSIG_WORDS >1 ) { if (__copy_to_user(frame->extramask, &set->sig[1],
@@ -384,6 +394,9 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, regs, set->sig[0]); err |=__copy_to_user (&frame->uc.uc_sigmask, set,sizeof (*set)); +if (!err) + err =setup_sigcontext_ext (ksig, fpstate); +if (err)return -EFAULT ;
arch/x86/include/asm/cet.h +5 -0
@@ -19,10 +19,15 @@ struct cet_status {int cet_setup_shstk (void );void cet_disable_shstk (void );void cet_disable_free_shstk (struct task_struct *p); +int cet_restore_signal (unsigned long ssp); +int cet_setup_signal (bool ia32,unsigned long rstor,unsigned long *new_ssp);#else static inline int cet_setup_shstk (void ) {return 0 ; }static inline void cet_disable_shstk (void ) {}static inline void cet_disable_free_shstk (struct task_struct *p) {} +static inline int cet_restore_signal (unsigned long ssp) {return 0 ; } +static inline int cet_setup_signal (bool ia32,unsigned long rstor, +unsigned long *new_ssp) {return 0 ; }#endif #endif /* __ASSEMBLY__ */
arch/x86/include/asm/sighandling.h +5 -0
@@ -17,4 +17,9 @@ void signal_fault(struct pt_regs *regs, void __user *frame, char *where);int setup_sigcontext (struct sigcontext __user *sc,void __user *fpstate,struct pt_regs *regs,unsigned long mask); +#ifdef CONFIG_X86_64 +int setup_sigcontext_ext (struct ksignal *ksig,void __user *fpu); +int restore_sigcontext_ext (void __user *fpu); +#endif +#endif /* _ASM_X86_SIGHANDLING_H */
arch/x86/include/uapi/asm/sigcontext.h +17 -0
@@ -196,6 +196,23 @@ struct _xstate {/* New processor state extensions go here: */ }; +#ifdef __x86_64__ +/* +* Sigcontext extension (struct sc_ext) is located after +* sigcontext->fpstate. Because currently only the shadow +* stack pointer is saved there and the shadow stack depends +* on XSAVES, we can find sc_ext from sigcontext->fpstate. +* +* The 64-bit fpstate has a size of fpu_user_xstate_size, plus +* FP_XSTATE_MAGIC2_SIZE when XSAVE* is used. The struct sc_ext +* is located at the end of sigcontext->fpstate, aligned to 8. +*/ +struct sc_ext { +unsigned long total_size; +unsigned long ssp; +}; +#endif +/* * The 32-bit signal frame: */
arch/x86/kernel/cet.c +115 -0
130 lines not on this page · raw message
arch/x86/kernel/signal.c +96 -0
132 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v3 21/24 x86/cet/shstk: ELF header parsing of Shadow Stack
In reply to Yu-cheng YuLook in .note.gnu.property of an ELF file and check if Shadow Stack needs to be enabled for the task.
--- arch/x86/Kconfig | 4 + arch/x86/include/asm/elf.h | 5 + arch/x86/include/uapi/asm/elf_property.h | 15 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/elf.c | 338 +++++++++++++++++++++++ fs/binfmt_elf.c | 15 + include/uapi/linux/elf.h | 1 + 7 files changed, 380 insertions(+) create mode 100644 arch/x86/include/uapi/asm/elf_property.h create mode 100644 arch/x86/kernel/elf.c
arch/x86/Kconfig +4 -0
@@ -1919,12 +1919,16 @@ config X86_INTEL_CET config ARCH_HAS_SHSTK def_bool n +config ARCH_HAS_PROGRAM_PROPERTIES + def_bool n + config X86_INTEL_SHADOW_STACK_USER prompt "Intel Shadow Stack for user-mode" def_bool n depends on CPU_SUP_INTEL && X86_64 select X86_INTEL_CET select ARCH_HAS_SHSTK + select ARCH_HAS_PROGRAM_PROPERTIES ---help--- Shadow stack provides hardware protection against program stack corruption. Only when all the following are true will an application
arch/x86/include/asm/elf.h +5 -0
@@ -382,4 +382,9 @@ struct va_alignment {extern struct va_alignment va_align;extern unsigned long align_vdso_addr (unsigned long ); + +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +extern int arch_setup_features (void *ehdr,void *phdr,struct file *file, +bool interp); +#endif #endif /* _ASM_X86_ELF_H */
arch/x86/include/uapi/asm/elf_property.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _UAPI_ASM_X86_ELF_PROPERTY_H +#define _UAPI_ASM_X86_ELF_PROPERTY_H + +/* +* pr_type +*/ +#define GNU_PROPERTY_X86_FEATURE_1_AND (0xc0000002) + +/* +* Bits for GNU_PROPERTY_X86_FEATURE_1_AND +*/ +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK (0x00000002) + +#endif /* _UAPI_ASM_X86_ELF_PROPERTY_H */
arch/x86/kernel/Makefile +2 -0
@@ -141,6 +141,8 @@ obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/elf.c new file +338 -0
338 lines not on this page · raw message
fs/binfmt_elf.c +15 -0
@@ -1081,6 +1081,21 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_dentry; } +#ifdef CONFIG_ARCH_HAS_PROGRAM_PROPERTIES +if (interpreter) { + retval =arch_setup_features (&loc->interp_elf_ex, + interp_elf_phdata, + interpreter, true); + }else { + retval =arch_setup_features (&loc->elf_ex, + elf_phdata, + bprm->file, false); + } + +if (retval <0 ) + goto out_free_dentry; +#endif +if (elf_interpreter) {unsigned long interp_map_addr =0 ;
include/uapi/linux/elf.h +1 -0
@@ -372,6 +372,7 @@ typedef struct elf64_shdr {#define NT_PRFPREG 2#define NT_PRPSINFO 3#define NT_TASKSTRUCT 4 +#define NT_GNU_PROPERTY_TYPE_0 5#define NT_AUXV 6/* * Note to userspace developers: size of NT_SIGINFO note may increase
Signature · 1 line
2.17.1
RFC PATCH v3 22/24 x86/cet/shstk: Handle thread shadow stack
In reply to Yu-cheng YuThe shadow stack for clone/fork is handled as the following:
(1) If ((clone_flags & (CLONE_VFORK | CLONE_VM)) == CLONE_VM), the kernel allocates (and frees on thread exit) a new SHSTK for the child.
It is possible for the kernel to complete the clone syscall and set the child's SHSTK pointer to NULL and let the child thread allocate a SHSTK for itself. There are two issues in this approach: It is not compatible with existing code that does inline syscall and it cannot handle signals before the child can successfully allocate a SHSTK.
(2) For (clone_flags & CLONE_VFORK), the child uses the existing SHSTK.
(3) For all other cases, the SHSTK is copied/reused whenever the parent or the child does a call/ret.
This patch handles cases (1) & (2). Case (3) is handled in the SHSTK page fault patches.
--- arch/x86/include/asm/cet.h | 2 ++ arch/x86/include/asm/mmu_context.h | 3 +++ arch/x86/kernel/cet.c | 34 ++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 1 + arch/x86/kernel/process_64.c | 7 ++++++ 5 files changed, 47 insertions(+)
arch/x86/include/asm/cet.h +2 -0
@@ -17,12 +17,14 @@ struct cet_status {#ifdef CONFIG_X86_INTEL_CET int cet_setup_shstk (void ); +int cet_setup_thread_shstk (struct task_struct *p);void cet_disable_shstk (void );void cet_disable_free_shstk (struct task_struct *p);int cet_restore_signal (unsigned long ssp);int cet_setup_signal (bool ia32,unsigned long rstor,unsigned long *new_ssp);#else static inline int cet_setup_shstk (void ) {return 0 ; } +static inline int cet_setup_thread_shstk (struct task_struct *p) {return 0 ; }static inline void cet_disable_shstk (void ) {}static inline void cet_disable_free_shstk (struct task_struct *p) {}static inline int cet_restore_signal (unsigned long ssp) {return 0 ; }
arch/x86/include/asm/mmu_context.h +3 -0
@@ -13,6 +13,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> +#include <asm/cet.h> extern atomic64_t last_mm_ctx_id;
@@ -223,6 +224,8 @@ do { \#else #define deactivate_mm (tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + cet_disable_free_shstk(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0)
arch/x86/kernel/cet.c +34 -0
@@ -134,6 +134,40 @@ int cet_setup_shstk(void)return 0 ; } +int cet_setup_thread_shstk (struct task_struct *tsk) +{ +unsigned long addr, size; +struct cet_user_state *state; + +if (!current->thread.cet.shstk_enabled) +return 0 ; + + state =get_xsave_addr (&tsk->thread.fpu.state.xsave, +XFEATURE_MASK_SHSTK_USER ); + +if (!state) +return -EINVAL ; + + size = tsk->thread.cet.shstk_size; +if (size ==0 ) + size =rlimit (RLIMIT_STACK ); + + addr =do_mmap_locked (0 , size,PROT_READ , +MAP_ANONYMOUS |MAP_PRIVATE ,VM_SHSTK ); + +if (addr >=TASK_SIZE_MAX ) { + tsk->thread.cet.shstk_base =0 ; + tsk->thread.cet.shstk_size =0 ; + tsk->thread.cet.shstk_enabled =0 ; +return -ENOMEM ; + } + + state->user_ssp = (u64)(addr + size -sizeof (u64)); + tsk->thread.cet.shstk_base = addr; + tsk->thread.cet.shstk_size = size; +return 0 ; +} +void cet_disable_shstk (void ) {u64 r;
arch/x86/kernel/process.c +1 -0
@@ -125,6 +125,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t); +cet_disable_free_shstk (tsk);fpu__drop (fpu); }
arch/x86/kernel/process_64.c +7 -0
@@ -317,6 +317,13 @@ int copy_thread_tls(unsigned long clone_flags, unsigned long sp,if (sp) childregs->sp = sp; +/* Allocate a new shadow stack for pthread */ +if ((clone_flags & (CLONE_VFORK |CLONE_VM )) ==CLONE_VM ) { + err =cet_setup_thread_shstk (p); +if (err) + goto out; + } + err = -ENOMEM ;if (unlikely (test_tsk_thread_flag (me,TIF_IO_BITMAP ))) { p->thread.io_bitmap_ptr = kmemdup(me->thread.io_bitmap_ptr,
Signature · 1 line
2.17.1
RFC PATCH v3 23/24 x86/cet/shstk: Add arch_prctl functions for Shadow Stack
In reply to Yu-cheng Yuarch_prctl(ARCH_CET_STATUS, unsigned long *addr)
Return CET feature status.
The parameter 'addr' is a pointer to a user buffer. On returning to the caller, the kernel fills the following information:
*addr = SHSTK/IBT status
*(addr + 1) = SHSTK base address
*(addr + 2) = SHSTK size
arch_prctl(ARCH_CET_DISABLE, unsigned long features)
Disable CET features specified in 'features'. Return -EPERM if CET is locked.
arch_prctl(ARCH_CET_LOCK)
Lock in CET feature.
arch_prctl(ARCH_CET_ALLOC_SHSTK, unsigned long *addr)
Allocate a new SHSTK.
The parameter 'addr' is a pointer to a user buffer and indicates the desired SHSTK size to allocate. On returning to the caller the buffer contains the address of the new SHSTK.
--- arch/x86/include/asm/cet.h | 5 ++ arch/x86/include/uapi/asm/prctl.h | 5 ++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet.c | 27 +++++++++++ arch/x86/kernel/cet_prctl.c | 79 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 5 ++ 6 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kernel/cet_prctl.c
arch/x86/include/asm/cet.h +5 -0
@@ -12,19 +12,24 @@ struct task_struct;struct cet_status {unsigned long shstk_base;unsigned long shstk_size; +unsigned int locked:1 ;unsigned int shstk_enabled:1 ; };#ifdef CONFIG_X86_INTEL_CET +int prctl_cet (int option,unsigned long arg2);int cet_setup_shstk (void );int cet_setup_thread_shstk (struct task_struct *p); +int cet_alloc_shstk (unsigned long *arg);void cet_disable_shstk (void );void cet_disable_free_shstk (struct task_struct *p);int cet_restore_signal (unsigned long ssp);int cet_setup_signal (bool ia32,unsigned long rstor,unsigned long *new_ssp);#else +static inline int prctl_cet (int option,unsigned long arg2) {return 0 ; }static inline int cet_setup_shstk (void ) {return 0 ; }static inline int cet_setup_thread_shstk (struct task_struct *p) {return 0 ; } +static inline int cet_alloc_shstk (unsigned long *arg) {return -EINVAL ; }static inline void cet_disable_shstk (void ) {}static inline void cet_disable_free_shstk (struct task_struct *p) {}static inline int cet_restore_signal (unsigned long ssp) {return 0 ; }
arch/x86/include/uapi/asm/prctl.h +5 -0
@@ -14,4 +14,9 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_CET_STATUS 0x3001 +#define ARCH_CET_DISABLE 0x3002 +#define ARCH_CET_LOCK 0x3003 +#define ARCH_CET_ALLOC_SHSTK 0x3004 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +1 -1
@@ -139,7 +139,7 @@ obj-$(CONFIG_UNWINDER_ORC) += unwind_orc.o obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o -obj-$(CONFIG_X86_INTEL_CET) += cet.o +obj-$(CONFIG_X86_INTEL_CET) += cet.o cet_prctl.o obj-$(CONFIG_ARCH_HAS_PROGRAM_PROPERTIES) += elf.o
arch/x86/kernel/cet.c +27 -0
@@ -110,6 +110,33 @@ static int create_rstor_token(bool ia32, unsigned long ssp,return 0 ; } +int cet_alloc_shstk (unsigned long *arg) +{ +unsigned long len = *arg; +unsigned long addr; +unsigned long token; +unsigned long ssp; + + addr =do_mmap_locked (0 , len,PROT_READ , +MAP_ANONYMOUS |MAP_PRIVATE ,VM_SHSTK ); +if (addr >=TASK_SIZE_MAX ) +return -ENOMEM ; + +/* Restore token is 8 bytes and aligned to 8 bytes */ + ssp = addr + len; + token = ssp; + +if (!in_ia32_syscall ()) + token |=1 ; + ssp -=8 ; + +if (write_user_shstk_64 (ssp, token)) +return -EINVAL ; + + *arg = addr; +return 0 ; +} +int cet_setup_shstk (void ) {unsigned long addr, size;
arch/x86/kernel/cet_prctl.c new file +79 -0
79 lines not on this page · raw message
arch/x86/kernel/process.c +5 -0
@@ -792,6 +792,11 @@ long do_arch_prctl_common(struct task_struct *task, int option,return get_cpuid_mode ();case ARCH_SET_CPUID :return set_cpuid_mode (task, cpuid_enabled); +case ARCH_CET_STATUS : +case ARCH_CET_DISABLE : +case ARCH_CET_LOCK : +case ARCH_CET_ALLOC_SHSTK : +return prctl_cet (option, cpuid_enabled); }return -EINVAL ;
Signature · 1 line
2.17.1
RFC PATCH v3 24/24 x86/cet/shstk: Add Shadow Stack instructions to opcode map
In reply to Yu-cheng YuAdd the following shadow stack management instructions.
INCSSP:
Increment shadow stack pointer by the steps specified.
RDSSP:
Read SSP register into a GPR.
SAVEPREVSSP:
Use "prev ssp" token at top of current shadow stack to create a "restore token" on previous shadow stack.
RSTORSSP:
Restore from a "restore token" pointed by a GPR to SSP.
WRSS:
Write to kernel-mode shadow stack (kernel-mode instruction).
WRUSS:
Write to user-mode shadow stack (kernel-mode instruction).
SETSSBSY:
Verify the "supervisor token" pointed by IA32_PL0_SSP MSR, if valid, set the token to busy, and set SSP to the value of IA32_PL0_SSP MSR.
CLRSSBSY:
Verify the "supervisor token" pointed by a GPR, if valid, clear the busy bit from the token.
--- arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ tools/objtool/arch/x86/lib/x86-opcode-map.txt | 26 +++++++++++++------ 2 files changed, 36 insertions(+), 16 deletions(-)
arch/x86/lib/x86-opcode-map.txt +18 -8
@@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd
@@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f
@@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff
@@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable
@@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable
tools/objtool/arch/x86/lib/x86-opcode-map.txt +18 -8
@@ -366,7 +366,7 @@ AVXcode: 1 1b: BNDCN Gv,Ev (F2) | BNDMOV Ev,Gv (66) | BNDMK Gv,Ev (F3) | BNDSTX Ev,Gv 1c: 1d: -1e: +1e: RDSSP Rd (F3),REX.W 1f: NOP Ev # 0x0f 0x20-0x2f 20: MOV Rd,Cd
@@ -610,7 +610,17 @@ fe: paddd Pq,Qq | vpaddd Vx,Hx,Wx (66),(v1) ff: UD0 EndTable -Table: 3-byte opcode 1 (0x0f 0x38) +Table: 3-byte opcode 1 (0x0f 0x01) +Referrer: +AVXcode: +# Skip 0x00-0xe7 +e8: SETSSBSY (f3) +e9: +ea: SAVEPREVSSP (f3) +# Skip 0xeb-0xff +EndTable + +Table: 3-byte opcode 2 (0x0f 0x38) Referrer: 3-byte escape 1 AVXcode: 2 # 0x0f 0x38 0x00-0x0f
@@ -789,12 +799,12 @@ f0: MOVBE Gy,My | MOVBE Gw,Mw (66) | CRC32 Gd,Eb (F2) | CRC32 Gd,Eb (66&F2) f1: MOVBE My,Gy | MOVBE Mw,Gw (66) | CRC32 Gd,Ey (F2) | CRC32 Gd,Ew (66&F2) f2: ANDN Gy,By,Ey (v) f3: Grp17 (1A) -f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) -f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) +f5: BZHI Gy,Ey,By (v) | PEXT Gy,By,Ey (F3),(v) | PDEP Gy,By,Ey (F2),(v) | WRUSS Pq,Qq (66),REX.W +f6: ADCX Gy,Ey (66) | ADOX Gy,Ey (F3) | MULX By,Gy,rDX,Ey (F2),(v) | WRSS Pq,Qq (66),REX.W f7: BEXTR Gy,Ey,By (v) | SHLX Gy,Ey,By (66),(v) | SARX Gy,Ey,By (F3),(v) | SHRX Gy,Ey,By (F2),(v) EndTable -Table: 3-byte opcode 2 (0x0f 0x3a) +Table: 3-byte opcode 3 (0x0f 0x3a) Referrer: 3-byte escape 2 AVXcode: 3 # 0x0f 0x3a 0x00-0xff
@@ -948,7 +958,7 @@ GrpTable: Grp7 2: LGDT Ms | XGETBV (000),(11B) | XSETBV (001),(11B) | VMFUNC (100),(11B) | XEND (101)(11B) | XTEST (110)(11B) 3: LIDT Ms 4: SMSW Mw/Rv -5: rdpkru (110),(11B) | wrpkru (111),(11B) +5: rdpkru (110),(11B) | wrpkru (111),(11B) | RSTORSSP Mq (F3) 6: LMSW Ew 7: INVLPG Mb | SWAPGS (o64),(000),(11B) | RDTSCP (001),(11B) EndTable
@@ -1019,8 +1029,8 @@ GrpTable: Grp15 2: vldmxcsr Md (v1) | WRFSBASE Ry (F3),(11B) 3: vstmxcsr Md (v1) | WRGSBASE Ry (F3),(11B) 4: XSAVE | ptwrite Ey (F3),(11B) -5: XRSTOR | lfence (11B) -6: XSAVEOPT | clwb (66) | mfence (11B) +5: XRSTOR | lfence (11B) | INCSSP Rd (F3),REX.W +6: XSAVEOPT | clwb (66) | mfence (11B) | CLRSSBSY Mq (F3) 7: clflush | clflushopt (66) | sfence (11B) EndTable
Signature · 1 line
2.17.1
Quoted 5 lines · Yu-cheng Yu
The previous version of CET patches can be found in the following link.
Quoted 9 lines
Summary of changes from v2:
Move Shadow Stack page fault handling logic to
arch/x86. Update can_follow_write_pte/pmd; move logic toarch/x86. Fix problems in WRUSS in-line assembly.
Fix issues in ELF parser.
Split out IBT/PTRACE patches to a second set. Other small fixes.
Quick question -- is there a simulator or some other way you've been testing this? Just curious, if it's possible to run these patches or just a review and internal hardware/simulator where they are run and posted
Balbir Singh.
Quoted 16 lines · Balbir Singh
Quoted 6 lines · Yu-cheng Yu
The previous version of CET patches can be found in the following link.
Quoted 9 lines
Summary of changes from v2:
Move Shadow Stack page fault handling logic to
arch/x86. Update can_follow_write_pte/pmd; move logic toarch/x86. Fix problems in WRUSS in-line assembly.
Fix issues in ELF parser.
Split out IBT/PTRACE patches to a second set. Other small fixes.
Quoted 6 lines
Quick question -- is there a simulator or some other way you've been testing this? Just curious, if it's possible to run these patches or just a review and internal hardware/simulator where they are run and posted
Balbir Singh.
Currently only for review.
Yu-cheng