Control-flow Enforcement: Shadow Stack
Subsystems (20)
v26 cover Control-flow Enforcement: Shadow Stack
Control-flow Enforcement (CET) is a new Intel processor feature that blocks return/jump-oriented programming attacks. Details are in "Intel 64 and IA-32 Architectures Software Developer's Manual" [1].
CET can protect applications and the kernel. This series enables only application-level protection, and has three parts:
- Shadow stack [2],
- Indirect branch tracking [3], and
- Selftests [4].
I have run tests on these patches for quite some time, and they have been very stable. Linux distributions with CET are available now, and Intel processors with CET are already on the market. It would be nice if CET support can be accepted into the kernel. I will be working to address any issues should they come up.
Changes in v26:
- Patch #19: Instead of passing vm_flags, pass vma pointer to can_follow_*().
- Patch #29: Instead of passing vma is anonymous, pass vma pointer to arch_validate_flags().
- Patch #30: Change PROT_SHSTK to PROT_SHADOW_STACK, remove redundant VM_SHARED check.
- Rebase to Linus tree v5.12.
Changes in v25:
- Remove Kconfig X86_CET and software-defined feature flag X86_FEATURE_CET. Use X86_SHADOW_STACK and X86_FEATURE_SHSTK directly. Update related areas accordingly.
- Patch #16: Make same changes to do_huge_pmd_numa_page() as to do_numa_page().
- Patch #25: Update signal handling, use restorer address already retrieved, update MSR restoring code.
- Smaller changes are called out in each patch.
- Rebase to Linus tree v5.12-rc7.
[1] Intel 64 and IA-32 Architectures Software Developer's Manual:
https://software.intel.com/en-us/download/intel-64-and-ia-32- architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4
[2] Shadow Stack patches v25:
https://lore.kernel.org/r/[email protected]/
[3] Indirect Branch Tracking patches v25
https://lore.kernel.org/r/[email protected]/
[4] I am holding off the selftests changes and working to get Reviewed-by's. The earlier version of the selftests patches:
https://lkml.kernel.org/r/[email protected]/
[5] The kernel ptrace patch is tested with an Intel-internal updated GDB. I am holding off the kernel ptrace patch to re-test it with my earlier patch for fixing regset holes.
Yu-cheng Yu (30):
Documentation/x86: Add CET description
x86/cet/shstk: Add Kconfig option for Shadow Stack x86/cpufeatures: Add CET CPU feature flags for Control-flow
Enforcement Technology (CET)
x86/cpufeatures: Introduce CPU setup and option parsing for CET x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states x86/cet: Add control-protection fault handler
x86/mm: Remove _PAGE_DIRTY from kernel RO pages x86/mm: Move pmd_write(), pud_write() up in the file x86/mm: Introduce _PAGE_COW
drm/i915/gvt: Change _PAGE_DIRTY to _PAGE_DIRTY_BITS x86/mm: Update pte_modify for _PAGE_COW
x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for
transition from _PAGE_DIRTY to _PAGE_COW
mm: Introduce VM_SHADOW_STACK for shadow stack memory x86/mm: Shadow Stack page fault error checking
x86/mm: Update maybe_mkwrite() for shadow stack
mm: Fixup places that call pte_mkwrite() directly
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting
mm: Update can_follow_write_pte() for shadow stack mm/mprotect: Exclude shadow stack from preserve_write
mm: Re-introduce vm_flags to do_mmap()
x86/cet/shstk: Add user-mode shadow stack support x86/cet/shstk: Handle thread shadow stack x86/cet/shstk: Introduce shadow stack token setup/verify routines x86/cet/shstk: Handle signals for shadow stack
ELF: Introduce arch_setup_elf_property() x86/cet/shstk: Add arch_prctl functions for shadow stack
mm: Move arch_calc_vm_prot_bits() to arch/x86/include/asm/mman.h
mm: Update arch_validate_flags() to include vma anonymous
mm: Introduce PROT_SHADOW_STACK for shadow stack
.../admin-guide/kernel-parameters.txt | 6 + Documentation/filesystems/proc.rst | 1 + Documentation/x86/index.rst | 1 + Documentation/x86/intel_cet.rst | 136 ++++++++ arch/arm64/include/asm/elf.h | 5 + arch/arm64/include/asm/mman.h | 4 +- arch/sparc/include/asm/mman.h | 4 +- arch/x86/Kconfig | 24 ++ arch/x86/Kconfig.assembler | 5 + arch/x86/ia32/ia32_signal.c | 24 +- arch/x86/include/asm/cet.h | 52 +++ arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/elf.h | 13 + arch/x86/include/asm/fpu/internal.h | 2 + arch/x86/include/asm/fpu/types.h | 23 +- arch/x86/include/asm/fpu/xstate.h | 6 +- arch/x86/include/asm/idtentry.h | 4 + arch/x86/include/asm/mman.h | 88 +++++ arch/x86/include/asm/mmu_context.h | 3 + arch/x86/include/asm/msr-index.h | 19 ++ arch/x86/include/asm/page_types.h | 7 + arch/x86/include/asm/pgtable.h | 299 +++++++++++++++-- arch/x86/include/asm/pgtable_types.h | 48 ++- arch/x86/include/asm/processor.h | 5 + arch/x86/include/asm/special_insns.h | 32 ++ arch/x86/include/asm/trap_pf.h | 2 + arch/x86/include/uapi/asm/mman.h | 28 +- arch/x86/include/uapi/asm/prctl.h | 4 + arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/include/uapi/asm/sigcontext.h | 9 + arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet_prctl.c | 60 ++++ arch/x86/kernel/cpu/common.c | 14 + arch/x86/kernel/cpu/cpuid-deps.c | 2 + arch/x86/kernel/fpu/signal.c | 137 +++++++- arch/x86/kernel/fpu/xstate.c | 10 +- arch/x86/kernel/idt.c | 4 + arch/x86/kernel/process.c | 21 +- arch/x86/kernel/process_64.c | 32 ++ arch/x86/kernel/shstk.c | 304 ++++++++++++++++++ arch/x86/kernel/signal.c | 9 + arch/x86/kernel/signal_compat.c | 2 +- arch/x86/kernel/traps.c | 63 ++++ arch/x86/mm/fault.c | 19 ++ arch/x86/mm/mmap.c | 48 +++ arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/mm/pgtable.c | 25 ++ drivers/gpu/drm/i915/gvt/gtt.c | 2 +- fs/aio.c | 2 +- fs/binfmt_elf.c | 4 + fs/proc/task_mmu.c | 3 + include/linux/elf.h | 6 + include/linux/mm.h | 18 +- include/linux/mman.h | 2 +- include/linux/pgtable.h | 7 + include/uapi/asm-generic/siginfo.h | 3 +- include/uapi/linux/elf.h | 9 + ipc/shm.c | 2 +- mm/gup.c | 16 +- mm/huge_memory.c | 27 +- mm/memory.c | 5 +- mm/migrate.c | 3 +- mm/mmap.c | 17 +- mm/mprotect.c | 11 +- mm/nommu.c | 4 +- mm/util.c | 2 +- 67 files changed, 1644 insertions(+), 119 deletions(-) create mode 100644 Documentation/x86/intel_cet.rst create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/include/asm/mman.h create mode 100644 arch/x86/kernel/cet_prctl.c create mode 100644 arch/x86/kernel/shstk.c
Signature · 1 line
2.21.0
PATCH v26 1/30 Documentation/x86: Add CET description
Explain no_user_shstk/no_user_ibt kernel parameters, and introduce a new document on Control-flow Enforcement Technology (CET).
---
v24:
- Update for Kconfig changes from X86_CET to X86_SHADOW_STACK, X86_IBT.
- Update for the change of VM_SHSTK to VM_SHADOW_STACK.
.../admin-guide/kernel-parameters.txt | 6 + Documentation/x86/index.rst | 1 + Documentation/x86/intel_cet.rst | 136 ++++++++++++++++++ 3 files changed, 143 insertions(+) create mode 100644 Documentation/x86/intel_cet.rst
Documentation/admin-guide/kernel-parameters.txt +6 -0
@@ -3220,6 +3220,12 @@ noexec=on: enable non-executable mappings (default) noexec=off: disable non-executable mappings + no_user_shstk [X86-64] Disable Shadow Stack for user-mode + applications + + no_user_ibt [X86-64] Disable Indirect Branch Tracking for user-mode + applications + nosmap [X86,PPC] Disable SMAP (Supervisor Mode Access Prevention) even if it is supported by processor.
Documentation/x86/index.rst +1 -0
@@ -21,6 +21,7 @@ x86-specific Documentation tlb mtrr pat + intel_cet intel-iommu intel_txt amd-memory-encryption
Documentation/x86/intel_cet.rst new file +136 -0
136 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v26 2/30 x86/cet/shstk: Add Kconfig option for Shadow Stack
In reply to Yu-cheng YuShadow Stack provides protection against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-Shadow Stack applications continue to work, but without protection.
---
v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.
v24:
- Update for the splitting X86_CET to X86_SHADOW_STACK and X86_IBT.
arch/x86/Kconfig | 22 ++++++++++++++++++++++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 27 insertions(+)
arch/x86/Kconfig +22 -0
@@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_USE_CMPXCHG_LOCKREF + select ARCH_HAS_SHADOW_STACK select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE
@@ -1941,6 +1942,27 @@ config X86_SGX If unsure, say N. +config ARCH_HAS_SHADOW_STACK + def_bool n + +config X86_SHADOW_STACK + prompt "Intel Shadow Stack" + def_bool n + depends on AS_WRUSS + depends on ARCH_HAS_SHADOW_STACK + select ARCH_USES_HIGH_VMA_FLAGS + help + Shadow Stack protection is a hardware feature that detects function + return address corruption. This helps mitigate ROP attacks. + Applications must be enabled to use it, and old userspace does not + get protection "for free". + Support for this feature is present on Tiger Lake family of + processors released in 2020 or later. Enabling this feature + increases kernel text size by 3.7 KB. + See Documentation/x86/intel_cet.rst for more information. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/Kconfig.assembler +5 -0
@@ -19,3 +19,8 @@ config AS_TPAUSE def_bool $(as-instr,tpause %ecx) help Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7 + +config AS_WRUSS + def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) + help + Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.21.0
PATCH v26 3/30 x86/cpufeatures: Add CET CPU feature flags for Control-flow Enforcement Technology (CET)
In reply to Yu-cheng YuAdd CPU feature flags for Control-flow Enforcement Technology (CET).
CPUID.(EAX=7,ECX=0):ECX[bit 7] Shadow stack CPUID.(EAX=7,ECX=0):EDX[bit 20] Indirect Branch Tracking
---
v25:
- Make X86_FEATURE_IBT depend on X86_FEATURE_SHSTK.
v24:
- Update for splitting CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK and CONFIG_X86_IBT.
- Move DISABLE_IBT definition to the IBT series.
arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 2 ++ 3 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -345,6 +345,7 @@#define X86_FEATURE_OSPKE (16*32+ 4)/* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5)/* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6)/* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7)/* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8)/* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9)/* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10)/* Carry-Less Multiplication Double Quadword */
@@ -377,6 +378,7 @@#define X86_FEATURE_TSXLDTRK (18*32+16)/* TSX Suspend Load Address Tracking */ #define X86_FEATURE_PCONFIG (18*32+18)/* Intel PCONFIG */ #define X86_FEATURE_ARCH_LBR (18*32+19)/* Intel ARCH LBR */ +#define X86_FEATURE_IBT (18*32+20)/* Indirect Branch Tracking */ #define X86_FEATURE_AVX512_FP16 (18*32+23)/* AVX512 FP16 */ #define X86_FEATURE_SPEC_CTRL (18*32+26)/* "" Speculation Control (IBRS + IBPB) */ #define X86_FEATURE_INTEL_STIBP (18*32+27)/* "" Single Thread Indirect Branch Predictors */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -68,6 +68,12 @@# define DISABLE_SGX (1 << (X86_FEATURE_SGX & 31))#endif +#ifdef CONFIG_X86_SHADOW_STACK +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1 << (X86_FEATURE_SHSTK & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -88,7 +94,7 @@#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_SHSTK)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK19 0
arch/x86/kernel/cpu/cpuid-deps.c +2 -0
@@ -72,6 +72,8 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_AVX512_FP16, X86_FEATURE_AVX512BW }, { X86_FEATURE_ENQCMD, X86_FEATURE_XSAVES }, { X86_FEATURE_PER_THREAD_MBA, X86_FEATURE_MBA }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, + { X86_FEATURE_IBT, X86_FEATURE_SHSTK }, {} };
Signature · 1 line
2.21.0
PATCH v26 4/30 x86/cpufeatures: Introduce CPU setup and option parsing for CET
In reply to Yu-cheng YuIntroduce CPU setup and boot option parsing for CET features.
---
v25:
- Remove software-defined X86_FEATURE_CET.
v24:
- Update #ifdef placement to reflect Kconfig changes of splitting shadow stack and ibt.
arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/common.c | 14 ++++++++++++++ 2 files changed, 16 insertions(+)
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -130,6 +130,8 @@#define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT)#define X86_CR4_PKE_BIT 22/* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_CET_BIT 23/* enable Control-flow Enforcement */ +#define X86_CR4_CET _BITUL(X86_CR4_CET_BIT)/* * x86-64 Task Priority Register, CR8
arch/x86/kernel/cpu/common.c +14 -0
@@ -510,6 +510,14 @@ static __init int setup_disable_pku(char *arg)__setup ("nopku" , setup_disable_pku);#endif /* CONFIG_X86_64 */ +static __always_inline void setup_cet(struct cpuinfo_x86 *c) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return ; + +cr4_set_bits (X86_CR4_CET ); +} +/* * Some CPU features depend on higher CPUID levels, which may not always * be available due to CPUID level capping or broken virtualization
@@ -1255,6 +1263,11 @@ static void __init cpu_parse_early_param(void)if (cmdline_find_option_bool (boot_command_line,"noxsaves" ))setup_clear_cpu_cap (X86_FEATURE_XSAVES ); +if (cmdline_find_option_bool (boot_command_line,"no_user_shstk" )) +setup_clear_cpu_cap (X86_FEATURE_SHSTK ); +if (cmdline_find_option_bool (boot_command_line,"no_user_ibt" )) +setup_clear_cpu_cap (X86_FEATURE_IBT ); + arglen =cmdline_find_option (boot_command_line,"clearcpuid" , arg,sizeof (arg));if (arglen <=0 )return ;
@@ -1594,6 +1607,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)x86_init_rdrand (c);setup_pku (c); +setup_cet (c);/* * Clear/Set all flags overridden by options, need do it
Signature · 1 line
2.21.0
PATCH v26 5/30 x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
In reply to Yu-cheng YuControl-flow Enforcement Technology (CET) introduces these MSRs:
MSR_IA32_U_CET (user-mode CET settings), MSR_IA32_PL3_SSP (user-mode shadow stack pointer),
MSR_IA32_PL0_SSP (kernel-mode shadow stack pointer), MSR_IA32_PL1_SSP (Privilege Level 1 shadow stack pointer), MSR_IA32_PL2_SSP (Privilege Level 2 shadow stack pointer), MSR_IA32_S_CET (kernel-mode CET settings), MSR_IA32_INT_SSP_TAB (exception shadow stack table).
The two user-mode MSRs belong to XFEATURE_CET_USER. The first three of kernel-mode MSRs belong to XFEATURE_CET_KERNEL. Both XSAVES states are supervisor states. This means that there is no direct, unprivileged access to these states, making it harder for an attacker to subvert CET.
For sigreturn and future ptrace() support, shadow stack address and MSR reserved bits are checked before written to the supervisor states.
---
v25:
- Update xsave_cpuid_features[]. Now CET XSAVES features depend on X86_FEATURE_SHSTK (vs. the software-defined X86_FEATURE_CET).
arch/x86/include/asm/fpu/types.h | 23 +++++++++++++++++++++-- arch/x86/include/asm/fpu/xstate.h | 6 ++++-- arch/x86/include/asm/msr-index.h | 19 +++++++++++++++++++ arch/x86/kernel/fpu/xstate.c | 10 +++++++++- 4 files changed, 53 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/types.h +21 -2
@@ -115,8 +115,8 @@ enum xfeature {XFEATURE_PT_UNIMPLEMENTED_SO_FAR ,XFEATURE_PKRU ,XFEATURE_PASID , -XFEATURE_RSRVD_COMP_11 , -XFEATURE_RSRVD_COMP_12 , +XFEATURE_CET_USER , +XFEATURE_CET_KERNEL ,XFEATURE_RSRVD_COMP_13 ,XFEATURE_RSRVD_COMP_14 , XFEATURE_LBR,
@@ -135,6 +135,8 @@ enum xfeature {#define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL)#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)#define XFEATURE_MASK_FPSSE (XFEATURE_MASK_FP | XFEATURE_MASK_SSE)
@@ -237,6 +239,23 @@ struct pkru_state {u32 pad; } __packed; +/* +* State component 11 is Control-flow Enforcement user states +*/ +struct cet_user_state { +u64 user_cet;/* user control-flow settings */ +u64 user_ssp;/* user shadow stack pointer */ +}; + +/* +* State component 12 is Control-flow Enforcement kernel states +*/ +struct cet_kernel_state { +u64 kernel_ssp;/* kernel shadow stack */ +u64 pl1_ssp;/* privilege level 1 shadow stack */ +u64 pl2_ssp;/* privilege level 2 shadow stack */ +}; +/* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth).
arch/x86/include/asm/fpu/xstate.h +4 -2
@@ -35,7 +35,8 @@ XFEATURE_MASK_BNDCSR)/* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER)/* * A supervisor state component may not always contain valuable information,
@@ -62,7 +63,8 @@* Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL)/* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
arch/x86/include/asm/msr-index.h +19 -0
@@ -933,4 +933,23 @@#define MSR_VM_IGNNE 0xc0010115#define MSR_VM_HSAVE_PA 0xc0010117 +/* Control-flow Enforcement Technology MSRs */ +#define MSR_IA32_U_CET 0x000006a0/* user mode cet setting */ +#define MSR_IA32_S_CET 0x000006a2/* kernel mode cet setting */ +#define CET_SHSTK_EN BIT_ULL(0) +#define CET_WRSS_EN BIT_ULL(1) +#define CET_ENDBR_EN BIT_ULL(2) +#define CET_LEG_IW_EN BIT_ULL(3) +#define CET_NO_TRACK_EN BIT_ULL(4) +#define CET_SUPPRESS_DISABLE BIT_ULL(5) +#define CET_RESERVED (BIT_ULL(6) | BIT_ULL(7) | BIT_ULL(8) | BIT_ULL(9)) +#define CET_SUPPRESS BIT_ULL(10) +#define CET_WAIT_ENDBR BIT_ULL(11) + +#define MSR_IA32_PL0_SSP 0x000006a4/* kernel shadow stack pointer */ +#define MSR_IA32_PL1_SSP 0x000006a5/* ring-1 shadow stack pointer */ +#define MSR_IA32_PL2_SSP 0x000006a6/* ring-2 shadow stack pointer */ +#define MSR_IA32_PL3_SSP 0x000006a7/* user shadow stack pointer */ +#define MSR_IA32_INT_SSP_TAB 0x000006a8/* exception shadow stack table */ +#endif /* _ASM_X86_MSR_INDEX_H */
arch/x86/kernel/fpu/xstate.c +9 -1
@@ -38,6 +38,8 @@ static const char *xfeature_names[] ="Processor Trace (unused)" ,"Protection Keys User registers" ,"PASID state" , +"Control-flow User registers" , +"Control-flow Kernel registers" , "unknown xstate feature" , };
@@ -53,6 +55,8 @@ static short xsave_cpuid_features[] __initdata = {X86_FEATURE_INTEL_PT ,X86_FEATURE_PKU ,X86_FEATURE_ENQCMD , +X86_FEATURE_SHSTK ,/* XFEATURE_CET_USER */ + X86_FEATURE_SHSTK, /* XFEATURE_CET_KERNEL */ };/*
@@ -321,6 +325,8 @@ static void __init print_xstate_features(void)print_xstate_feature (XFEATURE_MASK_Hi16_ZMM);print_xstate_feature (XFEATURE_MASK_PKRU );print_xstate_feature (XFEATURE_MASK_PASID ); +print_xstate_feature (XFEATURE_MASK_CET_USER ); +print_xstate_feature (XFEATURE_MASK_CET_KERNEL ); }/*
@@ -596,6 +602,8 @@ static void check_xstate_against_struct(int nr) XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state); + XCHECK_SZ(sz, nr, XFEATURE_CET_USER, struct cet_user_state); + XCHECK_SZ(sz, nr, XFEATURE_CET_KERNEL, struct cet_kernel_state);/* * Make *SURE* to add any feature numbers in below if
@@ -605,7 +613,7 @@ static void check_xstate_against_struct(int nr)if ((nr <XFEATURE_YMM ) || (nr >=XFEATURE_MAX ) || (nr ==XFEATURE_PT_UNIMPLEMENTED_SO_FAR ) || - ((nr >=XFEATURE_RSRVD_COMP_11 ) && (nr <=XFEATURE_LBR ))) { + ((nr >=XFEATURE_RSRVD_COMP_13 ) && (nr <=XFEATURE_LBR ))) {WARN_ONCE (1 ,"no structure for xstate: %d\n" , nr);XSTATE_WARN_ON (1 ); }
Signature · 1 line
2.21.0
PATCH v26 6/30 x86/cet: Add control-protection fault handler
In reply to Yu-cheng YuA control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack; or an indirect JMP instruction, without the NOTRACK prefix, arrives at a non-ENDBR opcode.
The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler.
---
v25:
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.
arch/x86/include/asm/idtentry.h | 4 ++ arch/x86/kernel/idt.c | 4 ++ arch/x86/kernel/signal_compat.c | 2 +- arch/x86/kernel/traps.c | 63 ++++++++++++++++++++++++++++++ include/uapi/asm-generic/siginfo.h | 3 +- 5 files changed, 74 insertions(+), 2 deletions(-)
arch/x86/include/asm/idtentry.h +4 -0
@@ -571,6 +571,10 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_SS, exc_stack_segment);DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_GP , exc_general_protection);DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_AC , exc_alignment_check); +#ifdef CONFIG_X86_SHADOW_STACK +DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_CP , exc_control_protection); +#endif +/* Raw exception entries which need extra work */ DECLARE_IDTENTRY_RAW (X86_TRAP_UD , exc_invalid_op);DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3);
arch/x86/kernel/idt.c +4 -0
@@ -105,6 +105,10 @@ static const __initconst struct idt_data def_idts[] = {#elif defined(CONFIG_X86_32 ) SYSG(IA32_SYSCALL_VECTOR, entry_INT80_32),#endif + +#ifdef CONFIG_X86_SHADOW_STACK + INTG(X86_TRAP_CP, asm_exc_control_protection), +#endif };/*
arch/x86/kernel/signal_compat.c +1 -1
@@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void)*/ BUILD_BUG_ON (NSIGILL !=11 );BUILD_BUG_ON (NSIGFPE !=15 ); -BUILD_BUG_ON (NSIGSEGV !=9 ); +BUILD_BUG_ON (NSIGSEGV !=10 );BUILD_BUG_ON (NSIGBUS !=5 );BUILD_BUG_ON (NSIGTRAP !=5 );BUILD_BUG_ON (NSIGCHLD !=6 );
arch/x86/kernel/traps.c +63 -0
75 lines not on this page · raw message
include/uapi/asm-generic/siginfo.h +2 -1
@@ -231,7 +231,8 @@ typedef struct siginfo {#define SEGV_ADIPERR 7/* Precise MCD exception */ #define SEGV_MTEAERR 8/* Asynchronous ARM MTE error */ #define SEGV_MTESERR 9/* Synchronous ARM MTE exception */ -#define NSIGSEGV 9 +#define SEGV_CPERR 10/* Control protection fault */ +#define NSIGSEGV 10/* * SIGBUS si_codes
Signature · 1 line
2.21.0
PATCH v26 7/30 x86/mm: Remove _PAGE_DIRTY from kernel RO pages
In reply to Yu-cheng YuThe x86 family of processors do not directly create read-only and Dirty PTEs. These PTEs are created by software. One such case is that kernel read-only pages are historically setup as Dirty.
New processors that support Shadow Stack regard read-only and Dirty PTEs as shadow stack pages. This results in ambiguity between shadow stack and kernel read-only pages. To resolve this, removed Dirty from kernel read- only pages.
--- arch/x86/include/asm/pgtable_types.h | 6 +++--- arch/x86/mm/pat/set_memory.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
@@ -192,10 +192,10 @@ enum page_cache_mode {#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC)#define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0)#define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) -#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0|___D| 0|___G) +#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) +#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) -#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_LARGE (__PP|__RW| 0|___A|__NX|___D|_PSE|___G)#define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW| 0|___A| 0|___D|_PSE|___G)#define __PAGE_KERNEL_WP (__PP|__RW| 0|___A|__NX|___D| 0|___G| __WP)
arch/x86/mm/pat/set_memory.c +1 -1
@@ -1932,7 +1932,7 @@ int set_memory_nx(unsigned long addr, int numpages)int set_memory_ro (unsigned long addr,int numpages) { -return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW),0 ); +return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW | _PAGE_DIRTY),0 ); }int set_memory_rw (unsigned long addr,int numpages)
Signature · 1 line
2.21.0
PATCH v26 8/30 x86/mm: Move pmd_write(), pud_write() up in the file
In reply to Yu-cheng YuTo prepare the introduction of _PAGE_COW, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below.
--- arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
arch/x86/include/asm/pgtable.h +12 -12
@@ -185,6 +185,18 @@ static inline int pte_write(pte_t pte)return pte_flags (pte) & _PAGE_RW; } +#define pmd_write pmd_write +static inline int pmd_write (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_RW; +} + +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pud_flags (pud) & _PAGE_RW; +} +static inline int pte_huge (pte_t pte) {return pte_flags (pte) & _PAGE_PSE;
@@ -1128,12 +1140,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); -#define pmd_write pmd_write -static inline int pmd_write (pmd_t pmd) -{ -return pmd_flags (pmd) & _PAGE_RW; -} -#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEARstatic inline pmd_t pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)
@@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); } -#define pud_write pud_write -static inline int pud_write (pud_t pud) -{ -return pud_flags (pud) & _PAGE_RW; -} -#ifndef pmdp_establish#define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
Signature · 1 line
2.21.0
PATCH v26 9/30 x86/mm: Introduce _PAGE_COW
In reply to Yu-cheng YuThere is essentially no room left in the x86 hardware PTEs on some OSes (not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0, Dirty=1.
The reason it's lightly used is that Dirty=1 is normally set by hardware and cannot normally be set by hardware on a Write=0 PTE. Software must normally be involved to create one of these PTEs, so software can simply opt to not create them.
In places where Linux normally creates Write=0, Dirty=1, it can use the software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0, Dirty=1, it instead creates Write=0, Cow=1, except for shadow stack, which is Write=0, Dirty=1. This clearly separates shadow stack from other data, and results in the following:
(a) A modified, copy-on-write (COW) page: (Write=0, Cow=1)
(b) A R/O page that has been COW'ed: (Write=0, Cow=1)
The user page is in a R/O VMA, and get_user_pages() needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
(c) A shadow stack PTE: (Write=0, Dirty=1)
(d) A shared shadow stack PTE: (Write=0, Cow=1)
When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.
(e) A page where the processor observed a Write=1 PTE, started a write, set Dirty=1, but then observed a Write=0 PTE. That's possible today, but will not happen on processors that support shadow stack.
Define _PAGE_COW and update pte_*() helpers and apply the same changes to pmd and pud.
After this, there are six free bits left in the 64-bit PTE, and no more free bits in the 32-bit PTE (except for PAE) and Shadow Stack is not implemented for the 32-bit kernel.
---
v24:
- Replace CONFIG_X86_CET with CONFIG_X86_SHADOW_STACK, to reflect the Kconfig changes.
arch/x86/include/asm/pgtable.h | 195 ++++++++++++++++++++++++--- arch/x86/include/asm/pgtable_types.h | 42 +++++- 2 files changed, 216 insertions(+), 21 deletions(-)
arch/x86/include/asm/pgtable.h +175 -20
@@ -121,11 +121,21 @@ extern pmdval_t early_pmd_flags;* The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty (pte_t pte) +static inline bool pte_dirty (pte_t pte) { -return pte_flags (pte) & _PAGE_DIRTY; +/* +* A dirty PTE has Dirty=1 or Cow=1. +*/ +return pte_flags (pte) & _PAGE_DIRTY_BITS; } +static inline bool pte_shstk (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return false; + +return (pte_flags (pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; +}static inline u32 read_pkru (void ) {
@@ -160,9 +170,20 @@ static inline int pte_young(pte_t pte)return pte_flags (pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty (pmd_t pmd) +static inline bool pmd_dirty (pmd_t pmd) +{ +/* +* A dirty PMD has Dirty=1 or Cow=1. +*/ +return pmd_flags (pmd) & _PAGE_DIRTY_BITS; +} + +static inline bool pmd_shstk (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_DIRTY; +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return false; + +return (pmd_flags (pmd) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; }static inline int pmd_young (pmd_t pmd)
@@ -170,9 +191,12 @@ static inline int pmd_young(pmd_t pmd)return pmd_flags (pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty (pud_t pud) +static inline bool pud_dirty (pud_t pud) { -return pud_flags (pud) & _PAGE_DIRTY; +/* +* A dirty PUD has Dirty=1 or Cow=1. +*/ +return pud_flags (pud) & _PAGE_DIRTY_BITS; }static inline int pud_young (pud_t pud)
@@ -182,13 +206,23 @@ static inline int pud_young(pud_t pud)static inline int pte_write (pte_t pte) { -return pte_flags (pte) & _PAGE_RW; +/* +* Shadow stack pages are always writable - but not by normal +* instructions, and only by shadow stack operations. Therefore, +* the W=0,D=1 test with pte_shstk(). +*/ +return (pte_flags (pte) & _PAGE_RW) ||pte_shstk (pte); }#define pmd_write pmd_writestatic inline int pmd_write (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_RW; +/* +* Shadow stack pages are always writable - but not by normal +* instructions, and only by shadow stack operations. Therefore, +* the W=0,D=1 test with pmd_shstk(). +*/ +return (pmd_flags (pmd) & _PAGE_RW) ||pmd_shstk (pmd); }#define pud_write pud_write
@@ -326,6 +360,24 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +static inline pte_t pte_mkcow (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_COW); +} + +static inline pte_t pte_clear_cow (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pte; + + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {
@@ -345,7 +397,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)static inline pte_t pte_mkclean (pte_t pte) { -return pte_clear_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_DIRTY_BITS); }static inline pte_t pte_mkold (pte_t pte)
@@ -355,7 +407,16 @@ static inline pte_t pte_mkold(pte_t pte)static inline pte_t pte_wrprotect (pte_t pte) { -return pte_clear_flags (pte, _PAGE_RW); + pte =pte_clear_flags (pte, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PTE (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pte_dirty (pte)) + pte =pte_mkcow (pte); +return pte; }static inline pte_t pte_mkexec (pte_t pte)
@@ -365,7 +426,18 @@ static inline pte_t pte_mkexec(pte_t pte)static inline pte_t pte_mkdirty (pte_t pte) { -return pte_set_flags (pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pteval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PTEs */ +if (cpu_feature_enabled (X86_FEATURE_SHSTK ) && !pte_write (pte)) + dirty = _PAGE_COW; + +return pte_set_flags (pte, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pte_t pte_mkwrite_shstk (pte_t pte) +{ +return pte_clear_cow (pte); }static inline pte_t pte_mkyoung (pte_t pte)
@@ -375,7 +447,12 @@ static inline pte_t pte_mkyoung(pte_t pte)static inline pte_t pte_mkwrite (pte_t pte) { -return pte_set_flags (pte, _PAGE_RW); + pte =pte_set_flags (pte, _PAGE_RW); + +if (pte_dirty (pte)) + pte =pte_clear_cow (pte); + +return pte; }static inline pte_t pte_mkhuge (pte_t pte)
@@ -422,6 +499,24 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +static inline pmd_t pmd_mkcow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_COW); +} + +static inline pmd_t pmd_clear_cow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd) {
@@ -446,17 +541,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd)static inline pmd_t pmd_mkclean (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_DIRTY_BITS); }static inline pmd_t pmd_wrprotect (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_RW); + pmd =pmd_clear_flags (pmd, _PAGE_RW); +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PMD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pmd_dirty (pmd)) + pmd =pmd_mkcow (pmd); +return pmd; }static inline pmd_t pmd_mkdirty (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pmdval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PMDs */ +if (cpu_feature_enabled (X86_FEATURE_SHSTK ) && !pmd_write (pmd)) + dirty = _PAGE_COW; + +return pmd_set_flags (pmd, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) +{ +return pmd_clear_cow (pmd); }static inline pmd_t pmd_mkdevmap (pmd_t pmd)
@@ -476,7 +590,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)static inline pmd_t pmd_mkwrite (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_RW); + pmd =pmd_set_flags (pmd, _PAGE_RW); + +if (pmd_dirty (pmd)) + pmd =pmd_clear_cow (pmd); +return pmd; }static inline pud_t pud_set_flags (pud_t pud,pudval_t set)
@@ -493,6 +611,24 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +static inline pud_t pud_mkcow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_COW); +} + +static inline pud_t pud_clear_cow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_COW); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);
@@ -500,17 +636,32 @@ static inline pud_t pud_mkold(pud_t pud)static inline pud_t pud_mkclean (pud_t pud) { -return pud_clear_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_DIRTY_BITS); }static inline pud_t pud_wrprotect (pud_t pud) { -return pud_clear_flags (pud, _PAGE_RW); + pud =pud_clear_flags (pud, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PUD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pud_dirty (pud)) + pud =pud_mkcow (pud); +return pud; }static inline pud_t pud_mkdirty (pud_t pud) { -return pud_set_flags (pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pudval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PUDs */ +if (cpu_feature_enabled (X86_FEATURE_SHSTK ) && !pud_write (pud)) + dirty = _PAGE_COW; + +return pud_set_flags (pud, dirty | _PAGE_SOFT_DIRTY); }static inline pud_t pud_mkdevmap (pud_t pud)
@@ -530,7 +681,11 @@ static inline pud_t pud_mkyoung(pud_t pud)static inline pud_t pud_mkwrite (pud_t pud) { -return pud_set_flags (pud, _PAGE_RW); + pud =pud_set_flags (pud, _PAGE_RW); + +if (pud_dirty (pud)) + pud =pud_clear_cow (pud); +return pud; }#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
arch/x86/include/asm/pgtable_types.h +41 -1
60 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v26 10/30 drm/i915/gvt: Change _PAGE_DIRTY to _PAGE_DIRTY_BITS
In reply to Yu-cheng YuAfter the introduction of _PAGE_COW, a modified page's PTE can have either _PAGE_DIRTY or _PAGE_COW. Change _PAGE_DIRTY to _PAGE_DIRTY_BITS.
--- drivers/gpu/drm/i915/gvt/gtt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/i915/gvt/gtt.c +1 -1
@@ -1216,7 +1216,7 @@ static int split_2MB_gtt_entry(struct intel_vgpu *vgpu, }/* Clear dirty field. */ - se->val64 &= ~_PAGE_DIRTY; + se->val64 &= ~_PAGE_DIRTY_BITS; ops->clear_pse (se); ops->clear_ips (se);
Signature · 1 line
2.21.0
PATCH v26 11/30 x86/mm: Update pte_modify for _PAGE_COW
In reply to Yu-cheng YuThe read-only and Dirty PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a read-only and Dirty PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_COW is created to replace _PAGE_DIRTY for the copy-on-write case, and pte_*() are updated.
Pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Introduce fixup_dirty_pte(), which sets a dirty PTE, based on _PAGE_RW, to either _PAGE_DIRTY or _PAGE_COW.
Apply the same changes to pmd_modify().
--- arch/x86/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
arch/x86/include/asm/pgtable.h +37 -0
@@ -799,6 +799,23 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd)static inline u64 flip_protnone_guard (u64 oldval,u64 val,u64 mask); +static inline pteval_t fixup_dirty_pte (pteval_t pteval) +{ +pte_t pte =__pte (pteval); + +/* +* Fix up potential shadow stack page flags because the RO, Dirty +* PTE is special. +*/ +if (cpu_feature_enabled (X86_FEATURE_SHSTK )) { +if (pte_dirty (pte)) { + pte =pte_mkclean (pte); + pte =pte_mkdirty (pte); + } + } +return pte_val (pte); +} +static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) {pteval_t val =pte_val (pte), oldval = val;
@@ -809,16 +826,36 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)*/ val &= _PAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_PAGE_CHG_MASK; + val =fixup_dirty_pte (val); val =flip_protnone_guard (oldval, val,PTE_PFN_MASK );return __pte (val); } +static inline int pmd_write (pmd_t pmd); +static inline pmdval_t fixup_dirty_pmd (pmdval_t pmdval) +{ +pmd_t pmd =__pmd (pmdval); + +/* +* Fix up potential shadow stack page flags because the RO, Dirty +* PMD is special. +*/ +if (cpu_feature_enabled (X86_FEATURE_SHSTK )) { +if (pmd_dirty (pmd)) { + pmd =pmd_mkclean (pmd); + pmd =pmd_mkdirty (pmd); + } + } +return pmd_val (pmd); +} +static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) {pmdval_t val =pmd_val (pmd), oldval = val; val &= _HPAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; + val =fixup_dirty_pmd (val); val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK );return __pmd (val); }
Signature · 1 line
2.21.0
PATCH v26 12/30 x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW
In reply to Yu-cheng YuWhen Shadow Stack is introduced, [R/O + _PAGE_DIRTY] PTE is reserved for shadow stack. Copy-on-write PTEs have [R/O + _PAGE_COW].
When a PTE goes from [R/W + _PAGE_DIRTY] to [R/O + _PAGE_COW], it could become a transient shadow stack PTE in two cases:
The first case is that some processors can start a write but end up seeing a read-only PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting Shadow Stack, and a TLB flush is not necessary.
The second case is that when _PAGE_DIRTY is replaced with _PAGE_COW non- atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.
Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.
--- arch/x86/include/asm/pgtable.h | 36 ++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
arch/x86/include/asm/pgtable.h +36 -0
@@ -1306,6 +1306,24 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +/* +* If Shadow Stack is enabled, pte_wrprotect() moves _PAGE_DIRTY +* to _PAGE_COW (see comments at pte_wrprotect()). +* When a thread reads a RW=1, Dirty=0 PTE and before changing it +* to RW=0, Dirty=0, another thread could have written to the page +* and the PTE is RW=1, Dirty=1 now. Use try_cmpxchg() to detect +* PTE changes and update old_pte, then try again. +*/ +if (cpu_feature_enabled (X86_FEATURE_SHSTK )) { +pte_t old_pte, new_pte; + + old_pte =READ_ONCE (*ptep); +do { + new_pte =pte_wrprotect (old_pte); + }while (!try_cmpxchg (&ptep->pte, &old_pte.pte, new_pte.pte)); + +return ; + }clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); }
@@ -1350,6 +1368,24 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +/* +* If Shadow Stack is enabled, pmd_wrprotect() moves _PAGE_DIRTY +* to _PAGE_COW (see comments at pmd_wrprotect()). +* When a thread reads a RW=1, Dirty=0 PMD and before changing it +* to RW=0, Dirty=0, another thread could have written to the page +* and the PMD is RW=1, Dirty=1 now. Use try_cmpxchg() to detect +* PMD changes and update old_pmd, then try again. +*/ +if (cpu_feature_enabled (X86_FEATURE_SHSTK )) { +pmd_t old_pmd, new_pmd; + + old_pmd =READ_ONCE (*pmdp); +do { + new_pmd =pmd_wrprotect (old_pmd); + }while (!try_cmpxchg ((pmdval_t *)pmdp, (pmdval_t *)&old_pmd,pmd_val (new_pmd))); + +return ; + }clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); }
Signature · 1 line
2.21.0
PATCH v26 13/30 mm: Introduce VM_SHADOW_STACK for shadow stack memory
In reply to Yu-cheng YuA shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.
---
v24:
- Change VM_SHSTK to VM_SHADOW_STACK.
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK to reflect Kconfig changes.
Documentation/filesystems/proc.rst | 1 + arch/x86/mm/mmap.c | 2 ++ fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 4 files changed, 14 insertions(+)
Documentation/filesystems/proc.rst +1 -0
@@ -549,6 +549,7 @@ encoded manner. The codes are the following: mg mergable advise flag bt arm64 BTI guarded page mt arm64 MTE allocation tags are enabled + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will
arch/x86/mm/mmap.c +2 -0
@@ -165,6 +165,8 @@ unsigned long get_mmap_base(int is_legacy)const char *arch_vma_name (struct vm_area_struct *vma) { +if (vma->vm_flags &VM_SHADOW_STACK ) +return "[shadow stack]" ;return NULL ; }
fs/proc/task_mmu.c +3 -0
@@ -661,6 +661,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) [ilog2(VM_PKEY_BIT4)] = "",#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_ARCH_HAS_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)]= "ss", +#endif };size_t i;
include/linux/mm.h +8 -0
@@ -312,11 +312,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -332,6 +334,12 @@ extern unsigned int kobjsize(const void *objp);#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_SHADOW_STACK +# define VM_SHADOW_STACK VM_HIGH_ARCH_5 +#else +# define VM_SHADOW_STACK VM_NONE +#endif +#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC )
Signature · 1 line
2.21.0
PATCH v26 14/30 x86/mm: Shadow Stack page fault error checking
In reply to Yu-cheng YuShadow stack accesses are those that are performed by the CPU where it expects to encounter a shadow stack mapping. These accesses are performed implicitly by CALL/RET at the site of the shadow stack pointer. These accesses are made explicitly by shadow stack management instructions like WRUSSQ.
Shadow stacks accesses to shadow-stack mapping can see faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write.
Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping.
In handling a shadow stack page fault, verify it occurs within a shadow stack mapping. It is always an error otherwise. For valid shadow stack accesses, set FAULT_FLAG_WRITE to effect copy-on-write. Because clearing _PAGE_DIRTY (vs. _PAGE_RW) is used to trigger the fault, shadow stack read fault and shadow stack write fault are not differentiated and both are handled as a write access.
---
v24:
- Change VM_SHSTK to VM_SHADOW_STACK.
arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 19 +++++++++++++++++++ 2 files changed, 21 insertions(+)
arch/x86/include/asm/trap_pf.h +2 -0
@@ -11,6 +11,7 @@* bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access +* bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code {
@@ -20,6 +21,7 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , +X86_PF_SHSTK =1 <<6 , X86_PF_SGX = 1 << 15, };
arch/x86/mm/fault.c +19 -0
@@ -1100,6 +1100,17 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; +/* +* Verify a shadow stack access is within a shadow stack VMA. +* It is always an error otherwise. Normal data access to a +* shadow stack area is checked in the case followed. +*/ +if (error_code &X86_PF_SHSTK ) { +if (!(vma->vm_flags &VM_SHADOW_STACK )) +return 1 ; +return 0 ; + } +if (error_code &X86_PF_WRITE ) {/* write, present and write, not present: */ if (unlikely (!(vma->vm_flags &VM_WRITE )))
@@ -1293,6 +1304,14 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* +* Clearing _PAGE_DIRTY is used to detect shadow stack access. +* This method cannot distinguish shadow stack read vs. write. +* For valid shadow stack accesses, set FAULT_FLAG_WRITE to effect +* copy-on-write. +*/ +if (error_code &X86_PF_SHSTK ) + flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_WRITE ) flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_INSTR )
Signature · 1 line
2.21.0
PATCH v26 15/30 x86/mm: Update maybe_mkwrite() for shadow stack
In reply to Yu-cheng YuWhen serving a page fault, maybe_mkwrite() makes a PTE writable if its vma has VM_WRITE.
A shadow stack vma has VM_SHADOW_STACK. Its PTEs have _PAGE_DIRTY, but not _PAGE_WRITE. In fork(), _PAGE_DIRTY is cleared to cause copy-on-write, and in the page fault handler, _PAGE_DIRTY is restored and the shadow stack page is writable again.
Introduce an x86 version of maybe_mkwrite(), which sets proper PTE bits according to VM flags.
Apply the same changes to maybe_pmd_mkwrite().
---
v24:
- Instead of doing arch_maybe_mkwrite(), overwrite maybe*_mkwrite() with x86 versions.
- Change VM_SHSTK to VM_SHADOW_STACK.
arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 20 ++++++++++++++++++++ include/linux/mm.h | 2 ++ mm/huge_memory.c | 2 ++ 4 files changed, 30 insertions(+)
arch/x86/include/asm/pgtable.h +6 -0
@@ -308,6 +308,9 @@ static inline int pmd_trans_huge(pmd_t pmd)return (pmd_val (pmd) & (_PAGE_PSE|_PAGE_DEVMAP)) == _PAGE_PSE; } +#define maybe_pmd_mkwrite maybe_pmd_mkwrite +extern pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static inline int pud_trans_huge (pud_t pud) {
@@ -1686,6 +1689,9 @@ static inline bool arch_faults_on_old_pte(void)return false; } +#define maybe_mkwrite maybe_mkwrite +extern pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma); +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_PGTABLE_H */
arch/x86/mm/pgtable.c +20 -0
@@ -610,6 +610,26 @@ int pmdp_clear_flush_young(struct vm_area_struct *vma, }#endif +pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +if (likely (vma->vm_flags &VM_WRITE )) + pte =pte_mkwrite (pte); +else if (likely (vma->vm_flags &VM_SHADOW_STACK )) + pte =pte_mkwrite_shstk (pte); +return pte; +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) +{ +if (likely (vma->vm_flags &VM_WRITE )) + pmd =pmd_mkwrite (pmd); +else if (likely (vma->vm_flags &VM_SHADOW_STACK )) + pmd =pmd_mkwrite_shstk (pmd); +return pmd; +} +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +/** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve
include/linux/mm.h +2 -0
@@ -993,12 +993,14 @@ void free_compound_page(struct page *page);* pte_mkwrite. But get_user_pages can cause write faults for mappings * that do not have writing enabled, when used by access_process_vm. */ +#ifndef maybe_mkwritestatic inline pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma) {if (likely (vma->vm_flags &VM_WRITE )) pte =pte_mkwrite (pte);return pte; } +#endif vm_fault_t do_set_pmd (struct vm_fault *vmf,struct page *page);void do_set_pte (struct vm_fault *vmf,struct page *page,unsigned long addr);
mm/huge_memory.c +2 -0
@@ -478,12 +478,14 @@ static int __init setup_transparent_hugepage(char *str) }__setup ("transparent_hugepage=" , setup_transparent_hugepage); +#ifndef maybe_pmd_mkwritepmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {if (likely (vma->vm_flags &VM_WRITE )) pmd =pmd_mkwrite (pmd);return pmd; } +#endif #ifdef CONFIG_MEMCG static inline struct deferred_split *get_deferred_split_queue (struct page *page)
Signature · 1 line
2.21.0
PATCH v26 16/30 mm: Fixup places that call pte_mkwrite() directly
In reply to Yu-cheng YuWhen serving a page fault, maybe_mkwrite() makes a PTE writable if it is in a writable vma. A shadow stack vma is writable, but its PTEs need _PAGE_DIRTY to be set to become writable. For this reason, maybe_mkwrite() has been updated.
There are a few places that call pte_mkwrite() directly, but have the same result as from maybe_mkwrite(). These sites need to be updated for shadow stack as well. Thus, change them to maybe_mkwrite():
- do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(), which is the same as maybe_mkwrite(). Change them to maybe_mkwrite().
- In do_numa_page(), if the numa entry was writable, then pte_mkwrite() is called directly. Fix it by doing maybe_mkwrite(). Make the same changes to do_huge_pmd_numa_page().
- In change_pte_range(), pte_mkwrite() is called directly. Replace it with maybe_mkwrite().
---
v25:
- Apply same changes to do_huge_pmd_numa_page() as to do_numa_page().
mm/huge_memory.c | 2 +- mm/memory.c | 5 ++--- mm/migrate.c | 3 +-- mm/mprotect.c | 2 +- 4 files changed, 5 insertions(+), 7 deletions(-)
mm/huge_memory.c +1 -1
@@ -1553,7 +1553,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) pmd =pmd_modify (pmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (was_writable) - pmd =pmd_mkwrite (pmd); + pmd =maybe_pmd_mkwrite (pmd, vma);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, pmd);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);unlock_page (page);
mm/memory.c +2 -3
@@ -3561,8 +3561,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)__SetPageUptodate (page); entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); + entry =maybe_mkwrite (pte_mkdirty (entry), vma); vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
@@ -4125,7 +4124,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) pte =pte_modify (old_pte, vma->vm_page_prot); pte =pte_mkyoung (pte);if (was_writable) - pte =pte_mkwrite (pte); + pte =maybe_mkwrite (pte, vma);ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte);update_mmu_cache (vma, vmf->address, vmf->pte);
mm/migrate.c +1 -2
@@ -2976,8 +2976,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } }else { entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); + entry =maybe_mkwrite (pte_mkdirty (entry), vma); } ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl);
mm/mprotect.c +1 -1
@@ -135,7 +135,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,if (dirty_accountable &&pte_dirty (ptent) && (pte_soft_dirty (ptent) || !(vma->vm_flags &VM_SOFTDIRTY ))) { - ptent =pte_mkwrite (ptent); + ptent =maybe_mkwrite (ptent, vma); }ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent); pages++;
Signature · 1 line
2.21.0
PATCH v26 17/30 mm: Add guard pages around a shadow stack.
In reply to Yu-cheng YuINCSSP(Q/D) increments shadow stack pointer and 'pops and discards' the first and the last elements in the range, effectively touches those memory areas.
The maximum moving distance by INCSSPQ is 255 * 8 = 2040 bytes and 255 * 4 = 1020 bytes by INCSSPD. Both ranges are far from PAGE_SIZE. Thus, putting a gap page on both ends of a shadow stack prevents INCSSP, CALL, and RET from going beyond.
---
v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.
v24:
- Instead changing vm_*_gap(), create x86-specific versions.
arch/x86/include/asm/page_types.h | 7 +++++ arch/x86/mm/mmap.c | 46 +++++++++++++++++++++++++++++++ include/linux/mm.h | 4 +++ 3 files changed, 57 insertions(+)
arch/x86/include/asm/page_types.h +7 -0
@@ -73,6 +73,13 @@ bool pfn_range_is_mapped(unsigned long start_pfn, unsigned long end_pfn);extern void initmem_init (void ); +#define vm_start_gap vm_start_gap +struct vm_area_struct ; +extern unsigned long vm_start_gap (struct vm_area_struct *vma); + +#define vm_end_gap vm_end_gap +extern unsigned long vm_end_gap (struct vm_area_struct *vma); +#endif /* !__ASSEMBLY__ */ #endif /* _ASM_X86_PAGE_DEFS_H */
arch/x86/mm/mmap.c +46 -0
49 lines not on this page · raw message
include/linux/mm.h +4 -0
@@ -2660,6 +2660,7 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * mreturn vma; } +#ifndef vm_start_gapstatic inline unsigned long vm_start_gap (struct vm_area_struct *vma) {unsigned long vm_start = vma->vm_start;
@@ -2671,7 +2672,9 @@ static inline unsigned long vm_start_gap(struct vm_area_struct *vma) }return vm_start; } +#endif +#ifndef vm_end_gapstatic inline unsigned long vm_end_gap (struct vm_area_struct *vma) {unsigned long vm_end = vma->vm_end;
@@ -2683,6 +2686,7 @@ static inline unsigned long vm_end_gap(struct vm_area_struct *vma) }return vm_end; } +#endif static inline unsigned long vma_pages (struct vm_area_struct *vma) {
Signature · 1 line
2.21.0
PATCH v26 18/30 mm/mmap: Add shadow stack pages to memory accounting
In reply to Yu-cheng YuAccount shadow stack pages to stack memory.
---
v26:
- Remove redundant #ifdef CONFIG_MMU.
v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().
v24:
- Change arch_shadow_stack_mapping() to is_shadow_stack_mapping().
- Change VM_SHSTK to VM_SHADOW_STACK.
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 5 +++++ include/linux/pgtable.h | 7 +++++++ mm/mmap.c | 5 +++++ 4 files changed, 20 insertions(+)
arch/x86/include/asm/pgtable.h +3 -0
@@ -1692,6 +1692,9 @@ static inline bool arch_faults_on_old_pte(void)#define maybe_mkwrite maybe_mkwriteextern pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma); +#define is_shadow_stack_mapping is_shadow_stack_mapping +extern bool is_shadow_stack_mapping (vm_flags_t vm_flags); +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_PGTABLE_H */
arch/x86/mm/pgtable.c +5 -0
@@ -897,3 +897,8 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +bool is_shadow_stack_mapping (vm_flags_t vm_flags) +{ +return vm_flags &VM_SHADOW_STACK ; +}
include/linux/pgtable.h +7 -0
@@ -1446,6 +1446,13 @@ static inline bool arch_has_pfn_modify_check(void) }#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shadow_stack_mapping +static inline bool is_shadow_stack_mapping (vm_flags_t vm_flags) +{ +return false; +} +#endif +/* * Architecture PAGE_KERNEL_* fallbacks *
mm/mmap.c +5 -0
@@ -1718,6 +1718,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags)if (file &&is_file_hugepages (file))return 0 ; +if (is_shadow_stack_mapping (vm_flags)) +return 1 ; +return (vm_flags & (VM_NORESERVE |VM_SHARED |VM_WRITE )) ==VM_WRITE ; }
@@ -3387,6 +3390,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; + else if (is_shadow_stack_mapping(flags)) + mm->stack_vm += npages; }static vm_fault_t special_mapping_fault (struct vm_fault *vmf);
Signature · 1 line
2.21.0
PATCH v26 19/30 mm: Update can_follow_write_pte() for shadow stack
In reply to Yu-cheng YuCan_follow_write_pte() ensures a read-only page is COWed by checking the FOLL_COW flag, and uses pte_dirty() to validate the flag is still valid.
Like a writable data page, a shadow stack page is writable, and becomes read-only during copy-on-write, but it is always dirty. Thus, in the can_follow_write_pte() check, it belongs to the writable page case and should be excluded from the read-only page pte_dirty() check. Apply the same changes to can_follow_write_pmd().
While at it, also split the long line into smaller ones.
---
v26:
- Instead of passing vm_flags, pass down vma pointer to can_follow_write_*().
v25:
- Split long line into smaller ones.
v24:
- Change arch_shadow_stack_mapping() to is_shadow_stack_mapping().
mm/gup.c | 16 ++++++++++++---- mm/huge_memory.c | 16 ++++++++++++---- 2 files changed, 24 insertions(+), 8 deletions(-)
mm/gup.c +12 -4
@@ -356,10 +356,18 @@ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address,* FOLL_FORCE can write to even unwritable pte's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pte (pte_t pte,unsigned int flags) +static inline bool can_follow_write_pte (pte_t pte,unsigned int flags, +struct vm_area_struct *vma) { -return pte_write (pte) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pte_dirty (pte)); +if (pte_write (pte)) +return true; +if ((flags & (FOLL_FORCE |FOLL_COW )) != (FOLL_FORCE |FOLL_COW )) +return false; +if (!pte_dirty (pte)) +return false; +if (is_shadow_stack_mapping (vma->vm_flags)) +return false; +return true; } static struct page *follow_page_pte(struct vm_area_struct *vma,
@@ -402,7 +410,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, }if ((flags &FOLL_NUMA ) &&pte_protnone (pte)) goto no_page; -if ((flags &FOLL_WRITE ) && !can_follow_write_pte (pte, flags)) { +if ((flags &FOLL_WRITE ) && !can_follow_write_pte (pte, flags, vma)) {pte_unmap_unlock (ptep, ptl);return NULL ; }
mm/huge_memory.c +12 -4
@@ -1338,10 +1338,18 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)* FOLL_FORCE can write to even unwritable pmd's, but only * after we've gone through a COW cycle and they are dirty. */ -static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags) +static inline bool can_follow_write_pmd (pmd_t pmd,unsigned int flags, +struct vm_area_struct *vma) { -return pmd_write (pmd) || - ((flags &FOLL_FORCE ) && (flags &FOLL_COW ) &&pmd_dirty (pmd)); +if (pmd_write (pmd)) +return true; +if ((flags & (FOLL_FORCE |FOLL_COW )) != (FOLL_FORCE |FOLL_COW )) +return false; +if (!pmd_dirty (pmd)) +return false; +if (is_shadow_stack_mapping (vma->vm_flags)) +return false; +return true; } struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
@@ -1354,7 +1362,7 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,assert_spin_locked (pmd_lockptr (mm, pmd)); -if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags)) +if (flags &FOLL_WRITE && !can_follow_write_pmd (*pmd, flags, vma)) goto out;/* Avoid dumping huge zero page */
Signature · 1 line
2.21.0
PATCH v26 20/30 mm/mprotect: Exclude shadow stack from preserve_write
In reply to Yu-cheng YuIn change_pte_range(), when a PTE is changed for prot_numa, _PAGE_RW is preserved to avoid the additional write fault after the NUMA hinting fault. However, pte_write() now includes both normal writable and shadow stack (RW=0, Dirty=1) PTEs, but the latter does not have _PAGE_RW and has no need to preserve it.
Exclude shadow stack from preserve_write test, and apply the same change to change_huge_pmd().
---
v25:
- Move is_shadow_stack_mapping() to a separate line.
v24:
- Change arch_shadow_stack_mapping() to is_shadow_stack_mapping().
mm/huge_memory.c | 7 +++++++ mm/mprotect.c | 7 +++++++ 2 files changed, 14 insertions(+)
mm/huge_memory.c +7 -0
@@ -1824,6 +1824,13 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,return 0 ; preserve_write = prot_numa &&pmd_write (*pmd); + +/* +* Preserve only normal writable huge PMD, but not shadow +* stack (RW=0, Dirty=1). +*/ +if (is_shadow_stack_mapping (vma->vm_flags)) + preserve_write = false; ret =1 ;#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
mm/mprotect.c +7 -0
@@ -77,6 +77,13 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,pte_t ptent;bool preserve_write = prot_numa &&pte_write (oldpte); +/* +* Preserve only normal writable PTE, but not shadow +* stack (RW=0, Dirty=1). +*/ +if (is_shadow_stack_mapping (vma->vm_flags)) + preserve_write = false; +/* * Avoid trapping faults against the zero or KSM * pages. See similar comment in change_huge_pmd.
Signature · 1 line
2.21.0
PATCH v26 21/30 mm: Re-introduce vm_flags to do_mmap()
In reply to Yu-cheng YuThere was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by:
commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()").
There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap().
---
v24:
- Change VM_SHSTK to VM_SHADOW_STACK.
- Update commit log.
fs/aio.c | 2 +- include/linux/mm.h | 3 ++- ipc/shm.c | 2 +- mm/mmap.c | 10 +++++----- mm/nommu.c | 4 ++-- mm/util.c | 2 +- 6 files changed, 12 insertions(+), 11 deletions(-)
fs/aio.c +1 -1
@@ -529,7 +529,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base =do_mmap (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE , -MAP_SHARED ,0 , &unused,NULL ); +MAP_SHARED ,0 ,0 , &unused,NULL );mmap_write_unlock (mm);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;
include/linux/mm.h +2 -1
@@ -2576,7 +2576,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf);extern unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flags, -unsigned long pgoff,unsigned long *populate,struct list_head *uf); +vm_flags_t vm_flags,unsigned long pgoff,unsigned long *populate, +struct list_head *uf);extern int __do_munmap (struct mm_struct *,unsigned long ,size_t ,struct list_head *uf,bool downgrade); extern int do_munmap(struct mm_struct *, unsigned long, size_t,
ipc/shm.c +1 -1
@@ -1556,7 +1556,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr =do_mmap (file, addr, size, prot, flags,0 , &populate,NULL ); + addr =do_mmap (file, addr, size, prot, flags,0 ,0 , &populate,NULL ); *raddr = addr; err =0 ;if (IS_ERR_VALUE (addr))
mm/mmap.c +5 -5
@@ -1401,11 +1401,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,*/ unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot, -unsigned long flags,unsigned long pgoff, -unsigned long *populate,struct list_head *uf) +unsigned long flags,vm_flags_t vm_flags, +unsigned long pgoff,unsigned long *populate, +struct list_head *uf) {struct mm_struct *mm = current->mm; -vm_flags_t vm_flags;int pkey =0 ; *populate =0 ;
@@ -1467,7 +1467,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,* to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags =calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | + vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ;if (flags &MAP_LOCKED )
@@ -3047,7 +3047,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, file =get_file (vma->vm_file); ret =do_mmap (vma->vm_file, start, size, - prot, flags, pgoff, &populate,NULL ); + prot, flags,0 , pgoff, &populate,NULL );fput (file); out:mmap_write_unlock (mm);
mm/nommu.c +2 -2
@@ -1071,6 +1071,7 @@ unsigned long do_mmap(struct file *file,unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf)
@@ -1078,7 +1079,6 @@ unsigned long do_mmap(struct file *file,struct vm_area_struct *vma;struct vm_region *region;struct rb_node *rb; -vm_flags_t vm_flags;unsigned long capabilities, result;int ret;
@@ -1097,7 +1097,7 @@ unsigned long do_mmap(struct file *file,/* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags =determine_vm_flags (file, prot, flags, capabilities); + vm_flags |=determine_vm_flags (file, prot, flags, capabilities);/* we're going to need to record the mapping */ region =kmem_cache_zalloc (vm_region_jar,GFP_KERNEL );
mm/util.c +1 -1
@@ -516,7 +516,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,if (!ret) {if (mmap_write_lock_killable (mm))return -EINTR ; - ret =do_mmap (file, addr, len, prot, flag, pgoff, &populate, + ret =do_mmap (file, addr, len, prot, flag,0 , pgoff, &populate, &uf);mmap_write_unlock (mm);userfaultfd_unmap_complete (mm, &uf);
Signature · 1 line
2.21.0
PATCH v26 22/30 x86/cet/shstk: Add user-mode shadow stack support
In reply to Yu-cheng YuIntroduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).
---
v25:
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Update alloc_shstk(), remove unused input flags.
v24:
- Rename cet.c to shstk.c, update related areas accordingly.
arch/x86/include/asm/cet.h | 29 ++++++++ arch/x86/include/asm/processor.h | 5 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/shstk.c | 123 +++++++++++++++++++++++++++++++ 4 files changed, 159 insertions(+) create mode 100644 arch/x86/include/asm/cet.h create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/cet.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CET_H +#define _ASM_X86_CET_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> + +struct task_struct ; +/* +* Per-thread CET status +*/ +struct cet_status { +unsigned long shstk_base; +unsigned long shstk_size; +}; + +#ifdef CONFIG_X86_SHADOW_STACK +int shstk_setup (void ); +void shstk_free (struct task_struct *p); +void shstk_disable (void ); +#else +static inline int shstk_setup (void ) {return 0 ; } +static inline void shstk_free (struct task_struct *p) {} +static inline void shstk_disable (void ) {} +#endif + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_CET_H */
arch/x86/include/asm/processor.h +5 -0
@@ -27,6 +27,7 @@ struct vm86;#include <asm/unwind_hints.h> #include <asm/vmxfeatures.h> #include <asm/vdso/processor.h> +#include <asm/cet.h> #include <linux/personality.h> #include <linux/cache.h>
@@ -535,6 +536,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; +#ifdef CONFIG_X86_SHADOW_STACK +struct cet_status cet; +#endif +/* Floating point and extended processor state */ struct fpu fpu;/*
arch/x86/kernel/Makefile +2 -0
@@ -150,6 +150,8 @@ obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o +obj-$(CONFIG_X86_SHADOW_STACK) += shstk.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/shstk.c new file +123 -0
123 lines not on this page · raw message
Signature · 1 line
2.21.0
Borislav Petkov
maintainerRe: [PATCH v26 22/30] x86/cet/shstk: Add user-mode shadow stack support
arch/x86/include/asm/processor.h struct thread_struct { · line 540
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/processor.h +2
-0@@ -535,6 +536,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; +#ifdef CONFIG_X86_SHADOW_STACK +struct cet_status cet;
A couple of versions ago I said:
" struct shstk_desc shstk;
or so"
but no movement here. That thing is still called cet_status even though there's nothing status-related with it.
So what's up?
arch/x86/kernel/shstk.c line 40
Quoted 5 lines
arch/x86/kernel/shstk.c new file +5
-0+static unsigned long alloc_shstk (unsigned long size) +{ +struct mm_struct *mm = current->mm; +unsigned long addr, populate; +int flags =MAP_ANONYMOUS |MAP_PRIVATE ;
The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order::
struct long_struct_name *descriptive_name; unsigned long foo, bar; unsigned int tmp; int ret;
The above is faster to parse than the reverse ordering::
int ret; unsigned int tmp; unsigned long foo, bar; struct long_struct_name *descriptive_name;
And even more so than random ordering::
unsigned long foo, bar; int ret; struct long_struct_name *descriptive_name; unsigned int tmp;
Please fix it up everywhere.
Quoted 18 lines
arch/x86/kernel/shstk.c new file +18
-0+mmap_write_lock (mm); + addr =do_mmap (NULL ,0 , size,PROT_READ , flags,VM_SHADOW_STACK ,0 , + &populate,NULL ); +mmap_write_unlock (mm); + +return addr; +} + +int shstk_setup (void ) +{ +unsigned long addr, size; +struct cet_status *cet = ¤t->thread.cet; + +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return -EOPNOTSUPP ; + + size = round_up(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G), PAGE_SIZE); + addr =alloc_shstk (size);
arch/x86/kernel/shstk.c line 69
Quoted 10 lines
arch/x86/kernel/shstk.c new file +10
-0+if (IS_ERR_VALUE (addr)) +return PTR_ERR ((void *)addr); + + cet->shstk_base = addr; + cet->shstk_size = size; + +start_update_msrs (); +wrmsrl (MSR_IA32_PL3_SSP , addr + size); +wrmsrl (MSR_IA32_U_CET ,CET_SHSTK_EN ); +end_update_msrs ();
<---- newline here.
Quoted 4 lines
arch/x86/kernel/shstk.c new file +4
-0+return 0 ; +} + +void shstk_free (struct task_struct *tsk)
arch/x86/kernel/shstk.c line 83
Quoted 10 lines
arch/x86/kernel/shstk.c new file +10
-0+{ +struct cet_status *cet = &tsk->thread.cet; + +if (!cpu_feature_enabled (X86_FEATURE_SHSTK ) || + !cet->shstk_size || + !cet->shstk_base) +return ; + +if (!tsk->mm) +return ;
Where are the comments you said you wanna add:
https://lkml.kernel.org/r/[email protected]
?
arch/x86/kernel/shstk.c line 88
Quoted 5 lines
arch/x86/kernel/shstk.c new file +5
-0+ +while (1 ) { +int r; + + r =vm_munmap (cet->shstk_base, cet->shstk_size);
int r = vm_munmap...
arch/x86/kernel/shstk.c line 100
Quoted 12 lines
arch/x86/kernel/shstk.c new file +12
-0+ +/* +* vm_munmap() returns -EINTR when mmap_lock is held by +* something else, and that lock should not be held for a +* long time. Retry it for the case. +*/ +if (r == -EINTR ) { +cond_resched (); +continue ; + } +break ; + }
vm_munmap() can return other negative error values, where are you handling those?
arch/x86/kernel/shstk.c line 108
Quoted 8 lines
arch/x86/kernel/shstk.c new file +8
-0+ + cet->shstk_base =0 ; + cet->shstk_size =0 ; +} + +void shstk_disable (void ) +{ +struct cet_status *cet = ¤t->thread.cet;
Same question as before: what guarantees that current doesn't change from under you here?
One of the worst thing to do is to ignore review comments. I'd strongly suggest you pay more attention and avoid that in the future.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 22/30] x86/cet/shstk: Add user-mode shadow stack support
Quoted 10 lines · Borislav Petkov
arch/x86/include/asm/processor.h struct thread_struct { · line 540
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/processor.h +2
-0@@ -535,6 +536,10 @@ struct thread_struct { unsigned int sig_on_uaccess_err:1; +#ifdef CONFIG_X86_SHADOW_STACK +struct cet_status cet;A couple of versions ago I said:
Quoted 9 lines
" struct
shstk_descshstk;or so"
but no movement here. That thing is still called
cet_statuseven though there's nothing status-related with it.So what's up?
Sorry about that. After that email thread, we went ahead to separate shadow stack and ibt into different files. I thought about the struct, the file names cet.h, etc. The struct still needs to include ibt status, and if it is shstk_desc, the name is not entirely true. One possible approach is, we don't make it a struct here, and put every item directly in thread_struct. However, the benefit of putting all in a struct is understandable (you might argue the opposite :-)). Please make the call, and I will do the change.
Quoted 21 lines
arch/x86/kernel/shstk.c line 40
Quoted 5 lines
arch/x86/kernel/shstk.c new file +5
-0+static unsigned long alloc_shstk (unsigned long size) +{ +struct mm_struct *mm = current->mm; +unsigned long addr, populate; +int flags =MAP_ANONYMOUS |MAP_PRIVATE ;The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order::
struct long_struct_name *descriptive_name; unsigned long foo, bar; unsigned int tmp; int ret;The above is faster to parse than the reverse ordering::
int ret; unsigned int tmp; unsigned long foo, bar; struct long_struct_name *descriptive_name;
Quoted 9 lines
And even more so than random ordering::
unsigned long foo, bar; int ret; struct long_struct_name *descriptive_name; unsigned int tmp;Please fix it up everywhere.
Ok!
Quoted 48 lines
Quoted 18 lines
arch/x86/kernel/shstk.c new file +18
-0+mmap_write_lock (mm); + addr =do_mmap (NULL ,0 , size,PROT_READ , flags,VM_SHADOW_STACK ,0 , + &populate,NULL ); +mmap_write_unlock (mm); + +return addr; +} + +int shstk_setup (void ) +{ +unsigned long addr, size; +struct cet_status *cet = ¤t->thread.cet; + +if (!cpu_feature_enabled (X86_FEATURE_SHSTK )) +return -EOPNOTSUPP ; + + size = round_up(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G), PAGE_SIZE); + addr =alloc_shstk (size);arch/x86/kernel/shstk.c line 69
Quoted 10 lines
arch/x86/kernel/shstk.c new file +10
-0+if (IS_ERR_VALUE (addr)) +return PTR_ERR ((void *)addr); + + cet->shstk_base = addr; + cet->shstk_size = size; + +start_update_msrs (); +wrmsrl (MSR_IA32_PL3_SSP , addr + size); +wrmsrl (MSR_IA32_U_CET ,CET_SHSTK_EN ); +end_update_msrs ();<---- newline here.
Quoted 4 lines
arch/x86/kernel/shstk.c new file +4
-0+return 0 ; +} + +void shstk_free (struct task_struct *tsk)arch/x86/kernel/shstk.c line 83
Quoted 10 lines
arch/x86/kernel/shstk.c new file +10
-0+{ +struct cet_status *cet = &tsk->thread.cet; + +if (!cpu_feature_enabled (X86_FEATURE_SHSTK ) || + !cet->shstk_size || + !cet->shstk_base) +return ; + +if (!tsk->mm) +return ;Where are the comments you said you wanna add:
Quoted 4 lines
https://lkml.kernel.org/r/[email protected]
?
Yes, the comments are in patch #23: Handle thread shadow stack. I wanted to add that in the patch that takes the path.
Quoted 8 lines
arch/x86/kernel/shstk.c line 88
Quoted 5 lines
arch/x86/kernel/shstk.c new file +5
-0+ +while (1 ) { +int r; + + r =vm_munmap (cet->shstk_base, cet->shstk_size);int r =
vm_munmap...
Quoted 16 lines
arch/x86/kernel/shstk.c line 100
Quoted 12 lines
arch/x86/kernel/shstk.c new file +12
-0+ +/* +* vm_munmap() returns -EINTR when mmap_lock is held by +* something else, and that lock should not be held for a +* long time. Retry it for the case. +*/ +if (r == -EINTR ) { +cond_resched (); +continue ; + } +break ; + }
vm_munmap()can return other negative error values, where are you handling those?
For other error values, the loop stops.
Quoted 11 lines
arch/x86/kernel/shstk.c line 108
Quoted 8 lines
arch/x86/kernel/shstk.c new file +8
-0+ + cet->shstk_base =0 ; + cet->shstk_size =0 ; +} + +void shstk_disable (void ) +{ +struct cet_status *cet = ¤t->thread.cet;Same question as before: what guarantees that current doesn't change from under you here?
The actual reading/writing MSRs are protected by fpregs_lock().
Quoted 6 lines
One of the worst thing to do is to ignore review comments. I'd strongly suggest you pay more attention and avoid that in the future.
Thx.
Borislav Petkov
maintainerRe: [PATCH v26 22/30] x86/cet/shstk: Add user-mode shadow stack support
Quoted 7 lines · Yu-cheng
Sorry about that. After that email thread, we went ahead to separate shadow stack and ibt into different files. I thought about the struct, the file names
cet.h, etc. The struct still needs to include ibt status, and if it isshstk_desc, the name is not entirely true. One possible approach is, we don't make it a struct here, and put every item directly inthread_struct. However, the benefit of putting all in a struct is understandable (you might argue the opposite :-)). Please make the call, and I will do the change.
/me looks forward into the patchset...
So this looks like the final version of it:
@@ -15,6 +15,7 @@ struct cet_status {
unsigned long shstk_base;
unsigned long shstk_size;
unsigned int locked:1;
+ unsigned int ibt_enabled:1;
};
If so, that thing should be simply:
struct cet {
unsigned long shstk_base;
unsigned long shstk_size;
unsigned int shstk_lock : 1,
ibt : 1;
}Is that ibt flag per thread or why is it here? I guess I'll find out.
/me greps...
ah yes, it is.
Quoted 2 lines
Yes, the comments are in patch #23: Handle thread shadow stack. I wanted to add that in the patch that takes the path.
That comes next, I'll look there.
Quoted 5 lines
Quoted 3 lines
vm_munmap()can return other negative error values, where are you handling those?For other error values, the loop stops.
And then what happens?
Quoted 2 lines
Quoted 2 lines
arch/x86/kernel/shstk.c line 103
Quoted 2 lines
arch/x86/kernel/shstk.c new file +2
-0+ cet->shstk_base =0 ; + cet->shstk_size =0 ;
You clear those here without even checking whether unmap failed somehow. And then stuff leaks but we don't care, right?
Someone else's problem, I'm sure.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 22/30] x86/cet/shstk: Add user-mode shadow stack support
Quoted 29 lines · Borislav Petkov
Quoted 7 lines · Yu-cheng
Sorry about that. After that email thread, we went ahead to separate shadow stack and ibt into different files. I thought about the struct, the file names
cet.h, etc. The struct still needs to include ibt status, and if it isshstk_desc, the name is not entirely true. One possible approach is, we don't make it a struct here, and put every item directly inthread_struct. However, the benefit of putting all in a struct is understandable (you might argue the opposite :-)). Please make the call, and I will do the change./me looks forward into the patchset...
So this looks like the final version of it:
(quoted diff) +1
-0@@ -15,6 +15,7 @@ struct cet_status { unsigned long shstk_base; unsigned long shstk_size; unsigned int locked:1; + unsigned int ibt_enabled:1; };If so, that thing should be simply:
struct cet { unsigned long shstk_base; unsigned long shstk_size; unsigned int shstk_lock : 1, ibt : 1; }
Quoted 6 lines
Is that ibt flag per thread or why is it here? I guess I'll find out.
/me greps...
ah yes, it is.
The lock applies to both shadow stack and ibt. So maybe just "locked"?
Quoted 13 lines
Quoted 2 lines
Yes, the comments are in patch #23: Handle thread shadow stack. I wanted to add that in the patch that takes the path.
That comes next, I'll look there.
Quoted 5 lines
Quoted 3 lines
vm_munmap()can return other negative error values, where are you handling those?For other error values, the loop stops.
And then what happens?
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
arch/x86/kernel/shstk.c line 103
Quoted 2 lines
arch/x86/kernel/shstk.c new file +2
-0+ cet->shstk_base =0 ; + cet->shstk_size =0 ;You clear those here without even checking whether unmap failed somehow. And then stuff leaks but we don't care, right?
Someone else's problem, I'm sure.
vm_munmap() returns error as the following:
(1) -EINVAL: address/size/alignment is wrong.
For shadow stack, the kernel keeps track of it, this cannot/should not
happen. Should it happen, it is a bug. The kernel can probably do WARN().
(2) -ENOMEM: when doing __split_vma()/__vma_adjust(), kmem_cache_alloc() fails. Not much we can do. Perhaps WARN()?
(3) -EINTR: mmap_write_lock_killable(mm) fails.
This should only happen to a pthread. When a thread is existing, its
siblings are holding mm->mmap_lock. This is handled here.
Right now, in the kernel, only the munmap() syscall returns __vm_munmap() error code, otherwise the error is not checked. Within the kernel and if -EINTR is not expected, this makes sense as explained above.
Thanks for questioning. This piece needs to be correct.
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 22/30] x86/cet/shstk: Add user-mode shadow stack support
Quoted 1 line · Yu-cheng
The lock applies to both shadow stack and ibt. So maybe just "locked"?
Sure.
Quoted 5 lines
vm_munmap()returns error as the following:(1)
-EINVAL: address/size/alignment is wrong.
For shadow stack, the kernel keeps track of it, this cannot/should not
happen.
You mean nothing might corrupt
cet->shstk_base
cet->shstk_size?
I can't count the ways I've heard "should not happen" before and then it happening anyway.
So probably not but we better catch stuff like that instead of leaking.
Quoted 1 line
Should it happen, it is a bug.
Ack.
Quoted 1 line
The kernel can probably do
WARN().
Most definitely WARN. You need to catch funsies like that. But WARN_ONCE should be enough for now.
Quoted 3 lines
(2)
-ENOMEM: when doing __split_vma()/__vma_adjust(),kmem_cache_alloc()fails.
Not much we can do. PerhapsWARN()?
You got it.
Bottom line is: if you can check for this and it is cheap, then definitely. Code changes, gets rewritten, reorganized, the old assertions change significance, and so on...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v26 23/30 x86/cet/shstk: Handle thread shadow stack
In reply to Yu-cheng YuFor clone() with CLONE_VM specified, the child and the parent must have separate shadow stacks. Thus, the kernel allocates, and frees on thread exit a new shadow stack for the child.
Use stack_size passed from clone3() syscall for thread shadow stack size, but cap it to min(RLIMIT_STACK, 4 GB). A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32- bit address space.
--- arch/x86/include/asm/cet.h | 5 +++ arch/x86/include/asm/mmu_context.h | 3 ++ arch/x86/kernel/process.c | 15 ++++++-- arch/x86/kernel/shstk.c | 57 +++++++++++++++++++++++++++++- 4 files changed, 76 insertions(+), 4 deletions(-)
arch/x86/include/asm/cet.h +5 -0
@@ -16,10 +16,15 @@ struct cet_status {#ifdef CONFIG_X86_SHADOW_STACK int shstk_setup (void ); +int shstk_setup_thread (struct task_struct *p,unsigned long clone_flags, +unsigned long stack_size);void shstk_free (struct task_struct *p);void shstk_disable (void );#else static inline int shstk_setup (void ) {return 0 ; } +static inline int shstk_setup_thread (struct task_struct *p, +unsigned long clone_flags, +unsigned long stack_size) {return 0 ; }static inline void shstk_free (struct task_struct *p) {}static inline void shstk_disable (void ) {}#endif
arch/x86/include/asm/mmu_context.h +3 -0
@@ -11,6 +11,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> +#include <asm/cet.h> #include <asm/debugreg.h> extern atomic64_t last_mm_ctx_id;
@@ -146,6 +147,8 @@ do { \#else #define deactivate_mm (tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + shstk_free(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0)
arch/x86/kernel/process.c +12 -3
@@ -43,6 +43,7 @@#include <asm/io_bitmap.h> #include <asm/proto.h> #include <asm/frame.h> +#include <asm/cet.h> #include "process.h"
@@ -109,6 +110,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t); +shstk_free (tsk);fpu__drop (fpu); }
@@ -122,8 +124,9 @@ static int set_new_tls(struct task_struct *p, unsigned long tls)return do_set_thread_area_64 (p,ARCH_SET_FS , tls); } -int copy_thread (unsigned long clone_flags,unsigned long sp,unsigned long arg, -struct task_struct *p,unsigned long tls) +int copy_thread (unsigned long clone_flags,unsigned long sp, +unsigned long stack_size,struct task_struct *p, +unsigned long tls) {struct inactive_task_frame *frame;struct fork_frame *fork_frame;
@@ -163,7 +166,7 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg,/* Kernel thread ? */ if (unlikely (p->flags & (PF_KTHREAD |PF_IO_WORKER ))) {memset (childregs,0 ,sizeof (struct pt_regs )); -kthread_frame_init (frame, sp, arg); +kthread_frame_init (frame, sp, stack_size);return 0 ; }
@@ -181,6 +184,12 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg,if (clone_flags &CLONE_SETTLS ) ret =set_new_tls (p, tls); +#ifdef CONFIG_X86_64 +/* Allocate a new shadow stack for pthread */ +if (!ret) + ret =shstk_setup_thread (p, clone_flags, stack_size); +#endif +if (!ret &&unlikely (test_tsk_thread_flag (current,TIF_IO_BITMAP )))io_bitmap_share (p);
arch/x86/kernel/shstk.c +56 -1
69 lines not on this page · raw message
Signature · 1 line
2.21.0
Borislav Petkov
maintainerRe: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
arch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 187
Quoted 5 lines · Yu-cheng Yu
arch/x86/kernel/process.c +1
-0@@ -181,6 +184,12 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg,if (clone_flags &CLONE_SETTLS ) ret =set_new_tls (p, tls); +#ifdef CONFIG_X86_64
IS_ENABLED
arch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 192
Quoted 5 lines
arch/x86/kernel/process.c +5
-0+/* Allocate a new shadow stack for pthread */ +if (!ret) + ret =shstk_setup_thread (p, clone_flags, stack_size); +#endif +
And why is this addition here...
arch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 194
Quoted 2 lines
arch/x86/kernel/process.c +0
-0if (!ret &&unlikely (test_tsk_thread_flag (current,TIF_IO_BITMAP )))io_bitmap_share (p);
... instead of here?
<---
arch/x86/kernel/shstk.c int shstk_setup(void) · line 73
Quoted 10 lines
arch/x86/kernel/shstk.c +1
-0@@ -70,6 +70,55 @@ int shstk_setup(void)return 0 ; } +int shstk_setup_thread(struct task_struct *tsk, unsigned long clone_flags,
Judging by what this function does, its name wants to be
shstk_alloc_thread_stack()
or so?
arch/x86/kernel/shstk.c int shstk_setup(void) · line 78
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+ unsigned long stack_size) +{ +unsigned long addr, size; +struct cet_user_state *state; +struct cet_status *cet = &tsk->thread.cet;
The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order::
struct long_struct_name *descriptive_name; unsigned long foo, bar; unsigned int tmp; int ret;
The above is faster to parse than the reverse ordering::
int ret; unsigned int tmp; unsigned long foo, bar; struct long_struct_name *descriptive_name;
And even more so than random ordering::
unsigned long foo, bar; int ret; struct long_struct_name *descriptive_name; unsigned int tmp;
arch/x86/kernel/shstk.c int shstk_setup(void) · line 82
Quoted 4 lines
arch/x86/kernel/shstk.c +4
-0+ +if (!cet->shstk_size) +return 0 ; +
This check needs a comment.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 87
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+if ((clone_flags & (CLONE_VFORK |CLONE_VM )) !=CLONE_VM ) +return 0 ; + + state =get_xsave_addr (&tsk->thread.fpu.state.xsave, +XFEATURE_CET_USER );
Let that line stick out.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 92
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+ +if (!state) +return -EINVAL ; + +if (stack_size ==0 )
if (!stack_size)
arch/x86/kernel/shstk.c int shstk_setup(void) · line 93
Quoted 1 line
arch/x86/kernel/shstk.c +1
-0+return -EINVAL ;
and that test needs to be done first in the function.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 95
Quoted 2 lines
arch/x86/kernel/shstk.c +2
-0+ +/* Cap shadow stack size to 4 GB */
Why?
arch/x86/kernel/shstk.c int shstk_setup(void) · line 105
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+ size = min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G); + size =min (size, stack_size); + +/* +* Compat-mode pthreads share a limited address space. +* If each function call takes an average of four slots +* stack space, allocate 1/4 of stack size for shadow stack. +*/ +if (in_compat_syscall ()) + size /=4 ;
<---- newline here.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 108
Quoted 3 lines
arch/x86/kernel/shstk.c +3
-0+ size =round_up (size,PAGE_SIZE ); + addr =alloc_shstk (size); +
^ Superfluous newline.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 116
Quoted 8 lines
arch/x86/kernel/shstk.c +8
-0+if (IS_ERR_VALUE (addr)) { + cet->shstk_base =0 ; + cet->shstk_size =0 ; +return PTR_ERR ((void *)addr); + } + +fpu__prepare_write (&tsk->thread.fpu); + state->user_ssp = (u64)(addr + size);
cet_user_state has u64, cet_status has unsigned longs. Make them all u64.
And since cet_status is per thread, but I had suggested struct shstk_desc, I think now that that should be called
struct thread_shstk
or so to denote *exactly* what it is.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
Quoted 1 line · Borislav Petkov
On Tue, Apr 27, 2021 at 01:43:08PM -0700, Yu-cheng Yu wrote:
[...]
Quoted 67 lines
arch/x86/kernel/shstk.c int shstk_setup(void) · line 73
Quoted 9 lines
arch/x86/kernel/shstk.c +1
-0@@ -70,6 +70,55 @@ int shstk_setup(void)return 0 ; } +int shstk_setup_thread(struct task_struct *tsk, unsigned long clone_flags,Judging by what this function does, its name wants to be
shstk_alloc_thread_stack()or so?
arch/x86/kernel/shstk.c int shstk_setup(void) · line 78
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+ unsigned long stack_size) +{ +unsigned long addr, size; +struct cet_user_state *state; +struct cet_status *cet = &tsk->thread.cet;The tip-tree preferred ordering of variable declarations at the beginning of a function is reverse fir tree order::
struct long_struct_name *descriptive_name; unsigned long foo, bar; unsigned int tmp; int ret;The above is faster to parse than the reverse ordering::
int ret; unsigned int tmp; unsigned long foo, bar; struct long_struct_name *descriptive_name;And even more so than random ordering::
unsigned long foo, bar; int ret; struct long_struct_name *descriptive_name; unsigned int tmp;arch/x86/kernel/shstk.c int shstk_setup(void) · line 82
Quoted 4 lines
arch/x86/kernel/shstk.c +4
-0+ +if (!cet->shstk_size) +return 0 ; +This check needs a comment.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 87
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+if ((clone_flags & (CLONE_VFORK |CLONE_VM )) !=CLONE_VM ) +return 0 ; + + state =get_xsave_addr (&tsk->thread.fpu.state.xsave, +XFEATURE_CET_USER );Let that line stick out.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 92
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+ +if (!state) +return -EINVAL ; + +if (stack_size ==0 )if (!stack_size)
Quoted 9 lines
arch/x86/kernel/shstk.c int shstk_setup(void) · line 93
Quoted 1 line
arch/x86/kernel/shstk.c +1
-0+return -EINVAL ;and that test needs to be done first in the function.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 95
Quoted 2 lines
arch/x86/kernel/shstk.c +2
-0+ +/* Cap shadow stack size to 4 GB */ Why?
This is not necessary. I will make it just stack_size, which is passed in from copy_thread().
Quoted 30 lines
arch/x86/kernel/shstk.c int shstk_setup(void) · line 105
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+ size = min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G); + size =min (size, stack_size); + +/* +* Compat-mode pthreads share a limited address space. +* If each function call takes an average of four slots +* stack space, allocate 1/4 of stack size for shadow stack. +*/ +if (in_compat_syscall ()) + size /=4 ;<---- newline here.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 108
Quoted 3 lines
arch/x86/kernel/shstk.c +3
-0+ size =round_up (size,PAGE_SIZE ); + addr =alloc_shstk (size); +^ Superfluous newline.
arch/x86/kernel/shstk.c int shstk_setup(void) · line 116
Quoted 8 lines
arch/x86/kernel/shstk.c +8
-0+if (IS_ERR_VALUE (addr)) { + cet->shstk_base =0 ; + cet->shstk_size =0 ; +return PTR_ERR ((void *)addr); + } + +fpu__prepare_write (&tsk->thread.fpu); + state->user_ssp = (u64)(addr + size);
cet_user_statehas u64,cet_statushas unsigned longs. Make them all u64.
Quoted 7 lines
And since
cet_statusis per thread, but I had suggested structshstk_desc, I think now that that should be calledstruct
thread_shstkor so to denote *exactly* what it is.
So this struct will be:
struct thread_shstk {
u64 shstk_base;
u64 shstk_size;
u64 locked:1;
u64 ibt:1;
};Ok?
Thanks,
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
Quoted 10 lines · Yu-cheng
So this struct will be:
struct thread_shstk { u64 shstk_base; u64 shstk_size; u64 locked:1; u64 ibt:1; };Ok?
Pretty much.
You can even remove the "shstk_" from the members and when you call the pointer "shstk", accessing the members will read
shstk->base shstk->size ...
and all is organic and readable :)
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRE: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
From: Borislav Petkov
Quoted 10 lines
Sent: 11 May 2021 18:10
Quoted 7 lines · Yu-cheng
So this struct will be:
struct thread_shstk { u64 shstk_base; u64 shstk_size; u64 locked:1; u64 ibt:1;
No point in bit fields?
Quoted 6 lines
Quoted 3 lines
};
Ok?
Pretty much.
Quoted 8 lines
You can even remove the "
shstk_" from the members and when you call the pointer "shstk", accessing the members will readshstk->base shstk->size ...and all is organic and readable :)
And entirely not greppable.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
In reply to Borislav PetkovQuoted 17 lines · Borislav Petkov
arch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 187
Quoted 5 lines · Yu-cheng Yu
arch/x86/kernel/process.c +1
-0@@ -181,6 +184,12 @@ int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg,if (clone_flags &CLONE_SETTLS ) ret =set_new_tls (p, tls); +#ifdef CONFIG_X86_64
IS_ENABLEDarch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 192
Quoted 5 lines
arch/x86/kernel/process.c +5
-0+/* Allocate a new shadow stack for pthread */ +if (!ret) + ret =shstk_setup_thread (p, clone_flags, stack_size); +#endif +And why is this addition here...
Quoted 7 lines
arch/x86/kernel/process.c int copy_thread(unsigned long clone_flags, unsigned long sp, unsigned long arg, · line 194
Quoted 2 lines
arch/x86/kernel/process.c +0
-0if (!ret &&unlikely (test_tsk_thread_flag (current,TIF_IO_BITMAP )))io_bitmap_share (p);... instead of here?
<---
io_bitmap_share() does refcount_inc(¤t->thread.io_bitmap->refcnt), and the function won't fail. However, shadow stack allocation can fail.
So, maybe leave io_bitmap_share() at the end?
Thanks,
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 23/30] x86/cet/shstk: Handle thread shadow stack
Quoted 3 lines · Yu-cheng
io_bitmap_share()doesrefcount_inc(¤t->thread.io_bitmap->refcnt), and the function won't fail. However, shadow stack allocation can fail. So, maybe leaveio_bitmap_share()at the end?
Yap, makes sense.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v26 24/30 x86/cet/shstk: Introduce shadow stack token setup/verify routines
In reply to Yu-cheng YuA shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.
The restore token can be used as an extra protection for signal handling. To deliver a signal, create a shadow stack restore token and put the token and the signal restorer address on the shadow stack. In sigreturn, verify the token and restore from it the shadow stack pointer.
Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack. It is used to construct user signal stack as described above.
---
v25:
- Update inline assembly syntax, use %[].
- Change token address from (unsigned long) to (u64/u32 __user *).
- Change -EPERM to -EFAULT.
arch/x86/include/asm/cet.h | 9 ++ arch/x86/include/asm/special_insns.h | 32 +++++++ arch/x86/kernel/shstk.c | 126 +++++++++++++++++++++++++++ 3 files changed, 167 insertions(+)
arch/x86/include/asm/cet.h +9 -0
@@ -20,6 +20,10 @@ int shstk_setup_thread(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size);void shstk_free (struct task_struct *p);void shstk_disable (void ); +int shstk_setup_rstor_token (bool ia32,unsigned long rstor, +unsigned long *token_addr,unsigned long *new_ssp); +int shstk_check_rstor_token (bool ia32,unsigned long token_addr, +unsigned long *new_ssp);#else static inline int shstk_setup (void ) {return 0 ; } static inline int shstk_setup_thread(struct task_struct *p,
@@ -27,6 +31,11 @@ static inline int shstk_setup_thread(struct task_struct *p, unsigned long stack_size) { return 0; }static inline void shstk_free (struct task_struct *p) {}static inline void shstk_disable (void ) {} +static inline int shstk_setup_rstor_token (bool ia32,unsigned long rstor, +unsigned long *token_addr, +unsigned long *new_ssp) {return 0 ; } +static inline int shstk_check_rstor_token (bool ia32,unsigned long token_addr, +unsigned long *new_ssp) {return 0 ; }#endif #endif /* __ASSEMBLY__ */
arch/x86/include/asm/special_insns.h +32 -0
@@ -234,6 +234,38 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_SHADOW_STACK +#if defined(CONFIG_IA32_EMULATION ) || defined(CONFIG_X86_X32 ) +static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ + asm_volatile_goto("1: wrussd %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); +return 0 ; +fail: +return -EFAULT ; +} +#else +static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ +WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); +return -EFAULT ; +} +#endif + +static inline int write_user_shstk_64 (u64 __user *addr,u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); +return 0 ; +fail: +return -EFAULT ; +} +#endif /* CONFIG_X86_SHADOW_STACK */ +#define nop () asm volatile ("nop")static inline void serialize (void )
arch/x86/kernel/shstk.c +126 -0
135 lines not on this page · raw message
Signature · 1 line
2.21.0
Borislav Petkov
maintainerRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 255
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/special_insns.h +6
-0+static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ +WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); +return -EFAULT ; +} +#endif
What is that supposed to catch? Any concrete (mis-)use cases?
Quoted 29 lines
arch/x86/include/asm/special_insns.h +13
-0+ +static inline int write_user_shstk_64 (u64 __user *addr,u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); +return 0 ; +fail: +return -EFAULT ; +} +#endif /* CONFIG_X86_SHADOW_STACK */ +#define nop () asm volatile ("nop")static inline void serialize (void )arch/x86/kernel/shstk.c +1
-0@@ -20,6 +20,7 @@#include <asm/fpu/xstate.h> #include <asm/fpu/types.h> #include <asm/cet.h> +#include <asm/special_insns.h> static void start_update_msrs (void ) {
arch/x86/kernel/shstk.c void shstk_disable(void) · line 181
Quoted 5 lines
arch/x86/kernel/shstk.c +2
-0@@ -176,3 +177,128 @@ void shstk_disable(void)shstk_free (current); } + +static unsigned long _get_user_shstk_addr (void )
What's the "_" prefix in the name supposed to denote?
Ditto for the other functions with "_" prefix you're adding.
Quoted 7 lines
arch/x86/kernel/shstk.c +7
-0+{ +struct fpu *fpu = ¤t->thread.fpu; +unsigned long ssp =0 ; + +fpregs_lock (); + +if (fpregs_state_valid (fpu,smp_processor_id ())) {
arch/x86/kernel/shstk.c void shstk_disable(void) · line 198
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+rdmsrl (MSR_IA32_PL3_SSP , ssp); + }else { +struct cet_user_state *p; + + p =get_xsave_addr (&fpu->state.xsave,XFEATURE_CET_USER ); +if (p) + ssp = p->user_ssp; + } + +fpregs_unlock ();
<---- newline here.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 205
Quoted 7 lines
arch/x86/kernel/shstk.c +7
-0+return ssp; +} + +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64 (token) (((token) & TOKEN_MODE_MASK) == TOKEN_MODE_64) +#define IS_TOKEN_32 (token) (((token) & TOKEN_MODE_MASK) == 0)
Why do you have to look at the second, busy bit, too in order to determine the mode?
Also, you don't need most of those defines - see below.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 216
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+/* +* Create a restore token on the shadow stack. A token is always 8-byte +* and aligned to 8. +*/ +static int _create_rstor_token (bool ia32,unsigned long ssp, +unsigned long *token_addr) +{ +unsigned long addr; + + *token_addr =0 ;
What for? Callers should check this function's retval and then interpret the validity of token_addr and it should not unconditionally write into it.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 217
Quoted 2 lines
arch/x86/kernel/shstk.c +2
-0+ +if ((!ia32 && !IS_ALIGNED (ssp,8 )) || !IS_ALIGNED (ssp,4 ))
Flip this logic:
if ((ia32 && !IS_ALIGNED(ssp, 4)) || !IS_ALIGNED(ssp, 8))
arch/x86/kernel/shstk.c void shstk_disable(void) · line 221
Quoted 3 lines
arch/x86/kernel/shstk.c +3
-0+return -EINVAL ; + + addr =ALIGN_DOWN (ssp,8 ) -8 ;
Yah, so this is weird. Why does the restore token need to be at -8 instead on the shadow stack address itself?
Looking at
Figure 18-2. RSTORSSP to Switch to New Shadow Stack Figure 18-3. SAVEPREVSSP to Save a Restore Point
in the SDM, it looks like unnecessarily more complex than it should be. But maybe there's some magic I'm missing.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 225
Quoted 4 lines
arch/x86/kernel/shstk.c +4
-0+ +/* Is the token for 64-bit? */ +if (!ia32) + ssp |=TOKEN_MODE_64 ;
|= BIT(0);
arch/x86/kernel/shstk.c void shstk_disable(void) · line 230
Quoted 5 lines
arch/x86/kernel/shstk.c +5
-0+ +if (write_user_shstk_64 ((u64 __user *)addr, (u64 )ssp)) +return -EFAULT ; + + *token_addr = addr;
<---- newline here.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 243
Quoted 13 lines
arch/x86/kernel/shstk.c +13
-0+return 0 ; +} + +/* +* Create a restore token on shadow stack, and then push the user-mode +* function return address. +*/ +int shstk_setup_rstor_token (bool ia32,unsigned long ret_addr, +unsigned long *token_addr,unsigned long *new_ssp) +{ +struct cet_status *cet = ¤t->thread.cet; +unsigned long ssp =0 ; +int err =0 ;
What are those cleared to 0 for?
arch/x86/kernel/shstk.c void shstk_disable(void) · line 244
Quoted 2 lines
arch/x86/kernel/shstk.c +2
-0+ +if (cet->shstk_size) {
Flip logic to save an indentation level:
if (!cet->shstk_size) return err; if (!ret_addr) ...
arch/x86/kernel/shstk.c void shstk_disable(void) · line 249
Quoted 4 lines
arch/x86/kernel/shstk.c +4
-0+if (!ret_addr) +return -EINVAL ; + + ssp =_get_user_shstk_addr ();
Needs to test retval for 0 here and return error if so.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 259
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+ err =_create_rstor_token (ia32, ssp, token_addr); +if (err) +return err; + +if (ia32) { + *new_ssp = *token_addr -sizeof (u32); + err =write_user_shstk_32 ((u32 __user *)*new_ssp, (u32 )ret_addr); + }else { + *new_ssp = *token_addr -sizeof (u64); + err =write_user_shstk_64 ((u64 __user *)*new_ssp, (u64 )ret_addr);
In both cases, you should write *new_ssp only when write_user_shstk_* functions have succeeded.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 267
Quoted 8 lines
arch/x86/kernel/shstk.c +8
-0+ } + } + +return err; +} + +/* +* Verify token_addr point to a valid token, and then set *new_ssp
points
arch/x86/kernel/shstk.c void shstk_disable(void) · line 274
Quoted 7 lines
arch/x86/kernel/shstk.c +7
-0+* according to the token. +*/ +int shstk_check_rstor_token (bool ia32,unsigned long token_addr,unsigned long *new_ssp) +{ +unsigned long token; + + *new_ssp =0 ;
Same as above.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 286
Quoted 12 lines
arch/x86/kernel/shstk.c +12
-0+ +if (!IS_ALIGNED (token_addr,8 )) +return -EINVAL ; + +if (get_user (token, (unsigned long __user *)token_addr)) +return -EFAULT ; + +/* Is 64-bit mode flag correct? */ +if (!ia32 && !IS_TOKEN_64 (token)) +return -EINVAL ; +else if (ia32 && !IS_TOKEN_32 (token)) +return -EINVAL ;
That test can be done using the XOR function - i.e., you want to return an error value when the two things are different.
In order to make this more readable, you call ia32 "proc32" to be clear what that variable denotes - a 32-bit process. Then, you do
bool shstk32 = !(token & BIT(0)); if (proc32 ^ shstk32) return -EINVAL;
Voila.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 293
Quoted 6 lines
arch/x86/kernel/shstk.c +6
-0+ token &= ~TOKEN_MODE_MASK ; + +/* +* Restore address properly aligned? +*/ +if ((!ia32 && !IS_ALIGNED (token,8 )) || !IS_ALIGNED (token,4 ))
Flip logic as above.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
Quoted 10 lines · Borislav Petkov
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 255
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/special_insns.h +6
-0+static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ +WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); +return -EFAULT ; +} +#endif What is that supposed to catch? Any concrete (mis-)use cases?
If 32-bit apps are not supported, there should be no need of 32-bit shadow stack write, otherwise there is a bug.
[...]
Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/shstk.c +1
-0@@ -20,6 +20,7 @@#include <asm/fpu/xstate.h> #include <asm/fpu/types.h> #include <asm/cet.h> +#include <asm/special_insns.h> static void start_update_msrs (void ) {@@ -176,3 +177,128 @@ void shstk_disable(void)
Quoted 9 lines
arch/x86/kernel/shstk.c void shstk_disable(void) · line 181
Quoted 4 lines
arch/x86/kernel/shstk.c +2
-0shstk_free (current); } + +static unsigned long _get_user_shstk_addr (void )What's the "_" prefix in the name supposed to denote?
Ditto for the other functions with "_" prefix you're adding.
These are static functions. I thought that would make the static scope clear. I can remove "_".
Quoted 20 lines
Quoted 7 lines
arch/x86/kernel/shstk.c +7
-0+{ +struct fpu *fpu = ¤t->thread.fpu; +unsigned long ssp =0 ; + +fpregs_lock (); + +if (fpregs_state_valid (fpu,smp_processor_id ())) {arch/x86/kernel/shstk.c void shstk_disable(void) · line 198
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+rdmsrl (MSR_IA32_PL3_SSP , ssp); + }else { +struct cet_user_state *p; + + p =get_xsave_addr (&fpu->state.xsave,XFEATURE_CET_USER ); +if (p) + ssp = p->user_ssp; + } + +fpregs_unlock ();<---- newline here.
Quoted 11 lines
arch/x86/kernel/shstk.c void shstk_disable(void) · line 205
Quoted 7 lines
arch/x86/kernel/shstk.c +7
-0+return ssp; +} + +#define TOKEN_MODE_MASK 3UL +#define TOKEN_MODE_64 1UL +#define IS_TOKEN_64 (token) (((token) & TOKEN_MODE_MASK) == TOKEN_MODE_64) +#define IS_TOKEN_32 (token) (((token) & TOKEN_MODE_MASK) == 0)Why do you have to look at the second, busy bit, too in order to determine the mode?
If the busy bit is set, it is only for SAVEPREVSSP, and invalid as a normal restore token.
Quoted 17 lines
Also, you don't need most of those defines - see below.
arch/x86/kernel/shstk.c void shstk_disable(void) · line 216
Quoted 10 lines
arch/x86/kernel/shstk.c +10
-0+/* +* Create a restore token on the shadow stack. A token is always 8-byte +* and aligned to 8. +*/ +static int _create_rstor_token (bool ia32,unsigned long ssp, +unsigned long *token_addr) +{ +unsigned long addr; + + *token_addr =0 ;What for? Callers should check this function's retval and then interpret the validity of
token_addrand it should not unconditionally write into it.
Ok.
Quoted 5 lines
arch/x86/kernel/shstk.c void shstk_disable(void) · line 217
Quoted 2 lines
arch/x86/kernel/shstk.c +2
-0+ +if ((!ia32 && !IS_ALIGNED (ssp,8 )) || !IS_ALIGNED (ssp,4 ))Flip this logic:
Quoted 8 lines
if ((ia32 && !IS_ALIGNED(ssp, 4)) || !IS_ALIGNED(ssp, 8))arch/x86/kernel/shstk.c void shstk_disable(void) · line 221
Quoted 3 lines
arch/x86/kernel/shstk.c +3
-0+return -EINVAL ; + + addr =ALIGN_DOWN (ssp,8 ) -8 ;Yah, so this is weird. Why does the restore token need to be at -8 instead on the shadow stack address itself?
With the lower two bits masked out, the restore token must point directly above itself.
Quoted 9 lines
Looking at
Figure 18-2. RSTORSSP to Switch to New Shadow Stack Figure 18-3. SAVEPREVSSP to Save a Restore Point
in the SDM, it looks like unnecessarily more complex than it should be. But maybe there's some magic I'm missing.
Quoted 7 lines
arch/x86/kernel/shstk.c void shstk_disable(void) · line 225
Quoted 4 lines
arch/x86/kernel/shstk.c +4
-0+ +/* Is the token for 64-bit? */ +if (!ia32) + ssp |=TOKEN_MODE_64 ;|= BIT(0);
Ok, then, we don't use #define's. I will put in comments about what it is doing, and fix the rest.
Thanks,
Yu-cheng
Re: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
Quoted 12 lines · Yu-cheng
Quoted 10 lines · Borislav Petkov
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 255
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/special_insns.h +6
-0+static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ +WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); +return -EFAULT ; +} +#endif What is that supposed to catch? Any concrete (mis-)use cases?
Quoted 2 lines
If 32-bit apps are not supported, there should be no need of 32-bit shadow stack write, otherwise there is a bug.
Speaking of which, I wonder what would happen if a 64-bit process makes a 32-bit system call (using int 0x80, for example), and gets a signal.
Borislav Petkov
maintainerRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
Quoted 2 lines · Eugene Syromiatnikov
Speaking of which, I wonder what would happen if a 64-bit process makes a 32-bit system call (using int
0x80, for example), and gets a signal.
I guess that's the next patch. And I see amluto has some concerns...
/me goes read.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Eugene Syromiatnikov
Speaking of which, I wonder what would happen if a 64-bit process makes a 32-bit system call (using int
0x80, for example), and gets a signal.I guess that's the next patch. And I see amluto has some concerns...
/me goes read.
In the next revision, there will be no "signal context extension" struct. However, the flow for 64, ia32 and x32 will be similar. I will send that out after some testing.
Thanks,
Yu-cheng
Re: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
In reply to Eugene SyromiatnikovQuoted 13 lines · Eugene Syromiatnikov
Quoted 12 lines · Yu-cheng
Quoted 10 lines · Borislav Petkov
arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 255
Quoted 6 lines · Yu-cheng Yu
arch/x86/include/asm/special_insns.h +6
-0+static inline int write_user_shstk_32 (u32 __user *addr,u32 val) +{ +WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); +return -EFAULT ; +} +#endif What is that supposed to catch? Any concrete (mis-)use cases?
Quoted 6 lines
Quoted 2 lines
If 32-bit apps are not supported, there should be no need of 32-bit shadow stack write, otherwise there is a bug.
Speaking of which, I wonder what would happen if a 64-bit process makes a 32-bit system call (using int
0x80, for example), and gets a signal.
Yes, that's right. Thanks! I should have said, if neither IA32 nor X32 is supported.
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
In reply to Yu, Yu-chengQuoted 2 lines · Yu-cheng
If 32-bit apps are not supported, there should be no need of 32-bit shadow stack write, otherwise there is a bug.
Aha, just a precaution. Then you can reduce the ifdeffery a bit (ontop of yours):
---
arch/x86/include/asm/special_insns.h +6 -8
@@ -235,9 +235,14 @@ static inline void clwb(volatile void *__p) } #ifdef CONFIG_X86_SHADOW_STACK -#if defined(CONFIG_IA32_EMULATION) || defined(CONFIG_X86_X32) static inline int write_user_shstk_32(u32 __user *addr, u32 val) { + if (WARN_ONCE(!IS_ENABLED(CONFIG_IA32_EMULATION) && + !IS_ENABLED(CONFIG_X86_X32), + "%s used but not supported.\n", __func__)) { + return -EFAULT; + } + asm_volatile_goto("1: wrussd %[val], (%[addr])\n" _ASM_EXTABLE(1b, %l[fail]) :: [addr] "r" (addr), [val] "r" (val)
@@ -246,13 +251,6 @@ static inline int write_user_shstk_32(u32 __user *addr, u32 val) fail:return -EFAULT ; } -#else -static inline int write_user_shstk_32 (u32 __user *addr,u32 val) -{ -WARN_ONCE (1 ,"%s used but not supported.\n" , __func__); -return -EFAULT ; -} -#endif static inline int write_user_shstk_64 (u64 __user *addr,u64 val) {
Quoted 2 lines
These are static functions. I thought that would make the static scope clear. I can remove "_".
No, "_" or "__" prefixed functions are generally supposed to denote internal interfaces which should not be used by other kernel facilities. In that case you have the external api <function_name> and the lower level helpers _<function_name>, __<function_name>, etc. They can be static but not necessarily.
This is not the case here so you can simply drop the "_" prefixes.
Quoted 2 lines
If the busy bit is set, it is only for SAVEPREVSSP, and invalid as a normal restore token.
Sure but the busy bit is independent from the mode.
Quoted 2 lines
With the lower two bits masked out, the restore token must point directly above itself.
That I know - I'm just questioning the design. It should be
addr = ALIGN_DOWN(ssp, 8);
Plain and simple.
Not this silly pushing and popping of stuff. But it is too late now anyway and it's not like hw people talk to software people who get to implement their shit.
Quoted 2 lines
Ok, then, we don't use #define's. I will put in comments about what it is doing, and fix the rest.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
In reply to Borislav PetkovQuoted 1 line · Borislav Petkov
On Tue, Apr 27, 2021 at 01:43:09PM -0700, Yu-cheng Yu wrote:
[...]
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + if ((!ia32 && !IS_ALIGNED(ssp, 8)) || !IS_ALIGNED(ssp, 4))Flip this logic:
if ((ia32 && !IS_ALIGNED(ssp, 4)) || !IS_ALIGNED(ssp, 8))
If !IS_ALIGNED(ssp, 4), then certainly !IS_ALIGNED(ssp, 8). This has to be as-is, I think.
Thanks,
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 24/30] x86/cet/shstk: Introduce shadow stack token setup/verify routines
Quoted 1 line · Yu-cheng
If !IS_ALIGNED(ssp, 4), then certainly !IS_ALIGNED(ssp, 8).
... but the reverse is true: when it is aligned by 8, it is already aligned by 4. Whoops, that's tricky. Pls put a comment over it so that we don't forget.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v26 25/30 x86/cet/shstk: Handle signals for shadow stack
In reply to Yu-cheng YuWhen shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Introduce a signal context extension struct 'sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt's wait_endbr status to be introduced later), and sc_ext.total_size field keeps track of total size.
Introduce routines for the allocation, save, and restore for sc_ext:
- fpu__alloc_sigcontext_ext(),
- save_extra_state_to_sigframe(),
- get_extra_state_from_sigframe(),
- restore_extra_state_to_xregs().
---
v25:
- Update commit log/comments for the sc_ext struct.
- Use restorer address already calculated.
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.
- Eliminate writing to MSR_IA32_U_CET for shadow stack.
- Change wrmsrl() to wrmsrl_safe() and handle error.
v24:
- Split out shadow stack token routines to a separate patch.
- Put signal frame save/restore routines to fpu/signal.c and re-name accordingly.
arch/x86/ia32/ia32_signal.c | 24 +++-- arch/x86/include/asm/cet.h | 2 + arch/x86/include/asm/fpu/internal.h | 2 + arch/x86/include/uapi/asm/sigcontext.h | 9 ++ arch/x86/kernel/fpu/signal.c | 137 ++++++++++++++++++++++++- arch/x86/kernel/signal.c | 9 ++ 6 files changed, 172 insertions(+), 11 deletions(-)
arch/x86/ia32/ia32_signal.c +15 -9
@@ -202,7 +202,8 @@ do { \*/ static void __user *get_sigframe (struct ksignal *ksig,struct pt_regs *regs,size_t frame_size, -void __user **fpstate) +void __user **fpstate, +void __user *restorer) {unsigned long sp, fx_aligned, math_size;
@@ -220,6 +221,10 @@ static void __user *get_sigframe(struct ksignal *ksig, struct pt_regs *regs, sp =fpu__alloc_mathframe (sp,1 , &fx_aligned, &math_size); *fpstate = (struct _fpstate_32 __user *) sp; + +if (save_extra_state_to_sigframe (1 , *fpstate, restorer)) +return (void __user *)-1L ; +if (copy_fpstate_to_sigframe (*fpstate, (void __user *)fx_aligned, math_size) <0 )return (void __user *)-1L ;
@@ -249,8 +254,6 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, 0x80cd, /* int $0x80 */ }; - frame =get_sigframe (ksig, regs,sizeof (*frame), &fp); -if (ksig->ka.sa.sa_flags &SA_RESTORER ) { restorer = ksig->ka.sa.sa_restorer; }else {
@@ -262,6 +265,8 @@ int ia32_setup_frame(int sig, struct ksignal *ksig, restorer = &frame->retcode; } + frame =get_sigframe (ksig, regs,sizeof (*frame), &fp, restorer); +if (!user_access_begin (frame,sizeof (*frame)))return -EFAULT ;
@@ -317,7 +322,13 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig, 0, }; - frame =get_sigframe (ksig, regs,sizeof (*frame), &fp); +if (ksig->ka.sa.sa_flags &SA_RESTORER ) + restorer = ksig->ka.sa.sa_restorer; +else + restorer = current->mm->context.vdso + + vdso_image_32.sym___kernel_rt_sigreturn; + + frame =get_sigframe (ksig, regs,sizeof (*frame), &fp, restorer);if (!user_access_begin (frame,sizeof (*frame)))return -EFAULT ;
@@ -334,11 +345,6 @@ int ia32_setup_rt_frame(int sig, struct ksignal *ksig,unsafe_put_user (0 , &frame->uc.uc_link, Efault);unsafe_compat_save_altstack (&frame->uc.uc_stack, regs->sp, Efault); -if (ksig->ka.sa.sa_flags &SA_RESTORER ) - restorer = ksig->ka.sa.sa_restorer; -else - restorer = current->mm->context.vdso + - vdso_image_32.sym___kernel_rt_sigreturn;unsafe_put_user (ptr_to_compat (restorer), &frame->pretcode, Efault);/*
arch/x86/include/asm/cet.h +2 -0
@@ -6,6 +6,8 @@#include <linux/types.h> struct task_struct ; +struct sc_ext ; +/* * Per-thread CET status */
arch/x86/include/asm/fpu/internal.h +2 -0
@@ -443,6 +443,8 @@ static inline void copy_kernel_to_fpregs(union fpregs_state *fpstate)__copy_kernel_to_fpregs (fpstate,-1 ); } +extern int save_extra_state_to_sigframe (int ia32,void __user *fp, +void __user *restorer);extern int copy_fpstate_to_sigframe (void __user *buf,void __user *fp,int size);/*
arch/x86/include/uapi/asm/sigcontext.h +9 -0
@@ -196,6 +196,15 @@ struct _xstate {/* New processor state extensions go here: */ }; +/* +* Located at the end of sigcontext->fpstate, aligned to 8. +* total_size keeps track of further extension of the struct. +*/ +struct sc_ext { +unsigned long total_size; +unsigned long ssp; +}; +/* * The 32-bit signal frame: */
arch/x86/kernel/fpu/signal.c +135 -2
180 lines not on this page · raw message
arch/x86/kernel/signal.c +9 -0
@@ -239,6 +239,9 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size,unsigned long buf_fx =0 ;int onsigstack =on_sig_stack (sp);int ret; +#ifdef CONFIG_X86_64 +void __user *restorer =NULL ; +#endif /* redzone */ if (IS_ENABLED (CONFIG_X86_64 ))
@@ -270,6 +273,12 @@ get_sigframe(struct k_sigaction *ka, struct pt_regs *regs, size_t frame_size,if (onsigstack && !likely (on_sig_stack (sp)))return (void __user *)-1L ; +#ifdef CONFIG_X86_64 +if (ka->sa.sa_flags &SA_RESTORER ) + restorer = ka->sa.sa_restorer; + ret =save_extra_state_to_sigframe (0 , *fpstate, restorer); +#endif +/* save i387 and extended state */ ret =copy_fpstate_to_sigframe (*fpstate, (void __user *)buf_fx, math_size);if (ret <0 )
Signature · 1 line
2.21.0
Andy Lutomirski
maintainerextending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.
I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe {
char __user *pretcode;
struct ucontext uc;
struct siginfo info;
/* fp state follows here */
};This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to &frame->info in RSI, anda pointer to &frame->uc in RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:
struct ucontext {
unsigned long uc_flags;
struct ucontext *uc_link;
stack_t uc_stack;
struct sigcontext uc_mcontext; <-- fp pointer is in here
sigset_t uc_sigmask; /* mask last for extensibility */
};The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes {
/*
* If set to FP_XSTATE_MAGIC1 then this is an xstate context.
* 0 if a legacy frame.
*/
__u32 magic1;
/*
* Total size of the fpstate area:
*
* - if magic1 == 0 then it's sizeof(struct _fpstate)
* - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate)
* plus extensions (if any)
*/
__u32 extended_size;
/*
* Feature bit mask (including FP/SSE/extended state) that is present
* in the memory layout:
*/
__u64 xfeatures;
/*
* Actual XSAVE state size, based on the xfeatures saved in the layout.
* 'extended_size' is greater than 'xstate_size':
*/
__u32 xstate_size;
/* For future use: */
__u32 padding[7];
};That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
This is *almost*, but not quite, easy: struct ucontext is already variable length! Unfortunately, the whole variable length portion is used up by uc_sigmask. So I propose that we introduce a brand new bona fide extension mechanism. It works like this:
First, we add a struct ucontext_extension at the end. It looks like:
struct ucontext_extension {
u64 length; /* sizeof(struct ucontext_extension) */ u64 flags; /* we will want this some day */ [CET stuff here]
[future stuff here]
};
And we locate it by scrounging a word somewhere in ucontext to give the offset from the beginning of struct ucontext to ucontext_extension. We indicate the presence of this feature using a new uc_flags bit. I can think of a couple of vaguely reasonable places:
a) the reserved word in sigcontext. This is fine for x86 but not so great if other architectures want to do this.
b) uc_link. Fine everywhere but powerpc. Oops.
c) use the high bits of uc_flags. After all, once we add extensions, we don't need new flags, so we can steal 16 high bits of uc_flags for this.
I think I'm in favor of (c). We do:
(uc_flags & 0xffff0000) == 0: extension not present
Otherwise the extension region is at ucontext + (uc_flags >> 16).
And sigreturn finds the extension the same way, because CRIU can already migrate a signal frame from one kernel to another, your patch breaks this, and having sigreturn hardcode the offset would also break it.
What do you think?
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 151 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes { /* * If set to FP_XSTATE_MAGIC1 then this is an xstate context. * 0 if a legacy frame. */ __u32 magic1; /* * Total size of the fpstate area: * * - if magic1 == 0 then it's sizeof(struct _fpstate) * - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate) * plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) that is present * in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved in the layout. * 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
This is *almost*, but not quite, easy: struct ucontext is already variable length! Unfortunately, the whole variable length portion is used up by
uc_sigmask. So I propose that we introduce a brand new bona fide extension mechanism. It works like this:First, we add a struct
ucontext_extensionat the end. It looks like:struct
ucontext_extension{
u64 length; /*sizeof(struct ucontext_extension)*/ u64 flags; /* we will want this some day */ [CET stuff here]
[future stuff here]
};And we locate it by scrounging a word somewhere in ucontext to give the offset from the beginning of struct ucontext to
ucontext_extension. We indicate the presence of this feature using a newuc_flagsbit. I can think of a couple of vaguely reasonable places:a) the reserved word in sigcontext. This is fine for x86 but not so great if other architectures want to do this.
b)
uc_link. Fine everywhere but powerpc. Oops.c) use the high bits of
uc_flags. After all, once we add extensions, we don't need new flags, so we can steal 16 high bits ofuc_flagsfor this.I think I'm in favor of (c). We do:
(
uc_flags&0xffff0000) == 0: extension not present
Quoted 9 lines
Otherwise the extension region is at ucontext + (
uc_flags>> 16).And sigreturn finds the extension the same way, because CRIU can already migrate a signal frame from one kernel to another, your patch breaks this, and having sigreturn hardcode the offset would also break it.
What do you think?
There are a lot of things in here. I think I could create some patches for ucontext_extension and send out for discussion. Thanks for explaining this!
Yu-cheng
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
In reply to Andy LutomirskiQuoted 28 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };
Quoted 3 lines
This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler
Well, not really. While indeed this is not declared as a part of API the structure is widely used for rt_sigreturn syscall (and we're using it inside criu thus any change here will simply break the restore procedure). Sorry out of time right now, I'll read your mail more carefully once time permit.
Andy Lutomirski
maintainerRe: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 34 lines · Cyrill Gorcunov
Quoted 28 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };Quoted 3 lines
This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler
Quoted 5 lines
Well, not really. While indeed this is not declared as a part of API the structure is widely used for
rt_sigreturnsyscall (and we're using it inside criu thus any change here will simply break the restore procedure). Sorry out of time right now, I'll read your mail more carefully once time permit.
I skimmed the CRIU code. You appear to declare struct rt_sigframe, and you use the offset from the start of rt_sigframe to uc. You also use the offset to the /* fp state follows here */ part, but that's unnecessary -- you could just as easily have put the fp state at any other address -- the kernel will happily follow the pointer you supply regardless of where it points. So the only issues I can see are if you write the fp state on top of something else or if you inadvertently fill in the proposed extension part of uc_flags. Right now you seem to be ignoring uc_flags, which I presume means that you are filling it in as zero. Even if the offset of the fp state in the kernel rt_sigframe changes, the kernel should still successfully parse the signal frame you generate.
I suppose there is another potential issue: would CRIU have issues if the *save* runs on a kernel that uses this proposed extension mechanism? Are you doing something with the saved state that would get confused?
--Andy
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 41 lines · Andy Lutomirski
Quoted 34 lines · Cyrill Gorcunov
Quoted 28 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };Quoted 3 lines
This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler
Quoted 5 lines
Well, not really. While indeed this is not declared as a part of API the structure is widely used for
rt_sigreturnsyscall (and we're using it inside criu thus any change here will simply break the restore procedure). Sorry out of time right now, I'll read your mail more carefully once time permit.
Quoted 6 lines
I skimmed the CRIU code. You appear to declare struct
rt_sigframe, and you use the offset from the start ofrt_sigframeto uc. You also use the offset to the /* fp state follows here */ part, but that's unnecessary -- you could just as easily have put the fp state at any other address -- the kernel will happily follow the pointer you supply regardless of where it points. So the only issues I can see are if
Yes, since rt_sigreturn is a very late stage of restore we've to preallocate memory early and surely doing it in one slab is a way more convenient, thus that's why fp context follows the structure declaration (well, it was at least I didn't touch this code for years already). Anyway, I meant that assuming "user code does not know the sizes of the fields" is not exactly true and strictly speaking current layout became a part of nondeclared api (simply because kernel use this structure run-time only and even if something get changed in kernel declaration for this structure the kernel itself will be fine because it doesn't save it on disk between kernel updates) but for tools like criu such change might require update. IOW, I mean that changing rt_sigreturn structure itself is not safe and we can't reorder the members and theirs size.
Or I misunderstood you, and you meant only the fact that fp may be arbitrary, pointing for any place in userspace? If so then indeed, fp can be pointing to context placed anywere since kernel will fetch it from the address during rt_sigreturn processing.
Quoted 4 lines
you write the fp state on top of something else or if you inadvertently fill in the proposed extension part of
uc_flags. Right now you seem to be ignoringuc_flags, which I presume means that you are filling it in as zero. Even if the offset of the fp state in the
yes
Quoted 7 lines
kernel
rt_sigframechanges, the kernel should still successfully parse the signal frame you generate.I suppose there is another potential issue: would CRIU have issues if the *save* runs on a kernel that uses this proposed extension mechanism? Are you doing something with the saved state that would get confused?
You know, actually I guess criu may not work with new extensions, we have a predefined max xsave frame size and any new extensions to cpu states will require additional support.
I must confess I didn't follow much on CET internals yet so I must be simply wrong. CC'ing Anrew just in case.
Cyrill
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
In reply to Andy Lutomirski* Andy Lutomirski:
Quoted 17 lines
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:
Quoted 7 lines
struct ucontext {
unsigned long uc_flags;
struct ucontext *uc_link;
stack_t uc_stack;
struct sigcontext uc_mcontext; <-- fp pointer is in here
sigset_t uc_sigmask; /* mask last for extensibility */
};I must say that I haven't reviewed this in detail, but for historic reasons, glibc userspace has a differently-sized sigset_t. So the kernel ucontext (used in signals) and user ucontext (used for swapcontext et al.) are already fully decoupled?
Thanks,
Florian
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
In reply to Andy LutomirskiQuoted 106 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes { /* * If set to FP_XSTATE_MAGIC1 then this is an xstate context. * 0 if a legacy frame. */ __u32 magic1; /* * Total size of the fpstate area: * * - if magic1 == 0 then it's sizeof(struct _fpstate) * - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate) * plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) that is present * in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved in the layout. * 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
However, the WAIT_ENDBR status needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.
I have tested the change, and will send out another version of the whole series.
Quoted 42 lines
This is *almost*, but not quite, easy: struct ucontext is already variable length! Unfortunately, the whole variable length portion is used up by
uc_sigmask. So I propose that we introduce a brand new bona fide extension mechanism. It works like this:First, we add a struct
ucontext_extensionat the end. It looks like:struct
ucontext_extension{
u64 length; /*sizeof(struct ucontext_extension)*/ u64 flags; /* we will want this some day */ [CET stuff here]
[future stuff here]
};And we locate it by scrounging a word somewhere in ucontext to give the offset from the beginning of struct ucontext to
ucontext_extension. We indicate the presence of this feature using a newuc_flagsbit. I can think of a couple of vaguely reasonable places:a) the reserved word in sigcontext. This is fine for x86 but not so great if other architectures want to do this.
b)
uc_link. Fine everywhere but powerpc. Oops.c) use the high bits of
uc_flags. After all, once we add extensions, we don't need new flags, so we can steal 16 high bits ofuc_flagsfor this.I think I'm in favor of (c). We do:
(
uc_flags&0xffff0000) == 0: extension not presentOtherwise the extension region is at ucontext + (
uc_flags>> 16).And sigreturn finds the extension the same way, because CRIU can already migrate a signal frame from one kernel to another, your patch breaks this, and having sigreturn hardcode the offset would also break it.
What do you think?
Andy Lutomirski
maintainerRe: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 121 lines · Yu-cheng
Quoted 106 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes { /* * If set to FP_XSTATE_MAGIC1 then this is an xstate context. * 0 if a legacy frame. */ __u32 magic1; /* * Total size of the fpstate area: * * - if magic1 == 0 then it's sizeof(struct _fpstate) * - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate) * plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) that is present * in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved in the layout. * 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 7 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 4 lines
However, the
WAIT_ENDBRstatus needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.
That seems like unnecessary ABI coupling. We have plenty of bits in uc_flags, and we have an entire reserved word in sigcontext. How about just sticking this bit in one of those places?
--Andy
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 134 lines · Andy Lutomirski
Quoted 121 lines · Yu-cheng
Quoted 106 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes { /* * If set to FP_XSTATE_MAGIC1 then this is an xstate context. * 0 if a legacy frame. */ __u32 magic1; /* * Total size of the fpstate area: * * - if magic1 == 0 then it's sizeof(struct _fpstate) * - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate) * plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) that is present * in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved in the layout. * 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 7 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
However, the
WAIT_ENDBRstatus needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.That seems like unnecessary ABI coupling. We have plenty of bits in
uc_flags, and we have an entire reserved word in sigcontext. How about just sticking this bit in one of those places?
Yes, I will make it UC_WAIT_ENDBR.
Thanks,
Yu-cheng
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 139 lines · Yu-cheng
Quoted 138 lines · Andy Lutomirski
Quoted 121 lines · Yu-cheng
Quoted 106 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct
rt_sigframe{
char__user*pretcode;
struct ucontext uc;
struct siginfo info;
/* fp state follows here */
};This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext {
unsigned longuc_flags;
struct ucontext*uc_link;
stack_tuc_stack;
struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct
_fpx_sw_bytes{
/*
* If set toFP_XSTATE_MAGIC1then this is an xstate context.
* 0 if a legacy frame.
*/
__u32 magic1;/*
* Total size of the fpstate area:
*
* - if magic1 == 0 then it'ssizeof(struct _fpstate)
* - if magic1 ==FP_XSTATE_MAGIC1then it'ssizeof(struct _xstate)* plus extensions (if any)
*/
__u32extended_size;/*
* Feature bit mask (including FP/SSE/extended state) that is present
* in the memory layout:
*/
__u64 xfeatures;/*
* Actual XSAVE state size, based on the xfeatures saved in the layout.
* 'extended_size' is greater than 'xstate_size':
*/
__u32xstate_size;/* For future use: */
__u32 padding[7];
};That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 7 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 4 lines
However, the
WAIT_ENDBRstatus needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.
Quoted 5 lines
Quoted 3 lines
That seems like unnecessary ABI coupling. We have plenty of bits in
uc_flags, and we have an entire reserved word in sigcontext. How about just sticking this bit in one of those places?Yes, I will make it
UC_WAIT_ENDBR.
Personally, I think an explicit flag is cleaner than using a reserved word somewhere. However, there is a small issue: ia32 has no uc_flags.
This series can support legacy apps up to now. But, instead of creating too many special cases, perhaps we should drop CET support of ia32?
Thoughts?
Thanks,
Yu-cheng
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 22 lines · Yu-cheng
Quoted 21 lines · Yu-cheng
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · Yu-cheng
Quoted 17 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
[...]
Quoted 50 lines
Quoted 50 lines
Quoted 49 lines
Quoted 30 lines
Quoted 17 lines
That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 7 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 4 lines
However, the
WAIT_ENDBRstatus needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.That seems like unnecessary ABI coupling. We have plenty of bits in
uc_flags, and we have an entire reserved word in sigcontext. How about just sticking this bit in one of those places?
Quoted 8 lines
Quoted 1 line
Yes, I will make it
UC_WAIT_ENDBR.Personally, I think an explicit flag is cleaner than using a reserved word somewhere. However, there is a small issue: ia32 has no
uc_flags.This series can support legacy apps up to now. But, instead of creating too many special cases, perhaps we should drop CET support of ia32?
Thoughts?
Once we have UC_WAIT_ENDBR, IBT signal handling becomes quite simple. Like the following:
arch/x86/include/uapi/asm/ucontext.h +5 -0
@@ -49,6 +49,11 @@*/ #define UC_SIGCONTEXT_SS 0x2#define UC_STRICT_RESTORE_SS 0x4 + +/* +* UC_WAIT_ENDBR indicates the task is in wait-ENDBR status. +*/ +#define UC_WAIT_ENDBR 0x08#endif #include <asm-generic/ucontext.h>
arch/x86/kernel/ibt.c +29 -0
@@ -66,3 +66,32 @@ void ibt_disable(void)ibt_set_clear_msr_bits (0 ,CET_ENDBR_EN ); current->thread.cet.ibt =0 ; } + +int ibt_get_clear_wait_endbr (void ) +{ +u64 msr_val =0 ; + +if (!current->thread.cet.ibt) +return 0 ; + +fpregs_lock (); + +if (test_thread_flag (TIF_NEED_FPU_LOAD )) +__fpregs_load_activate (); + +if (!rdmsrl_safe (MSR_IA32_U_CET , &msr_val)) +wrmsrl (MSR_IA32_U_CET , msr_val & ~CET_WAIT_ENDBR ); + +fpregs_unlock (); + +return msr_val &CET_WAIT_ENDBR ; +} + +int ibt_set_wait_endbr (void ) +{ +if (!current->thread.cet.ibt) +return 0 ; + + +return ibt_set_clear_msr_bits (CET_WAIT_ENDBR ,0 ); +}
arch/x86/kernel/signal.c +1 -0
@@ -46,6 +46,7 @@#include <asm/syscall.h> #include <asm/sigframe.h> #include <asm/signal.h> +#include <asm/cet.h> #ifdef CONFIG_X86_64 /*
@@ -134,6 +135,9 @@ static int restore_sigcontext(struct pt_regs *regs,*/ if (unlikely(!(uc_flags & UC_STRICT_RESTORE_SS) &&
user_64bit_mode(regs)))
force_valid_ss(regs);
+
+ if (uc_flags & UC_WAIT_ENDBR)
+ ibt_set_wait_endbr();
#endif
return fpu__restore_sig((void __user *)sc.fpstate,
@@ -433,6 +437,9 @@ static unsigned long frame_uc_flags(struct pt_regs *regs)
if (likely(user_64bit_mode(regs)))
flags |= UC_STRICT_RESTORE_SS;
+ if (ibt_get_clear_wait_endbr())
+ flags |= UC_WAIT_ENDBR;
+
return flags;
}
However, this cannot handle ia32 with no SA_SIGINFO. For that, can we create a synthetic token on the shadow stack?
- The token points to itself with reserved bit[1] set, and cannot be used for RSTORSSP.
- The token only exists for ia32 with no SA_SIGINFO *AND* when the task is in wait-endbr.
The signal shadow stack will look like this:
--> ssp before signal
synthetic IBT token (for ia32 no SA_SIGINFO)
shadow stack restore token
sigreturn address
The synthetic token is not valid in other situations. How is that?
Thanks,
Yu-cheng
Andy Lutomirski
maintainerRe: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 92 lines · Yu-cheng
Quoted 33 lines · Yu-cheng
Quoted 32 lines · Yu-cheng
Quoted 31 lines · Andy Lutomirski
Quoted 29 lines · Yu-cheng
Quoted 27 lines · Andy Lutomirski
Quoted 13 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved
along with the signal context and later restored in sigreturn. However,
currently there is no systematic facility for extending a signal context.
There is some space left in the ucontext, but changing ucontext is likely
to create compatibility issues and there is not enough space for further
extensions.Quoted 9 lines
Introduce a signal context extension struct '
sc_ext', which is used to save
shadow stack restore token address. The extension is located above the fpu
states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field
keeps track of total size.I still don't like this.
[...]
Quoted 54 lines
Quoted 53 lines
Quoted 44 lines
Quoted 31 lines
Quoted 17 lines
That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we
don't stick more data at the end, and take the relocating-fpu approach?Quoted 8 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at
the restore token. There is no need to put that in ucontext.That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
However, the
WAIT_ENDBRstatus needs to be saved/restored for signals. Since IBT is now dependent on shadow stack, we can use a spare bit of the shadow stack restore token for that.That seems like unnecessary ABI coupling. We have plenty of bits in
uc_flags, and we have an entire reserved word in sigcontext. How about just sticking this bit in one of those places?Yes, I will make it
UC_WAIT_ENDBR.
Quoted 7 lines
Quoted 7 lines
Personally, I think an explicit flag is cleaner than using a reserved word somewhere. However, there is a small issue: ia32 has no
uc_flags.This series can support legacy apps up to now. But, instead of creating too many special cases, perhaps we should drop CET support of ia32?
Thoughts?
I'm really not thrilled about coupling IBT and SHSTK like this.
Here are a couple of possible solutions:
- Don't support IBT in 32-bit mode, or maybe just don't support IBT with legacy 32-bit signals. The actual mechanics of this could be awkward. Maybe we would reject the sigaction() call or the IBT-enabling request if they conflict?
- Find some space in the signal frame for these flags. Looking around a bit, sigframe_ia32 has fpstate_unused, but I can imagine things like CRIU getting very confused if it stops being unused. sigframe_ia32 uses sigcontext_32, which has a bunch of reserved space in __gsh, __fsh, etc.
rt_sigframe_ia32 has uc_flags, so this isn't a real problem.
I don't have a brilliant solution here.
Andy Lutomirski
maintainerRe: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
In reply to Andy LutomirskiQuoted 127 lines · Andy Lutomirski
Quoted 125 lines · Yu-cheng
Quoted 106 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not have this struct declared, and user code does not know the sizes of the fields. So it's accessed in a nonsensical way. The signal handler function is passed a pointer to the whole sigframe implicitly in RSP, a pointer to
&frame->infoin RSI, anda pointer to&frame->ucin RDX. User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility */ };The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the benefit of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses uc accordingly. And the kernel follows the pointer in mcontext to find the fp state. The latter bit is quite important later. The kernel does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly (?) gave us a software defined area between the "legacy" x87 region and the modern supposedly extensible part. Linux sticks the following structure in that hole:
struct _fpx_sw_bytes { /* * If set to FP_XSTATE_MAGIC1 then this is an xstate context. * 0 if a legacy frame. */ __u32 magic1; /* * Total size of the fpstate area: * * - if magic1 == 0 then it's sizeof(struct _fpstate) * - if magic1 == FP_XSTATE_MAGIC1 then it's sizeof(struct _xstate) * plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) that is present * in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved in the layout. * 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Quoted 6 lines
Quoted 3 lines
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Wait, what's the actual shadow stack token format? And is the token on the new stack or the old stack when sigaltstack is in use? For that matter, is there any support for an alternate shadow stack for signals?
--Andy
H. Peter Anvin
reviewerRe: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
<[email protected]>,Dave Martin <[email protected]>,Weijiang Yang <[email protected]>,Pengfei Xu <[email protected]>,Haitao Huang <[email protected]>
From: "H. Peter Anvin" <[email protected]>
Message-ID: <[email protected]>
There are a few words at the end of the FXSAVE area for software use for this reason: so we can extend the signal frame – originally to enable saving the XSAVE state but that doesn't use up the whole software available area.
Quoted 190 lines · Andy Lutomirski
Quoted 2 lines · Andy Lutomirski
On Fri, Apr 30, 2021 at 10:00 AM Yu, Yu-cheng <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines · Andy Lutomirski
Quoted 1 line
On Tue, Apr 27, 2021 at 1:44 PM Yu-cheng Yu
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
When shadow stack is enabled, a task's shadow stack states must
be saved
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
along with the signal context and later restored in sigreturn.
However,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
currently there is no systematic facility for extending a signal
context.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
There is some space left in the ucontext, but changing ucontext
is likely
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to create compatibility issues and there is not enough space for
further
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
extensions.
Introduce a signal context extension struct '
sc_ext', which isused to save
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
shadow stack restore token address. The extension is located
above the fpu
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
states, plus alignment. The struct can be extended (such as the
ibt's
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
wait_endbrstatus to be introduced later), and sc_ext.total_sizefield
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
keeps track of total size.
I still don't like this.
Here's how the signal layout works, for better or for worse:
The kernel has:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
struct rt_sigframe { char __user *pretcode; struct ucontext uc; struct siginfo info; /* fp state follows here */ };This is roughly the actual signal frame. But userspace does not
have
Quoted 1 line
Quoted 1 line
Quoted 1 line
this struct declared, and user code does not know the sizes of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
fields. So it's accessed in a nonsensical way. The signal
handler
Quoted 1 line
Quoted 1 line
Quoted 1 line
function is passed a pointer to the whole sigframe implicitly in
RSP,
Quoted 1 line
Quoted 1 line
Quoted 1 line
a pointer to
&frame->infoin RSI, anda pointer to&frame->ucinRDX.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
User code can *find* the fp state by following a pointer from mcontext, which is, in turn, found via uc:
struct ucontext { unsigned long uc_flags; struct ucontext *uc_link; stack_t uc_stack; struct sigcontext uc_mcontext; <-- fp pointer is in here sigset_t uc_sigmask; /* mask last for extensibility*/
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
};
The kernel, in sigreturn, works a bit differently. The sigreturn variants know the base address of the frame but don't have the
benefit
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
of receiving pointers to the fields. So instead the kernel takes advantage of the fact that it knows the offset to uc and parses
uc
Quoted 1 line
Quoted 1 line
Quoted 1 line
accordingly. And the kernel follows the pointer in mcontext to
find
Quoted 1 line
Quoted 1 line
Quoted 1 line
the fp state. The latter bit is quite important later. The
kernel
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
does not parse info at all.
The fp state is its own mess. When XSAVE happened, Intel kindly
(?)
Quoted 1 line
Quoted 1 line
Quoted 1 line
gave us a software defined area between the "legacy" x87 region
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
the modern supposedly extensible part. Linux sticks the
following
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
structure in that hole:
struct
_fpx_sw_bytes{
/*
* If set toFP_XSTATE_MAGIC1then this is an xstatecontext.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
* 0 if a legacy frame.
*/
__u32magic1;/*
* Total size of the fpstate area:
*
* - if magic1 == 0 then it'ssizeof(struct _fpstate)
* - if magic1 ==FP_XSTATE_MAGIC1then it's sizeof(struct
_xstate)Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
* plus extensions (if any) */ __u32 extended_size; /* * Feature bit mask (including FP/SSE/extended state) thatis present
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
* in the memory layout: */ __u64 xfeatures; /* * Actual XSAVE state size, based on the xfeatures saved inthe layout.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
* 'extended_size' is greater than 'xstate_size': */ __u32 xstate_size; /* For future use: */ __u32 padding[7]; };That's where we are right now upstream. The kernel has a parser
for
Quoted 1 line
Quoted 1 line
Quoted 1 line
the FPU state that is bugs piled upon bugs and is going to have
to be
Quoted 1 line
Quoted 1 line
Quoted 1 line
rewritten sometime soon. On top of all this, we have two
upcoming
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years
ago,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With
AMX
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU
state
Quoted 1 line
Quoted 1 line
Quoted 1 line
vary depending on which features are INIT, i.e. making it more
compact
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature
with
Quoted 1 line
Quoted 1 line
Quoted 1 line
no space allocated will make the structure bigger, and the stack
won't
Quoted 1 line
Quoted 1 line
Quoted 1 line
have room. Fortunately, one can relocate the entire FPU state,
update
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap?
very
Quoted 1 line
Quoted 1 line
Quoted 1 line
awkwardly on the stack?) and changing the pointer. For all we
know,
Quoted 1 line
Quoted 1 line
Quoted 1 line
some code already fiddles with the pointer. This is great,
except
Quoted 1 line
Quoted 1 line
Quoted 1 line
that your patch sticks more data at the end of the FPU block that
no
Quoted 1 line
Quoted 1 line
Quoted 1 line
one is expecting, and your sigreturn code follows that pointer,
and
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
will read off into lala land.
Then, what about we don't do that at all. Is it possible from now
on we
Quoted 1 line
Quoted 1 line
don't stick more data at the end, and take the relocating-fpu
approach?
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
2. CET. CET wants us to find a few more bytes somewhere, and
those
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When
the
Quoted 1 line
Quoted 1 line
kernel handles sigreturn, the user-mode shadow stack pointer is
right at
Quoted 4 lines
Quoted 1 line
the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Wait, what's the actual shadow stack token format? And is the token on the new stack or the old stack when sigaltstack is in use? For that matter, is there any support for an alternate shadow stack for signals?
--Andy
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
In reply to Andy LutomirskiQuoted 24 lines · Andy Lutomirski
Quoted 23 lines · Andy Lutomirski
Quoted 21 lines · Yu-cheng
Quoted 19 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
[...]
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 17 lines
That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
Quoted 8 lines
Quoted 2 lines
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Wait, what's the actual shadow stack token format? And is the token on the new stack or the old stack when sigaltstack is in use? For that matter, is there any support for an alternate shadow stack for signals?
The restore token is a pointer pointing directly above itself and bit[0] indicates 64-bit mode.
Because the shadow stack stores only return addresses, there is no alternate shadow stack. However, the application can allocate and switch to a new shadow stack.
Yu-cheng
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 71 lines · Yu-cheng
Quoted 24 lines · Andy Lutomirski
Quoted 24 lines · Andy Lutomirski
Quoted 21 lines · Yu-cheng
Quoted 19 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
[...]
Quoted 40 lines
Quoted 40 lines
Quoted 30 lines
Quoted 17 lines
That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 7 lines
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Quoted 8 lines
Quoted 4 lines
Wait, what's the actual shadow stack token format? And is the token on the new stack or the old stack when sigaltstack is in use? For that matter, is there any support for an alternate shadow stack for signals?
The restore token is a pointer pointing directly above itself and bit[0] indicates 64-bit mode.
Because the shadow stack stores only return addresses, there is no alternate shadow stack. However, the application can allocate and switch to a new shadow stack.
I think we should make the ABI support an alternate shadow stack even if we don’t implement it initially. After all, some day someone might want to register a handler for shadow stack overflow.
Quoted 2 lines
Yu-cheng
Re: extending ucontext (Re: [PATCH v26 25/30] x86/cet/shstk: Handle signals for shadow stack)
Quoted 76 lines · Andy Lutomirski
Quoted 76 lines · Yu-cheng
Quoted 24 lines · Andy Lutomirski
Quoted 24 lines · Andy Lutomirski
Quoted 21 lines · Yu-cheng
Quoted 19 lines · Andy Lutomirski
Quoted 8 lines · Yu-cheng Yu
When shadow stack is enabled, a task's shadow stack states must be saved along with the signal context and later restored in sigreturn. However, currently there is no systematic facility for extending a signal context. There is some space left in the ucontext, but changing ucontext is likely to create compatibility issues and there is not enough space for further extensions.
Quoted 5 lines
Introduce a signal context extension struct '
sc_ext', which is used to save shadow stack restore token address. The extension is located above the fpu states, plus alignment. The struct can be extended (such as the ibt'swait_endbrstatus to be introduced later), and sc_ext.total_size field keeps track of total size.I still don't like this.
Here's how the signal layout works, for better or for worse:
[...]
Quoted 44 lines
Quoted 34 lines
Quoted 34 lines
Quoted 17 lines
That's where we are right now upstream. The kernel has a parser for the FPU state that is bugs piled upon bugs and is going to have to be rewritten sometime soon. On top of all this, we have two upcoming features, both of which require different kinds of extensions:
1. AVX-512. (Yeah, you thought this story was over a few years ago, but no. And AMX makes it worse.) To make a long story short, we promised user code many years ago that a signal frame fit in 2048 bytes with some room to spare. With AVX-512 this is false. With AMX it's so wrong it's not even funny. The only way out of the mess anyone has come up with involves making the length of the FPU state vary depending on which features are INIT, i.e. making it more compact than "compact" mode is. This has a side effect: it's no longer possible to modify the state in place, because enabling a feature with no space allocated will make the structure bigger, and the stack won't have room. Fortunately, one can relocate the entire FPU state, update
Quoted 9 lines
the pointer in mcontext, and the kernel will happily follow the pointer. So new code on a new kernel using a super-compact state could expand the state by allocating new memory (on the heap? very awkwardly on the stack?) and changing the pointer. For all we know, some code already fiddles with the pointer. This is great, except that your patch sticks more data at the end of the FPU block that no one is expecting, and your sigreturn code follows that pointer, and will read off into lala land.
Then, what about we don't do that at all. Is it possible from now on we don't stick more data at the end, and take the relocating-fpu approach?
Quoted 3 lines
2. CET. CET wants us to find a few more bytes somewhere, and those bytes logically belong in ucontext, and here we are.
Quoted 6 lines
Quoted 3 lines
Fortunately, we can spare CET the need of ucontext extension. When the kernel handles sigreturn, the user-mode shadow stack pointer is right at the restore token. There is no need to put that in ucontext.
That seems entirely reasonable. This might also avoid needing to teach CRIU about CET at all.
Wait, what's the actual shadow stack token format? And is the token on the new stack or the old stack when sigaltstack is in use? For that matter, is there any support for an alternate shadow stack for signals?
Quoted 6 lines
Quoted 3 lines
The restore token is a pointer pointing directly above itself and bit[0] indicates 64-bit mode.
Because the shadow stack stores only return addresses, there is no alternate shadow stack. However, the application can allocate and switch to a new shadow stack.
I think we should make the ABI support an alternate shadow stack even if we don’t implement it initially. After all, some day someone might want to register a handler for shadow stack overflow.
Agree. We can probably add something in parallel of sigaltstack(), and let the user choose separately alternate normal/shadow stacks.
Thanks,
Yu-cheng
PATCH v26 26/30 ELF: Introduce arch_setup_elf_property()
In reply to Yu-cheng YuAn ELF file's .note.gnu.property indicates arch features supported by the file. These features are extracted by arch_parse_elf_property() and stored in 'arch_elf_state'.
Introduce x86 feature definitions and arch_setup_elf_property(), which enables such features. The first use-case of this function is Shadow Stack.
ARM64 is the other arch that has ARCH_USE_GNU_PROPERTY and arch_parse_elf_ property(). Add arch_setup_elf_property() for it.
---
v24:
- Change cet_setup_shstk() to shstk_setup() to reflect function name changes relating to the splitting of shadow stack and ibt.
arch/arm64/include/asm/elf.h | 5 +++++ arch/x86/Kconfig | 2 ++ arch/x86/include/asm/elf.h | 13 +++++++++++++ arch/x86/kernel/process_64.c | 32 ++++++++++++++++++++++++++++++++ fs/binfmt_elf.c | 4 ++++ include/linux/elf.h | 6 ++++++ include/uapi/linux/elf.h | 9 +++++++++ 7 files changed, 71 insertions(+)
arch/arm64/include/asm/elf.h +5 -0
@@ -281,6 +281,11 @@ static inline int arch_parse_elf_property(u32 type, const void *data,return 0 ; } +static inline int arch_setup_elf_property (struct arch_elf_state *arch) +{ +return 0 ; +} +static inline int arch_elf_pt_proc (void *ehdr,void *phdr,struct file *f,bool is_interp,struct arch_elf_state *state)
arch/x86/Kconfig +2 -0
@@ -1951,6 +1951,8 @@ config X86_SHADOW_STACK depends on AS_WRUSS depends on ARCH_HAS_SHADOW_STACK select ARCH_USES_HIGH_VMA_FLAGS + select ARCH_USE_GNU_PROPERTY + select ARCH_BINFMT_ELF_STATE help Shadow Stack protection is a hardware feature that detects function return address corruption. This helps mitigate ROP attacks.
arch/x86/include/asm/elf.h +13 -0
@@ -390,6 +390,19 @@ extern int compat_arch_setup_additional_pages(struct linux_binprm *bprm,extern bool arch_syscall_is_vdso_sigreturn (struct pt_regs *regs); +#ifdef CONFIG_ARCH_BINFMT_ELF_STATE +struct arch_elf_state { +unsigned int gnu_property; +}; + +#define INIT_ARCH_ELF_STATE { \ + .gnu_property = 0, \ +} + +#define arch_elf_pt_proc (ehdr, phdr, elf, interp, state) (0) +#define arch_check_elf (ehdr, interp, interp_ehdr, state) (0) +#endif +/* Do not change the values. See get_align_mask() */ enum align_flags {ALIGN_VA_32 =BIT (0 ),
arch/x86/kernel/process_64.c +32 -0
@@ -835,3 +835,35 @@ unsigned long KSTK_ESP(struct task_struct *task) {return task_pt_regs (task)->sp; } + +#ifdef CONFIG_ARCH_USE_GNU_PROPERTY +int arch_parse_elf_property (u32 type,const void *data,size_t datasz, +bool compat,struct arch_elf_state *state) +{ +if (type !=GNU_PROPERTY_X86_FEATURE_1_AND ) +return 0 ; + +if (datasz !=sizeof (unsigned int )) +return -ENOEXEC ; + + state->gnu_property = *(unsigned int *)data; +return 0 ; +} + +int arch_setup_elf_property (struct arch_elf_state *state) +{ +int r =0 ; + +if (!IS_ENABLED (CONFIG_X86_SHADOW_STACK )) +return r; + +memset (¤t->thread.cet,0 ,sizeof (struct cet_status )); + +if (static_cpu_has (X86_FEATURE_SHSTK )) { +if (state->gnu_property &GNU_PROPERTY_X86_FEATURE_1_SHSTK ) + r =shstk_setup (); + } + +return r; +} +#endif
fs/binfmt_elf.c +4 -0
@@ -1248,6 +1248,10 @@ static int load_elf_binary(struct linux_binprm *bprm)set_binfmt (&elf_format); + retval =arch_setup_elf_property (&arch_state); +if (retval <0 ) + goto out; +#ifdef ARCH_HAS_SETUP_ADDITIONAL_PAGES retval =ARCH_SETUP_ADDITIONAL_PAGES (bprm, elf_ex, !!interpreter);if (retval <0 )
include/linux/elf.h +6 -0
@@ -92,9 +92,15 @@ static inline int arch_parse_elf_property(u32 type, const void *data, {return 0 ; } + +static inline int arch_setup_elf_property (struct arch_elf_state *arch) +{ +return 0 ; +}#else extern int arch_parse_elf_property (u32 type,const void *data,size_t datasz,bool compat,struct arch_elf_state *arch); +extern int arch_setup_elf_property (struct arch_elf_state *arch);#endif #ifdef CONFIG_ARCH_HAVE_ELF_PROT
include/uapi/linux/elf.h +9 -0
@@ -455,4 +455,13 @@ typedef struct elf64_note {/* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0) +/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002 + +/* Bits for GNU_PROPERTY_X86_FEATURE_1_AND */ +#define GNU_PROPERTY_X86_FEATURE_1_IBT 0x00000001 +#define GNU_PROPERTY_X86_FEATURE_1_SHSTK 0x00000002 +#define GNU_PROPERTY_X86_FEATURE_1_VALID (GNU_PROPERTY_X86_FEATURE_1_IBT | \ + GNU_PROPERTY_X86_FEATURE_1_SHSTK) +#endif /* _UAPI_LINUX_ELF_H */
Signature · 1 line
2.21.0
Borislav Petkov
maintainerRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
arch/x86/Kconfig config X86_SHADOW_STACK · line 1955
Quoted 6 lines · Yu-cheng Yu
arch/x86/Kconfig +2
-0@@ -1951,6 +1951,8 @@ config X86_SHADOW_STACK depends on AS_WRUSS depends on ARCH_HAS_SHADOW_STACK select ARCH_USES_HIGH_VMA_FLAGS + select ARCH_USE_GNU_PROPERTY + select ARCH_BINFMT_ELF_STATE
^^^^^^^^
What's that for? Isn't ARCH_USE_GNU_PROPERTY enough?
arch/x86/kernel/process_64.c unsigned long KSTK_ESP(struct task_struct *task) · line 862
Quoted 10 lines
arch/x86/kernel/process_64.c +10
-0+int arch_setup_elf_property (struct arch_elf_state *state) +{ +int r =0 ; + +if (!IS_ENABLED (CONFIG_X86_SHADOW_STACK )) +return r; + +memset (¤t->thread.cet,0 ,sizeof (struct cet_status )); + +if (static_cpu_has (X86_FEATURE_SHSTK )) {
cpu_feature_enabled
Quoted 7 lines
arch/x86/kernel/process_64.c +7
-0+if (state->gnu_property &GNU_PROPERTY_X86_FEATURE_1_SHSTK ) + r = shstk_setup(); + } + +return r; +} +#endif
...
include/uapi/linux/elf.h typedef struct elf64_note { · line 459
Quoted 10 lines
include/uapi/linux/elf.h +2
-0@@ -455,4 +455,13 @@ typedef struct elf64_note {/* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0) +/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002
Why not 0xc0000001? ARM64 is 0xc0000000...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 11 lines · Borislav Petkov
arch/x86/Kconfig config X86_SHADOW_STACK · line 1955
Quoted 6 lines · Yu-cheng Yu
arch/x86/Kconfig +2
-0@@ -1951,6 +1951,8 @@ config X86_SHADOW_STACK depends on AS_WRUSS depends on ARCH_HAS_SHADOW_STACK select ARCH_USES_HIGH_VMA_FLAGS + select ARCH_USE_GNU_PROPERTY + select ARCH_BINFMT_ELF_STATE^^^^^^^^What's that for? Isn't
ARCH_USE_GNU_PROPERTYenough?
ARCH_USE_GNU_PROPERTY is for defining parsing functions, e.g.
arch_parse_elf_property(), arch_setup_property().
ARCH_BINFMT_ELF_STATE is for defining "struct arch_elf_state".
However, those parsing functions take (struct arch_elf_state *) as an input. It probably makes sense to have ARCH_USE_GNU_PROPERTY dependent on ARCH_BINFMT_ELF_STATE. It would be ok as-is too. ARM people might have other plans in mind.
[...]
Quoted 9 lines
Quoted 9 lines
include/uapi/linux/elf.h +0
-0@@ -455,4 +455,13 @@ typedef struct elf64_note {/* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0)
Quoted 5 lines
include/uapi/linux/elf.h typedef struct elf64_note { · line 459
Quoted 2 lines
include/uapi/linux/elf.h +2
-0+/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Why not
0xc0000001? ARM64 is0xc0000000...
I just looked at the ABI document.
ARM has GNU_PROPERTY_AARCH64_FEATURE_1_AND 0xc0000000
X86 has:
GNU_PROPERTY_X86_ISA_1_USED 0xc0000000 GNU_PROPERTY_X86_ISA_1_NEEDED 0xc0000001 GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 1 line · Yu-cheng
However, those parsing functions take (struct
arch_elf_state*) as an input.
Exactly.
Quoted 3 lines
It probably makes sense to have
ARCH_USE_GNU_PROPERTYdependent onARCH_BINFMT_ELF_STATE. It would be ok as-is too. ARM people might have other plans in mind.
Well, let's look at ARM, ARM64 in particular. They have defined struct arch_elf_state without the ifdeffery in
arch/arm64/include/asm/elf.h
and are using that struct in arch_parse_elf_property().
And they have selected ARCH_BINFMT_ELF_STATE just so that they disable those dummy accessors in fs/binfmt_elf.c
And you're practically glueing together ARCH_BINFMT_ELF_STATE and ARCH_USE_GNU_PROPERTY. However, all the functionality is for adding the gnu property note so I think you should select both but only use ARCH_USE_GNU_PROPERTY in all the ifdeffery in your patch to at least have this as simple as possible.
Quoted 1 line
I just looked at the ABI document.
Which document is that? Link?
Quoted 6 lines
ARM has
GNU_PROPERTY_AARCH64_FEATURE_1_AND0xc0000000X86 has:
GNU_PROPERTY_X86_ISA_1_USED 0xc0000000 GNU_PROPERTY_X86_ISA_1_NEEDED 0xc0000001 GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002
Our defines should at least have a comment pointing to that document.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 16 lines · Borislav Petkov
Quoted 1 line · Yu-cheng
However, those parsing functions take (struct
arch_elf_state*) as an input.Exactly.
Quoted 3 lines
It probably makes sense to have
ARCH_USE_GNU_PROPERTYdependent onARCH_BINFMT_ELF_STATE. It would be ok as-is too. ARM people might have other plans in mind.Well, let's look at ARM, ARM64 in particular. They have defined struct
arch_elf_statewithout the ifdeffery in
arch/arm64/include/asm/elf.hand are using that struct in
arch_parse_elf_property().
Quoted 9 lines
And they have selected
ARCH_BINFMT_ELF_STATEjust so that they disable those dummy accessors infs/binfmt_elf.cAnd you're practically glueing together
ARCH_BINFMT_ELF_STATEandARCH_USE_GNU_PROPERTY. However, all the functionality is for adding the gnu property note so I think you should select both but only useARCH_USE_GNU_PROPERTYin all the ifdeffery in your patch to at least have this as simple as possible.
ARM64 has ARCH_USE_GNU_PROPERTY and ARCH_BINFMT_ELF_STATE selected unconditionally. We will do the same for X86_64 and remove the ifdeffery.
Quoted 6 lines
Quoted 1 line
I just looked at the ABI document.
Which document is that? Link?
Quoted 2 lines
ARM has
GNU_PROPERTY_AARCH64_FEATURE_1_AND0xc0000000
Quoted 9 lines
Quoted 4 lines
X86 has:
GNU_PROPERTY_X86_ISA_1_USED 0xc0000000 GNU_PROPERTY_X86_ISA_1_NEEDED 0xc0000001 GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Our defines should at least have a comment pointing to that document.
Thx.
The latest pdf's are posted here.
https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-psABI
Thanks,
Yu-cheng
Borislav Petkov
maintainerRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 3 lines · Yu-cheng
The latest pdf's are posted here.
https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-psABI
Ah, that document.
Please make sure it is specified over those defines from which document they're coming from.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Yu-cheng
The latest pdf's are posted here.
https://gitlab.com/x86-psABIs/x86-64-ABI/-/wikis/x86-64-psABI
Ah, that document.
Please make sure it is specified over those defines from which document they're coming from.
I will do that.
Re: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
In reply to Yu, Yu-chengQuoted 10 lines · Yu-cheng
Quoted 9 lines
include/uapi/linux/elf.h typedef struct elf64_note { · line 458
Quoted 6 lines
+++
b/include/uapi/linux/elf.hinclude/uapi/linux/elf.h +2
-0@@ -455,4 +455,13 @@ typedef struct elf64_note {/* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0) +/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Why not
0xc0000001? ARM64 is0xc0000000...
Quoted 8 lines
I just looked at the ABI document.
ARM has
GNU_PROPERTY_AARCH64_FEATURE_1_AND0xc0000000X86 has:
GNU_PROPERTY_X86_ISA_1_USED 0xc0000000 GNU_PROPERTY_X86_ISA_1_NEEDED 0xc0000001 GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002
Please add all three, not just the last one.
Re: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 13 lines · Matthew Wilcox
Quoted 12 lines · Yu-cheng
Quoted 9 lines
include/uapi/linux/elf.h typedef struct elf64_note { · line 458
Quoted 6 lines
+++
b/include/uapi/linux/elf.hinclude/uapi/linux/elf.h +2
-0@@ -455,4 +455,13 @@ typedef struct elf64_note {/* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0) +/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Why not
0xc0000001? ARM64 is0xc0000000...I just looked at the ABI document.
Quoted 9 lines
Quoted 6 lines
ARM has
GNU_PROPERTY_AARCH64_FEATURE_1_AND0xc0000000X86 has:
GNU_PROPERTY_X86_ISA_1_USED 0xc0000000 GNU_PROPERTY_X86_ISA_1_NEEDED 0xc0000001 GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Please add all three, not just the last one.
Ok!
Re: [PATCH v26 26/30] ELF: Introduce arch_setup_elf_property()
Quoted 21 lines · Yu-cheng
Quoted 20 lines · Matthew Wilcox
Quoted 18 lines · Yu-cheng
Quoted 9 lines
include/uapi/linux/elf.h typedef struct elf64_note { · line 458
Quoted 6 lines
+++
b/include/uapi/linux/elf.hinclude/uapi/linux/elf.h +2
-0@@ -455,4 +455,13 @@ typedef struct elf64_note { /* Bits for GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ #define GNU_PROPERTY_AARCH64_FEATURE_1_BTI (1U << 0) +/* .note.gnu.property types for x86: */ +#define GNU_PROPERTY_X86_FEATURE_1_AND 0xc0000002Why not
0xc0000001? ARM64 is0xc0000000...I just looked at the ABI document.
ARM has
GNU_PROPERTY_AARCH64_FEATURE_1_AND0xc0000000X86 has:
GNU_PROPERTY_X86_ISA_1_USED0xc0000000
GNU_PROPERTY_X86_ISA_1_NEEDED0xc0000001
GNU_PROPERTY_X86_FEATURE_1_AND0xc0000002
Quoted 4 lines
Quoted 2 lines
Please add all three, not just the last one.
Ok!
Just found out, I have been reading an older version. Now 0xc0000000 and 0xc0000001 have become reserved/not-used. Maybe I will just put a note about that along with the link to the ABI document.
Yu-cheng
PATCH v26 27/30 x86/cet/shstk: Add arch_prctl functions for shadow stack
In reply to Yu-cheng Yuarch_prctl(ARCH_X86_CET_STATUS, u64 *args)
Get CET feature status.
The parameter 'args' is a pointer to a user buffer. The kernel returns the following information:
*args = shadow stack/IBT status
*(args + 1) = shadow stack base address
*(args + 2) = shadow stack size
32-bit binaries use the same interface, but only lower 32-bits of each item.
arch_prctl(ARCH_X86_CET_DISABLE, unsigned int features)
Disable CET features specified in 'features'. Return -EPERM if CET is locked.
arch_prctl(ARCH_X86_CET_LOCK)
Lock in CET features.
Also change do_arch_prctl_common()'s parameter 'cpuid_enabled' to 'arg2', as it is now also passed to prctl_cet().
---
v25:
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.
v24:
- Update #ifdef placement relating to shadow stack and ibt split.
- Update function names.
arch/x86/include/asm/cet.h | 7 ++++ arch/x86/include/uapi/asm/prctl.h | 4 +++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/cet_prctl.c | 60 +++++++++++++++++++++++++++++++ arch/x86/kernel/process.c | 6 ++-- 5 files changed, 75 insertions(+), 4 deletions(-) create mode 100644 arch/x86/kernel/cet_prctl.c
arch/x86/include/asm/cet.h +7 -0
@@ -14,6 +14,7 @@ struct sc_ext;struct cet_status {unsigned long shstk_base;unsigned long shstk_size; +unsigned int locked:1 ; };#ifdef CONFIG_X86_SHADOW_STACK
@@ -40,6 +41,12 @@ static inline int shstk_check_rstor_token(bool ia32, unsigned long token_addr, unsigned long *new_ssp) { return 0; }#endif +#ifdef CONFIG_X86_SHADOW_STACK +int prctl_cet (int option,u64 arg2); +#else +static inline int prctl_cet (int option,u64 arg2) {return -EINVAL ; } +#endif +#endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_CET_H */
arch/x86/include/uapi/asm/prctl.h +4 -0
@@ -14,4 +14,8 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_X86_CET_STATUS 0x3001 +#define ARCH_X86_CET_DISABLE 0x3002 +#define ARCH_X86_CET_LOCK 0x3003 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +1 -1
@@ -150,7 +150,7 @@ obj-$(CONFIG_UNWINDER_FRAME_POINTER) += unwind_frame.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += sev-es.o -obj-$(CONFIG_X86_SHADOW_STACK) += shstk.o +obj-$(CONFIG_X86_SHADOW_STACK) += shstk.o cet_prctl.o ### # 64 bit specific files
arch/x86/kernel/cet_prctl.c new file +60 -0
60 lines not on this page · raw message
arch/x86/kernel/process.c +3 -3
@@ -980,14 +980,14 @@ unsigned long get_wchan(struct task_struct *p) }long do_arch_prctl_common (struct task_struct *task,int option, -unsigned long cpuid_enabled) +unsigned long arg2) {switch (option) {case ARCH_GET_CPUID :return get_cpuid_mode ();case ARCH_SET_CPUID : -return set_cpuid_mode (task, cpuid_enabled); +return set_cpuid_mode (task, arg2); } -return -EINVAL ; +return prctl_cet (option, arg2); }
Signature · 1 line
2.21.0
PATCH v26 28/30 mm: Move arch_calc_vm_prot_bits() to arch/x86/include/asm/mman.h
In reply to Yu-cheng YuTo prepare the introduction of PROT_SHADOW_STACK and be consistent with other architectures, move arch_vm_get_page_prot() and arch_calc_vm_prot_bits() to arch/x86/include/asm/mman.h.
--- arch/x86/include/asm/mman.h | 30 ++++++++++++++++++++++++++++++ arch/x86/include/uapi/asm/mman.h | 28 +++------------------------- 2 files changed, 33 insertions(+), 25 deletions(-) create mode 100644 arch/x86/include/asm/mman.h
arch/x86/include/asm/mman.h new file +30 -0
@@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_MMAN_H +#define _ASM_X86_MMAN_H + +#include <linux/mm.h> +#include <uapi/asm/mman.h> + +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS +/* +* Take the 4 protection key bits out of the vma->vm_flags +* value and turn them in to the bits that we can put in +* to a pte. +* +* Only override these if Protection Keys are available +* (which is only on 64-bit). +*/ +#define arch_vm_get_page_prot (vm_flags) __pgprot( \ + ((vm_flags) & VM_PKEY_BIT0 ? _PAGE_PKEY_BIT0 : 0) | \ + ((vm_flags) & VM_PKEY_BIT1 ? _PAGE_PKEY_BIT1 : 0) | \ + ((vm_flags) & VM_PKEY_BIT2 ? _PAGE_PKEY_BIT2 : 0) | \ + ((vm_flags) & VM_PKEY_BIT3 ? _PAGE_PKEY_BIT3 : 0)) + +#define arch_calc_vm_prot_bits (prot, key) ( \ + ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ + ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ + ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ + ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) +#endif + +#endif /* _ASM_X86_MMAN_H */
arch/x86/include/uapi/asm/mman.h +3 -25
@@ -1,31 +1,9 @@/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _ASM_X86_MMAN_H -#define _ASM_X86_MMAN_H +#ifndef _UAPI_ASM_X86_MMAN_H +#define _UAPI_ASM_X86_MMAN_H#define MAP_32BIT 0x40/* only give out 32bit addresses */ -#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS -/* -* Take the 4 protection key bits out of the vma->vm_flags -* value and turn them in to the bits that we can put in -* to a pte. -* -* Only override these if Protection Keys are available -* (which is only on 64-bit). -*/ -#define arch_vm_get_page_prot (vm_flags) __pgprot( \ - ((vm_flags) & VM_PKEY_BIT0 ? _PAGE_PKEY_BIT0 : 0) | \ - ((vm_flags) & VM_PKEY_BIT1 ? _PAGE_PKEY_BIT1 : 0) | \ - ((vm_flags) & VM_PKEY_BIT2 ? _PAGE_PKEY_BIT2 : 0) | \ - ((vm_flags) & VM_PKEY_BIT3 ? _PAGE_PKEY_BIT3 : 0)) - -#define arch_calc_vm_prot_bits (prot, key) ( \ - ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ - ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ - ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ - ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) -#endif -#include <asm-generic/mman.h> -#endif /* _ASM_X86_MMAN_H */ +#endif /* _UAPI_ASM_X86_MMAN_H */
Signature · 1 line
2.21.0
PATCH v26 29/30 mm: Update arch_validate_flags() to test vma anonymous
In reply to Yu-cheng YuWhen newer VM flags are being created, such as VM_MTE, it becomes necessary for mmap/mprotect to verify if certain flags are being applied to an anonymous VMA.
To solve this, one approach is adding a VM flag to track that MAP_ANONYMOUS is specified [1], and then using the flag in arch_validate_flags().
Another approach is passing the VMA to arch_validate_flags(), and check vma_is_anonymous().
To prepare the introduction of PROT_SHADOW_STACK, which creates a shadow stack mapping and can be applied only to an anonymous VMA, update arch_validate_flags() to pass in the VMA.
[1] commit 9f3419315f3c ("arm64: mte: Add PROT_MTE support to mmap() and mprotect()"),
---
v26:
- Instead of passing vma is anonymous, pass the VMA to arch_validate_flags().
arch/arm64/include/asm/mman.h | 4 ++-- arch/sparc/include/asm/mman.h | 4 ++-- include/linux/mman.h | 2 +- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/include/asm/mman.h +2 -2
@@ -74,7 +74,7 @@ static inline bool arch_validate_prot(unsigned long prot, }#define arch_validate_prot (prot, addr) arch_validate_prot(prot, addr) -static inline bool arch_validate_flags (unsigned long vm_flags) +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long vm_flags) {if (!system_supports_mte ())return true;
@@ -82,6 +82,6 @@ static inline bool arch_validate_flags(unsigned long vm_flags)/* only allow VM_MTE if VM_MTE_ALLOWED has been set previously */ return !(vm_flags & VM_MTE) || (vm_flags & VM_MTE_ALLOWED); } -#define arch_validate_flags (vm_flags) arch_validate_flags(vm_flags) +#define arch_validate_flags (vma, vm_flags) arch_validate_flags(vma, vm_flags)#endif /* ! __ASM_MMAN_H__ */
arch/sparc/include/asm/mman.h +2 -2
@@ -60,11 +60,11 @@ static inline int sparc_validate_prot(unsigned long prot, unsigned long addr)return 1 ; } -#define arch_validate_flags (vm_flags) arch_validate_flags(vm_flags) +#define arch_validate_flags (vma, vm_flags) arch_validate_flags(vma, vm_flags)/* arch_validate_flags() - Ensure combination of flags is valid for a * VMA. */ -static inline bool arch_validate_flags (unsigned long vm_flags) +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long vm_flags) {/* If ADI is being enabled on this VMA, check for ADI * capability on the platform and ensure VMA is suitable
include/linux/mman.h +1 -1
@@ -114,7 +114,7 @@ static inline bool arch_validate_prot(unsigned long prot, unsigned long addr)* * Returns true if the VM_* flags are valid. */ -static inline bool arch_validate_flags (unsigned long flags) +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long flags) {return true; }
mm/mmap.c +1 -1
@@ -1850,7 +1850,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, }/* Allow architectures to sanity-check the vm_flags */ -if (!arch_validate_flags (vma->vm_flags)) { +if (!arch_validate_flags (vma, vma->vm_flags)) { error = -EINVAL ;if (file) goto unmap_and_free_vma;
mm/mprotect.c +1 -1
@@ -611,7 +611,7 @@ static int do_mprotect_pkey(unsigned long start, size_t len, }/* Allow architectures to sanity-check the new flags */ -if (!arch_validate_flags (newflags)) { +if (!arch_validate_flags (vma, newflags)) { error = -EINVAL ; goto out; }
Signature · 1 line
2.21.0
Re: [PATCH v26 29/30] mm: Update arch_validate_flags() to test vma anonymous
Quoted 10 lines · Yu-cheng Yu
When newer VM flags are being created, such as
VM_MTE, it becomes necessary for mmap/mprotect to verify if certain flags are being applied to an anonymous VMA.To solve this, one approach is adding a VM flag to track that
MAP_ANONYMOUSis specified [1], and then using the flag inarch_validate_flags().Another approach is passing the VMA to
arch_validate_flags(), and checkvma_is_anonymous().
Quoted 7 lines
To prepare the introduction of
PROT_SHADOW_STACK, which creates a shadow stack mapping and can be applied only to an anonymous VMA, updatearch_validate_flags()to pass in the VMA.[1] commit
9f3419315f3c("arm64: mte: AddPROT_MTEsupport tommap()andmprotect()"),Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kirill A. Shutemov
PATCH v26 30/30 mm: Introduce PROT_SHADOW_STACK for shadow stack
In reply to Yu-cheng YuThere are three possible options to create a shadow stack allocation API:
an arch_prctl, a new syscall, or adding PROT_SHADOW_STACK to mmap() and mprotect(). Each has its advantages and compromises.
An arch_prctl() is the least intrusive. However, the existing x86 arch_prctl() takes only two parameters. Multiple parameters must be passed in a memory buffer. There is a proposal to pass more parameters in registers [1], but no active discussion on that.
A new syscall minimizes compatibility issues and offers an extensible frame work to other architectures, but this will likely result in some overlap of mmap()/mprotect().
The introduction of PROT_SHADOW_STACK to mmap()/mprotect() takes advantage of existing APIs. The x86-specific PROT_SHADOW_STACK is translated to VM_SHADOW_STACK and a shadow stack mapping is created without reinventing the wheel. There are potential pitfalls though. The most obvious one would be using this as a bypass to shadow stack protection. However, the attacker would have to get to the syscall first.
[1] https://lore.kernel.org/lkml/[email protected]/
---
v26:
- Change PROT_SHSTK to PROT_SHADOW_STACK.
- Remove (vm_flags & VM_SHARED) check, since it is covered by !vma_is_anonymous().
v24:
- Update arch_calc_vm_prot_bits(), leave PROT* checking to arch_validate_prot().
- Update arch_validate_prot(), leave vma flags checking to arch_validate_flags().
- Add arch_validate_flags().
arch/x86/include/asm/mman.h | 60 +++++++++++++++++++++++++++++++- arch/x86/include/uapi/asm/mman.h | 2 ++ include/linux/mm.h | 1 + 3 files changed, 62 insertions(+), 1 deletion(-)
arch/x86/include/asm/mman.h +59 -1
@@ -20,11 +20,69 @@ ((vm_flags) & VM_PKEY_BIT2 ? _PAGE_PKEY_BIT2 : 0) | \ ((vm_flags) & VM_PKEY_BIT3 ? _PAGE_PKEY_BIT3 : 0)) -#define arch_calc_vm_prot_bits (prot, key) ( \ +#define pkey_vm_prot_bits (prot, key) ( \ ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) +#else +#define pkey_vm_prot_bits (prot, key) (0)#endif +static inline unsigned long arch_calc_vm_prot_bits (unsigned long prot, +unsigned long pkey) +{ +unsigned long vm_prot_bits =pkey_vm_prot_bits (prot, pkey); + +if (prot &PROT_SHADOW_STACK ) + vm_prot_bits |=VM_SHADOW_STACK ; + +return vm_prot_bits; +} + +#define arch_calc_vm_prot_bits (prot, pkey) arch_calc_vm_prot_bits(prot, pkey) + +#ifdef CONFIG_X86_SHADOW_STACK +static inline bool arch_validate_prot (unsigned long prot,unsigned long addr) +{ +unsigned long valid =PROT_READ |PROT_WRITE |PROT_EXEC |PROT_SEM | +PROT_SHADOW_STACK ; + +if (prot & ~valid) +return false; + +if (prot &PROT_SHADOW_STACK ) { +if (!current->thread.cet.shstk_size) +return false; + +/* +* A shadow stack mapping is indirectly writable by only +* the CALL and WRUSS instructions, but not other write +* instructions). PROT_SHADOW_STACK and PROT_WRITE are +* mutually exclusive. +*/ +if (prot &PROT_WRITE ) +return false; + } + +return true; +} + +#define arch_validate_prot arch_validate_prot + +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long vm_flags) +{ +/* +* Shadow stack must be anonymous and not shared. +*/ +if ((vm_flags &VM_SHADOW_STACK ) && !vma_is_anonymous (vma)) +return false; + +return true; +} + +#define arch_validate_flags (vma, vm_flags) arch_validate_flags(vma, vm_flags) + +#endif /* CONFIG_X86_SHADOW_STACK */ +#endif /* _ASM_X86_MMAN_H */
arch/x86/include/uapi/asm/mman.h +2 -0
@@ -4,6 +4,8 @@#define MAP_32BIT 0x40/* only give out 32bit addresses */ +#define PROT_SHADOW_STACK 0x10/* shadow stack pages */ +#include <asm-generic/mman.h> #endif /* _UAPI_ASM_X86_MMAN_H */
include/linux/mm.h +1 -0
@@ -342,6 +342,7 @@ extern unsigned int kobjsize(const void *objp);#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ +# define VM_ARCH_CLEAR VM_SHADOW_STACK#elif defined(CONFIG_PPC )# define VM_SAO VM_ARCH_1/* Strong Access Ordering (powerpc) */ #elif defined(CONFIG_PARISC )
Signature · 1 line
2.21.0
Re: [PATCH v26 30/30] mm: Introduce PROT_SHADOW_STACK for shadow stack
Quoted 136 lines · Yu-cheng Yu
There are three possible options to create a shadow stack allocation API:
anarch_prctl, a new syscall, or addingPROT_SHADOW_STACKtommap()andmprotect(). Each has its advantages and compromises.An
arch_prctl()is the least intrusive. However, the existing x86arch_prctl()takes only two parameters. Multiple parameters must be passed in a memory buffer. There is a proposal to pass more parameters in registers [1], but no active discussion on that.A new syscall minimizes compatibility issues and offers an extensible frame work to other architectures, but this will likely result in some overlap of mmap()/mprotect().
The introduction of
PROT_SHADOW_STACKto mmap()/mprotect() takes advantage of existing APIs. The x86-specificPROT_SHADOW_STACKis translated toVM_SHADOW_STACKand a shadow stack mapping is created without reinventing the wheel. There are potential pitfalls though. The most obvious one would be using this as a bypass to shadow stack protection. However, the attacker would have to get to the syscall first.[1] https://lore.kernel.org/lkml/[email protected]/
Signed-off-by Yu-cheng Yu <[email protected]>Cc Kees Cook <[email protected]>Cc Kirill A. Shutemov <[email protected]>---
v26:
- ChangePROT_SHSTKtoPROT_SHADOW_STACK.
- Remove (vm_flags&VM_SHARED) check, since it is covered by !vma_is_anonymous().v24:
- Updatearch_calc_vm_prot_bits(), leave PROT* checking toarch_validate_prot().
- Updatearch_validate_prot(), leave vma flags checking toarch_validate_flags().
- Addarch_validate_flags().arch/x86/include/asm/mman.h | 60 +++++++++++++++++++++++++++++++- arch/x86/include/uapi/asm/mman.h | 2 ++ include/linux/mm.h | 1 + 3 files changed, 62 insertions(+), 1 deletion(-)arch/x86/include/asm/mman.h +59
-1@@ -20,11 +20,69 @@ ((vm_flags) & VM_PKEY_BIT2 ? _PAGE_PKEY_BIT2 : 0) | \ ((vm_flags) & VM_PKEY_BIT3 ? _PAGE_PKEY_BIT3 : 0)) -#define arch_calc_vm_prot_bits (prot, key) ( \ +#define pkey_vm_prot_bits (prot, key) ( \ ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) +#else +#define pkey_vm_prot_bits (prot, key) (0)#endif +static inline unsigned long arch_calc_vm_prot_bits (unsigned long prot, +unsigned long pkey) +{ +unsigned long vm_prot_bits =pkey_vm_prot_bits (prot, pkey); + +if (prot &PROT_SHADOW_STACK ) + vm_prot_bits |=VM_SHADOW_STACK ; + +return vm_prot_bits; +} + +#define arch_calc_vm_prot_bits (prot, pkey) arch_calc_vm_prot_bits(prot, pkey) + +#ifdef CONFIG_X86_SHADOW_STACK +static inline bool arch_validate_prot (unsigned long prot,unsigned long addr) +{ +unsigned long valid =PROT_READ |PROT_WRITE |PROT_EXEC |PROT_SEM | +PROT_SHADOW_STACK ; + +if (prot & ~valid) +return false; + +if (prot &PROT_SHADOW_STACK ) { +if (!current->thread.cet.shstk_size) +return false; + +/* +* A shadow stack mapping is indirectly writable by only +* the CALL and WRUSS instructions, but not other write +* instructions). PROT_SHADOW_STACK and PROT_WRITE are +* mutually exclusive. +*/ +if (prot &PROT_WRITE ) +return false; + } + +return true; +} + +#define arch_validate_prot arch_validate_prot + +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long vm_flags) +{ +/* +* Shadow stack must be anonymous and not shared. +*/ +if ((vm_flags &VM_SHADOW_STACK ) && !vma_is_anonymous (vma)) +return false; + +return true; +} + +#define arch_validate_flags (vma, vm_flags) arch_validate_flags(vma, vm_flags) + +#endif /* CONFIG_X86_SHADOW_STACK */ +#endif /* _ASM_X86_MMAN_H */ arch/x86/include/uapi/asm/mman.h +2
-0@@ -4,6 +4,8 @@#define MAP_32BIT 0x40/* only give out 32bit addresses */ +#define PROT_SHADOW_STACK 0x10/* shadow stack pages */ +#include <asm-generic/mman.h> #endif /* _UAPI_ASM_X86_MMAN_H */ include/linux/mm.h +0
-0
include/linux/mm.h extern unsigned int kobjsize(const void *objp); · line 345
Quoted 4 lines
include/linux/mm.h +1
-0@@ -342,6 +342,7 @@ extern unsigned int kobjsize(const void *objp);#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ +# define VM_ARCH_CLEAR VM_SHADOW_STACK
Nit: you can put VM_SHADOW_STACK directly into VM_FLAGS_CLEAR. It's already conditinal on the feature enabled and VM_NONE otherwise.
Up to you.
Signature · 1 line
Kirill A. Shutemov
Re: [PATCH v26 30/30] mm: Introduce PROT_SHADOW_STACK for shadow stack
Quoted 145 lines · Kirill A. Shutemov
Quoted 136 lines · Yu-cheng Yu
There are three possible options to create a shadow stack allocation API:
anarch_prctl, a new syscall, or addingPROT_SHADOW_STACKtommap()andmprotect(). Each has its advantages and compromises.An
arch_prctl()is the least intrusive. However, the existing x86arch_prctl()takes only two parameters. Multiple parameters must be passed in a memory buffer. There is a proposal to pass more parameters in registers [1], but no active discussion on that.A new syscall minimizes compatibility issues and offers an extensible frame work to other architectures, but this will likely result in some overlap of mmap()/mprotect().
The introduction of
PROT_SHADOW_STACKto mmap()/mprotect() takes advantage of existing APIs. The x86-specificPROT_SHADOW_STACKis translated toVM_SHADOW_STACKand a shadow stack mapping is created without reinventing the wheel. There are potential pitfalls though. The most obvious one would be using this as a bypass to shadow stack protection. However, the attacker would have to get to the syscall first.[1] https://lore.kernel.org/lkml/[email protected]/
Signed-off-by Yu-cheng Yu <[email protected]>Cc Kees Cook <[email protected]>Cc Kirill A. Shutemov <[email protected]>---
v26:
- ChangePROT_SHSTKtoPROT_SHADOW_STACK.
- Remove (vm_flags&VM_SHARED) check, since it is covered by !vma_is_anonymous().v24:
- Updatearch_calc_vm_prot_bits(), leave PROT* checking toarch_validate_prot().
- Updatearch_validate_prot(), leave vma flags checking toarch_validate_flags().
- Addarch_validate_flags().arch/x86/include/asm/mman.h | 60 +++++++++++++++++++++++++++++++- arch/x86/include/uapi/asm/mman.h | 2 ++ include/linux/mm.h | 1 + 3 files changed, 62 insertions(+), 1 deletion(-)arch/x86/include/asm/mman.h +59
-1@@ -20,11 +20,69 @@ ((vm_flags) & VM_PKEY_BIT2 ? _PAGE_PKEY_BIT2 : 0) | \ ((vm_flags) & VM_PKEY_BIT3 ? _PAGE_PKEY_BIT3 : 0)) -#define arch_calc_vm_prot_bits (prot, key) ( \ +#define pkey_vm_prot_bits (prot, key) ( \ ((key) & 0x1 ? VM_PKEY_BIT0 : 0) | \ ((key) & 0x2 ? VM_PKEY_BIT1 : 0) | \ ((key) & 0x4 ? VM_PKEY_BIT2 : 0) | \ ((key) & 0x8 ? VM_PKEY_BIT3 : 0)) +#else +#define pkey_vm_prot_bits (prot, key) (0)#endif +static inline unsigned long arch_calc_vm_prot_bits (unsigned long prot, +unsigned long pkey) +{ +unsigned long vm_prot_bits =pkey_vm_prot_bits (prot, pkey); + +if (prot &PROT_SHADOW_STACK ) + vm_prot_bits |=VM_SHADOW_STACK ; + +return vm_prot_bits; +} + +#define arch_calc_vm_prot_bits (prot, pkey) arch_calc_vm_prot_bits(prot, pkey) + +#ifdef CONFIG_X86_SHADOW_STACK +static inline bool arch_validate_prot (unsigned long prot,unsigned long addr) +{ +unsigned long valid =PROT_READ |PROT_WRITE |PROT_EXEC |PROT_SEM | +PROT_SHADOW_STACK ; + +if (prot & ~valid) +return false; + +if (prot &PROT_SHADOW_STACK ) { +if (!current->thread.cet.shstk_size) +return false; + +/* +* A shadow stack mapping is indirectly writable by only +* the CALL and WRUSS instructions, but not other write +* instructions). PROT_SHADOW_STACK and PROT_WRITE are +* mutually exclusive. +*/ +if (prot &PROT_WRITE ) +return false; + } + +return true; +} + +#define arch_validate_prot arch_validate_prot + +static inline bool arch_validate_flags (struct vm_area_struct *vma,unsigned long vm_flags) +{ +/* +* Shadow stack must be anonymous and not shared. +*/ +if ((vm_flags &VM_SHADOW_STACK ) && !vma_is_anonymous (vma)) +return false; + +return true; +} + +#define arch_validate_flags (vma, vm_flags) arch_validate_flags(vma, vm_flags) + +#endif /* CONFIG_X86_SHADOW_STACK */ +#endif /* _ASM_X86_MMAN_H */ arch/x86/include/uapi/asm/mman.h +2
-0@@ -4,6 +4,8 @@#define MAP_32BIT 0x40/* only give out 32bit addresses */ +#define PROT_SHADOW_STACK 0x10/* shadow stack pages */ +#include <asm-generic/mman.h> #endif /* _UAPI_ASM_X86_MMAN_H */ include/linux/mm.h +0
-0include/linux/mm.h extern unsigned int kobjsize(const void *objp); · line 345
Quoted 4 lines
include/linux/mm.h +1
-0@@ -342,6 +342,7 @@ extern unsigned int kobjsize(const void *objp);#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ +# define VM_ARCH_CLEAR VM_SHADOW_STACKNit: you can put
VM_SHADOW_STACKdirectly intoVM_FLAGS_CLEAR. It's already conditinal on the feature enabled andVM_NONEotherwise.
Quoted 4 lines
Up to you.
Reviewed-by Kirill A. Shutemov <[email protected]>
Thanks!
Borislav Petkov
maintainerQuoted 2 lines · Yu-cheng Yu
I have run tests on these patches for quite some time, and they have been very stable.
While at it, how about adding a simple selftest too, so that people can play with it and perhaps it could even run a bunch of tests?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 7 lines · Borislav Petkov
Quoted 2 lines · Yu-cheng Yu
I have run tests on these patches for quite some time, and they have been very stable.
While at it, how about adding a simple selftest too, so that people can play with it and perhaps it could even run a bunch of tests?
I did send out some selftest patches (link is in the cover letter). However, those need to be updated to the Linux standard, and I would like to do it separately.
Yu-cheng
Borislav Petkov
maintainerQuoted 3 lines · Yu-cheng
I did send out some selftest patches (link is in the cover letter). However, those need to be updated to the Linux standard, and I would like to do it separately.
Ok, and they look exactly like what I had in mind.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette