Shadow stacks for userspace
Subsystems (23)
v5 cover Shadow stacks for userspace
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.
The main use case for shadow stack is providing protection against return oriented programming attacks. It works by maintaining a secondary (shadow) stack using a special memory type that has protections against modification. When executing a CALL instruction, the processor pushes the return address to both the normal stack and to the special permission shadow stack. Upon RET, the processor pops the shadow stack copy and compares it to the normal stack copy. For more details, see the coverletter from v1 [0].
The main change in this version is the removal of the attempt to prevent 32 bit signals from being registered with shadow stack enabled. Peterz originally raised the issue that shadow stack support in 32 bit signals was in a half working state. The reason for that was 32 bit signals are not easy to support for shadow stack, and also there is not a huge demand for shadow stack support in 32 bit apps using 32 bit emulation on 64 bit kernels. At that point the solution was to prevent shadow stack from being enabled on 32 bit processes. But Peterz pointed that 64 bit apps can transition to 32 bit outside of kernel interaction by making a far call to a 32 bit segment.
So the next solution was to prevent 32 bit signals from being registered when shadow stack was enabled. This turned out to be hard to do, due to signals being per-process and shadow stack being per task.
But it turns out this far call scenario was already mostly not possible due to the HW not supporting shadow stacks located outside of the 32 bit address space when in 32 bit mode. During the transition to 32 bit mode with an SSP pointing outside of the 32 bit address space, HW generates a #GP which in turn triggers a segfault. So basically there is already a barrier in place for this far call scenario for the most part. Creation of shadow stack memory is tightly controlled, so the solution in this version is just to *ensure* that shadow stacks can never be allocated in the 32 bit address space. For more information see the new patch: "x86/mm: Introduce MAP_ABOVE4G", and the documentation in patch 1.
Additionally:
- A smattering of small changes from Boris and Kees
- Fixed my spellcheck setup and then fixed a bunch of spelling issues in the commit logs.
- An update to the pte_modify() PAGE_COW solution
I left tested-by tags in place per discussion with testers. Testers, please retest.
Previous version [1].
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
Kirill A. Shutemov (1):
x86: Introduce userspace API for shadow stack
Mike Rapoport (1):
x86/shstk: Add ARCH_SHSTK_UNLOCK
Rick Edgecombe (14):
x86/fpu: Add helper for modifying xstate x86/mm: Introduce _PAGE_COW
x86/mm: Start actually marking _PAGE_COW
mm: Handle faultless write upgrades for shstk
mm: Don't allow write GUPs to shadow stack memory x86/mm: Introduce MAP_ABOVE4G
mm: Warn on shadow stack memory in wrong vma x86/shstk: Introduce map_shadow_stack syscall x86/shstk: Support WRSS for userspace
x86: Expose thread features in /proc/$PID/status x86/shstk: Wire in shadow stack interface selftests/x86: Add shadow stack test x86/fpu: Add helper for initing features x86/shstk: Add ARCH_SHSTK_STATUS
Yu-cheng Yu (23):
Documentation/x86: Add CET shadow stack description x86/shstk: Add Kconfig option for shadow stack x86/cpufeatures: Add CPU feature flags for shadow stacks x86/cpufeatures: Enable CET CR4 bit for shadow stack x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
x86: Add user control-protection fault handler x86/mm: Remove _PAGE_DIRTY from kernel RO pages x86/mm: Move pmd_write(), pud_write() up in the file x86/mm: Update pte_modify for _PAGE_COW
x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for
transition from _PAGE_DIRTY to _PAGE_COW
mm: Move VM_UFFD_MINOR_BIT from 37 to 38
mm: Introduce VM_SHADOW_STACK for shadow stack memory x86/mm: Check shadow stack page fault errors
x86/mm: Update maybe_mkwrite() for shadow stack
mm: Fixup places that call pte_mkwrite() directly
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting
mm: Re-introduce vm_flags to do_mmap()
x86/shstk: Add user-mode shadow stack support x86/shstk: Handle thread shadow stack
x86/shstk: Introduce routines modifying shstk x86/shstk: Handle signals for shadow stack
x86: Add PTRACE interface for shadow stack
Documentation/filesystems/proc.rst | 1 + Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 176 +++++ arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/Kconfig | 24 + arch/x86/Kconfig.assembler | 5 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/disabled-features.h | 16 +- arch/x86/include/asm/fpu/api.h | 9 + arch/x86/include/asm/fpu/regset.h | 7 +- arch/x86/include/asm/fpu/sched.h | 3 +- arch/x86/include/asm/fpu/types.h | 16 +- arch/x86/include/asm/fpu/xstate.h | 6 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/mmu_context.h | 2 + arch/x86/include/asm/msr.h | 11 + arch/x86/include/asm/pgtable.h | 338 ++++++++- arch/x86/include/asm/pgtable_types.h | 65 +- arch/x86/include/asm/processor.h | 8 + arch/x86/include/asm/shstk.h | 40 ++ arch/x86/include/asm/special_insns.h | 13 + arch/x86/include/asm/tlbflush.h | 3 +- arch/x86/include/asm/trap_pf.h | 2 + arch/x86/include/asm/traps.h | 12 + arch/x86/include/uapi/asm/mman.h | 4 + arch/x86/include/uapi/asm/prctl.h | 12 + arch/x86/kernel/Makefile | 4 + arch/x86/kernel/cet.c | 152 ++++ arch/x86/kernel/cpu/common.c | 35 +- arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/proc.c | 23 + arch/x86/kernel/fpu/core.c | 59 +- arch/x86/kernel/fpu/regset.c | 87 +++ arch/x86/kernel/fpu/xstate.c | 148 ++-- arch/x86/kernel/fpu/xstate.h | 6 + arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/process.c | 18 +- arch/x86/kernel/process_64.c | 9 +- arch/x86/kernel/ptrace.c | 12 + arch/x86/kernel/shstk.c | 492 +++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 8 +- arch/x86/kernel/sys_x86_64.c | 6 +- arch/x86/kernel/traps.c | 87 --- arch/x86/mm/fault.c | 38 + arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/mm/pgtable.c | 6 + arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- fs/aio.c | 2 +- fs/proc/array.c | 6 + fs/proc/task_mmu.c | 3 + include/linux/mm.h | 59 +- include/linux/mman.h | 4 + include/linux/pgtable.h | 35 + include/linux/proc_fs.h | 2 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/siginfo.h | 3 +- include/uapi/asm-generic/unistd.h | 2 +- include/uapi/linux/elf.h | 2 + ipc/shm.c | 2 +- kernel/sys_ni.c | 1 + mm/gup.c | 2 +- mm/huge_memory.c | 12 +- mm/memory.c | 7 +- mm/migrate_device.c | 4 +- mm/mmap.c | 12 +- mm/nommu.c | 4 +- mm/userfaultfd.c | 10 +- mm/util.c | 2 +- tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 667 ++++++++++++++++++ 78 files changed, 2578 insertions(+), 259 deletions(-) create mode 100644 Documentation/x86/shstk.rst create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/cet.c create mode 100644 arch/x86/kernel/shstk.c create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
Signature · 1 line
2.17.1
PATCH v5 1/39 Documentation/x86: Add CET shadow stack description
From: Yu-cheng Yu <[email protected]>
Introduce a new document on Control-flow Enforcement Technology (CET).
---
v5:
- Literal format tweaks (Bagas Sanjaya)
- Update EOPNOTSUPP text due to unification after comment from (Kees)
- Update 32 bit signal support with new behavior
- Remove capitalization on shadow stack (Boris)
- Fix typo
v4:
- Drop clearcpuid piece (Boris)
- Add some info about 32 bit
v3:
- Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper)
- Clarify which arch_prctl's can take multiple bits. (Kees)
- Describe ASLR characteristics of thread shadow stacks. (Kees)
- Add exec section. (Andrew Cooper)
- Fix some capitalization (Bagas Sanjaya)
- Update new location of enablement status proc.
- Add info about new user_shstk software capability.
- Add more info about what the kernel pushes to the shadow stack on signal.
v2:
- Updated to new arch_prctl() API
- Add bit about new proc status
Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 Documentation/x86/shstk.rst
Documentation/x86/index.rst +1 -0
@@ -22,6 +22,7 @@ x86-specific Documentation mtrr pat intel-hfi + shstk iommu intel_txt amd-memory-encryption
Documentation/x86/shstk.rst new file +166 -0
166 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 01/39] Documentation/x86: Add CET shadow stack description
Quoted 7 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Introduce a new document on Control-flow Enforcement Technology (CET).
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 2/39 x86/shstk: Add Kconfig option for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection.
Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create CONFIG_CET to signify that at least one CET feature is configured.
---
v5:
- Remove capitalization of shadow stack (Boris)
v3:
- Add X86_CET (Kees)
- Add back WRUSS dependency (Kees)
- Fix verbiage (Dave)
- Change from promt to bool (Kirill)
- Add more to commit log
v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)
Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.
arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 29 insertions(+)
arch/x86/Kconfig +24 -0
@@ -1851,6 +1851,11 @@ config CC_HAS_IBT (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \ $(as-instr,endbr64) +config X86_CET + def_bool n + help + CET features configured (Shadow stack or IBT) + config X86_KERNEL_IBT prompt "Indirect Branch Tracking" def_bool y
@@ -1858,6 +1863,7 @@ config X86_KERNEL_IBT # https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f depends on !LD_IS_LLD || LLD_VERSION >= 140000 select OBJTOOL + select X86_CET help Build the kernel with support for Indirect Branch Tracking, a hardware support course-grain forward-edge Control Flow Integrity
@@ -1952,6 +1958,24 @@ config X86_SGX If unsure, say N. +config X86_USER_SHADOW_STACK + bool "X86 userspace shadow stack" + depends on AS_WRUSS + depends on X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_CET + help + Shadow stack protection is a hardware feature that detects function + return address corruption. This helps mitigate ROP attacks. + Applications must be enabled to use it, and old userspace does not + get protection "for free". + + CPUs supporting shadow stacks were first released in 2020. + + See Documentation/x86/shstk.rst for more information. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/Kconfig.assembler +5 -0
@@ -19,3 +19,8 @@ config AS_TPAUSE def_bool $(as-instr,tpause %ecx) help Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7 + +config AS_WRUSS + def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) + help + Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.17.1
Re: [PATCH v5 02/39] x86/shstk: Add Kconfig option for shadow stack
Quoted 9 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has
CONFIG_X86_SHADOW_STACKenabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection.
Quoted 7 lines
Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create
CONFIG_CETto signify that at least one CET feature is configured.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 3/39 x86/cpufeatures: Add CPU feature flags for shadow stacks
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).
To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.
The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches.
---
v5:
- Drop "shstk" from cpuinfo (Boris)
- Remove capitalization on shadow stack (Boris)
v3:
- Add user specific shadow stack cpu cap (Andrew Cooper)
- Drop reviewed-bys from Boris and Kees due to the above change.
v2:
- Remove IBT reference in commit log (Kees)
- Describe xsaves dependency using text from (Dave)
v1:
- Remove IBT, can be added in a follow on IBT series.
arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -307,6 +307,7 @@#define X86_FEATURE_SGX_EDECCSSA (11*32+18)/* "" SGX EDECCSSA user leaf function */ #define X86_FEATURE_CALL_DEPTH (11*32+19)/* "" Call depth tracking for RSB stuffing */ #define X86_FEATURE_MSR_TSX_CTRL (11*32+20)/* "" MSR IA32_TSX_CTRL (Intel) implemented */ +#define X86_FEATURE_USER_SHSTK (11*32+21)/* Shadow stack support for user mode applications */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
@@ -373,6 +374,7 @@#define X86_FEATURE_OSPKE (16*32+ 4)/* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5)/* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6)/* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7)/* "" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8)/* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9)/* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10)/* Carry-Less Multiplication Double Quadword */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -99,6 +99,12 @@# define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31))#endif +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define DISABLE_USER_SHSTK 0 +#else +#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -114,7 +120,7 @@#define DISABLED_MASK9 (DISABLE_SGX)#define DISABLED_MASK10 0#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ - DISABLE_CALL_DEPTH_TRACKING) + DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK)#define DISABLED_MASK12 0#define DISABLED_MASK13 0#define DISABLED_MASK14 0
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
@@ -79,6 +79,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, {} };
Signature · 1 line
2.17.1
Re: [PATCH v5 03/39] x86/cpufeatures: Add CPU feature flags for shadow stacks
Quoted 20 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).
To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make
X86_FEATURE_SHSTKdepend on XSAVES.The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches.
Quoted 3 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 4/39 x86/cpufeatures: Enable CET CR4 bit for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs.
Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.
Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.
---
v5:
- Remove #ifdeffery (Boris)
v4:
- Add back dedicated command line disable: "nousershtk" (Boris)
v3:
- Remove stay new line (Boris)
- Simplify commit log (Andrew Cooper)
v2:
- In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support.
- Clear MSR_IA32_U_CET as well. (PeterZ)
arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/common.c +27 -8
@@ -599,27 +599,43 @@ __noendbr void ibt_restore(u64 save) static __always_inline void setup_cet(struct cpuinfo_x86 *c) { -u64 msr =CET_ENDBR_EN ; +bool user_shstk, kernel_ibt; -if (!HAS_KERNEL_IBT || - !cpu_feature_enabled (X86_FEATURE_IBT )) +if (!IS_ENABLED (CONFIG_X86_CET ))return ; -wrmsrl (MSR_IA32_S_CET , msr); + kernel_ibt =HAS_KERNEL_IBT &&cpu_feature_enabled (X86_FEATURE_IBT ); + user_shstk =cpu_feature_enabled (X86_FEATURE_SHSTK ) && +IS_ENABLED (CONFIG_X86_USER_SHADOW_STACK ); + +if (!kernel_ibt && !user_shstk) +return ; + +if (user_shstk) +set_cpu_cap (c,X86_FEATURE_USER_SHSTK ); + +if (kernel_ibt) +wrmsrl (MSR_IA32_S_CET ,CET_ENDBR_EN ); +else +wrmsrl (MSR_IA32_S_CET ,0 ); +cr4_set_bits (X86_CR4_CET ); -if (!ibt_selftest ()) { +if (kernel_ibt && !ibt_selftest ()) {pr_err ("IBT selftest: Failed!\n" );wrmsrl (MSR_IA32_S_CET ,0 );setup_clear_cpu_cap (X86_FEATURE_IBT ); -return ; } } __noendbr void cet_disable(void) { -if (cpu_feature_enabled (X86_FEATURE_IBT )) -wrmsrl (MSR_IA32_S_CET ,0 ); +if (!(cpu_feature_enabled (X86_FEATURE_IBT ) || +cpu_feature_enabled (X86_FEATURE_SHSTK ))) +return ; + +wrmsrl (MSR_IA32_S_CET ,0 ); +wrmsrl (MSR_IA32_U_CET ,0 ); }/*
@@ -1476,6 +1492,9 @@ static void __init cpu_parse_early_param(void)if (cmdline_find_option_bool (boot_command_line,"noxsaves" ))setup_clear_cpu_cap (X86_FEATURE_XSAVES ); +if (cmdline_find_option_bool (boot_command_line,"nousershstk" )) +setup_clear_cpu_cap (X86_FEATURE_USER_SHSTK ); + arglen =cmdline_find_option (boot_command_line,"clearcpuid" , arg,sizeof (arg));if (arglen <=0 )return ;
Signature · 1 line
2.17.1
Re: [PATCH v5 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack
Quoted 10 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs.
Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.
Quoted 6 lines
Clear
MSR_IA32_U_CETincet_disable()so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 5/39 x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operation
The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size.
Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required.
Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable.
While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks.
Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET.
To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches.
---
v5:
- Move comments from end of lines in cet_user_state struct (Boris)
v3:
- Add missing "is" in commit log (Boris)
- Change to case statement for struct size checking (Boris)
- Adjust commas on xfeature_names (Kees, Boris)
v2:
- Change name to XFEATURE_CET_KERNEL_UNUSED (peterz)
KVM refresh:
- Reword commit log using some verbiage posted by Dave Hansen
- Remove unlikely to be used supervisor cet xsave struct
- Clarify that supervisor cet state is not saved by xsave
- Remove unused supervisor MSRs
arch/x86/include/asm/fpu/types.h | 16 +++++- arch/x86/include/asm/fpu/xstate.h | 6 ++- arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++---------------- 3 files changed, 61 insertions(+), 51 deletions(-)
arch/x86/include/asm/fpu/types.h +14 -2
@@ -115,8 +115,8 @@ enum xfeature {XFEATURE_PT_UNIMPLEMENTED_SO_FAR ,XFEATURE_PKRU ,XFEATURE_PASID , -XFEATURE_RSRVD_COMP_11 , -XFEATURE_RSRVD_COMP_12 , +XFEATURE_CET_USER , +XFEATURE_CET_KERNEL_UNUSED ,XFEATURE_RSRVD_COMP_13 ,XFEATURE_RSRVD_COMP_14 , XFEATURE_LBR,
@@ -138,6 +138,8 @@ enum xfeature {#define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL_UNUSED)#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)#define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG)#define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA)
@@ -252,6 +254,16 @@ struct pkru_state {u32 pad; } __packed; +/* +* State component 11 is Control-flow Enforcement user states +*/ +struct cet_user_state { +/* user control-flow settings */ +u64 user_cet; +/* user shadow stack pointer */ +u64 user_ssp; +}; +/* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth).
arch/x86/include/asm/fpu/xstate.h +4 -2
@@ -50,7 +50,8 @@#define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA/* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER)/* * A supervisor state component may not always contain valuable information,
@@ -77,7 +78,8 @@* Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL)/* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
arch/x86/kernel/fpu/xstate.c +43 -47
137 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
Quoted 39 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operationThe architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size.
Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required.
Adding this exacerbates the already unwieldy if statement in
check_xstate_against_struct()that handles warning about un-implemented xfeatures. So refactor these check's by havingXCHECK_SZ()set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear thatXCHECK_SZ()can change the variable.While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in
XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done forXFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks.Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET.
Quoted 7 lines
To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 6/39 x86/fpu: Add helper for modifying xstate
In reply to Rick EdgecombeJust like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.
When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.
The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.
Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.
---
v5:
- Fix spelling error (Boris)
- Don't export fpregs_lock_and_load() (Boris)
v3:
- Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees)
- Elaborate in comment about helper. (Dave)
v2:
- Drop optimization of writing directly the buffer, and change API accordingly.
- fpregs_lock_and_load() suggested by tglx
- Some commit log verbiage from dhansen
v1:
- New patch.
arch/x86/include/asm/fpu/api.h | 9 +++++++++ arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+)
arch/x86/include/asm/fpu/api.h +9 -0
@@ -82,6 +82,15 @@ static inline void fpregs_unlock(void)preempt_enable (); } +/* +* FPU state gets lazily restored before returning to userspace. So when in the +* kernel, the valid FPU state may be kept in the buffer. This function will force +* restore all the fpu state to the registers early if needed, and lock them from +* being automatically saved/restored. Then FPU state can be modified safely in the +* registers, before unlocking with fpregs_unlock(). +*/ +void fpregs_lock_and_load (void ); +#ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent (void );#else
arch/x86/kernel/fpu/core.c +18 -0
@@ -753,6 +753,24 @@ void switch_fpu_return(void) }EXPORT_SYMBOL_GPL (switch_fpu_return); +void fpregs_lock_and_load (void ) +{ +/* +* fpregs_lock() only disables preemption (mostly). So modifying state +* in an interrupt could screw up some in progress fpregs operation, +* but appear to work. Warn about it. +*/ +WARN_ON_ONCE (!irq_fpu_usable ()); +WARN_ON_ONCE (current->flags &PF_KTHREAD ); + +fpregs_lock (); + +fpregs_assert_state_consistent (); + +if (test_thread_flag (TIF_NEED_FPU_LOAD )) +fpregs_restore_userregs (); +} +#ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this
Signature · 1 line
2.17.1
Re: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate
Quoted 17 lines · Rick Edgecombe
Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.
When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.
The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.
Quoted 8 lines
Add a new function
fpregs_lock_and_load()that can simultaneously callfpregs_lock()and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
Borislav Petkov
maintainerRe: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate
In reply to Rick Edgecombearch/x86/kernel/fpu/core.c void switch_fpu_return(void) · line 761
Quoted 6 lines · Rick Edgecombe
arch/x86/kernel/fpu/core.c +6
-0+void fpregs_lock_and_load (void ) +{ +/* +* fpregs_lock() only disables preemption (mostly). So modifying state +* in an interrupt could screw up some in progress fpregs operation, +* but appear to work. Warn about it.
I don't like comments where it sounds like we don't know what we're doing. "Appear to work"?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate
Quoted 12 lines · Borislav Petkov
Quoted 8 lines · Rick Edgecombe
arch/x86/kernel/fpu/core.c +4
-0+void fpregs_lock_and_load (void ) +{ +/* +* fpregs_lock() only disables preemption (mostly). So modifying state
+ * in an interrupt could screw up some in progress fpregs
operation,
+ * but appear to work. Warn about it.I don't like comments where it sounds like we don't know what we're doing. "Appear to work"?
I can change it. This patch started with the observation that modifying xstate from the kernel had been gotten wrong a couple times in the past, so that is what this is referencing. Since then, the fancy automatic solution got boiled down to this helper and a couple warnings.
Borislav Petkov
maintainerRe: [PATCH v5 06/39] x86/fpu: Add helper for modifying xstate
Quoted 14 lines · Rick P
Quoted 12 lines · Borislav Petkov
Quoted 8 lines · Rick Edgecombe
arch/x86/kernel/fpu/core.c +4
-0+void fpregs_lock_and_load (void ) +{ +/* +* fpregs_lock() only disables preemption (mostly). So modifying state
+ * in an interrupt could screw up some in progress fpregs
operation,
+ * but appear to work. Warn about it.I don't like comments where it sounds like we don't know what we're doing. "Appear to work"?
Quoted 5 lines
I can change it. This patch started with the observation that modifying xstate from the kernel had been gotten wrong a couple times in the past, so that is what this is referencing. Since then, the fancy automatic solution got boiled down to this helper and a couple warnings.
Yeah, but that comment right now reads like: modifying in interrupt context can corrupt fpregs and you should not do it but it kinda works, by chance. Thus encouraging people to keep doing that.
I guess "but appear to work" can go and then it is fine.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v5 7/39 x86: Add user control-protection fault handler
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.
There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file cet.c, which is compiled in the case of either of the two CET features supported in the
kernel: kernel IBT or user mode shadow stack. Move some static inline functions from traps.c into a header so they can be used in cet.c.
Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it.
Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable.
The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler.
---
v5:
- Move to separate file to advoid ifdeffery (Boris)
- Improvements to commit log (Boris)
- Rename control_protection_err (Boris)
- Move comment from end of line in IBT fault handler (Boris)
v3:
- Shorten user/kernel #CP handler function names (peterz)
- Restore CP_ENDBR check to kernel handler (peterz)
- Utilize CONFIG_X86_CET (Kees)
- Unify "unexpected" warnings (Andrew Cooper)
- Use 2d array for error code chars (Andrew Cooper)
- Add comment about why to read SSP MSR before enabling interrupts
v2:
- Integrate with kernel IBT fault handler
- Update printed messages. (Dave)
- Remove array_index_nospec() usage. (Dave)
- Remove IBT messages. (Dave)
- Add enclave error code bit processing it case it can get triggered somehow.
- Add extra "unknown" in control_protection_err.
v1:
- Update static asserts for NSIGSEGV
arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/traps.h | 12 ++ arch/x86/kernel/Makefile | 2 + arch/x86/kernel/cet.c | 152 +++++++++++++++++++++++ arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 2 +- arch/x86/kernel/traps.c | 87 ------------- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- include/uapi/asm-generic/siginfo.h | 3 +- 17 files changed, 186 insertions(+), 100 deletions(-) create mode 100644 arch/x86/kernel/cet.c
arch/arm/kernel/signal.c +1 -1
@@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/arm64/kernel/signal.c +1 -1
@@ -1176,7 +1176,7 @@ void __init minsigstksz_setup(void)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/arm64/kernel/signal32.c +1 -1
@@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/sparc/kernel/signal32.c +1 -1
@@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/sparc/kernel/signal_64.c +1 -1
@@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/include/asm/disabled-features.h +7 -1
@@ -105,6 +105,12 @@#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31))#endif +#ifdef CONFIG_X86_KERNEL_IBT +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -128,7 +134,7 @@#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ DISABLE_ENQCMD)#define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT)#define DISABLED_MASK19 0#define DISABLED_MASK20 0#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 21)
arch/x86/include/asm/idtentry.h +1 -1
@@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF, xenpv_exc_double_fault);#endif /* #CP */ -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_CP , exc_control_protection);#endif
arch/x86/include/asm/traps.h +12 -0
@@ -47,4 +47,16 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, struct stack_info *info);#endif +static inline void cond_local_irq_enable (struct pt_regs *regs) +{ +if (regs->flags &X86_EFLAGS_IF ) +local_irq_enable (); +} + +static inline void cond_local_irq_disable (struct pt_regs *regs) +{ +if (regs->flags &X86_EFLAGS_IF ) +local_irq_disable (); +} +#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/Makefile +2 -0
@@ -144,6 +144,8 @@ obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_CALL_THUNKS) += callthunks.o +obj-$(CONFIG_X86_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/cet.c new file +152 -0
152 lines not on this page · raw message
arch/x86/kernel/idt.c +1 -1
@@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = { ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),#endif -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET INTG(X86_TRAP_CP, asm_exc_control_protection),#endif
arch/x86/kernel/signal_32.c +1 -1
@@ -402,7 +402,7 @@ int ia32_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/kernel/signal_64.c +1 -1
@@ -403,7 +403,7 @@ void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/kernel/traps.c +0 -87
99 lines not on this page · raw message
arch/x86/xen/enlighten_pv.c +1 -1
@@ -640,7 +640,7 @@ static struct trap_array_entry trap_array[] = {TRAP_ENTRY (exc_coprocessor_error, false ),TRAP_ENTRY (exc_alignment_check, false ),TRAP_ENTRY (exc_simd_coprocessor_error, false ), -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET TRAP_ENTRY(exc_control_protection, false ), #endif };
arch/x86/xen/xen-asm.S +1 -1
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET xen_pv_trap asm_exc_control_protection #endif #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +2 -1
@@ -242,7 +242,8 @@ typedef struct siginfo {#define SEGV_ADIPERR 7/* Precise MCD exception */ #define SEGV_MTEAERR 8/* Asynchronous ARM MTE error */ #define SEGV_MTESERR 9/* Synchronous ARM MTE exception */ -#define NSIGSEGV 9 +#define SEGV_CPERR 10/* Control protection fault */ +#define NSIGSEGV 10/* * SIGBUS si_codes
Signature · 1 line
2.17.1
Re: [PATCH v5 07/39] x86: Add user control-protection fault handler
Quoted 24 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.
There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file
cet.c, which is compiled in the case of either of the two CET features supported in the
kernel: kernel IBT or user mode shadow stack. Move some static inline functions fromtraps.cinto a header so they can be used incet.c.Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it.
Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable.
Quoted 7 lines
The control-protection fault handler works in a similar way as the general protection fault handler. It provides the
si_codeSEGV_CPERRto the signal handler.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
This diff would have been a bit easier to review if the file move was separate from the addition of the handler, but regardless:
Signature · 1 line
Kees Cook
Borislav Petkov
maintainerRe: [PATCH v5 07/39] x86: Add user control-protection fault handler
In reply to Rick EdgecombeQuoted 1 line · Rick Edgecombe
Subject: Re: [PATCH v5 07/39] x86: Add user control-protection fault handler
Subject: x86/shstk: Add...
Quoted 6 lines
From: Yu-cheng Yu <[email protected]>
A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.
...
Quoted 6 lines
arch/x86/kernel/cet.c new file +0
-0
arch/x86/kernel/cet.c line 8
Quoted 8 lines
arch/x86/kernel/cet.c new file +8
-0@@ -0,0 +1,152 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/ptrace.h> +#include <asm/bugs.h> +#include <asm/traps.h> + +enum cp_error_code { +CP_EC = (1 <<15 ) -1 ,
That looks like a mask, so
CP_EC_MASK
I guess.
Quoted 9 lines
arch/x86/kernel/cet.c new file +9
-0+ +CP_RET =1 , +CP_IRET =2 , +CP_ENDBR =3 , +CP_RSTRORSSP =4 , +CP_SETSSBSY =5 , + + CP_ENCL = 1 << 15, +};
...
Quoted 6 lines
arch/x86/kernel/cet.c new file +6
-0+static void do_user_cp_fault (struct pt_regs *regs,unsigned long error_code) +{ +struct task_struct *tsk; +unsigned long ssp; + +/*
arch/x86/kernel/cet.c line 62
Quoted 10 lines
arch/x86/kernel/cet.c new file +10
-0+* An exception was just taken from userspace. Since interrupts are disabled +* here, no scheduling should have messed with the registers yet and they +* will be whatever is live in userspace. So read the SSP before enabling +* interrupts so locking the fpregs to do it later is not required. +*/ +rdmsrl (MSR_IA32_PL3_SSP , ssp); + +cond_local_irq_enable (regs); + + tsk = current;
Hmm, should you read current before you enable interrupts? Not that it changes from under us...
Quoted 18 lines
arch/x86/kernel/cet.c new file +18
-0+ tsk->thread.error_code = error_code; + tsk->thread.trap_nr =X86_TRAP_CP ; + +/* Ratelimit to prevent log spamming. */ +if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && +__ratelimit (&cpf_rate)) { +pr_emerg ("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s" , + tsk->comm,task_pid_nr (tsk), + regs->ip, regs->sp, ssp, error_code, +cp_err_string (error_code), + error_code &CP_ENCL ?" in enclave" :"" ); +print_vma_addr (KERN_CONT " in " , regs->ip); +pr_cont ("\n" ); + } + +force_sig_fault (SIGSEGV ,SEGV_CPERR , (void __user *)0 ); +cond_local_irq_disable (regs); +}
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 07/39] x86: Add user control-protection fault handler
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Rick Edgecombe
Subject: Re: [PATCH v5 07/39] x86: Add user control-protection fault handler
Subject: x86/shstk: Add...
Sure.
Quoted 30 lines
Quoted 10 lines
From: Yu-cheng Yu <[email protected]>
A control-protection fault is triggered when a control-flow transfer
attempt violates Shadow Stack or Indirect Branch Tracking constraints.
For example, the return address for a RET instruction differs from the copy
on the shadow stack....
Quoted 6 lines
arch/x86/kernel/cet.c new file +0
-0arch/x86/kernel/cet.c line 8
Quoted 8 lines
arch/x86/kernel/cet.c new file +8
-0@@ -0,0 +1,152 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/ptrace.h> +#include <asm/bugs.h> +#include <asm/traps.h> + +enum cp_error_code { +CP_EC = (1 <<15 ) -1 ,That looks like a mask, so
Quoted 3 lines
CP_EC_MASKI guess.
The name seems better, but this is actually from the existing kernel IBT control protection exception code. So it seems like an separate change. Would you like to see it snuck into the user shadow stack handler, or could we leave this for future cleanups?
Kees pointed out that adding to the handler and moving it in the same patch makes it difficult to see where the changes are. I'm splitting this one into two patches for the next version.
Quoted 30 lines
Quoted 10 lines
arch/x86/kernel/cet.c new file +9
-0+ +CP_RET =1 , +CP_IRET =2 , +CP_ENDBR =3 , +CP_RSTRORSSP =4 , +CP_SETSSBSY =5 , + + CP_ENCL = 1 << 15, +};...
Quoted 17 lines
+static void
do_user_cp_fault(struct pt_regs *regs, unsigned long error_code)arch/x86/kernel/cet.c new file +6
-0+{ +struct task_struct *tsk; +unsigned long ssp; + +/* +* An exception was just taken from userspace. Since interrupts are disabled
arch/x86/kernel/cet.c new file +1
-0+* here, no scheduling should have messed with the registers yet and they
arch/x86/kernel/cet.c new file +1
-0+* will be whatever is live in userspace. So read the SSP before enabling
arch/x86/kernel/cet.c new file +1
-0+* interrupts so locking the fpregs to do it later is not required.
arch/x86/kernel/cet.c new file +2
-0+*/ +rdmsrl (MSR_IA32_PL3_SSP , ssp);
Quoted 8 lines
arch/x86/kernel/cet.c line 62
Quoted 4 lines
arch/x86/kernel/cet.c new file +4
-0+ +cond_local_irq_enable (regs); + + tsk = current;Hmm, should you read current before you enable interrupts? Not that it
changes from under us...
I think we have to read it before we enable interrupts or use fpregs_lock(). So reading it before saves disabling preemption later.
Quoted 22 lines
Quoted 20 lines
arch/x86/kernel/cet.c new file +7
-0+ tsk->thread.error_code = error_code; + tsk->thread.trap_nr =X86_TRAP_CP ; + +/* Ratelimit to prevent log spamming. */ +if (show_unhandled_signals &&unhandled_signal (tsk,SIGSEGV ) && +__ratelimit (&cpf_rate)) { + pr_emerg("%s[%d] control protection ip:%lx sp:%lxssp:%lx error:%lx(%s)%s",
arch/x86/kernel/cet.c new file +11
-0+ tsk->comm,task_pid_nr (tsk), + regs->ip, regs->sp, ssp, error_code, +cp_err_string (error_code), + error_code & CP_ENCL ? " in enclave" : ""); +print_vma_addr (KERN_CONT " in " , regs->ip); +pr_cont ("\n" ); + } + +force_sig_fault (SIGSEGV ,SEGV_CPERR , (void __user *)0 ); +cond_local_irq_disable (regs); +}
Borislav Petkov
maintainerRe: [PATCH v5 07/39] x86: Add user control-protection fault handler
Quoted 8 lines · Rick P
The name seems better, but this is actually from the existing kernel IBT control protection exception code. So it seems like an separate change. Would you like to see it snuck into the user shadow stack handler, or could we leave this for future cleanups?
Kees pointed out that adding to the handler and moving it in the same patch makes it difficult to see where the changes are. I'm splitting this one into two patches for the next version.
Yap, that's the right way to do it.
Quoted 2 lines
I think we have to read it before we enable interrupts or use
fpregs_lock(). So reading it before saves disabling preemption later.
So I'm a bit confused - there's that cond_local_irq_enable() which will enable interrupts if they were enabled before.
So if they were enabled before and you reenable them here, then that current could be the wrong one if we schedule in between, right?
IOW, shouldn't those two lines be swapped so that it says:
tsk = current;
cond_local_irq_enable(regs);and you can be sure that tsk is always the right current which caused the #CP? Or am I way off again?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 07/39] x86: Add user control-protection fault handler
Quoted 11 lines · Borislav Petkov
Quoted 3 lines
I think we have to read it before we enable interrupts or use
fpregs_lock(). So reading it before saves disabling preemption later.So I'm a bit confused - there's that
cond_local_irq_enable()which will
enable interrupts if they were enabled before.So if they were enabled before and you reenable them here, then that current could be the wrong one if we schedule in between, right?
Quoted 8 lines
IOW, shouldn't those two lines be swapped so that it says:
tsk = current; cond_local_irq_enable(regs);and you can be sure that tsk is always the right current which caused the #CP? Or am I way off again?
Since this path is only for exceptions coming from userspace, I think it should be valid either way. It can't be during a task switch. I can swap the lines if it looks odd, but unless I'm wrong about the 'current' validity I think it's negligibly better as is because it is preemptible for as long as possible.
Borislav Petkov
maintainerRe: [PATCH v5 07/39] x86: Add user control-protection fault handler
Quoted 5 lines · Rick P
Since this path is only for exceptions coming from userspace, I think it should be valid either way. It can't be during a task switch. I can swap the lines if it looks odd, but unless I'm wrong about the 'current' validity I think it's negligibly better as is because it is preemptible for as long as possible.
Nah, all good. I was confused here. Sorry for the noise.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v5 8/39 x86/mm: Remove _PAGE_DIRTY from kernel RO pages
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages.
In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory.
When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs.
---
v5:
- Spelling and grammar in commit log (Boris)
v3:
- Update commit log (Andrew Cooper, Peterz)
v2:
- Normalize PTE bit descriptions between patches
arch/x86/include/asm/pgtable_types.h | 6 +++--- arch/x86/mm/pat/set_memory.c | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
@@ -192,10 +192,10 @@ enum page_cache_mode {#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC)#define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0)#define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) -#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0|___D| 0|___G) +#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) +#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) -#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_LARGE (__PP|__RW| 0|___A|__NX|___D|_PSE|___G)#define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW| 0|___A| 0|___D|_PSE|___G)#define __PAGE_KERNEL_WP (__PP|__RW| 0|___A|__NX|___D| 0|___G| __WP)
arch/x86/mm/pat/set_memory.c +1 -1
@@ -2073,7 +2073,7 @@ int set_memory_nx(unsigned long addr, int numpages)int set_memory_ro (unsigned long addr,int numpages) { -return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW),0 ); +return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW | _PAGE_DIRTY),0 ); }int set_memory_rox (unsigned long addr,int numpages)
Signature · 1 line
2.17.1
Re: [PATCH v5 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages
Quoted 19 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages.
In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory.
When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs.
Quoted 3 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 9/39 x86/mm: Move pmd_write(), pud_write() up in the file
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
To prepare the introduction of _PAGE_COW, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes.
--- arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
arch/x86/include/asm/pgtable.h +12 -12
@@ -160,6 +160,18 @@ static inline int pte_write(pte_t pte)return pte_flags (pte) & _PAGE_RW; } +#define pmd_write pmd_write +static inline int pmd_write (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_RW; +} + +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pud_flags (pud) & _PAGE_RW; +} +static inline int pte_huge (pte_t pte) {return pte_flags (pte) & _PAGE_PSE;
@@ -1120,12 +1132,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); -#define pmd_write pmd_write -static inline int pmd_write (pmd_t pmd) -{ -return pmd_flags (pmd) & _PAGE_RW; -} -#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEARstatic inline pmd_t pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)
@@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); } -#define pud_write pud_write -static inline int pud_write (pud_t pud) -{ -return pud_flags (pud) & _PAGE_RW; -} -#ifndef pmdp_establish#define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
Signature · 1 line
2.17.1
PATCH v5 10/39 x86/mm: Introduce _PAGE_COW
In reply to Rick EdgecombeSome OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:
(Write=0,Cow=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW via fork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This happens in copy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.
(Write=0,Cow=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support set Dirty=1.
There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.
Implement only the infrastructure for _PAGE_COW. Changes to start creating _PAGE_COW PTEs will follow once other pieces are in place.
---
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)
v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability
v3:
- Add comment around _PAGE_TABLE in response to comment from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)
v2:
- Update commit log with comments (Dave Hansen)
- Add comments in code to explain pte modification code better (Dave)
- Clarify info on the meaning of various Write,Cow,Dirty combinations
arch/x86/include/asm/pgtable.h | 78 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 134 insertions(+), 6 deletions(-)
arch/x86/include/asm/pgtable.h +78 -0
@@ -301,6 +301,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +/* +* Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case +* of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the +* Dirty=1,Write=0 will result in the memory being treated as shadow stack +* by the HW. So when creating COW memory, a software bit is used +* _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow() +* take a PTE marked conventionally COW (Dirty=1) and transition it to the +* shadow stack compatible version of COW (Cow=1). +*/ +static inline pte_t pte_mkcow (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_COW); +} + +static inline pte_t pte_clear_cow (pte_t pte) +{ +/* +* _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK kernels, since +* the HW dirty bit can be used without creating shadow stack memory. +* See the _PAGE_COW definition for more details. +*/ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + +/* +* PTE is getting copied-on-write, so it will be dirtied +* if writable, or made shadow stack if shadow stack and +* being copied on access. Set the dirty bit for both +* cases. +*/ + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {
@@ -413,6 +451,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_mkcow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_clear_cow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd) {
@@ -484,6 +542,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pud_t pud_mkcow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pud_t pud_clear_cow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_COW); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);
arch/x86/include/asm/pgtable_types.h +54 -5
85 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +2 -1
@@ -273,7 +273,8 @@ static inline bool pte_flags_need_flush(unsigned long oldflags,const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED;const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | - _PAGE_SOFTW3 | _PAGE_SOFTW4; + _PAGE_SOFTW3 | _PAGE_SOFTW4 | + _PAGE_COW;const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT | _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT | _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 |
Signature · 1 line
2.17.1
Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW
Quoted 46 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined
_PAGE_COWin place of the hardware_PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:(Write=0,Cow=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW viafork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This happens incopy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, andget_user_pages(FOLL_FORCE)needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens atfork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.(Write=0,Cow=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support set Dirty=1.There are six bits left available to software in the 64-bit PTE after consuming a bit for
_PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.
Quoted 7 lines
Implement only the infrastructure for
_PAGE_COW. Changes to start creating_PAGE_COWPTEs will follow once other pieces are in place.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW
In reply to Rick EdgecombeQuoted 219 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined
_PAGE_COWin place of the hardware_PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:(Write=0,Cow=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW viafork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This happens incopy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, andget_user_pages(FOLL_FORCE)needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens atfork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.(Write=0,Cow=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support setDirty=1.There are six bits left available to software in the 64-bit PTE after consuming a bit for
_PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.Implement only the infrastructure for
_PAGE_COW. Changes to start creating_PAGE_COWPTEs will follow once other pieces are in place.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)v4:
- Teachpte_flags_need_flush()about_PAGE_COWbit
- Break apart patch for better bisectabilityv3:
- Add comment around_PAGE_TABLEin response to comment from (Andrew Cooper)
- Check for PSE inpmd_shstk(Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)v2:
- Update commit log with comments (Dave Hansen)
- Add comments in code to explain pte modification code better (Dave)
- Clarify info on the meaning of various Write,Cow,Dirty combinationsarch/x86/include/asm/pgtable.h | 78 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 134 insertions(+), 6 deletions(-)arch/x86/include/asm/pgtable.h +78
-0@@ -301,6 +301,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +/* +* Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case +* of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the +* Dirty=1,Write=0 will result in the memory being treated as shadow stack +* by the HW. So when creating COW memory, a software bit is used +* _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow() +* take a PTE marked conventionally COW (Dirty=1) and transition it to the +* shadow stack compatible version of COW (Cow=1). +*/ +static inline pte_t pte_mkcow (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_COW); +} + +static inline pte_t pte_clear_cow (pte_t pte) +{ +/* +* _PAGE_COW is unnecessary on !X86_FEATURE_USER_SHSTK kernels, since +* the HW dirty bit can be used without creating shadow stack memory. +* See the _PAGE_COW definition for more details. +*/ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + +/* +* PTE is getting copied-on-write, so it will be dirtied +* if writable, or made shadow stack if shadow stack and +* being copied on access. Set the dirty bit for both +* cases. +*/ + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {@@ -413,6 +451,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_mkcow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pmd_t pmd_clear_cow (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_COW); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd) {@@ -484,6 +542,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +/* See comments above pte_mkcow() */ +static inline pud_t pud_mkcow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_COW); +} + +/* See comments above pte_mkcow() */ +static inline pud_t pud_clear_cow (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_COW); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);arch/x86/include/asm/pgtable_types.h +11
-1@@ -21,7 +21,8 @@#define _PAGE_BIT_SOFTW2 10/* " */ #define _PAGE_BIT_SOFTW3 11/* " */ #define _PAGE_BIT_PAT_LARGE 12/* On 2MB or 1GB pages */ -#define _PAGE_BIT_SOFTW4 58/* available for programmer */ +#define _PAGE_BIT_SOFTW4 57/* available for programmer */ +#define _PAGE_BIT_SOFTW5 58/* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59/* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60/* Protection Keys, bit 2/4 */ #define _PAGE_BIT_PKEY_BIT2 61/* Protection Keys, bit 3/4 */ @@ -34,6 +35,15 @@#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3/* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +/* +* Indicates a copy-on-write page. +*/ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_BIT_COW _PAGE_BIT_SOFTW5/* copy-on-write */ +#else +#define _PAGE_BIT_COW 0 +#endif +/* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ #define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
arch/x86/include/asm/pgtable_types.h line 133
Quoted 7 lines
arch/x86/include/asm/pgtable_types.h +4
-0@@ -117,6 +127,40 @@#define _PAGE_SOFTW4 (_AT(pteval_t, 0))#endif +/* +* The hardware requires shadow stack to be read-only and Dirty. +* _PAGE_COW is a software-only bit used to separate copy-on-write PTEs +* from shadow stack PTEs:
Is that really required?
For anon pages, we have PG_anon_exclusive, that can tell you whether the page is "certainly exclusive" (now cow necessary) vs. "maybe shared" (cow maybe necessary).
Why isn't that sufficient to make the same decisions here?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW
In reply to Rick EdgecombeQuoted 92 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined
_PAGE_COWin place of the hardware_PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:(Write=0,Cow=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW viafork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This happens incopy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, andget_user_pages(FOLL_FORCE)needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens atfork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.(Write=0,Cow=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support setDirty=1.There are six bits left available to software in the 64-bit PTE after consuming a bit for
_PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.Implement only the infrastructure for
_PAGE_COW. Changes to start creating_PAGE_COWPTEs will follow once other pieces are in place.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)v4:
- Teachpte_flags_need_flush()about_PAGE_COWbit
- Break apart patch for better bisectabilityv3:
- Add comment around_PAGE_TABLEin response to comment from (Andrew Cooper)
- Check for PSE inpmd_shstk(Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)v2:
- Update commit log with comments (Dave Hansen)
- Add comments in code to explain pte modification code better (Dave)
- Clarify info on the meaning of various Write,Cow,Dirty combinationsarch/x86/include/asm/pgtable.h | 78 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 59 +++++++++++++++++++-- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 134 insertions(+), 6 deletions(-)arch/x86/include/asm/pgtable.h +0
-0@@ -301,6 +301,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); }
arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 312
Quoted 10 lines
arch/x86/include/asm/pgtable.h +9
-0+/* +* Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case +* of X86_FEATURE_USER_SHSTK, the software COW bit is used, since the +* Dirty=1,Write=0 will result in the memory being treated as shadow stack +* by the HW. So when creating COW memory, a software bit is used +* _PAGE_BIT_COW. The following functions pte_mkcow() and pte_clear_cow() +* take a PTE marked conventionally COW (Dirty=1) and transition it to the +* shadow stack compatible version of COW (Cow=1). +*/
TBH, I find that all highly confusing.
Dirty=1,Write=0 does not indicate a COW page reliably. You could have both, false negatives and false positives.
False negative: fork() on a clean anon page.
False positives: wrpotect() of a dirty anon page.
I wonder if it really has to be that complicated: what you really want to achieve is to disallow "Dirty=1,Write=0" if it's not a shadow stack page, correct?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW
Trying to answer both questions to this patch on this one.
Quoted 23 lines · David Hildenbrand
Quoted 6 lines
+/*
+ * Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case
+ * ofX86_FEATURE_USER_SHSTK, the software COW bit is used, since the
+ * Dirty=1,Write=0 will result in the memory being treated asQuoted 8 lines
shadow stack
+ * by the HW. So when creating COW memory, a software bit is used
+ *_PAGE_BIT_COW. The following functionspte_mkcow()andpte_clear_cow()
+ * take a PTE marked conventionally COW (Dirty=1) and transition it to the
+ * shadow stack compatible version of COW (Cow=1).
+ */TBH, I find that all highly confusing.
Dirty=1,Write=0 does not indicate a COW page reliably. You could have
both, false negatives and false positives.False negative:
fork()on a clean anon page.
Quoted 8 lines
False positives:
wrpotect()of a dirty anon page.I wonder if it really has to be that complicated: what you really want
to achieve is to disallow "Dirty=1,Write=0" if it's not a shadow stack
page, correct?
The other thing is to save that the PTE is/was Dirty=1 somewhere (for non-shadow stack memory). A slightly different but related thing. But losing that information would would introduce differences for pte_dirty() between when shadow stack was enabled or not. GUP/COW doesn't need this anymore but there are lots of other places it gets checked.
Perhaps following your GUP changes, _PAGE_COW is just now the wrong name for it. _PAGE_SAVED_DIRTY maybe?
Re: [PATCH v5 10/39] x86/mm: Introduce _PAGE_COW
Quoted 42 lines · Rick P
Trying to answer both questions to this patch on this one.
Quoted 23 lines · David Hildenbrand
Quoted 6 lines
+/*
+ * Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the case
+ * ofX86_FEATURE_USER_SHSTK, the software COW bit is used, since the
+ * Dirty=1,Write=0 will result in the memory being treated asQuoted 8 lines
shadow stack
+ * by the HW. So when creating COW memory, a software bit is used
+ *_PAGE_BIT_COW. The following functionspte_mkcow()andpte_clear_cow()
+ * take a PTE marked conventionally COW (Dirty=1) and transition it to the
+ * shadow stack compatible version of COW (Cow=1).
+ */TBH, I find that all highly confusing.
Dirty=1,Write=0 does not indicate a COW page reliably. You could have
both, false negatives and false positives.False negative:
fork()on a clean anon page.Quoted 8 lines
False positives:
wrpotect()of a dirty anon page.I wonder if it really has to be that complicated: what you really want
to achieve is to disallow "Dirty=1,Write=0" if it's not a shadow stack
page, correct?The other thing is to save that the PTE is/was Dirty=1 somewhere (for non-shadow stack memory). A slightly different but related thing. But losing that information would would introduce differences for
pte_dirty()between when shadow stack was enabled or not. GUP/COW doesn't need this anymore but there are lots of other places it gets checked.
Quoted 2 lines
Perhaps following your GUP changes,
_PAGE_COWis just now the wrong name for it._PAGE_SAVED_DIRTYmaybe?
It goes into the direction of my other proposal/idea, yes. Not sure if _PAGE_SAVED_DIRTY would currently mimic what's happening here ... _PAGE_COW is certainly wrong and misleading.
Signature · 3 lines
Thanks, David / dhildenb
PATCH v5 11/39 x86/mm: Update pte_modify for _PAGE_COW
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this.
pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one.
However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Do this by using the pte_mkdirty() helper. Since pte_mkdirty() also sets the soft dirty bit, extract a helper that optionally doesn't set _PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move _PAGE_DIRTY to _PAGE_COW can live in one place.
Apply the same changes to pmd_modify().
---
v5:
- Fix pte_modify() again, to not lose _PAGE_DIRTY, but still not set _PAGE_SOFT_DIRTY as was fixed in v4.
v4:
- Fix an issue in soft-dirty test, where pte_modify() would detect _PAGE_COW in pte_dirty() and set the soft dirty bit in pte_mkdirty().
v2:
- Update commit log with text and suggestions from (Dave Hansen)
- Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen)
arch/x86/include/asm/pgtable.h | 64 +++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-)
arch/x86/include/asm/pgtable.h +56 -8
@@ -392,9 +392,19 @@ static inline pte_t pte_mkexec(pte_t pte)return pte_clear_flags (pte, _PAGE_NX); } +static inline pte_t __pte_mkdirty (pte_t pte,bool soft) +{ +pteval_t dirty = _PAGE_DIRTY; + +if (soft) + dirty |= _PAGE_SOFT_DIRTY; + +return pte_set_flags (pte, dirty); +} +static inline pte_t pte_mkdirty (pte_t pte) { -return pte_set_flags (pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +return __pte_mkdirty (pte, true); }static inline pte_t pte_mkyoung (pte_t pte)
@@ -503,9 +513,19 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)return pmd_clear_flags (pmd, _PAGE_RW); } +static inline pmd_t __pmd_mkdirty (pmd_t pmd,bool soft) +{ +pmdval_t dirty = _PAGE_DIRTY; + +if (soft) + dirty |= _PAGE_SOFT_DIRTY; + +return pmd_set_flags (pmd, dirty); +} +static inline pmd_t pmd_mkdirty (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +return __pmd_mkdirty (pmd, true); }static inline pmd_t pmd_mkdevmap (pmd_t pmd)
@@ -715,26 +735,54 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask);static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) { +pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY;pteval_t val =pte_val (pte), oldval = val; +pte_t pte_result;/* * Chop off the NX bit (if present), and add the NX portion of * the newprot (if present): */ - val &= _PAGE_CHG_MASK; - val |=check_pgprot (newprot) & ~_PAGE_CHG_MASK; + val &= _page_chg_mask_no_dirty; + val |=check_pgprot (newprot) & ~_page_chg_mask_no_dirty; val =flip_protnone_guard (oldval, val,PTE_PFN_MASK ); -return __pte (val); + + pte_result =__pte (val); + +/* +* Dirty bit is not preserved above so it can be done +* in a special way for the shadow stack case, where it +* may need to set _PAGE_COW. __pte_mkdirty() will do this in +* the case of shadow stack. +*/ +if (pte_dirty (pte)) + pte_result =__pte_mkdirty (pte_result, false); + +return pte_result; }static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) { +pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY;pmdval_t val =pmd_val (pmd), oldval = val; +pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; - val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; + val &= _hpage_chg_mask_no_dirty; + val |=check_pgprot (newprot) & ~_hpage_chg_mask_no_dirty; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK ); -return __pmd (val); + + pmd_result =__pmd (val); + +/* +* Dirty bit is not preserved above so it can be done +* in a special way for the shadow stack case, where it +* may need to set _PAGE_COW. __pmd_mkdirty() will do this in +* the case of shadow stack. +*/ +if (pmd_dirty (pmd)) + pmd_result =__pmd_mkdirty (pmd_result, false); + +return pmd_result; }/*
Signature · 1 line
2.17.1
Re: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 20 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined
_PAGE_DIRTYis changed to_PAGE_COWfor the copy-on-write case, and pte_*() are updated to do this.
pte_modify()takes a "raw"pgprot_twhich was not necessarily created with any of the existing PTE bit helpers. That means that it can return apte_twith Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one.However
pte_modify()changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move_PAGE_DIRTYto_PAGE_COW. Do this by using thepte_mkdirty()helper. Sincepte_mkdirty()also sets the soft dirty bit, extract a helper that optionally doesn't set_PAGE_SOFT_DIRTY. This helper will allow future logic for deciding when to move_PAGE_DIRTYto_PAGE_COWcan live in one place.
Quoted 5 lines
Apply the same changes to
pmd_modify().Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
Borislav Petkov
maintainerRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
In reply to Rick EdgecombeQuoted 7 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined
_PAGE_DIRTYis changed to_PAGE_COWfor the copy-on-write case, and pte_*() are updated to do this.
"In order to separate the two, change the software-defined ..."
From section "2) Describe your changes" inDocumentation/process/submitting-patches.rst:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
arch/x86/include/asm/pgtable.h static inline pte_t pte_mkexec(pte_t pte) · line 403
Quoted 9 lines
arch/x86/include/asm/pgtable.h +9
-0+static inline pte_t __pte_mkdirty (pte_t pte,bool soft) +{ +pteval_t dirty = _PAGE_DIRTY; + +if (soft) + dirty |= _PAGE_SOFT_DIRTY; + +return pte_set_flags (pte, dirty); +}
Dunno, do you even need that __pte_mkdirty() helper?
AFAIU, pte_mkdirty() will always set _PAGE_SOFT_DIRTY too so whatever the __pte_mkdirty() thing needs to do, you can simply do it by foot in the two callsites.
And this way you won't have the confusion: should I use pte_mkdirty() or __pte_mkdirty()?
Ditto for the pmd variants.
Otherwise, this is starting to make more sense now.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 14 lines · Borislav Petkov
Quoted 10 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages.
However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a
shadow stack page. In order to separate the two, the software- defined_PAGE_DIRTYis changed to_PAGE_COWfor the copy-on-write case, and pte_*() are updated to do this."In order to separate the two, change the software-defined ..."
Quoted 7 lines
From section "2) Describe your changes" in
Documentation/process/submitting-patches.rst:"Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
Yea, this is ambiguous. It's actually trying to say that "the software- defined..." *were* changed in previous patches. I'll change it to make that clear.
Quoted 18 lines
arch/x86/include/asm/pgtable.h static inline pte_t pte_mkexec(pte_t pte) · line 403
Quoted 10 lines
arch/x86/include/asm/pgtable.h +9
-0+static inline pte_t __pte_mkdirty (pte_t pte,bool soft) +{ +pteval_t dirty = _PAGE_DIRTY; + +if (soft) + dirty |= _PAGE_SOFT_DIRTY; + +return pte_set_flags (pte, dirty); +}Dunno, do you even need that
__pte_mkdirty()helper?AFAIU,
pte_mkdirty()will always set_PAGE_SOFT_DIRTYtoo so whatever the__pte_mkdirty()thing needs to do, you can simply do it by foot in
the two callsites.
Quoted 7 lines
And this way you won't have the confusion: should I use
pte_mkdirty()or__pte_mkdirty()?Ditto for the pmd variants.
Otherwise, this is starting to make more sense now.
The thing is it would need to duplicate the pte_write() and shadow stack enablement check and know when to set the Cow(soon to be SavedDirty) bit.
I see that having a similar helper is not ideal, but isn't it nice that this special critical logic for setting the Cow bit is all in one place? I actually tried it the other way, but thought that it was nicer to have a helper that might drive future people to not miss the Cow bit part.
What do you think, can we leave it or give it a new name? Maybe pte_set_dirty() to be more like the x86-only pte_set_flags() family of functions? Then we have:
static inline pte_t pte_mkdirty(pte_t pte)
{
pte = pte_set_flags(pte, _PAGE_SOFT_DIRTY);
return pte_set_dirty(pte);
}And...
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
...
/*
* Dirty bit is not preserved above so it can be done
* in a special way for the shadow stack case, where it
* may need to set _PAGE_SAVED_DIRTY. __pte_mkdirty() will do
* this in the case of shadow stack.
*/
if (oldval & _PAGE_DIRTY)
pte_result = pte_set_dirty(pte_result);
return pte_result;
}
Borislav Petkov
maintainerRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 3 lines · Rick P
What do you think, can we leave it or give it a new name? Maybe
pte_set_dirty()to be more like the x86-onlypte_set_flags()family of functions?
I'd do this (ontop of yours, not built, not tested, etc). It is short and sweet:
pte_mkdirty() set both dirty flagspte_modifl() sets only _PAGE_DIRTY
No special helpers to lookup what they do, no nothing. Plain and simple.
---
arch/x86/include/asm/pgtable.h +3 -18
@@ -392,19 +392,10 @@ static inline pte_t pte_mkexec(pte_t pte)return pte_clear_flags (pte, _PAGE_NX); } -static inline pte_t __pte_mkdirty (pte_t pte,bool soft) -{ -pteval_t dirty = _PAGE_DIRTY; - -if (soft) - dirty |= _PAGE_SOFT_DIRTY; - -return pte_set_flags (pte, dirty); -} - +/* Set _PAGE_SOFT_DIRTY for shadow stack pages. */ static inline pte_t pte_mkdirty (pte_t pte) { -return __pte_mkdirty (pte, true); +return pte_set_flags (pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); }static inline pte_t pte_mkyoung (pte_t pte)
@@ -749,14 +740,8 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) pte_result =__pte (val); -/* -* Dirty bit is not preserved above so it can be done -* in a special way for the shadow stack case, where it -* may need to set _PAGE_COW. __pte_mkdirty() will do this in -* the case of shadow stack. -*/ if (pte_dirty (pte)) - pte_result =__pte_mkdirty (pte_result, false); + pte_result =pte_set_flags (pte_result, _PAGE_DIRTY);return pte_result; }
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 16 lines · Borislav Petkov
arch/x86/include/asm/pgtable.h +1
-7@@ -749,14 +740,8 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) pte_result =__pte (val); -/* -* Dirty bit is not preserved above so it can be done -* in a special way for the shadow stack case, where it -* may need to set _PAGE_COW. __pte_mkdirty() will do this in -* the case of shadow stack. -*/ if (pte_dirty (pte)) - pte_result =__pte_mkdirty (pte_result, false); + pte_result =pte_set_flags (pte_result, _PAGE_DIRTY);return pte_result;
Quoted 2 lines
}
Oh, I see what you are seeing now. Did you notice that the __pte_mkdirty() logic got expanded in "x86/mm: Start actually marking _PAGE_COW"? So if we don't put that logic in a usable helper, it ends up open coded with pte_modify() looking something like this:
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
pteval_t val = pte_val(pte), oldval = val;
pte_t pte_result;
/*
* Chop off the NX bit (if present), and add the NX portion of
* the newprot (if present):
*/
val &= (_PAGE_CHG_MASK & ~_PAGE_DIRTY);
val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK;
val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);
pte_result = __pte(val);
/*
* Dirty bit is not preserved above so it can be done
* in a special way for the shadow stack case, where it
* may need to set _PAGE_SAVED_DIRTY. __pte_mkdirty() will do
* this in the case of shadow stack.
*/
if (oldval & _PAGE_DIRTY)
if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) &&
!pte_write(pte_result))
pte_set_flags(pte_result, _PAGE_SAVED_DIRTY);
else
pte_set_flags(pte_result, _PAGE_DIRTY);
}
return pte_result;
}So the later logic of doing the _PAGE_SAVED_DIRTY (_PAGE_COW) part is not centralized. It's ok?
Borislav Petkov
maintainerRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 17 lines · Rick P
/* * Dirty bit is not preserved above so it can be done * in a special way for the shadow stack case, where it * may need to set _PAGE_SAVED_DIRTY. __pte_mkdirty() will do * this in the case of shadow stack. */ if (oldval & _PAGE_DIRTY) if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pte_write(pte_result)) pte_set_flags(pte_result, _PAGE_SAVED_DIRTY); else pte_set_flags(pte_result, _PAGE_DIRTY); } return pte_result; }
Quoted 2 lines
So the later logic of doing the
_PAGE_SAVED_DIRTY(_PAGE_COW) part is not centralized. It's ok?
I think so.
1. If you have a single pte_mkdirty() and not also a __ helper, then there's less confusion for callers as to which interface they should be using
2. The not centralized part is a single conditional so it's not like you're saving on gazillion code lines
So I'd prefer that.
If we end up needing this in more places then we can carve it out into a proper helper which is not in a header file such that anyone can use it but move the whole functionality into cet.c or so where we can control its visibility to the rest of the kernel.
I'd say.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v5 11/39] x86/mm: Update pte_modify for _PAGE_COW
Quoted 31 lines · Borislav Petkov
Quoted 19 lines · Rick P
/*
* Dirty bit is not preserved above so it can be done
* in a special way for the shadow stack case, where it
* may need to set_PAGE_SAVED_DIRTY.__pte_mkdirty()will
do* this in the case of shadow stack. */ if (oldval & _PAGE_DIRTY) if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pte_write(pte_result)) pte_set_flags(pte_result, _PAGE_SAVED_DIRTY); else pte_set_flags(pte_result, _PAGE_DIRTY); } return pte_result; }Quoted 3 lines
So the later logic of doing the
_PAGE_SAVED_DIRTY(_PAGE_COW) part is
not centralized. It's ok?I think so.
1. If you have a single
pte_mkdirty()and not also a __ helper, then there's less confusion for callers as to which interface they
should be
using
Quoted 6 lines
2. The not centralized part is a single conditional so it's not like you're saving on gazillion code lines
So I'd prefer that.
Fair enough, I'll adjust it. Thanks!
PATCH v5 12/39 x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,Cow=1.
When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could become a transient shadow stack PTE in two cases:
1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary.
2. When _PAGE_DIRTY is replaced with _PAGE_COW non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.
In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature.
Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy.
Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.
---
v5:
- Commit log verbiage and formatting (Boris)
- Remove capitalization on shadow stack (Boris)
- Fix i386 warning on recent clang
v3:
- Remove unnecessary #ifdef (Dave Hansen)
v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)
Yu-cheng v30:
- Replace (pmdval_t) cast with CONFIG_PGTABLE_LEVELES > 2 (Borislav Petkov).
arch/x86/include/asm/pgtable.h | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
arch/x86/include/asm/pgtable.h +37 -0
@@ -1232,6 +1232,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK +/* +* Avoid accidentally creating shadow stack PTEs +* (Write=0,Dirty=1). Use cmpxchg() to prevent races with +* the hardware setting Dirty=1. +*/ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) { +pte_t old_pte, new_pte; + + old_pte =READ_ONCE (*ptep); +do { + new_pte =pte_wrprotect (old_pte); + }while (!try_cmpxchg (&ptep->pte, &old_pte.pte, new_pte.pte)); + +return ; + } +#endif clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); }
@@ -1284,6 +1301,26 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK +/* +* If shadow stack is enabled, pmd_wrprotect() moves _PAGE_DIRTY +* to _PAGE_COW (see comments at pmd_wrprotect()). +* When a thread reads a RW=1, Dirty=0 PMD and before changing it +* to RW=0, Dirty=0, another thread could have written to the page +* and the PMD is RW=1, Dirty=1 now. +*/ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) { +pmd_t old_pmd, new_pmd; + + old_pmd =READ_ONCE (*pmdp); +do { + new_pmd =pmd_wrprotect (old_pmd); + }while (!try_cmpxchg (&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + +return ; + } +#endif +clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); }
Signature · 1 line
2.17.1
Re: [PATCH v5 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW
Quoted 31 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,Cow=1.
When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could become a transient shadow stack PTE in two cases:
1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary.
2. When
_PAGE_DIRTYis replaced with_PAGE_COWnon-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.In the case of
pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave thecpu_feature_enabled()check so that the functionality also gets disabled based on runtime detection of the feature.Similarly, compile it out in
ptep_set_wrprotect()due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy.
Quoted 6 lines
Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 13/39 x86/mm: Start actually marking _PAGE_COW
In reply to Rick EdgecombeThe recently introduced _PAGE_COW should be used instead of the HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply the same changes to pmd and pud.
---
v4:
- Break part patch for better bisectability
arch/x86/include/asm/pgtable.h | 125 ++++++++++++++++++++++++++++----- 1 file changed, 107 insertions(+), 18 deletions(-)
arch/x86/include/asm/pgtable.h +107 -18
@@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags;* The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty (pte_t pte) +static inline bool pte_dirty (pte_t pte) { -return pte_flags (pte) & _PAGE_DIRTY; +return pte_flags (pte) & _PAGE_DIRTY_BITS; +} + +static inline bool pte_shstk (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return false; + +return (pte_flags (pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; }static inline int pte_young (pte_t pte)
@@ -134,9 +142,18 @@ static inline int pte_young(pte_t pte)return pte_flags (pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty (pmd_t pmd) +static inline bool pmd_dirty (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_DIRTY; +return pmd_flags (pmd) & _PAGE_DIRTY_BITS; +} + +static inline bool pmd_shstk (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return false; + +return (pmd_flags (pmd) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); }#define pmd_young pmd_young
@@ -145,9 +162,9 @@ static inline int pmd_young(pmd_t pmd)return pmd_flags (pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty (pud_t pud) +static inline bool pud_dirty (pud_t pud) { -return pud_flags (pud) & _PAGE_DIRTY; +return pud_flags (pud) & _PAGE_DIRTY_BITS; }static inline int pud_young (pud_t pud)
@@ -157,13 +174,21 @@ static inline int pud_young(pud_t pud)static inline int pte_write (pte_t pte) { -return pte_flags (pte) & _PAGE_RW; +/* +* Shadow stack pages are logically writable, but do not have +* _PAGE_RW. Check for them separately from _PAGE_RW itself. +*/ +return (pte_flags (pte) & _PAGE_RW) ||pte_shstk (pte); }#define pmd_write pmd_writestatic inline int pmd_write (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_RW; +/* +* Shadow stack pages are logically writable, but do not have +* _PAGE_RW. Check for them separately from _PAGE_RW itself. +*/ +return (pmd_flags (pmd) & _PAGE_RW) ||pmd_shstk (pmd); }#define pud_write pud_write
@@ -374,7 +399,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)static inline pte_t pte_mkclean (pte_t pte) { -return pte_clear_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_DIRTY_BITS); }static inline pte_t pte_mkold (pte_t pte)
@@ -384,7 +409,16 @@ static inline pte_t pte_mkold(pte_t pte)static inline pte_t pte_wrprotect (pte_t pte) { -return pte_clear_flags (pte, _PAGE_RW); + pte =pte_clear_flags (pte, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PTE (Write=0,Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pte_dirty (pte)) + pte =pte_mkcow (pte); +return pte; }static inline pte_t pte_mkexec (pte_t pte)
@@ -396,6 +430,10 @@ static inline pte_t __pte_mkdirty(pte_t pte, bool soft) {pteval_t dirty = _PAGE_DIRTY; +/* Avoid creating Dirty=1,Write=0 PTEs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pte_write (pte)) + dirty = _PAGE_COW; +if (soft) dirty |= _PAGE_SOFT_DIRTY;
@@ -407,6 +445,12 @@ static inline pte_t pte_mkdirty(pte_t pte)return __pte_mkdirty (pte, true); } +static inline pte_t pte_mkwrite_shstk (pte_t pte) +{ +/* pte_clear_cow() also sets Dirty=1 */ +return pte_clear_cow (pte); +} +static inline pte_t pte_mkyoung (pte_t pte) {return pte_set_flags (pte, _PAGE_ACCESSED);
@@ -414,7 +458,12 @@ static inline pte_t pte_mkyoung(pte_t pte)static inline pte_t pte_mkwrite (pte_t pte) { -return pte_set_flags (pte, _PAGE_RW); + pte =pte_set_flags (pte, _PAGE_RW); + +if (pte_dirty (pte)) + pte =pte_clear_cow (pte); + +return pte; }static inline pte_t pte_mkhuge (pte_t pte)
@@ -505,18 +554,30 @@ static inline pmd_t pmd_mkold(pmd_t pmd)static inline pmd_t pmd_mkclean (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_DIRTY_BITS); }static inline pmd_t pmd_wrprotect (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_RW); + pmd =pmd_clear_flags (pmd, _PAGE_RW); +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PMD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pmd_dirty (pmd)) + pmd =pmd_mkcow (pmd); +return pmd; }static inline pmd_t __pmd_mkdirty (pmd_t pmd,bool soft) {pmdval_t dirty = _PAGE_DIRTY; +/* Avoid creating (HW)Dirty=1, Write=0 PMDs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pmd_write (pmd)) + dirty = _PAGE_COW; +if (soft) dirty |= _PAGE_SOFT_DIRTY;
@@ -528,6 +589,11 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)return __pmd_mkdirty (pmd, true); } +static inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) +{ +return pmd_clear_cow (pmd); +} +static inline pmd_t pmd_mkdevmap (pmd_t pmd) {return pmd_set_flags (pmd, _PAGE_DEVMAP);
@@ -545,7 +611,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)static inline pmd_t pmd_mkwrite (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_RW); + pmd =pmd_set_flags (pmd, _PAGE_RW); + +if (pmd_dirty (pmd)) + pmd =pmd_clear_cow (pmd); +return pmd; }static inline pud_t pud_set_flags (pud_t pud,pudval_t set)
@@ -589,17 +659,32 @@ static inline pud_t pud_mkold(pud_t pud)static inline pud_t pud_mkclean (pud_t pud) { -return pud_clear_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_DIRTY_BITS); }static inline pud_t pud_wrprotect (pud_t pud) { -return pud_clear_flags (pud, _PAGE_RW); + pud =pud_clear_flags (pud, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PUD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pud_dirty (pud)) + pud =pud_mkcow (pud); +return pud; }static inline pud_t pud_mkdirty (pud_t pud) { -return pud_set_flags (pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pudval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PUDs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pud_write (pud)) + dirty = _PAGE_COW; + +return pud_set_flags (pud, dirty | _PAGE_SOFT_DIRTY); }static inline pud_t pud_mkdevmap (pud_t pud)
@@ -619,7 +704,11 @@ static inline pud_t pud_mkyoung(pud_t pud)static inline pud_t pud_mkwrite (pud_t pud) { -return pud_set_flags (pud, _PAGE_RW); + pud =pud_set_flags (pud, _PAGE_RW); + +if (pud_dirty (pud)) + pud =pud_clear_cow (pud); +return pud; }#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
Signature · 1 line
2.17.1
PATCH v5 14/39 mm: Move VM_UFFD_MINOR_BIT from 37 to 38
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Future patches will introduce a new VM flag VM_SHADOW_STACK that will be VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake of order, make all VM_HIGH_ARCH_BITs stay together by moving VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be introduced as 37.
--- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
@@ -366,7 +366,7 @@ extern unsigned int kobjsize(const void *objp);#endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR -# define VM_UFFD_MINOR_BIT 37 +# define VM_UFFD_MINOR_BIT 38# define VM_UFFD_MINOR BIT(VM_UFFD_MINOR_BIT)/* UFFD minor faults */ #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ # define VM_UFFD_MINOR VM_NONE
Signature · 1 line
2.17.1
PATCH v5 15/39 mm: Introduce VM_SHADOW_STACK for shadow stack memory
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.
---
v3:
- Drop arch specific change in arch_vma_name(). The memory can show as anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags() (Boris)
Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+)
Documentation/filesystems/proc.rst +1 -0
@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will
fs/proc/task_mmu.c +3 -0
@@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui",#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif };size_t i;
include/linux/mm.h +8 -0
@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp);#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +# define VM_SHADOW_STACK VM_HIGH_ARCH_5 +#else +# define VM_SHADOW_STACK VM_NONE +#endif +#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC )
Signature · 1 line
2.17.1
PATCH v5 16/39 x86/mm: Check shadow stack page fault errors
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.
The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.
For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes.
Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.
---
v5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)
v4:
- Further improve comment talking about FAULT_FLAG_WRITE (Peterz)
v3:
- Improve comment talking about using FAULT_FLAG_WRITE (Peterz)
v2:
- Update commit log with verbiage/feedback from Dave Hansen
- Clarify reasoning for FAULT_FLAG_WRITE for all shadow stack accesses
- Update comments with some verbiage from Dave Hansen
arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
arch/x86/include/asm/trap_pf.h +2 -0
@@ -11,6 +11,7 @@* bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access +* bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code {
@@ -20,6 +21,7 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , +X86_PF_SHSTK =1 <<6 , X86_PF_SGX = 1 << 15, };
arch/x86/mm/fault.c +38 -0
@@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; +/* +* Shadow stack accesses (PF_SHSTK=1) are only permitted to +* shadow stack VMAs. All other accesses result in an error. +*/ +if (error_code &X86_PF_SHSTK ) { +if (unlikely (!(vma->vm_flags &VM_SHADOW_STACK ))) +return 1 ; +if (unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; +return 0 ; + } +if (error_code &X86_PF_WRITE ) {/* write, present and write, not present: */ +if (unlikely (vma->vm_flags &VM_SHADOW_STACK )) +return 1 ;if (unlikely (!(vma->vm_flags &VM_WRITE )))return 1 ;return 0 ;
@@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* +* When a page becomes COW it changes from a shadow stack permission +* page (Write=0,Dirty=1) to (Write=0,Dirty=0,CoW=1), which is simply +* read-only to the CPU. When shadow stack is enabled, a RET would +* normally pop the shadow stack by reading it with a "shadow stack +* read" access. However, in the COW case the shadow stack memory does +* not have shadow stack permissions, it is read-only. So it will +* generate a fault. +* +* For conventionally writable pages, a read can be serviced with a +* read only PTE, and COW would not have to happen. But for shadow +* stack, there isn't the concept of read-only shadow stack memory. +* If it is shadow stack permission, it can be modified via CALL and +* RET instructions. So COW needs to happen before any memory can be +* mapped with shadow stack permissions. +* +* Shadow stack accesses (read or write) need to be serviced with +* shadow stack permission memory, so in the case of a shadow stack +* read access, treat it as a WRITE fault so both COW will happen and +* the write fault path will tickle maybe_mkwrite() and map the memory +* shadow stack. +*/ +if (error_code &X86_PF_SHSTK ) + flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_WRITE ) flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_INSTR )
Signature · 1 line
2.17.1
Re: [PATCH v5 16/39] x86/mm: Check shadow stack page fault errors
Quoted 34 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.
The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set
FAULT_FLAG_WRITEfor shadow stack accesses, even if the access was a shadow stack read.For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes.
Quoted 7 lines
Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 17/39 x86/mm: Update maybe_mkwrite() for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When serving a page fault, maybe_mkwrite() makes a PTE writable if there is a write access to it, and its vma has VM_WRITE. Shadow stack accesses to shadow stack vma's are also treated as write accesses by the fault handler. This is because setting shadow stack memory makes it writable via some instructions, so COW has to happen even for shadow stack reads.
So maybe_mkwrite() should continue to set VM_WRITE vma's as normally writable, but also set VM_WRITE|VM_SHADOW_STACK vma's as shadow stack.
Do this by adding a pte_mkwrite_shstk() and a cross-arch stub. Check for VM_SHADOW_STACK in maybe_mkwrite() and call pte_mkwrite_shstk() accordingly.
Apply the same changes to maybe_pmd_mkwrite().
---
v3:
- Remove unneeded define for maybe_mkwrite (Peterz)
- Switch to cleaner version of maybe_mkwrite() (Peterz)
v2:
- Change to handle shadow stacks that are VM_WRITE|VM_SHADOW_STACK
- Ditch arch specific maybe_mkwrite(), and make the code generic
- Move do_anonymous_page() to next patch (Kirill)
Yu-cheng v29:
- Remove likely()'s.
arch/x86/include/asm/pgtable.h | 2 ++ include/linux/mm.h | 13 ++++++++++--- include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 10 +++++++--- 4 files changed, 33 insertions(+), 6 deletions(-)
arch/x86/include/asm/pgtable.h +2 -0
@@ -445,6 +445,7 @@ static inline pte_t pte_mkdirty(pte_t pte)return __pte_mkdirty (pte, true); } +#define pte_mkwrite_shstk pte_mkwrite_shstkstatic inline pte_t pte_mkwrite_shstk (pte_t pte) {/* pte_clear_cow() also sets Dirty=1 */
@@ -589,6 +590,7 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)return __pmd_mkdirty (pmd, true); } +#define pmd_mkwrite_shstk pmd_mkwrite_shstkstatic inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) {return pmd_clear_cow (pmd);
include/linux/mm.h +10 -3
@@ -1106,12 +1106,19 @@ void free_compound_page(struct page *page);* servicing faults for write access. In the normal case, do always want * pte_mkwrite. But get_user_pages can cause write faults for mappings * that do not have writing enabled, when used by access_process_vm. +* +* If a vma is shadow stack (a type of writable memory), mark the pte shadow +* stack. */ static inline pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma) { -if (likely (vma->vm_flags &VM_WRITE )) - pte =pte_mkwrite (pte); -return pte; +if (!(vma->vm_flags &VM_WRITE )) +return pte; + +if (vma->vm_flags &VM_SHADOW_STACK ) +return pte_mkwrite_shstk (pte); + +return pte_mkwrite (pte); }vm_fault_t do_set_pmd (struct vm_fault *vmf,struct page *page);
include/linux/pgtable.h +14 -0
@@ -532,6 +532,20 @@ static inline pte_t pte_sw_mkyoung(pte_t pte)#define pte_sw_mkyoung pte_sw_mkyoung#endif +#ifndef pte_mkwrite_shstk +static inline pte_t pte_mkwrite_shstk (pte_t pte) +{ +return pte; +} +#endif + +#ifndef pmd_mkwrite_shstk +static inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) +{ +return pmd; +} +#endif +#ifndef __HAVE_ARCH_PMDP_SET_WRPROTECT#ifdef CONFIG_TRANSPARENT_HUGEPAGE static inline void pmdp_set_wrprotect(struct mm_struct *mm,
mm/huge_memory.c +7 -3
@@ -553,9 +553,13 @@ __setup("transparent_hugepage=", setup_transparent_hugepage);pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) { -if (likely (vma->vm_flags &VM_WRITE )) - pmd =pmd_mkwrite (pmd); -return pmd; +if (!(vma->vm_flags &VM_WRITE )) +return pmd; + +if (vma->vm_flags &VM_SHADOW_STACK ) +return pmd_mkwrite_shstk (pmd); + +return pmd_mkwrite (pmd); }#ifdef CONFIG_MEMCG
Signature · 1 line
2.17.1
PATCH v5 18/39 mm: Handle faultless write upgrades for shstk
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Since shadow stack memory can be changed from userspace, is both VM_SHADOW_STACK and VM_WRITE. But it should not be made conventionally writable (i.e. pte_mkwrite()). So some code that calls pte_mkwrite() needs to be adjusted.
One such case is when memory is made writable without an actual write fault. This happens in some mprotect operations, and also prot_numa faults. In both cases code checks whether it should be made (conventionally) writable by calling vma_wants_manual_pte_write_upgrade().
One way to fix this would be have code actually check if memory is also VM_SHADOW_STACK and in that case call pte_mkwrite_shstk(). But since most memory won't be shadow stack, just have simpler logic and skip this optimization by changing vma_wants_manual_pte_write_upgrade() to not return true for VM_SHADOW_STACK_MEMORY. This will simply handle all cases of this type.
---
v5:
- Update solution after the recent removal of pte_savedwrite()
v4:
- Add "why" to comments in code (Peterz)
Yu-cheng v25:
- Move is_shadow_stack_mapping() to a separate line.
Yu-cheng v24:
- Change arch_shadow_stack_mapping() to is_shadow_stack_mapping().
include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
@@ -2181,7 +2181,7 @@ static inline bool vma_wants_manual_pte_write_upgrade(struct vm_area_struct *vma*/ if (vma->vm_flags &VM_SHARED )return vma_wants_writenotify (vma, vma->vm_page_prot); -return !!(vma->vm_flags &VM_WRITE ); +return (vma->vm_flags &VM_WRITE ) && !(vma->vm_flags &VM_SHADOW_STACK ); } bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
Signature · 1 line
2.17.1
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 22 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Since shadow stack memory can be changed from userspace, is both
VM_SHADOW_STACKandVM_WRITE. But it should not be made conventionally writable (i.e.pte_mkwrite()). So some code that callspte_mkwrite()needs to be adjusted.One such case is when memory is made writable without an actual write fault. This happens in some mprotect operations, and also
prot_numafaults. In both cases code checks whether it should be made (conventionally) writable by callingvma_wants_manual_pte_write_upgrade().One way to fix this would be have code actually check if memory is also
VM_SHADOW_STACKand in that case callpte_mkwrite_shstk(). But since most memory won't be shadow stack, just have simpler logic and skip this optimization by changingvma_wants_manual_pte_write_upgrade()to not return true forVM_SHADOW_STACK_MEMORY. This will simply handle all cases of this type.
Quoted 7 lines
Cc David Hildenbrand <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
Instead of having these x86-shadow stack details all over the MM space, was the option explored to handle this more in arch specific code?
IIUC, one way to get it working would be
1) Have a SW "shadowstack" PTE flag.
2) Have an "SW-dirty" PTE flag, to store "dirty=1" when "write=0".
pte_mkwrite(), pte_write(), pte_dirty ... can then make decisions based on the "shadowstack" PTE flag and hide all these details from core-mm.
When mapping a shadowstack page (new page, migration, swapin, ...), which can be obtained by looking at the VMA flags, the first thing you'd do is set the "shadowstack" PTE flag.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 42 lines · David Hildenbrand
Quoted 33 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Since shadow stack memory can be changed from userspace, is both
VM_SHADOW_STACKandVM_WRITE. But it should not be made conventionally
writable (i.e.pte_mkwrite()). So some code that callspte_mkwrite()needs
to be adjusted.One such case is when memory is made writable without an actual write
fault. This happens in some mprotect operations, and alsoprot_numafaults.
In both cases code checks whether it should be made (conventionally)
writable by callingvma_wants_manual_pte_write_upgrade().One way to fix this would be have code actually check if memory is also
VM_SHADOW_STACKand in that case callpte_mkwrite_shstk(). But since
most memory won't be shadow stack, just have simpler logic and skip this
optimization by changingvma_wants_manual_pte_write_upgrade()to not
return true forVM_SHADOW_STACK_MEMORY. This will simply handle all cases of this type.Quoted 7 lines
Cc David Hildenbrand <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
Quoted 8 lines
Instead of having these x86-shadow stack details all over the MM space,
was the option explored to handle this more in arch specific code?IIUC, one way to get it working would be
1) Have a SW "shadowstack" PTE flag.
2) Have an "SW-dirty" PTE flag, to store "dirty=1" when "write=0".
I don't think that idea came up. So vma->vm_page_prot would have the SW shadow stack flag for VM_SHADOW_STACK, and pte_mkwrite() could do Write=0,Dirty=1 part. It seems like it should work.
Quoted 10 lines
pte_mkwrite(),pte_write(),pte_dirty... can then make decisions based
on the "shadowstack" PTE flag and hide all these details from core- mm.When mapping a shadowstack page (new page, migration, swapin, ...), which can be obtained by looking at the VMA flags, the first thing you'd
do is set the "shadowstack" PTE flag.
I guess the downside is that it uses an extra software bit. But the other positive is that it's less error prone, so that someone writing core-mm code won't introduce a change that makes shadow stack VMAs Write=1 if they don't know to also check for VM_SHADOW_STACK.
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 47 lines · Rick P
Quoted 46 lines · David Hildenbrand
Quoted 33 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Since shadow stack memory can be changed from userspace, is both
VM_SHADOW_STACKandVM_WRITE. But it should not be made conventionally
writable (i.e.pte_mkwrite()). So some code that callspte_mkwrite()needs
to be adjusted.One such case is when memory is made writable without an actual write
fault. This happens in some mprotect operations, and alsoprot_numafaults.
In both cases code checks whether it should be made (conventionally)
writable by callingvma_wants_manual_pte_write_upgrade().One way to fix this would be have code actually check if memory is also
VM_SHADOW_STACKand in that case callpte_mkwrite_shstk(). But since
most memory won't be shadow stack, just have simpler logic and skip this
optimization by changingvma_wants_manual_pte_write_upgrade()to not
return true forVM_SHADOW_STACK_MEMORY. This will simply handle all cases of this type.Quoted 7 lines
Cc David Hildenbrand <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
Instead of having these x86-shadow stack details all over the MM space,
was the option explored to handle this more in arch specific code?
Quoted 9 lines
Quoted 4 lines
IIUC, one way to get it working would be
1) Have a SW "shadowstack" PTE flag.
2) Have an "SW-dirty" PTE flag, to store "dirty=1" when "write=0".I don't think that idea came up. So
vma->vm_page_protwould have the SW shadow stack flag forVM_SHADOW_STACK, andpte_mkwrite()could do Write=0,Dirty=1 part. It seems like it should work.
Right, if we include it in vma->vm_page_prot, we'd immediately let mk_pte() just handle that.
Otherwise, we'd have to refactor e.g., mk_pte() to consume a vma instead of the vma->vm_page_prot. Let's see if we can avoid that for now.
Quoted 11 lines
Quoted 10 lines
pte_mkwrite(),pte_write(),pte_dirty... can then make decisions based
on the "shadowstack" PTE flag and hide all these details from core- mm.When mapping a shadowstack page (new page, migration, swapin, ...), which can be obtained by looking at the VMA flags, the first thing you'd
do is set the "shadowstack" PTE flag.
Quoted 4 lines
I guess the downside is that it uses an extra software bit. But the other positive is that it's less error prone, so that someone writing core-mm code won't introduce a change that makes shadow stack VMAs Write=1 if they don't know to also check for
VM_SHADOW_STACK.
Right. And I think this mimics the what I would have expected HW to provide: a dedicated HW bit, not somehow mangling this into semantics of existing bits.
Roughly speaking: if we abstract it that way and get all of the "how to set it writable now?" out of core-MM, it not only is cleaner and less error prone, it might even allow other architectures that implement something comparable (e.g., using a dedicated HW bit) to actually reuse some of that work. Otherwise most of that "shstk" is really just x86 specific ...
I guess the only cases we have to special case would be page pinning code where pte_write() would indicate that the PTE is writable (well, it is, just not by "ordinary CPU instruction" context directly): but you do that already, so ... :)
Sorry for stumbling over that this late, I only started looking into this when you CCed me on that one patch.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 98 lines · David Hildenbrand
Quoted 60 lines · Rick P
Quoted 59 lines · David Hildenbrand
Quoted 42 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.Since shadow stack memory can be changed from userspace, is both
VM_SHADOW_STACKandVM_WRITE. But it should not be made conventionally
writable (i.e.pte_mkwrite()). So some code that callspte_mkwrite()needs
to be adjusted.One such case is when memory is made writable without an actual write
fault. This happens in some mprotect operations, and alsoprot_numa
faults.
In both cases code checks whether it should be made (conventionally)
writable by callingvma_wants_manual_pte_write_upgrade().One way to fix this would be have code actually check if memory is
alsoVM_SHADOW_STACKand in that case callpte_mkwrite_shstk(). But since
most memory won't be shadow stack, just have simpler logic and skip
this
optimization by changingvma_wants_manual_pte_write_upgrade()to
not
return true forVM_SHADOW_STACK_MEMORY. This will simply handle all
cases of this type.Quoted 8 lines
Cc David Hildenbrand <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <Signed-off-by Rick Edgecombe <[email protected]>---
Instead of having these x86-shadow stack details all over the MM space,
was the option explored to handle this more in arch specific code?IIUC, one way to get it working would be
Quoted 9 lines
Quoted 3 lines
1) Have a SW "shadowstack" PTE flag.
2) Have an "SW-dirty" PTE flag, to store "dirty=1" when "write=0".I don't think that idea came up. So
vma->vm_page_protwould have the SW
shadow stack flag forVM_SHADOW_STACK, andpte_mkwrite()could do Write=0,Dirty=1 part. It seems like it should work.Right, if we include it in
vma->vm_page_prot, we'd immediately letmk_pte()just handle that.Otherwise, we'd have to refactor e.g.,
mk_pte()to consume a vma instead
of thevma->vm_page_prot. Let's see if we can avoid that for now.Quoted 14 lines
Quoted 7 lines
pte_mkwrite(),pte_write(),pte_dirty... can then make decisions based
on the "shadowstack" PTE flag and hide all these details from core-
mm.Quoted 6 lines
When mapping a shadowstack page (new page, migration, swapin, ...),
which can be obtained by looking at the VMA flags, the first thing
you'd
do is set the "shadowstack" PTE flag.Quoted 5 lines
I guess the downside is that it uses an extra software bit. But the other positive is that it's less error prone, so that someone writing
core-mm code won't introduce a change that makes shadow stack VMAs Write=1 if they don't know to also check forVM_SHADOW_STACK.
Quoted 4 lines
Right. And I think this mimics the what I would have expected HW to
provide: a dedicated HW bit, not somehow mangling this into semantics of
existing bits.
Yea.
Quoted 18 lines
Roughly speaking: if we abstract it that way and get all of the "how to
set it writable now?" out of core-MM, it not only is cleaner and less
error prone, it might even allow other architectures that implement something comparable (e.g., using a dedicated HW bit) to actually reuse
some of that work. Otherwise most of that "shstk" is really just x86 specific ...I guess the only cases we have to special case would be page pinning code where
pte_write()would indicate that the PTE is writable (well, it
is, just not by "ordinary CPU instruction" context directly): but you do
that already, so ... :)
Quoted 2 lines
Sorry for stumbling over that this late, I only started looking into this when you CCed me on that one patch.
Sorry for not calling more attention to it earlier. Appreciate your comments.
Previously versions of this series had changed some of these pte_mkwrite() calls to maybe_mkwrite(), which of course takes a vma. This way an x86 implementation could use the VM_SHADOW_STACK vma flag to decide between pte_mkwrite() and pte_mkwrite_shstk(). The feedback was that in some of these code paths "maybe" isn't really an option, it *needs* to make it writable. Even though the logic was the same, the name of the function made it look wrong.
But another option could be to change pte_mkwrite() to take a vma. This would save using another software bit on x86, but instead requires a small change to each arch's pte_mkwrite().
x86's pte_mkwrite() would then be pretty close to maybe_mkwrite(), but maybe it could additionally warn if the vma is not writable. It also seems more aligned with your changes to stop taking hints from PTE bits and just look at the VMA? (I'm thinking about the dropping of the dirty check in GUP and dropping pte_saved_write())
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 138 lines · Rick P
Quoted 98 lines · David Hildenbrand
Quoted 60 lines · Rick P
Quoted 59 lines · David Hildenbrand
Quoted 42 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes
a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.Since shadow stack memory can be changed from userspace, is both
VM_SHADOW_STACKandVM_WRITE. But it should not be made conventionally
writable (i.e.pte_mkwrite()). So some code that callspte_mkwrite()needs
to be adjusted.One such case is when memory is made writable without an actual write
fault. This happens in some mprotect operations, and alsoprot_numa
faults.
In both cases code checks whether it should be made (conventionally)
writable by callingvma_wants_manual_pte_write_upgrade().One way to fix this would be have code actually check if memory is
alsoVM_SHADOW_STACKand in that case callpte_mkwrite_shstk(). But since
most memory won't be shadow stack, just have simpler logic and skip
this
optimization by changingvma_wants_manual_pte_write_upgrade()to
not
return true forVM_SHADOW_STACK_MEMORY. This will simply handle all
cases of this type.Quoted 8 lines
Cc David Hildenbrand <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <Signed-off-by Rick Edgecombe <[email protected]>---
Instead of having these x86-shadow stack details all over the MM space,
was the option explored to handle this more in arch specific code?IIUC, one way to get it working would be
Quoted 9 lines
Quoted 3 lines
1) Have a SW "shadowstack" PTE flag.
2) Have an "SW-dirty" PTE flag, to store "dirty=1" when "write=0".I don't think that idea came up. So
vma->vm_page_protwould have the SW
shadow stack flag forVM_SHADOW_STACK, andpte_mkwrite()could do Write=0,Dirty=1 part. It seems like it should work.Right, if we include it in
vma->vm_page_prot, we'd immediately letmk_pte()just handle that.Otherwise, we'd have to refactor e.g.,
mk_pte()to consume a vma instead
of thevma->vm_page_prot. Let's see if we can avoid that for now.Quoted 14 lines
Quoted 7 lines
pte_mkwrite(),pte_write(),pte_dirty... can then make decisions based
on the "shadowstack" PTE flag and hide all these details from core-
mm.Quoted 6 lines
When mapping a shadowstack page (new page, migration, swapin, ...),
which can be obtained by looking at the VMA flags, the first thing
you'd
do is set the "shadowstack" PTE flag.Quoted 5 lines
I guess the downside is that it uses an extra software bit. But the other positive is that it's less error prone, so that someone writing
core-mm code won't introduce a change that makes shadow stack VMAs Write=1 if they don't know to also check forVM_SHADOW_STACK.Quoted 4 lines
Right. And I think this mimics the what I would have expected HW to
provide: a dedicated HW bit, not somehow mangling this into semantics of
existing bits.Yea.
Quoted 18 lines
Roughly speaking: if we abstract it that way and get all of the "how to
set it writable now?" out of core-MM, it not only is cleaner and less
error prone, it might even allow other architectures that implement something comparable (e.g., using a dedicated HW bit) to actually reuse
some of that work. Otherwise most of that "shstk" is really just x86 specific ...I guess the only cases we have to special case would be page pinning code where
pte_write()would indicate that the PTE is writable (well, it
is, just not by "ordinary CPU instruction" context directly): but you do
that already, so ... :)Quoted 2 lines
Sorry for stumbling over that this late, I only started looking into this when you CCed me on that one patch.
Sorry for not calling more attention to it earlier. Appreciate your comments.
Previously versions of this series had changed some of these
pte_mkwrite()calls tomaybe_mkwrite(), which of course takes a vma. This way an x86 implementation could use theVM_SHADOW_STACKvma flag to decide betweenpte_mkwrite()andpte_mkwrite_shstk(). The feedback was that in some of these code paths "maybe" isn't really an option, it *needs* to make it writable. Even though the logic was the same, the name of the function made it look wrong.
Quoted 3 lines
But another option could be to change
pte_mkwrite()to take a vma. This would save using another software bit on x86, but instead requires a small change to each arch'spte_mkwrite().
I played with that idea shortly as well, but discarded it. I was not able to convince myself that it wouldn't be required to pass in the VMA as well for things like pte_dirty(), pte_mkdirty(), pte_write(), ... which would end up fairly ugly (or even impossible in thing slike GUP-fast).
For example, I wonder how we'd be handling stuff like do_numa_page() cleanly correctly, where we use pte_modify() + pte_mkwrite(), and either call might set the PTE writable and maintain dirty bit ...
Having that said, maybe it could work with only a single saved-dirty bit and passing in the VMA for pte_mkwrite() only.
pte_wrprotect() would detect "writable=0,dirty=1" and move the dirty bit to the soft-dirty bit instead, resulting in "writable=0,dirty=0,saved-dirty=1",
pte_dirty() would return dirty==1||saved-dirty==1.
pte_mkdirty() would set either set dirty=1 or saved-dirty=1, depending on the writable bit.
pte_mkclean() would clean both bits.
pte_write() would detect "writable == 1 || (writable==0 && dirty==1)"
pte_mkwrite() would act according to the VMA, and in addition, merge the saved-dirty bit into the dirty bit.
pte_modify() and mk_pte() .... would require more thought ...
Further, ptep_modify_prot_commit() might have to be adjusted to properly flush in all relevant cases IIRC.
Quoted 6 lines
x86's
pte_mkwrite()would then be pretty close tomaybe_mkwrite(), but maybe it could additionally warn if the vma is not writable. It also seems more aligned with your changes to stop taking hints from PTE bits and just look at the VMA? (I'm thinking about the dropping of the dirty check in GUP and droppingpte_saved_write())
The soft-shstk bit wouldn't be a hint, it would be logically changing the "type" of the PTE such that any other PTE functions can do the right thing without having to consume the VMA.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 56 lines · David Hildenbrand
Quoted 43 lines
Quoted 23 lines
Roughly speaking: if we abstract it that way and get all of the "how
to
set it writable now?" out of core-MM, it not only is cleaner and less
error prone, it might even allow other architectures that implement
something comparable (e.g., using a dedicated HW bit) to actually reuse
some of that work. Otherwise most of that "shstk" is really just x86
specific ...I guess the only cases we have to special case would be page pinning
code wherepte_write()would indicate that the PTE is writable (well,
it
is, just not by "ordinary CPU instruction" context directly): but you
do
that already, so ... :)Quoted 3 lines
Sorry for stumbling over that this late, I only started looking into
this when you CCed me on that one patch.Sorry for not calling more attention to it earlier. Appreciate your comments.
Previously versions of this series had changed some of these
pte_mkwrite()calls tomaybe_mkwrite(), which of course takes a vma.
This way an x86 implementation could use theVM_SHADOW_STACKvma flag
to decide betweenpte_mkwrite()andpte_mkwrite_shstk(). The feedback
was that in some of these code paths "maybe" isn't really an option, it
*needs* to make it writable. Even though the logic was the same, the
name of the function made it look wrong.Quoted 5 lines
But another option could be to change
pte_mkwrite()to take a vma. This
would save using another software bit on x86, but instead requires a
small change to each arch'spte_mkwrite().I played with that idea shortly as well, but discarded it. I was not able to convince myself that it wouldn't be required to pass in the VMA
as well for things likepte_dirty(),pte_mkdirty(),pte_write(), ... which would end up fairly ugly (or even impossible in thing slike GUP-fast).
Quoted 4 lines
For example, I wonder how we'd be handling stuff like
do_numa_page()cleanly correctly, where we usepte_modify()+pte_mkwrite(), and either
call might set the PTE writable and maintain dirty bit ...
pte_modify() is handled like this currently:
https://lore.kernel.org/lkml/[email protected]/
There has been a couple iterations on that. The current solution is to do the Dirty->SavedDirty fixup if needed after the new prots are added.
Of course pte_modify() can't know whether you are are attempting to create a shadow stack PTE with the prot you are passing in. But the callers today explicitly call pte_mkwrite() after filling in the other bits with pte_modify(). Today this patch causes the pte_mkwrite() to be skipped and another fault may be required in the mprotect() and numa cases, but if we change pte_mkwrite() to take a VMA we can just make it shadow stack to start.
It might be worth mentioning, there was a suggestion in the past to try to have the shadow stack bits come out of vm_get_page_prot(), but MM code would then try to map the zero page as (shadow stack) writable when there was a normal (non-shadow stack) read access. So I had to abandon that approach and rely on explicit calls to pte_mkwrite/shstk() to make it shadow stack.
Quoted 18 lines
Having that said, maybe it could work with only a single saved-dirty bit
and passing in the VMA forpte_mkwrite()only.
pte_wrprotect()would detect "writable=0,dirty=1" and move the dirty bit
to the soft-dirty bit instead, resulting in "writable=0,dirty=0,saved-dirty=1",
pte_dirty()would return dirty==1||saved-dirty==1.
pte_mkdirty()would set either set dirty=1 or saved-dirty=1, depending
on the writable bit.
pte_mkclean()would clean both bits.
Quoted 7 lines
pte_write()would detect "writable == 1 || (writable==0 && dirty==1)"
pte_mkwrite()would act according to the VMA, and in addition, merge the
saved-dirty bit into the dirty bit.
pte_modify()andmk_pte().... would require more thought ...
Not sure I'm following what the mk_pte() problem would be. You mean if Write=0,Dirty=1 is manually added to the prot?
Shouldn't people generally use the pte_mkwrite() helpers unless they are drawing from a prot that was already created with the helpers or vm_get_page_prot()? I think they can't manually create prot's from bits in core mm code, right? And x86 arch code already has to be aware of shadow stack. It's a bit of an assumption I guess, but I think maybe not too crazy of one?
Quoted 5 lines
Further,
ptep_modify_prot_commit()might have to be adjusted to properly
flush in all relevant cases IIRC.
Sorry, I'm not following. Can you elaborate? There is an adjustment made in pte_flags_need_flush().
Quoted 12 lines
Quoted 11 lines
x86's
pte_mkwrite()would then be pretty close tomaybe_mkwrite(), but
maybe it could additionally warn if the vma is not writable. It also
seems more aligned with your changes to stop taking hints from PTE bits
and just look at the VMA? (I'm thinking about the dropping of the dirty
check in GUP and droppingpte_saved_write())
Quoted 5 lines
The soft-shstk bit wouldn't be a hint, it would be logically changing
the "type" of the PTE such that any other PTE functions can do the right
thing without having to consume the VMA.
Yea, true.
Thanks for your comments and ideas here, I'll give the:pte_t pte_mkwrite(struct vm_area_struct *vma, pte_t pte) ...solution a try.
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.
Well, it turns out there are some pte_mkwrite() callers in other arch's that operate on kernel memory and don't have a VMA. So it needed a new function that can be overridden in arch code. I ended up with x86 versions of these, like this:pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
{
if (!(vma->vm_flags & VM_WRITE))
return pte;
if (vma->vm_flags & VM_SHADOW_STACK)
return pte_mkwrite_shstk(pte);
return pte_mkwrite(pte);
}pte_t pte_mkwrite_vma(pte_t pte, struct vm_area_struct *vma)
{
if (vma->vm_flags & VM_SHADOW_STACK)
return pte_mkwrite_shstk(pte);
return pte_mkwrite(pte);
}#ifdef CONFIG_TRANSPARENT_HUGEPAGEpmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
{
if (!(vma->vm_flags & VM_WRITE))
return pmd;
if (vma->vm_flags & VM_SHADOW_STACK)
return pmd_mkwrite_shstk(pmd);
return pmd_mkwrite(pmd);
}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
All the other pte_mkdirty()s, etc remain the same.
Previously, there was a suggestion to not override the maybe_mkwrite()'s and put the logic in core MM by having a generic version of pte_mkwrite_shstk() that does nothing. But given what we are trying to do with pte_mkwrite_vma() it seemed better to hide all the shadow stack PTE changes in arch code again.
After the changes, the only shadow stack specific bits in core mm are the bit in GUP to require FOLL_FORCE, the memory accounting, and these warnings:
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 7 lines · Rick P
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.Well, it turns out there are some
pte_mkwrite()callers in other arch's that operate on kernel memory and don't have a VMA. So it needed a new
Why not pass in NULL as VMA then and document the semantics? The less similarly named but slightly different functions, the better :)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Rick P
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.
Quoted 8 lines
Quoted 4 lines
Well, it turns out there are some
pte_mkwrite()callers in other arch's
that operate on kernel memory and don't have a VMA. So it needed a newWhy not pass in NULL as VMA then and document the semantics? The less
similarly named but slightly different functions, the better :)
Hmm. The x86 and generic versions should probably have the same semantics, so then if you pass a NULL, it would do a regular pte_mkwrite() I guess?
I see another benefit of requiring the vma argument, such that raw pte_mkwrite()s are less likely to appear in core MM code. But I think the NULL is awkward because it's not obvious, to me at least, what the implications of that should be.
So it will be confusing to read in the NULL cases for the other archs. We also have some warnings to catch miss cases in the PTE tear down code, so the scenario of new code accidentally marking shadow stack PTEs as writable is not totally unchecked.
The three functions that do slightly different things are:
pte_mkwrite():
Makes a PTE conventionally writable, only takes a PTE. Very clear that it is a low level helper and what it does.
maybe_mkwrite():
Might make a PTE writable if the VMA allows it.
pte_mkwrite_vma():
Makes a PTE writable in a specific way depending on the VMA
I wonder if the name pte_mkwrite_vma() is maybe just not clear enough. It takes a VMA, yes, but what does it do with it?
What if it was called pte_mkwrite_type() instead? Some arch's have additional types of writable memory and this function creates them. Of course they also have the normal type of writable memory, and pte_mkwrite() creates that like usual. Doesn't it seem more readable?
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 42 lines · Rick P
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Rick P
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.Quoted 8 lines
Quoted 4 lines
Well, it turns out there are some
pte_mkwrite()callers in other arch's
that operate on kernel memory and don't have a VMA. So it needed a newWhy not pass in NULL as VMA then and document the semantics? The less
similarly named but slightly different functions, the better :)Hmm. The x86 and generic versions should probably have the same semantics, so then if you pass a NULL, it would do a regular
pte_mkwrite()I guess?I see another benefit of requiring the vma argument, such that raw pte_mkwrite()s are less likely to appear in core MM code. But I think the NULL is awkward because it's not obvious, to me at least, what the implications of that should be.
So it will be confusing to read in the NULL cases for the other archs. We also have some warnings to catch miss cases in the PTE tear down code, so the scenario of new code accidentally marking shadow stack PTEs as writable is not totally unchecked.
The three functions that do slightly different things are:
pte_mkwrite():
Makes a PTE conventionally writable, only takes a PTE. Very clear that it is a low level helper and what it does.
maybe_mkwrite():
Might make a PTE writable if the VMA allows it.
pte_mkwrite_vma():
Makes a PTE writable in a specific way depending on the VMA
Quoted 7 lines
I wonder if the name
pte_mkwrite_vma()is maybe just not clear enough. It takes a VMA, yes, but what does it do with it?What if it was called
pte_mkwrite_type()instead? Some arch's have additional types of writable memory and this function creates them. Of course they also have the normal type of writable memory, andpte_mkwrite()creates that like usual. Doesn't it seem more readable?
The issue is, the more variants we provide the easier it is to make mistakes and introduce new buggy code.
It's tempting to simply use pte_mkwrite() and call it a day, where people actually should use pte_mkwrite_vma().
Then, they at least have to investigate what to do about the second VMA parameter.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 63 lines · David Hildenbrand
Quoted 52 lines · Rick P
Quoted 13 lines · David Hildenbrand
Quoted 11 lines · Rick P
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.Well, it turns out there are some
pte_mkwrite()callers in other
arch's
that operate on kernel memory and don't have a VMA. So it needed a
newQuoted 3 lines
Why not pass in NULL as VMA then and document the semantics? The less
similarly named but slightly different functions, the better :)Hmm. The x86 and generic versions should probably have the same semantics, so then if you pass a NULL, it would do a regular
pte_mkwrite()I guess?I see another benefit of requiring the vma argument, such that raw pte_mkwrite()s are less likely to appear in core MM code. But I think
the NULL is awkward because it's not obvious, to me at least, what the
implications of that should be.So it will be confusing to read in the NULL cases for the other archs.
We also have some warnings to catch miss cases in the PTE tear down code, so the scenario of new code accidentally marking shadow stack PTEs as writable is not totally unchecked.The three functions that do slightly different things are:
pte_mkwrite():
Makes a PTE conventionally writable, only takes a PTE. Very clear that
it is a low level helper and what it does.
maybe_mkwrite():
Might make a PTE writable if the VMA allows it.
pte_mkwrite_vma():
Makes a PTE writable in a specific way depending on the VMAI wonder if the name
pte_mkwrite_vma()is maybe just not clear enough.
It takes a VMA, yes, but what does it do with it?Quoted 6 lines
What if it was called
pte_mkwrite_type()instead? Some arch's have additional types of writable memory and this function creates them. Of
course they also have the normal type of writable memory, andpte_mkwrite()creates that like usual. Doesn't it seem more readable?The issue is, the more variants we provide the easier it is to make mistakes and introduce new buggy code.
Quoted 6 lines
It's tempting to simply use
pte_mkwrite()and call it a day, where people actually should usepte_mkwrite_vma().Then, they at least have to investigate what to do about the second VMA
parameter.
Ok, I'll give it a spin. So far it looks ok. The downside is the giant tree-wide pte_mkwrite() signature change, but once that is over with there are other advantages. Like getting rid of maybe_mkwrite()'s awareness of shadow stack so the logic is more centralized. Please let me know if you don't feel comfortable with a suggested-by credit tag.
Thanks,
Rick
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 71 lines · Rick P
Quoted 63 lines · David Hildenbrand
Quoted 52 lines · Rick P
Quoted 13 lines · David Hildenbrand
Quoted 11 lines · Rick P
Quoted 3 lines · Rick Edgecombe
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.Well, it turns out there are some
pte_mkwrite()callers in other
arch's
that operate on kernel memory and don't have a VMA. So it needed a
newQuoted 3 lines
Why not pass in NULL as VMA then and document the semantics? The less
similarly named but slightly different functions, the better :)Hmm. The x86 and generic versions should probably have the same semantics, so then if you pass a NULL, it would do a regular
pte_mkwrite()I guess?I see another benefit of requiring the vma argument, such that raw pte_mkwrite()s are less likely to appear in core MM code. But I think
the NULL is awkward because it's not obvious, to me at least, what the
implications of that should be.So it will be confusing to read in the NULL cases for the other archs.
We also have some warnings to catch miss cases in the PTE tear down code, so the scenario of new code accidentally marking shadow stack PTEs as writable is not totally unchecked.The three functions that do slightly different things are:
pte_mkwrite():
Makes a PTE conventionally writable, only takes a PTE. Very clear that
it is a low level helper and what it does.
maybe_mkwrite():
Might make a PTE writable if the VMA allows it.
pte_mkwrite_vma():
Makes a PTE writable in a specific way depending on the VMAI wonder if the name
pte_mkwrite_vma()is maybe just not clear enough.
It takes a VMA, yes, but what does it do with it?Quoted 6 lines
What if it was called
pte_mkwrite_type()instead? Some arch's have additional types of writable memory and this function creates them. Of
course they also have the normal type of writable memory, andpte_mkwrite()creates that like usual. Doesn't it seem more readable?The issue is, the more variants we provide the easier it is to make mistakes and introduce new buggy code.
Quoted 6 lines
It's tempting to simply use
pte_mkwrite()and call it a day, where people actually should usepte_mkwrite_vma().Then, they at least have to investigate what to do about the second VMA
parameter.
Quoted 5 lines
Ok, I'll give it a spin. So far it looks ok. The downside is the giant tree-wide
pte_mkwrite()signature change, but once that is over with there are other advantages. Like getting rid of maybe_mkwrite()'s awareness of shadow stack so the logic is more centralized. Please let me know if you don't feel comfortable with a suggested-by credit tag.
Sure ...
but I reconsidered :)
Maybe there is a cleaner way to do it and avoid the "NULL" argument.
What about having (while you're going over everything already):
pte_mkwrite(pte, vma)pte_mkwrite_kernel(pte)
The latter would only be used in that arch code where we're working on kernel pgtables. We already have pte_offset_kernel() and pte_alloc_kernel_track(), so it's not too weird.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 8 lines · David Hildenbrand
Sure ...
but I reconsidered :)
Maybe there is a cleaner way to do it and avoid the "NULL" argument.
What about having (while you're going over everything already):
Quoted 7 lines
pte_mkwrite(pte, vma)pte_mkwrite_kernel(pte)The latter would only be used in that arch code where we're working on
kernel pgtables. We already havepte_offset_kernel()andpte_alloc_kernel_track(), so it's not too weird.
Hmm, one downside is the "mk" part might lead people to guess pte_mkwrite_kernel() would make it writable AND a kernel page (like U/S=0 on x86). Instead of being a mkwrite() that's useful for setting on kernel PTEs.
The other problem is that one of NULL passers is not for kernel memory. huge_pte_mkwrite() calls pte_mkwrite(). Shadow stack memory can't be created with MAP_HUGETLB, so it is not needed. Using pte_mkwrite_kernel() would look weird in this case, but making huge_pte_mkwrite() take a VMA would be for no reason. Maybe making huge_pte_mkwrite() take a VMA is the better of those two options. Or keep the NULL semantics... Any thoughts?
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 17 lines · Rick P
Quoted 8 lines · David Hildenbrand
Sure ...
but I reconsidered :)
Maybe there is a cleaner way to do it and avoid the "NULL" argument.
What about having (while you're going over everything already):
Quoted 7 lines
pte_mkwrite(pte, vma)pte_mkwrite_kernel(pte)The latter would only be used in that arch code where we're working on
kernel pgtables. We already havepte_offset_kernel()andpte_alloc_kernel_track(), so it's not too weird.
Quoted 4 lines
Hmm, one downside is the "mk" part might lead people to guess
pte_mkwrite_kernel()would make it writable AND a kernel page (like U/S=0 on x86). Instead of being amkwrite()that's useful for setting on kernel PTEs.
At least I wouldn't worry about that too much. We handle nowhere in common code user vs. supervisor access that way explicitly (e.g., mkkernel), and it wouldn't even apply on architectures where we cannot make such a decision on a per-PTE basis.
Quoted 8 lines
The other problem is that one of NULL passers is not for kernel memory.
huge_pte_mkwrite()callspte_mkwrite(). Shadow stack memory can't be created withMAP_HUGETLB, so it is not needed. Usingpte_mkwrite_kernel()would look weird in this case, but makinghuge_pte_mkwrite()take a VMA would be for no reason. Maybe makinghuge_pte_mkwrite()take a VMA is the better of those two options. Or keep the NULL semantics... Any thoughts?
Well, the reason would be consistency. From a core-mm point of view it makes sense to handle this all consistency, even if the single user (x86) wouldn't strictly require it right now.
I'd just pass in the VMA and call it a day :)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
The other problem is that one of NULL passers is not for kernel memory.
huge_pte_mkwrite()callspte_mkwrite(). Shadow stack memory can't be
created withMAP_HUGETLB, so it is not needed. Usingpte_mkwrite_kernel()would look weird in this case, but makinghuge_pte_mkwrite()take a VMA would be for no reason. Maybe makinghuge_pte_mkwrite()take a VMA is the better of those two options. Or
keep the NULL semantics... Any thoughts?
Quoted 6 lines
Well, the reason would be consistency. From a core-mm point of view it
makes sense to handle this all consistency, even if the single user (x86) wouldn't strictly require it right now.I'd just pass in the VMA and call it a day :)
Ok, I'll give it a spin.
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 13 lines · Rick P
Quoted 12 lines · David Hildenbrand
Quoted 11 lines
The other problem is that one of NULL passers is not for kernel memory.
huge_pte_mkwrite()callspte_mkwrite(). Shadow stack memory can't be
created withMAP_HUGETLB, so it is not needed. Usingpte_mkwrite_kernel()would look weird in this case, but makinghuge_pte_mkwrite()take a VMA would be for no reason. Maybe makinghuge_pte_mkwrite()take a VMA is the better of those two options. Or
keep the NULL semantics... Any thoughts?
Quoted 8 lines
Quoted 6 lines
Well, the reason would be consistency. From a core-mm point of view it
makes sense to handle this all consistency, even if the single user (x86) wouldn't strictly require it right now.I'd just pass in the VMA and call it a day :)
Ok, I'll give it a spin.
It would be good to get more opinions on that, but I'm afraid we won't get more deep down in this thread :)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
In reply to Edgecombe, Rick PQuoted 66 lines · Rick P
Quoted 56 lines · David Hildenbrand
Quoted 43 lines
Quoted 23 lines
Roughly speaking: if we abstract it that way and get all of the "how
to
set it writable now?" out of core-MM, it not only is cleaner and less
error prone, it might even allow other architectures that implement
something comparable (e.g., using a dedicated HW bit) to actually reuse
some of that work. Otherwise most of that "shstk" is really just x86
specific ...I guess the only cases we have to special case would be page pinning
code wherepte_write()would indicate that the PTE is writable (well,
it
is, just not by "ordinary CPU instruction" context directly): but you
do
that already, so ... :)Quoted 3 lines
Sorry for stumbling over that this late, I only started looking into
this when you CCed me on that one patch.Sorry for not calling more attention to it earlier. Appreciate your comments.
Previously versions of this series had changed some of these
pte_mkwrite()calls tomaybe_mkwrite(), which of course takes a vma.
This way an x86 implementation could use theVM_SHADOW_STACKvma flag
to decide betweenpte_mkwrite()andpte_mkwrite_shstk(). The feedback
was that in some of these code paths "maybe" isn't really an option, it
*needs* to make it writable. Even though the logic was the same, the
name of the function made it look wrong.Quoted 5 lines
But another option could be to change
pte_mkwrite()to take a vma. This
would save using another software bit on x86, but instead requires a
small change to each arch'spte_mkwrite().I played with that idea shortly as well, but discarded it. I was not able to convince myself that it wouldn't be required to pass in the VMA
as well for things likepte_dirty(),pte_mkdirty(),pte_write(), ... which would end up fairly ugly (or even impossible in thing slike GUP-fast).Quoted 4 lines
For example, I wonder how we'd be handling stuff like
do_numa_page()cleanly correctly, where we usepte_modify()+pte_mkwrite(), and either
call might set the PTE writable and maintain dirty bit ...
pte_modify()is handled like this currently:
Quoted 7 lines
There has been a couple iterations on that. The current solution is to do the
Dirty->SavedDirtyfixup if needed after the new prots are added.Of course
pte_modify()can't know whether you are are attempting to create a shadow stack PTE with the prot you are passing in. But the callers today explicitly callpte_mkwrite()after filling in the other bits withpte_modify().
See below on my MAP_PRIVATE vs. MAP_SHARED comment.
Quoted 11 lines
Today this patch causes the
pte_mkwrite()to be
skipped and another fault may be required in themprotect()and numa cases, but if we changepte_mkwrite()to take a VMA we can just make it shadow stack to start.It might be worth mentioning, there was a suggestion in the past to try to have the shadow stack bits come out of
vm_get_page_prot(), but MM code would then try to map the zero page as (shadow stack) writable when there was a normal (non-shadow stack) read access. So I had to abandon that approach and rely on explicit calls to pte_mkwrite/shstk() to make it shadow stack.
Thanks, do you have a pointer?
Quoted 25 lines
Quoted 25 lines
Having that said, maybe it could work with only a single saved-dirty bit
and passing in the VMA forpte_mkwrite()only.
pte_wrprotect()would detect "writable=0,dirty=1" and move the dirty bit
to the soft-dirty bit instead, resulting in "writable=0,dirty=0,saved-dirty=1",
pte_dirty()would return dirty==1||saved-dirty==1.
pte_mkdirty()would set either set dirty=1 or saved-dirty=1, depending
on the writable bit.
pte_mkclean()would clean both bits.
pte_write()would detect "writable == 1 || (writable==0 && dirty==1)"
pte_mkwrite()would act according to the VMA, and in addition, merge the
saved-dirty bit into the dirty bit.
Quoted 8 lines
Quoted 1 line
pte_modify()andmk_pte().... would require more thought ...Not sure I'm following what the
mk_pte()problem would be. You mean if Write=0,Dirty=1 is manually added to the prot?Shouldn't people generally use the
pte_mkwrite()helpers unless they are drawing from a prot that was already created with the helpers orvm_get_page_prot()?
pte_mkwrite() is mostly only used (except for writenotify ...) for MAP_PRIVATE memory ("COW-able"). For MAP_SHARED memory, vma->vm_page_prot in a VM_WRITE mapping already contains the write permissions. pte_mkwrite() is not necessary (again, unless writenotify is active).
I assume shstk VMAs don't apply to MAP_SHARED VMAs, which is why you didn't stumble over that issue yet? Because I don't see how it could work with MAP_SHARED VMAs.
The other thing I had in mind was that we have to make sure that we're not accidentally setting "Write=0,Dirty=1" in mk_pte() / pte_modify().
Assume we had a "Write=1,Dirty=1" PTE, and we effectively wrprotect using pte_modify(), we have to make sure to move the dirty bit to the saved_dirty bit.
Quoted 4 lines
I think they can't manually create prot's from bits
in core mm code, right? And x86 arch code already has to be aware of shadow stack. It's a bit of an assumption I guess, but I think maybe not too crazy of one?
I think that's true. Arch code is supposed to deal with that IIRC.
Quoted 9 lines
Quoted 6 lines
Further,
ptep_modify_prot_commit()might have to be adjusted to properly
flush in all relevant cases IIRC.Sorry, I'm not following. Can you elaborate? There is an adjustment made in
pte_flags_need_flush().
Note that I did not fully review all bits of this patch set, just throwing out what was on my mind. If already handled, great.
Quoted 19 lines
Quoted 13 lines
Quoted 12 lines
x86's
pte_mkwrite()would then be pretty close tomaybe_mkwrite(), but
maybe it could additionally warn if the vma is not writable. It also
seems more aligned with your changes to stop taking hints from PTE bits
and just look at the VMA? (I'm thinking about the dropping of the dirty
check in GUP and droppingpte_saved_write())Quoted 5 lines
The soft-shstk bit wouldn't be a hint, it would be logically changing
the "type" of the PTE such that any other PTE functions can do the right
thing without having to consume the VMA.
Quoted 5 lines
Yea, true.
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.
Good!
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 79 lines · David Hildenbrand
Quoted 78 lines · Rick P
Quoted 76 lines · David Hildenbrand
Quoted 53 lines
Quoted 29 lines
Roughly speaking: if we abstract it that way and get all of the
"how
to
set it writable now?" out of core-MM, it not only is cleaner and
less
error prone, it might even allow other architectures that implement
something comparable (e.g., using a dedicated HW bit) to actually
reuse
some of that work. Otherwise most of that "shstk" is really just
x86
specific ...I guess the only cases we have to special case would be page pinning
code wherepte_write()would indicate that the PTE is writable
(well,
it
is, just not by "ordinary CPU instruction" context directly):
but
you
do
that already, so ... :)Quoted 4 lines
Sorry for stumbling over that this late, I only started looking
into
this when you CCed me on that one patch.Sorry for not calling more attention to it earlier. Appreciate your
comments.Previously versions of this series had changed some of these
pte_mkwrite()calls tomaybe_mkwrite(), which of course takes a vma.
This way an x86 implementation could use theVM_SHADOW_STACKvma
flag
to decide betweenpte_mkwrite()andpte_mkwrite_shstk(). The feedback
was that in some of these code paths "maybe" isn't really an option, it
*needs* to make it writable. Even though the logic was the same,
the
name of the function made it look wrong.Quoted 7 lines
But another option could be to change
pte_mkwrite()to take a vma.
This
would save using another software bit on x86, but instead requires
a
small change to each arch'spte_mkwrite().I played with that idea shortly as well, but discarded it. I was not
able to convince myself that it wouldn't be required to pass in the
VMA
as well for things likepte_dirty(),pte_mkdirty(),pte_write(),
...
which would end up fairly ugly (or even impossible in thing slike GUP-fast).For example, I wonder how we'd be handling stuff like
do_numa_page()
cleanly correctly, where we usepte_modify()+pte_mkwrite(), and either
call might set the PTE writable and maintain dirty bit ...
Quoted 3 lines
Quoted 3 lines
pte_modify()is handled like this currently:
https://lore.kernel.org/lkml/[email protected]/
Quoted 6 lines
Quoted 6 lines
There has been a couple iterations on that. The current solution is to
do theDirty->SavedDirtyfixup if needed after the new prots are added.
Quoted 7 lines
Quoted 5 lines
Of course
pte_modify()can't know whether you are are attempting to create a shadow stack PTE with the prot you are passing in. But the callers today explicitly callpte_mkwrite()after filling in the other
bits withpte_modify().See below on my
MAP_PRIVATEvs.MAP_SHAREDcomment.
Yep, MAP_SHARED support was dropped with the reboot of the series. It did have some problems IIRC.
Now shadow stack memory creation is tightly controlled. Either created via special syscall or automatically with a new thread.
Quoted 18 lines
Quoted 8 lines
Today this patch causes the
pte_mkwrite()to be skipped and another fault may be required in themprotect()and numa
cases, but if we changepte_mkwrite()to take a VMA we can just make it
shadow stack to start.Quoted 9 lines
It might be worth mentioning, there was a suggestion in the past to try
to have the shadow stack bits come out ofvm_get_page_prot(), but MM
code would then try to map the zero page as (shadow stack) writable when there was a normal (non-shadow stack) read access. So I had to abandon that approach and rely on explicit calls to pte_mkwrite/shstk()
to make it shadow stack.
Quoted 1 line
Thanks, do you have a pointer?
I never posted it because it didn't work out. This was the comment that prompted the exploration in that direction:
https://lore.kernel.org/lkml/[email protected]/
Shadow stack memory also used to not be VM_WRITE (VM_SHADOW_STACK only), but this was changed for other reasons. In v2 there were some updates to how shadow stack memory was handled, and the cover letter had a writeup of the reasons and general design:
https://lore.kernel.org/lkml/[email protected]/
Quoted 42 lines
Quoted 36 lines
Quoted 25 lines
Having that said, maybe it could work with only a single saved- dirty
bit
and passing in the VMA forpte_mkwrite()only.
pte_wrprotect()would detect "writable=0,dirty=1" and move the dirty
bit
to the soft-dirty bit instead, resulting in "writable=0,dirty=0,saved-dirty=1",
pte_dirty()would return dirty==1||saved-dirty==1.
pte_mkdirty()would set either set dirty=1 or saved-dirty=1, depending
on the writable bit.
pte_mkclean()would clean both bits.
pte_write()would detect "writable == 1 || (writable==0 && dirty==1)"Quoted 6 lines
pte_mkwrite()would act according to the VMA, and in addition, merge
the
saved-dirty bit into the dirty bit.
pte_modify()andmk_pte().... would require more thought ...Not sure I'm following what the
mk_pte()problem would be. You mean if
Write=0,Dirty=1 is manually added to the prot?Quoted 5 lines
Shouldn't people generally use the
pte_mkwrite()helpers unless they
are drawing from a prot that was already created with the helpers orvm_get_page_prot()?
Quoted 6 lines
pte_mkwrite()is mostly only used (except for writenotify ...) forMAP_PRIVATEmemory ("COW-able"). ForMAP_SHAREDmemory,vma->vm_page_protin aVM_WRITEmapping already contains the write permissions.pte_mkwrite()is not necessary (again, unless writenotify
is active).
Oh, interesting.
Quoted 4 lines
I assume shstk VMAs don't apply to
MAP_SHAREDVMAs, which is why you didn't stumble over that issue yet? Because I don't see how it could work withMAP_SHAREDVMAs.
Yep, it doesn't support MAP_SHARED.
Quoted 11 lines
The other thing I had in mind was that we have to make sure that we're
not accidentally setting "Write=0,Dirty=1" inmk_pte()/pte_modify().Assume we had a "Write=1,Dirty=1" PTE, and we effectively wrprotect using
pte_modify(), we have to make sure to move the dirty bit to thesaved_dirtybit.
For the mk_pte() case, I don't think a Write=0,Dirty=1 prot could come from anywhere. I guess the MAP_SHARED case is a little less bounded. We could maybe add a warning for this case.
For the pte_modify() case, this does happen. There are two scenarios considered:
1. A Write=0,Dirty=0 PTE is made dirty. This can't happen today as Dirty is filtered via _PAGE_CHG_MASK. Basically pte_modify() doesn't support it.
2. A Write=1,Dirty=1 PTE gets write protected. This does happen because the Write=0 prot comes from protection_map, and pte_modify() would leave the Dirty=1 bit alone. The main case I know of is mprotect(). It is handled by changes to pte_modify() by doing the Dirty->SoftDirty fixup if needed.
So pte_modify()s job should not be too tricky. What you can't do with it though, is create shadow stack PTEs. But it is ok for our uses because of the explicit mkwrite().
Quoted 54 lines
Quoted 7 lines
I think they can't manually create prot's from bits
in core mm code, right? And x86 arch code already has to be aware of
shadow stack. It's a bit of an assumption I guess, but I think maybe
not too crazy of one?I think that's true. Arch code is supposed to deal with that IIRC.
Quoted 9 lines
Quoted 6 lines
Further,
ptep_modify_prot_commit()might have to be adjusted to properly
flush in all relevant cases IIRC.Sorry, I'm not following. Can you elaborate? There is an adjustment made in
pte_flags_need_flush().Note that I did not fully review all bits of this patch set, just throwing out what was on my mind. If already handled, great.
Quoted 23 lines
Quoted 16 lines
Quoted 15 lines
x86's
pte_mkwrite()would then be pretty close tomaybe_mkwrite(),
but
maybe it could additionally warn if the vma is not writable. It also
seems more aligned with your changes to stop taking hints from PTE
bits
and just look at the VMA? (I'm thinking about the dropping of the
dirty
check in GUP and droppingpte_saved_write())Quoted 6 lines
The soft-shstk bit wouldn't be a hint, it would be logically changing
the "type" of the PTE such that any other PTE functions can do the
right
thing without having to consume the VMA.Quoted 5 lines
Yea, true.
Thanks for your comments and ideas here, I'll give the:
pte_tpte_mkwrite(struct vm_area_struct *vma, pte_t pte)...solution a try.Good!
Re: [PATCH v5 18/39] mm: Handle faultless write upgrades for shstk
Quoted 3 lines
Now shadow stack memory creation is tightly controlled. Either created via special syscall or automatically with a new thread.
Good, it would be valuable to document that somewhere ("Neve rapplies to VM_SHARED|VM_MAYSHARE VMAs").
[...]
Quoted 6 lines
Quoted 6 lines
The other thing I had in mind was that we have to make sure that we're
not accidentally setting "Write=0,Dirty=1" inmk_pte()/pte_modify().
Quoted 8 lines
Quoted 4 lines
Assume we had a "Write=1,Dirty=1" PTE, and we effectively wrprotect using
pte_modify(), we have to make sure to move the dirty bit to thesaved_dirtybit.For the
mk_pte()case, I don't think a Write=0,Dirty=1 prot could come from anywhere. I guess theMAP_SHAREDcase is a little less bounded. We could maybe add a warning for this case.
Right, Write=0,Dirty=1 shouldn't apply at that point if shstk are always wrprotected as default.
Quoted 6 lines
For the
pte_modify()case, this does happen. There are two scenarios considered:
1. A Write=0,Dirty=0 PTE is made dirty. This can't happen today as Dirty is filtered via_PAGE_CHG_MASK. Basicallypte_modify()doesn't support it.
It should simply set the saved_dirty bit I guess. But I don't think pte_modify() is actually supposed to set PTEs dirty (primary goal is to change protection IIRC).
Quoted 5 lines
2. A Write=1,Dirty=1 PTE gets write protected. This does happen because the Write=0 prot comes from
protection_map, andpte_modify()would leave the Dirty=1 bit alone. The main case I know of ismprotect(). It is handled by changes topte_modify()by doing theDirty->SoftDirtyfixup if needed.
Right, we'd have to move the dirty bit to the saved_dirty bit. (we have to handle soft-dirty, too, whenever setting the PTE dirty -- either via the dirty bit or via the saved_dirty bit)
Quoted 4 lines
So pte_modify()s job should not be too tricky. What you can't do with it though, is create shadow stack PTEs. But it is ok for our uses because of the explicit
mkwrite().
I think you are correct.
Signature · 3 lines
Thanks, David / dhildenb
PATCH v5 19/39 mm: Fixup places that call pte_mkwrite() directly
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.
In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite() or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places where pte_mkwrite() is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.
- do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(). Teach it about pte_mkwrite_shstk()
- When userfaultfd is creating a PTE after userspace handles the fault it calls pte_mkwrite() directly. Teach it about pte_mkwrite_shstk()
To make the code cleaner, introduce is_shstk_write() which simplifies checking for VM_WRITE | VM_SHADOW_STACK together.
In other cases where pte_mkwrite() is called directly, the VMA will not be VM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case of pte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization in mprotect(), shadow stack VMA's won't be VM_SHARED, so it is not necessary.
---
v5:
- Fix typo in commit log
v3:
- Restore do_anonymous_page() that accidetally moved commits (Kirill)
- Open code maybe_mkwrite() cases from v2, so the behavior doesn't change to mark that non-writable PTEs dirty. (Nadav)
v2:
- Updated commit log with comment's from Dave Hansen
- Dave also suggested (I understood) to maybe tweak vm_get_page_prot() to avoid having to call maybe_mkwrite(). After playing around with this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be writable ended up mapping the zero page as writable and other surprises. So creating shadow stack memory needs to be done with manual logic like pte_mkwrite().
- Drop change in change_pte_range() because it couldn't actually trigger for shadow stack VMAs.
- Clarify reasoning for skipped cases of pte_mkwrite().
Yu-cheng v25:
- Apply same changes to do_huge_pmd_numa_page() as to do_numa_page().
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 7 +++++++ mm/memory.c | 5 ++++- mm/migrate_device.c | 4 +++- mm/userfaultfd.c | 10 +++++++--- 6 files changed, 30 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable.h +3 -0
@@ -951,6 +951,9 @@ static inline pgd_t pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) }#endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#define is_shstk_write is_shstk_write +extern bool is_shstk_write (unsigned long vm_flags); +#endif /* __ASSEMBLY__ */
arch/x86/mm/pgtable.c +6 -0
@@ -880,3 +880,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +bool is_shstk_write (unsigned long vm_flags) +{ +return (vm_flags & (VM_SHADOW_STACK |VM_WRITE )) == + (VM_SHADOW_STACK |VM_WRITE ); +}
include/linux/pgtable.h +7 -0
@@ -1578,6 +1578,13 @@ static inline bool arch_has_pfn_modify_check(void) }#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shstk_write +static inline bool is_shstk_write (unsigned long vm_flags) +{ +return false; +} +#endif +/* * Architecture PAGE_KERNEL_* fallbacks *
mm/memory.c +4 -1
@@ -4088,7 +4088,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot); entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) + +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
mm/migrate_device.c +3 -1
@@ -645,7 +645,9 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, goto abort; } entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); }
mm/userfaultfd.c +7 -3
@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,int ret;pte_t _dst_pte, *dst_pte;bool writable = dst_vma->vm_flags &VM_WRITE ; +bool shstk = dst_vma->vm_flags &VM_SHADOW_STACK ;bool vm_shared = dst_vma->vm_flags &VM_SHARED ;bool page_in_cache =page_mapping (page);spinlock_t *ptl;
@@ -84,9 +85,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, writable = false; } -if (writable) - _dst_pte =pte_mkwrite (_dst_pte); -else +if (writable) { +if (shstk) + _dst_pte =pte_mkwrite_shstk (_dst_pte); +else + _dst_pte =pte_mkwrite (_dst_pte); + }else /* * We need this to make sure write bit removed; as mk_pte() * could return a pte with write bit set.
Signature · 1 line
2.17.1
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
Quoted 31 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.
In past patches,
maybe_mkwrite()has been updated to applypte_mkwrite()orpte_mkwrite_shstk()depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places wherepte_mkwrite()is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.-
do_anonymous_page()andmigrate_vma_insert_page()checkVM_WRITEdirectly and callpte_mkwrite(). Teach it aboutpte_mkwrite_shstk()- When userfaultfd is creating a PTE after userspace handles the fault it calls
pte_mkwrite()directly. Teach it aboutpte_mkwrite_shstk()To make the code cleaner, introduce
is_shstk_write()which simplifies checking forVM_WRITE|VM_SHADOW_STACKtogether.In other cases where
pte_mkwrite()is called directly, the VMA will not beVM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case ofpte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization inmprotect(), shadow stack VMA's won't beVM_SHARED, so it is not necessary.
Quoted 3 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
In reply to Rick EdgecombeSince I've a general question on outcome of discussion of how to handle pte_mkwrite, so I am top posting.
I have posted patches yesterday targeting riscv zisslpcfi extension.
https://lore.kernel.org/lkml/[email protected]/
Since there're similarities in extension(s), patches have similarity too. One of the similarity was updating maybe_mkwrite. I was asked (by dhildenb on my patch #11) to look at x86 approach on how to approach this so that core-mm approach fits multiple architectures along with the need to update pte_mkwrite to consume vma flags.
In x86 CET patch series, I see that locations where pte_mkwrite is invoked are updated to check for shadow stack vma and not necessarily pte_mkwrite itself is updated to consume vma flags. Let me know if my understanding is correct and that's the current direction (to update call sites for vma check where pte_mkwrite is invoked)
Being said that as I've mentioned in my patch series that there're similarities between x86, arm and now riscv for implementing shadow stack and indirect branch tracking, overall it'll be a good thing if we can collaborate and come up with common bits.
Rest inline.
Quoted 90 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.
In past patches,
maybe_mkwrite()has been updated to applypte_mkwrite()orpte_mkwrite_shstk()depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places wherepte_mkwrite()is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.-
do_anonymous_page()andmigrate_vma_insert_page()checkVM_WRITEdirectly and callpte_mkwrite(). Teach it aboutpte_mkwrite_shstk()- When userfaultfd is creating a PTE after userspace handles the fault it calls
pte_mkwrite()directly. Teach it aboutpte_mkwrite_shstk()To make the code cleaner, introduce
is_shstk_write()which simplifies checking forVM_WRITE|VM_SHADOW_STACKtogether.In other cases where
pte_mkwrite()is called directly, the VMA will not beVM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case ofpte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization inmprotect(), shadow stack VMA's won't beVM_SHARED, so it is not necessary.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v5:
- Fix typo in commit logv3:
- Restoredo_anonymous_page()that accidetally moved commits (Kirill)
- Open codemaybe_mkwrite()cases from v2, so the behavior doesn't change to mark that non-writable PTEs dirty. (Nadav)v2:
- Updated commit log with comment's from Dave Hansen
- Dave also suggested (I understood) to maybe tweakvm_get_page_prot()to avoid having to callmaybe_mkwrite(). After playing around with this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be writable ended up mapping the zero page as writable and other surprises. So creating shadow stack memory needs to be done with manual logic likepte_mkwrite().
- Drop change inchange_pte_range()because it couldn't actually trigger for shadow stack VMAs.
- Clarify reasoning for skipped cases ofpte_mkwrite().Yu-cheng v25:
- Apply same changes todo_huge_pmd_numa_page()as todo_numa_page().arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 7 +++++++ mm/memory.c | 5 ++++- mm/migrate_device.c | 4 +++- mm/userfaultfd.c | 10 +++++++--- 6 files changed, 30 insertions(+), 5 deletions(-)arch/x86/include/asm/pgtable.h +3
-0@@ -951,6 +951,9 @@ static inline pgd_t pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) }#endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#define is_shstk_write is_shstk_write +extern bool is_shstk_write (unsigned long vm_flags); +#endif /* __ASSEMBLY__ */ arch/x86/mm/pgtable.c +0
-0
arch/x86/mm/pgtable.c int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) · line 888
Quoted 9 lines
arch/x86/mm/pgtable.c +6
-0@@ -880,3 +880,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +bool is_shstk_write (unsigned long vm_flags) +{ +return (vm_flags & (VM_SHADOW_STACK |VM_WRITE )) == + (VM_SHADOW_STACK |VM_WRITE ); +}
Can we call this function something along the lines is_shadow_stack_vma? Reason being, we're actually checking for vma property here.
Also can we move this into common code? Common code can then further call arch_is_shadow_stack_vma. Respective arch can implement their own shadow stack encoding. I see that x86 is using one of the arch bit. Current riscv
implementation uses presence of only VM_WRITE as shadow stack encoding.
Please see patch #11 and #12 in the series I posted (URL at the top of this e-mail).
Quoted 80 lines
include/linux/pgtable.h +7
-0@@ -1578,6 +1578,13 @@ static inline bool arch_has_pfn_modify_check(void) }#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shstk_write +static inline bool is_shstk_write (unsigned long vm_flags) +{ +return false; +} +#endif +/* * Architecture PAGE_KERNEL_* fallbacks * mm/memory.c +4
-1@@ -4088,7 +4088,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot); entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) + +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,mm/migrate_device.c +3
-1@@ -645,7 +645,9 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, goto abort; } entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); }mm/userfaultfd.c +7
-3@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,int ret;pte_t _dst_pte, *dst_pte;bool writable = dst_vma->vm_flags &VM_WRITE ; +bool shstk = dst_vma->vm_flags &VM_SHADOW_STACK ;bool vm_shared = dst_vma->vm_flags &VM_SHARED ;bool page_in_cache =page_mapping (page);spinlock_t *ptl;@@ -84,9 +85,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, writable = false; } -if (writable) - _dst_pte =pte_mkwrite (_dst_pte); -else +if (writable) { +if (shstk) + _dst_pte =pte_mkwrite_shstk (_dst_pte); +else + _dst_pte =pte_mkwrite (_dst_pte); + }else /* * We need this to make sure write bit removed; as mk_pte() * could return a pte with write bit set. Signature · 1 line
2.17.1
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
Quoted 6 lines · Deepak Gupta
Since I've a general question on outcome of discussion of how to handle
pte_mkwrite, so I am top posting.I have posted patches yesterday targeting riscv zisslpcfi extension.
https://lore.kernel.org/lkml/[email protected]/
Quoted 16 lines
Since there're similarities in extension(s), patches have similarity too.
One of the similarity was updatingmaybe_mkwrite. I was asked (by dhildenb
on my patch #11) to look at x86 approach on how to approach this so that
core-mm approach fits multiple architectures along with the need to updatepte_mkwriteto consume vma flags.
In x86 CET patch series, I see that locations wherepte_mkwriteis invoked are updated to check for shadow stack vma and not necessarilypte_mkwriteitself is updated to consume vma flags. Let me know if my
understanding is correct and that's the current direction (to update call sites for vma check wherepte_mkwriteis invoked)
Quoted 5 lines
Being said that as I've mentioned in my patch series that there're similarities between x86, arm and now riscv for implementing shadow stack
and indirect branch tracking, overall it'll be a good thing if we can collaborate and come up with common bits.
Oh interesting. I've made the changes to have pte_mkwrite() take a VMA. It seems to work pretty well with the core MM code, but I'm letting 0- day chew on it for a bit because it touched so many arch's. I'll include you when I send it out, hopefully later this week.
From just a quick look, I see some design aspects that have been problematic on the x86 implementation.
There was something like PROT_SHADOW_STACK before, but there were two problems:
1. Writable windows while provisioning restore tokens (maybe this is just an x86 thing)
2. Adding guard pages when a shadow stack was mprotect()ed to change it from writable to shadow stack. Again this might be an x86 need, since it needed to have it writable to add a restore token, and the guard pages help with security.
So instead this series creates a map_shadow_stack syscall that maps a shadow stack and writes the token from the kernel side. Then mprotect() is prevented from making shadow stack's conventionally writable.
another difference is enabling shadow stack based on elf header bits instead of the arch_prctl()s. See the history and reasoning here (section "Switch Enabling Interface"):
https://lore.kernel.org/lkml/[email protected]/
Not sure if those two issues would be problems on riscv or not.
For sharing the prctl() interface. The other thing is that x86 also has this "wrss" instruction that can be enabled with shadow stack. The current arch_prctl() interface supports both. I'm thinking it's probably a pretty arch-specific thing.
ABI-wise, are you planning to automatically allocate shadow stacks for new tasks? If the ABI is completely different it might be best to not share user interfaces. But also, I wonder why is it different.
Quoted 135 lines
Rest inline.
Quoted 118 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.With the introduction of shadow stack memory there are two ways a pte can
be writable: regular writable memory and shadow stack memory.In past patches,
maybe_mkwrite()has been updated to applypte_mkwrite()
orpte_mkwrite_shstk()depending on the VMA flag. This covers most cases
where a PTE is made writable. However, there are places wherepte_mkwrite()
is called directly and the logic should now also create a shadow stack PTE
in the case of a shadow stack VMA.-
do_anonymous_page()andmigrate_vma_insert_page()checkVM_WRITEdirectly and callpte_mkwrite(). Teach it aboutpte_mkwrite_shstk()- When userfaultfd is creating a PTE after userspace handles the fault
it callspte_mkwrite()directly. Teach it aboutpte_mkwrite_shstk()To make the code cleaner, introduce
is_shstk_write()which simplifies
checking forVM_WRITE|VM_SHADOW_STACKtogether.In other cases where
pte_mkwrite()is called directly, the VMA will not
beVM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case ofpte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization inmprotect(),
shadow stack VMA's won't beVM_SHARED, so it is not necessary.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v5:
- Fix typo in commit logv3:
- Restoredo_anonymous_page()that accidetally moved commits (Kirill)
- Open codemaybe_mkwrite()cases from v2, so the behavior doesn't change
to mark that non-writable PTEs dirty. (Nadav)v2:
- Updated commit log with comment's from Dave Hansen
- Dave also suggested (I understood) to maybe tweakvm_get_page_prot()
to avoid having to callmaybe_mkwrite(). After playing around
with
this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be
writable
ended up mapping the zero page as writable and other surprises.
So
creating shadow stack memory needs to be done with manual logic likepte_mkwrite().
- Drop change inchange_pte_range()because it couldn't actually trigger
for shadow stack VMAs.
- Clarify reasoning for skipped cases ofpte_mkwrite().Yu-cheng v25:
- Apply same changes todo_huge_pmd_numa_page()as todo_numa_page().arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 7 +++++++ mm/memory.c | 5 ++++- mm/migrate_device.c | 4 +++- mm/userfaultfd.c | 10 +++++++--- 6 files changed, 30 insertions(+), 5 deletions(-)arch/x86/include/asm/pgtable.h +3
-0@@ -951,6 +951,9 @@ static inline pgd_t pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) }#endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#define is_shstk_write is_shstk_write +extern bool is_shstk_write (unsigned long vm_flags); +#endif /* __ASSEMBLY__ */ arch/x86/mm/pgtable.c +0
-0@@ -880,3 +880,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /*
CONFIG_HAVE_ARCH_HUGE_VMAP*/arch/x86/mm/pgtable.c int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) · line 888
Quoted 6 lines
arch/x86/mm/pgtable.c +6
-0+ +bool is_shstk_write (unsigned long vm_flags) +{ +return (vm_flags & (VM_SHADOW_STACK |VM_WRITE )) == + (VM_SHADOW_STACK |VM_WRITE ); +}Can we call this function something along the lines
is_shadow_stack_vma?
Reason being, we're actually checking for vma property here.
Quoted 8 lines
Also can we move this into common code? Common code can then further call
arch_is_shadow_stack_vma. Respective arch can implement their own shadow
stack encoding. I see that x86 is using one of the arch bit. Current riscv
implementation uses presence of onlyVM_WRITEas shadow stack encoding.
In the next version I've successfully moved all of the shadow stack bits out of core MM. It doesn't need is_shstk_write() after the pte_mkwrite() change, and changing this other one:
https://lore.kernel.org/lkml/[email protected]/
For that I added an arch_check_zapped_pte() which an arch can use to add warnings.
So I wonder if riscv won't need anything either?
Quoted 93 lines
Please see patch #11 and #12 in the series I posted (URL at the top of
this e-mail).Quoted 87 lines
include/linux/pgtable.h +7
-0@@ -1578,6 +1578,13 @@ static inline bool arch_has_pfn_modify_check(void) }#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shstk_write +static inline bool is_shstk_write (unsigned long vm_flags) +{ +return false; +} +#endif +/* * Architecture PAGE_KERNEL_* fallbacks *
mm/memory.c +4
-1@@ -4088,7 +4088,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot); entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) + +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf-Quoted 1 line
address,
mm/migrate_device.c +3
-1@@ -645,7 +645,9 @@ static void migrate_vma_insert_page(structmigrate_vma *migrate, goto abort; } entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); }mm/userfaultfd.c +1
-0@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, int ret;pte_t _dst_pte, *dst_pte;bool writable = dst_vma->vm_flags &VM_WRITE ; +bool shstk = dst_vma->vm_flags &VM_SHADOW_STACK ;bool vm_shared = dst_vma->vm_flags &VM_SHARED ;bool page_in_cache =page_mapping (page);spinlock_t *ptl;mm/userfaultfd.c +6
-3@@ -84,9 +85,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, writable = false; } -if (writable) - _dst_pte =pte_mkwrite (_dst_pte); -else +if (writable) { +if (shstk) + _dst_pte =pte_mkwrite_shstk (_dst_pte); +else + _dst_pte =pte_mkwrite (_dst_pte); + }else /* * We need this to make sure write bit removed; as
mk_pte()
* could return a pte with write bit set.Signature · 1 line
2.17.1
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
Quoted 30 lines · Rick P
Quoted 6 lines · Deepak Gupta
Since I've a general question on outcome of discussion of how to handle
pte_mkwrite, so I am top posting.I have posted patches yesterday targeting riscv zisslpcfi extension.
https://lore.kernel.org/lkml/[email protected]/
Quoted 16 lines
Since there're similarities in extension(s), patches have similarity too.
One of the similarity was updatingmaybe_mkwrite. I was asked (by dhildenb
on my patch #11) to look at x86 approach on how to approach this so that
core-mm approach fits multiple architectures along with the need to updatepte_mkwriteto consume vma flags.
In x86 CET patch series, I see that locations wherepte_mkwriteis invoked are updated to check for shadow stack vma and not necessarilypte_mkwriteitself is updated to consume vma flags. Let me know if my
understanding is correct and that's the current direction (to update call sites for vma check wherepte_mkwriteis invoked)Quoted 5 lines
Being said that as I've mentioned in my patch series that there're similarities between x86, arm and now riscv for implementing shadow stack
and indirect branch tracking, overall it'll be a good thing if we can collaborate and come up with common bits.
Quoted 4 lines
Oh interesting. I've made the changes to have
pte_mkwrite()take a VMA. It seems to work pretty well with the core MM code, but I'm letting 0- day chew on it for a bit because it touched so many arch's. I'll include you when I send it out, hopefully later this week.
Thanks.
Quoted 4 lines
From just a quick look, I see some design aspects that have been problematic on the x86 implementation.
Quoted 8 lines
There was something like
PROT_SHADOW_STACKbefore, but there were two problems:
1. Writable windows while provisioning restore tokens (maybe this is just an x86 thing)
2. Adding guard pages when a shadow stack was mprotect()ed to change it from writable to shadow stack. Again this might be an x86 need, since it needed to have it writable to add a restore token, and the guard pages help with security.
I've not seen your earlier patch but I am assuming when you say window you mean that shadow stack was open to regular stores (or I may be missing something here)
I am wondering if mapping it as shadow stack (instead of having temporary writeable mapping) and using wruss was an option to put the token or you wanted to avoid it?
And yes on riscv, architecture itself doesn't define token or its format. Since it's RISC, software can define the token format and thus can use either sspush or ssamoswap to put a token on shadow stack virtual memory.
Quoted 5 lines
So instead this series creates a
map_shadow_stacksyscall that maps a shadow stack and writes the token from the kernel side. Thenmprotect()is prevented from making shadow stack's conventionally writable.
Quoted 7 lines
another difference is enabling shadow stack based on elf header bits instead of the arch_prctl()s. See the history and reasoning here (section "Switch Enabling Interface"):
https://lore.kernel.org/lkml/[email protected]/
Not sure if those two issues would be problems on riscv or not.
Apart from mapping and window issue that you mentioned, I couldn't understand on why elf header bit is an issue only in this case for x86 shadow stack and not an issue for let's say aarch64. I can see that aarch64 pretty much uses elf header bit for BTI. Eventually indirect branch tracking also needs to be enabled which is analogous to BTI.
BTW eventually riscv binaries plan to use .riscv.attributes section in riscv elf binary instead of .gnu.note.property. So I am hoping that part will go into arch specific code of elf parsing for riscv and will be contained.
Quoted 5 lines
For sharing the
prctl()interface. The other thing is that x86 also has this "wrss" instruction that can be enabled with shadow stack. The currentarch_prctl()interface supports both. I'm thinking it's probably a pretty arch-specific thing.
yes ability to perform writes on shadow stack absolutely are prevented on x86. So enabling that should be a arch specific prctl.
Quoted 4 lines
ABI-wise, are you planning to automatically allocate shadow stacks for new tasks? If the ABI is completely different it might be best to not share user interfaces. But also, I wonder why is it different.
Yes as of now planning both:
- allocate shadow stack for new task based on elf header
- task can create them using prctls (from glibc)
And yes fork will get the all cfi properties (shdow stack and branch tracking) from parent.
Quoted 248 lines
Quoted 136 lines
Rest inline.
Quoted 118 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.With the introduction of shadow stack memory there are two ways a pte can
be writable: regular writable memory and shadow stack memory.In past patches,
maybe_mkwrite()has been updated to applypte_mkwrite()
orpte_mkwrite_shstk()depending on the VMA flag. This covers most cases
where a PTE is made writable. However, there are places wherepte_mkwrite()
is called directly and the logic should now also create a shadow stack PTE
in the case of a shadow stack VMA.-
do_anonymous_page()andmigrate_vma_insert_page()checkVM_WRITEdirectly and callpte_mkwrite(). Teach it aboutpte_mkwrite_shstk()- When userfaultfd is creating a PTE after userspace handles the fault
it callspte_mkwrite()directly. Teach it aboutpte_mkwrite_shstk()To make the code cleaner, introduce
is_shstk_write()which simplifies
checking forVM_WRITE|VM_SHADOW_STACKtogether.In other cases where
pte_mkwrite()is called directly, the VMA will not
beVM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case ofpte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization inmprotect(),
shadow stack VMA's won't beVM_SHARED, so it is not necessary.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v5:
- Fix typo in commit logv3:
- Restoredo_anonymous_page()that accidetally moved commits (Kirill)
- Open codemaybe_mkwrite()cases from v2, so the behavior doesn't change
to mark that non-writable PTEs dirty. (Nadav)v2:
- Updated commit log with comment's from Dave Hansen
- Dave also suggested (I understood) to maybe tweakvm_get_page_prot()
to avoid having to callmaybe_mkwrite(). After playing around
with
this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be
writable
ended up mapping the zero page as writable and other surprises.
So
creating shadow stack memory needs to be done with manual logic likepte_mkwrite().
- Drop change inchange_pte_range()because it couldn't actually trigger
for shadow stack VMAs.
- Clarify reasoning for skipped cases ofpte_mkwrite().Yu-cheng v25:
- Apply same changes todo_huge_pmd_numa_page()as todo_numa_page().arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 6 ++++++ include/linux/pgtable.h | 7 +++++++ mm/memory.c | 5 ++++- mm/migrate_device.c | 4 +++- mm/userfaultfd.c | 10 +++++++--- 6 files changed, 30 insertions(+), 5 deletions(-)arch/x86/include/asm/pgtable.h +3
-0@@ -951,6 +951,9 @@ static inline pgd_t pti_set_user_pgtbl(pgd_t *pgdp, pgd_t pgd) }#endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#define is_shstk_write is_shstk_write +extern bool is_shstk_write (unsigned long vm_flags); +#endif /* __ASSEMBLY__ */ arch/x86/mm/pgtable.c +0
-0@@ -880,3 +880,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /*
CONFIG_HAVE_ARCH_HUGE_VMAP*/arch/x86/mm/pgtable.c int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) · line 888
Quoted 6 lines
arch/x86/mm/pgtable.c +6
-0+ +bool is_shstk_write (unsigned long vm_flags) +{ +return (vm_flags & (VM_SHADOW_STACK |VM_WRITE )) == + (VM_SHADOW_STACK |VM_WRITE ); +}Can we call this function something along the lines
is_shadow_stack_vma?
Reason being, we're actually checking for vma property here.Quoted 8 lines
Also can we move this into common code? Common code can then further call
arch_is_shadow_stack_vma. Respective arch can implement their own shadow
stack encoding. I see that x86 is using one of the arch bit. Current riscv
implementation uses presence of onlyVM_WRITEas shadow stack encoding.In the next version I've successfully moved all of the shadow stack bits out of core MM. It doesn't need
is_shstk_write()after thepte_mkwrite()change, and changing this other one:https://lore.kernel.org/lkml/[email protected]/
For that I added anarch_check_zapped_pte()which an arch can use to add warnings.So I wonder if riscv won't need anything either?
Quoted 93 lines
Please see patch #11 and #12 in the series I posted (URL at the top of
this e-mail).Quoted 87 lines
include/linux/pgtable.h +7
-0@@ -1578,6 +1578,13 @@ static inline bool arch_has_pfn_modify_check(void) }#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */ +#ifndef is_shstk_write +static inline bool is_shstk_write (unsigned long vm_flags) +{ +return false; +} +#endif +/* * Architecture PAGE_KERNEL_* fallbacks *
mm/memory.c +4
-1@@ -4088,7 +4088,10 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot); entry =pte_sw_mkyoung (entry); -if (vma->vm_flags &VM_WRITE ) + +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry));vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf-Quoted 1 line
address,
mm/migrate_device.c +3
-1@@ -645,7 +645,9 @@ static void migrate_vma_insert_page(structmigrate_vma *migrate, goto abort; } entry =mk_pte (page, vma->vm_page_prot); -if (vma->vm_flags &VM_WRITE ) +if (is_shstk_write (vma->vm_flags)) + entry =pte_mkwrite_shstk (pte_mkdirty (entry)); +else if (vma->vm_flags &VM_WRITE ) entry =pte_mkwrite (pte_mkdirty (entry)); }mm/userfaultfd.c +1
-0@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, int ret;pte_t _dst_pte, *dst_pte;bool writable = dst_vma->vm_flags &VM_WRITE ; +bool shstk = dst_vma->vm_flags &VM_SHADOW_STACK ;bool vm_shared = dst_vma->vm_flags &VM_SHARED ;bool page_in_cache =page_mapping (page);spinlock_t *ptl;mm/userfaultfd.c +6
-3@@ -84,9 +85,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, writable = false; } -if (writable) - _dst_pte =pte_mkwrite (_dst_pte); -else +if (writable) { +if (shstk) + _dst_pte =pte_mkwrite_shstk (_dst_pte); +else + _dst_pte =pte_mkwrite (_dst_pte); + }else /* * We need this to make sure write bit removed; as
mk_pte()
* could return a pte with write bit set.
--
2.17.1
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
Quoted 11 lines · Deepak Gupta
Quoted 10 lines · Rick P
Quoted 7 lines · Deepak Gupta
Since I've a general question on outcome of discussion of how to handle
pte_mkwrite, so I am top posting.I have posted patches yesterday targeting riscv zisslpcfi extension.
https://lore.kernel.org/lkml/[email protected]/
Quoted 70 lines
Quoted 33 lines
Quoted 24 lines
Since there're similarities in extension(s), patches have similarity
too.
One of the similarity was updatingmaybe_mkwrite. I was asked (by
dhildenb
on my patch #11) to look at x86 approach on how to approach this so
that
core-mm approach fits multiple architectures along with the need to
updatepte_mkwriteto consume vma flags.
In x86 CET patch series, I see that locations wherepte_mkwriteis
invoked are updated to check for shadow stack vma and not necessarilypte_mkwriteitself is updated to consume vma flags. Let me know if
my
understanding is correct and that's the current direction (to update
call sites for vma check wherepte_mkwriteis invoked)Quoted 8 lines
Being said that as I've mentioned in my patch series that there're
similarities between x86, arm and now riscv for implementing shadow
stack
and indirect branch tracking, overall it'll be a good thing if we can
collaborate and come up with common bits.Quoted 6 lines
Oh interesting. I've made the changes to have
pte_mkwrite()take a VMA.
It seems to work pretty well with the core MM code, but I'm letting 0-
day chew on it for a bit because it touched so many arch's. I'll include you when I send it out, hopefully later this week.Thanks.
Quoted 4 lines
From just a quick look, I see some design aspects that have been problematic on the x86 implementation.
Quoted 12 lines
There was something like
PROT_SHADOW_STACKbefore, but there were two
problems:
1. Writable windows while provisioning restore tokens (maybe this is
just an x86 thing)
2. Adding guard pages when a shadow stack was mprotect()ed to change it
from writable to shadow stack. Again this might be an x86 need, since
it needed to have it writable to add a restore token, and the guard pages help with security.I've not seen your earlier patch but I am assuming when you say window you
mean that shadow stack was open to regular stores (or I may be missing
something here)I am wondering if mapping it as shadow stack (instead of having temporary
writeable mapping) and usingwrusswas an option to put the token or
you wanted to avoid it?
Quoted 7 lines
And yes on riscv, architecture itself doesn't define token or its format.
Since it's RISC, software can define the token format and thus can use
eithersspushorssamoswapto put a token onshadow stackvirtual
memory.
With WRSS a token could be created via software, but x86 shadow stack includes instructions to create and switch to tokens in limited ways (RSTORSSP, SAVEPREVSSP), where WRSS lets you write anything. These other instructions are enough for glibc, except for writing a restore token on a brand new shadow stack.
So WRSS is made optional since it weakens the protection of the shadow stack. Some apps may prefer to use it to do exotic things, but the glibc implementation didn't require it.
Quoted 8 lines
Quoted 8 lines
So instead this series creates a
map_shadow_stacksyscall that maps a
shadow stack and writes the token from the kernel side. Thenmprotect()
is prevented from making shadow stack's conventionally writable.
Quoted 6 lines
Quoted 6 lines
another difference is enabling shadow stack based on elf header bits
instead of the arch_prctl()s. See the history and reasoning here (section "Switch Enabling Interface"):
https://lore.kernel.org/lkml/[email protected]/
Quoted 9 lines
Quoted 2 lines
Not sure if those two issues would be problems on riscv or not.
Apart from mapping and window issue that you mentioned, I couldn't understand on why elf header bit is an issue only in this case for x86
shadow stack and not an issue for let's say aarch64. I can see that aarch64 pretty much uses elf header bit for BTI. Eventually indirect branch tracking also needs to be enabled which is analogous to BTI.
Well for one, we had to deal with those old glibc's. But doesn't BTI text need to be mapped with a special PROT as well? So it doesn't just turn on enforcement automatically if it detects the elf bit.
Quoted 26 lines
BTW eventually riscv binaries plan to use
.riscv.attributessection in riscv elf binary instead of.gnu.note.property. So I am hoping that
part will go into arch specific code of elf parsing for riscv and will be
contained.Quoted 6 lines
For sharing the
prctl()interface. The other thing is that x86 also has
this "wrss" instruction that can be enabled with shadow stack. The currentarch_prctl()interface supports both. I'm thinking it's probably a pretty arch-specific thing.yes ability to perform writes on shadow stack absolutely are prevented on
x86. So enabling that should be a arch specific prctl.Quoted 6 lines
ABI-wise, are you planning to automatically allocate shadow stacks for
new tasks? If the ABI is completely different it might be best to not
share user interfaces. But also, I wonder why is it different.
Quoted 7 lines
Yes as of now planning both:
- allocate shadow stack for new task based on elf header
- task can create them usingprctls(from glibc)And yes
forkwill get the all cfi properties (shdow stack and branch tracking)
from parent.
Have you looked at a riscv libc implementation yet? For unifying ABI I think that might be best interface to target, for app developers. Then each arch can implement enough kernel functionality to support libc (for example map_shadow_stack).
Re: [PATCH v5 19/39] mm: Fixup places that call pte_mkwrite() directly
Quoted 98 lines · Rick P
Quoted 11 lines · Deepak Gupta
Quoted 10 lines · Rick P
Quoted 7 lines · Deepak Gupta
Since I've a general question on outcome of discussion of how to handle
pte_mkwrite, so I am top posting.I have posted patches yesterday targeting riscv zisslpcfi extension.
https://lore.kernel.org/lkml/[email protected]/
Quoted 70 lines
Quoted 33 lines
Quoted 24 lines
Since there're similarities in extension(s), patches have similarity
too.
One of the similarity was updatingmaybe_mkwrite. I was asked (by
dhildenb
on my patch #11) to look at x86 approach on how to approach this so
that
core-mm approach fits multiple architectures along with the need to
updatepte_mkwriteto consume vma flags.
In x86 CET patch series, I see that locations wherepte_mkwriteis
invoked are updated to check for shadow stack vma and not necessarilypte_mkwriteitself is updated to consume vma flags. Let me know if
my
understanding is correct and that's the current direction (to update
call sites for vma check wherepte_mkwriteis invoked)Quoted 8 lines
Being said that as I've mentioned in my patch series that there're
similarities between x86, arm and now riscv for implementing shadow
stack
and indirect branch tracking, overall it'll be a good thing if we can
collaborate and come up with common bits.Quoted 6 lines
Oh interesting. I've made the changes to have
pte_mkwrite()take a VMA.
It seems to work pretty well with the core MM code, but I'm letting 0-
day chew on it for a bit because it touched so many arch's. I'll include you when I send it out, hopefully later this week.Thanks.
Quoted 4 lines
From just a quick look, I see some design aspects that have been problematic on the x86 implementation.
Quoted 12 lines
There was something like
PROT_SHADOW_STACKbefore, but there were two
problems:
1. Writable windows while provisioning restore tokens (maybe this is
just an x86 thing)
2. Adding guard pages when a shadow stack was mprotect()ed to change it
from writable to shadow stack. Again this might be an x86 need, since
it needed to have it writable to add a restore token, and the guard pages help with security.I've not seen your earlier patch but I am assuming when you say window you
mean that shadow stack was open to regular stores (or I may be missing
something here)I am wondering if mapping it as shadow stack (instead of having temporary
writeable mapping) and usingwrusswas an option to put the token or
you wanted to avoid it?Quoted 7 lines
And yes on riscv, architecture itself doesn't define token or its format.
Since it's RISC, software can define the token format and thus can use
eithersspushorssamoswapto put a token onshadow stackvirtual
memory.With WRSS a token could be created via software, but x86 shadow stack includes instructions to create and switch to tokens in limited ways (RSTORSSP, SAVEPREVSSP), where WRSS lets you write anything. These other instructions are enough for glibc, except for writing a restore token on a brand new shadow stack.
Quoted 4 lines
So WRSS is made optional since it weakens the protection of the shadow stack. Some apps may prefer to use it to do exotic things, but the glibc implementation didn't require it.
Yes, I understand WRSS in user mode is not safe and defeat the purpose as well.
I actually had meant why WRUSS couldn't be used in the kernel to manufacture the token when the kernel
creates the shadow stack while parsing elf bits. But then I went through you earlier patch series now and I've a
a little bit of context now. There is a lot of history and context (and mess) here.
Quoted 69 lines
Quoted 8 lines
Quoted 8 lines
So instead this series creates a
map_shadow_stacksyscall that maps a
shadow stack and writes the token from the kernel side. Thenmprotect()
is prevented from making shadow stack's conventionally writable.Quoted 6 lines
Quoted 6 lines
another difference is enabling shadow stack based on elf header bits
instead of the arch_prctl()s. See the history and reasoning here (section "Switch Enabling Interface"):https://lore.kernel.org/lkml/[email protected]/
Quoted 9 lines
Quoted 2 lines
Not sure if those two issues would be problems on riscv or not.
Apart from mapping and window issue that you mentioned, I couldn't understand on why elf header bit is an issue only in this case for x86
shadow stack and not an issue for let's say aarch64. I can see that aarch64 pretty much uses elf header bit for BTI. Eventually indirect branch tracking also needs to be enabled which is analogous to BTI.Well for one, we had to deal with those old glibc's. But doesn't BTI text need to be mapped with a special PROT as well? So it doesn't just turn on enforcement automatically if it detects the elf bit.
Quoted 26 lines
BTW eventually riscv binaries plan to use
.riscv.attributessection in riscv elf binary instead of.gnu.note.property. So I am hoping that
part will go into arch specific code of elf parsing for riscv and will be
contained.Quoted 6 lines
For sharing the
prctl()interface. The other thing is that x86 also has
this "wrss" instruction that can be enabled with shadow stack. The currentarch_prctl()interface supports both. I'm thinking it's probably a pretty arch-specific thing.yes ability to perform writes on shadow stack absolutely are prevented on
x86. So enabling that should be a arch specific prctl.Quoted 6 lines
ABI-wise, are you planning to automatically allocate shadow stacks for
new tasks? If the ABI is completely different it might be best to not
share user interfaces. But also, I wonder why is it different.Quoted 7 lines
Yes as of now planning both:
- allocate shadow stack for new task based on elf header
- task can create them usingprctls(from glibc)And yes
forkwill get the all cfi properties (shdow stack and branch tracking)
from parent.Have you looked at a riscv libc implementation yet? For unifying ABI I think that might be best interface to target, for app developers. Then each arch can implement enough kernel functionality to support libc (for example
map_shadow_stack).
PATCH v5 20/39 mm: Add guard pages around a shadow stack.
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks.
Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.
The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:
addq $0x80, %rsp
However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:
READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stackREAD_ONCE(ssp-8); // read+discard last popped stack element
The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.
This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work.
---
v5:
- Fix typo in commit log
v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)
v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)
Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.
include/linux/mm.h | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)
include/linux/mm.h +26 -5
@@ -2987,15 +2987,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)return mtree_load (&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap (struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_GROWSDOWN ) +return stack_guard_gap; + +/* +* Shadow stack pointer is moved by CALL, RET, and INCSSPQ. +* INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB +* and touches the first and the last element in the range, which +* triggers a page fault if the range is not in a shadow stack. +* Because of this, creating 4-KB guard pages around a shadow +* stack prevents these instructions from going beyond. +* +* Creation of VM_SHADOW_STACK is tightly controlled, so a vma +* can't be both VM_GROWSDOWN and VM_SHADOW_STACK +*/ +if (vma->vm_flags &VM_SHADOW_STACK ) +return PAGE_SIZE ; + +return 0 ; +} +static inline unsigned long vm_start_gap (struct vm_area_struct *vma) { +unsigned long gap =stack_guard_start_gap (vma);unsigned long vm_start = vma->vm_start; -if (vma->vm_flags &VM_GROWSDOWN ) { - vm_start -= stack_guard_gap; -if (vm_start > vma->vm_start) - vm_start =0 ; - } + vm_start -= gap; +if (vm_start > vma->vm_start) + vm_start =0 ;return vm_start; }
Signature · 1 line
2.17.1
PATCH v5 21/39 mm/mmap: Add shadow stack pages to memory accounting
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Account shadow stack pages to stack memory.
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().
mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+)
mm/mmap.c +2 -0
@@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; }
Signature · 1 line
2.17.1
PATCH v5 22/39 mm: Re-introduce vm_flags to do_mmap()
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by:
commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()").
There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap().
--- fs/aio.c | 2 +- include/linux/mm.h | 3 ++- ipc/shm.c | 2 +- mm/mmap.c | 10 +++++----- mm/nommu.c | 4 ++-- mm/util.c | 2 +- 6 files changed, 12 insertions(+), 11 deletions(-)
fs/aio.c +1 -1
@@ -554,7 +554,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base =do_mmap (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE , -MAP_SHARED ,0 , &unused,NULL ); +MAP_SHARED ,0 ,0 , &unused,NULL );mmap_write_unlock (mm);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;
include/linux/mm.h +2 -1
@@ -2897,7 +2897,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf);extern unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flags, -unsigned long pgoff,unsigned long *populate,struct list_head *uf); +vm_flags_t vm_flags,unsigned long pgoff,unsigned long *populate, +struct list_head *uf);extern int do_mas_munmap (struct ma_state *mas,struct mm_struct *mm,unsigned long start,size_t len,struct list_head *uf,bool downgrade);
ipc/shm.c +1 -1
@@ -1662,7 +1662,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr =do_mmap (file, addr, size, prot, flags,0 , &populate,NULL ); + addr =do_mmap (file, addr, size, prot, flags,0 ,0 , &populate,NULL ); *raddr = addr; err =0 ;if (IS_ERR_VALUE (addr))
mm/mmap.c +5 -5
@@ -1238,11 +1238,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,*/ unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot, -unsigned long flags,unsigned long pgoff, -unsigned long *populate,struct list_head *uf) +unsigned long flags,vm_flags_t vm_flags, +unsigned long pgoff,unsigned long *populate, +struct list_head *uf) {struct mm_struct *mm = current->mm; -vm_flags_t vm_flags;int pkey =0 ;validate_mm (mm);
@@ -1303,7 +1303,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,* to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags =calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | + vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ;if (flags &MAP_LOCKED )
@@ -2877,7 +2877,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, file =get_file (vma->vm_file); ret =do_mmap (vma->vm_file, start, size, - prot, flags, pgoff, &populate,NULL ); + prot, flags,0 , pgoff, &populate,NULL );fput (file); out:mmap_write_unlock (mm);
mm/nommu.c +2 -2
@@ -1042,6 +1042,7 @@ unsigned long do_mmap(struct file *file,unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf)
@@ -1049,7 +1050,6 @@ unsigned long do_mmap(struct file *file,struct vm_area_struct *vma;struct vm_region *region;struct rb_node *rb; -vm_flags_t vm_flags;unsigned long capabilities, result;int ret;MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );
@@ -1069,7 +1069,7 @@ unsigned long do_mmap(struct file *file,/* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags =determine_vm_flags (file, prot, flags, capabilities); + vm_flags |=determine_vm_flags (file, prot, flags, capabilities);/* we're going to need to record the mapping */
mm/util.c +1 -1
@@ -517,7 +517,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,if (!ret) {if (mmap_write_lock_killable (mm))return -EINTR ; - ret =do_mmap (file, addr, len, prot, flag, pgoff, &populate, + ret =do_mmap (file, addr, len, prot, flag,0 , pgoff, &populate, &uf);mmap_write_unlock (mm);userfaultfd_unmap_complete (mm, &uf);
Signature · 1 line
2.17.1
PATCH v5 23/39 mm: Don't allow write GUPs to shadow stack memory
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.
Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.
---
v3:
- Add comment in __pte_access_permitted() (Dave)
- Remove unneeded shadow stack specific check in
__pte_access_permitted() (Jann)
arch/x86/include/asm/pgtable.h | 5 +++++ mm/gup.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +5 -0
@@ -1671,6 +1671,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write) {unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER; +/* +* Write=0,Dirty=1 PTEs are shadow stack, which the kernel +* shouldn't generally allow access to, but since they +* are already Write=0, the below logic covers both cases. +*/ if (write) need_pte_bits |= _PAGE_RW;
mm/gup.c +1 -1
@@ -982,7 +982,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)return -EFAULT ;if (write) { -if (!(vm_flags &VM_WRITE )) { +if (!(vm_flags &VM_WRITE ) || (vm_flags &VM_SHADOW_STACK )) {if (!(gup_flags &FOLL_FORCE ))return -EFAULT ;/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
Signature · 1 line
2.17.1
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 5 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Quoted 8 lines
Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via
get_user_pages(, FOLL_WRITE)operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.Still allow
FOLL_FORCEto write through shadow stack protections, as it does for read-only protections.
So an app can simply modify the shadow stack itself by writing to /proc/self/mem ?
Is that really intended? Looks like security hole to me at first sight, but maybe I am missing something important.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
* David Hildenbrand:
Quoted 15 lines · Rick Edgecombe
Quoted 5 lines
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Shadow stack memory is writable only in very specific, controlledQuoted 8 lines
ways.
However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's viaget_user_pages(, FOLL_WRITE)operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs. Still allowFOLL_FORCEto write through shadow stack protections, as it
does for read-only protections.
Quoted 5 lines
So an app can simply modify the shadow stack itself by writing to
/proc/self/mem?Is that really intended? Looks like security hole to me at first sight, but maybe I am missing something important.
Isn't it possible to overwrite GOT pointers using the same vector? So I think it's merely reflecting the status quo.
Thanks,
Florian
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 24 lines · Florian Weimer
* David Hildenbrand:
Quoted 22 lines · Rick Edgecombe
Quoted 20 lines
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result
there remain many ways for userspace to trigger the kernel to write to
shadow stack's viaget_user_pages(, FOLL_WRITE)operations. To make this a
little less exposed, block writable GUPs for shadow stack VMAs. Still allowFOLL_FORCEto write through shadow stack protections, as
it
does for read-only protections.
Quoted 8 lines
Quoted 5 lines
So an app can simply modify the shadow stack itself by writing to
/proc/self/mem?Is that really intended? Looks like security hole to me at first sight, but maybe I am missing something important.
Isn't it possible to overwrite GOT pointers using the same vector? So I think it's merely reflecting the status quo.
There was some debate on this. /proc/self/mem can currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite executable code?
The consensus seemed to lean towards not making special rules for this case, and there was some discussion that /proc/self/mem should maybe be hardened generally.
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 31 lines · Rick P
Quoted 30 lines · Florian Weimer
* David Hildenbrand:
Quoted 27 lines · Rick Edgecombe
Quoted 12 lines
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.
Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result
there remain many ways for userspace to trigger the kernel toQuoted 8 lines
write to
shadow stack's viaget_user_pages(, FOLL_WRITE)operations. To make this a
little less exposed, block writable GUPs for shadow stack VMAs. Still allowFOLL_FORCEto write through shadow stack protections, as
it
does for read-only protections.So an app can simply modify the shadow stack itself by writing to
/proc/self/mem?Is that really intended? Looks like security hole to me at first sight, but maybe I am missing something important.
Quoted 8 lines
Quoted 2 lines
Isn't it possible to overwrite GOT pointers using the same vector? So I think it's merely reflecting the status quo.
There was some debate on this.
/proc/self/memcan currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite executable code?
The question is, if there is reasonable debugging reason to keep it. I assume if a debugger would adjust the ordinary stack, it would have to adjust the shadow stack as well (oh my ...). So it sounds reasonable to have it in theory at least ... not sure when debugger would support that, but maybe they already do.
Quoted 3 lines
The consensus seemed to lean towards not making special rules for this case, and there was some discussion that
/proc/self/memshould maybe be hardened generally.
I agree with that. It's a debugging mechanism that a process can abuse to do nasty stuff to its memory that it maybe shouldn't be able to do ...
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Ping Cristina regarding GDB.
Ping Kees regarding /proc/self/mem.
Quoted 11 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Isn't it possible to overwrite GOT pointers using the same vector?
So I think it's merely reflecting the status quo.There was some debate on this.
/proc/self/memcan currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite
executable code?
Quoted 8 lines
The question is, if there is reasonable debugging reason to keep it. I
assume if a debugger would adjust the ordinary stack, it would have to
adjust the shadow stack as well (oh my ...). So it sounds reasonable to
have it in theory at least ... not sure when debugger would support that, but maybe they already do.
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc. But yea ptrace poke will still need to use FOLL_FORCE and be able to write through shadow stacks.
Quoted 11 lines
Quoted 6 lines
The consensus seemed to lean towards not making special rules for this
case, and there was some discussion that/proc/self/memshould maybe be
hardened generally.I agree with that. It's a debugging mechanism that a process can abuse
to do nasty stuff to its memory that it maybe shouldn't be able to do
...
Ok.
Kees Cook
maintainerRe: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 25 lines · Rick P
Ping Cristina regarding GDB.
Ping Kees regarding
/proc/self/mem.Quoted 11 lines · David Hildenbrand
Quoted 10 lines
Quoted 3 lines
Isn't it possible to overwrite GOT pointers using the same vector?
So I think it's merely reflecting the status quo.There was some debate on this.
/proc/self/memcan currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite
executable code?Quoted 8 lines
The question is, if there is reasonable debugging reason to keep it. I
assume if a debugger would adjust the ordinary stack, it would have to
adjust the shadow stack as well (oh my ...). So it sounds reasonable to
have it in theory at least ... not sure when debugger would support that, but maybe they already do.
Quoted 4 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc. But yea ptrace poke will still need to use
FOLL_FORCEand be able to write through shadow stacks.
I'd prefer to avoid adding more FOLL_FORCE if we can. If gdb can do stack manipulations through a ptrace interface then let's leave off FOLL_FORCE.
-Kees
Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
The consensus seemed to lean towards not making special rules for this
case, and there was some discussion that/proc/self/memshould maybe be
hardened generally.Quoted 4 lines
I agree with that. It's a debugging mechanism that a process can abuse
to do nasty stuff to its memory that it maybe shouldn't be able to do
...Ok.
Signature · 1 line
Kees Cook
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 10 lines · Kees Cook
Quoted 6 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc.
But yea ptrace poke will still need to useFOLL_FORCEand be able to
write through shadow stacks.I'd prefer to avoid adding more
FOLL_FORCEif we can. If gdb can do stack manipulations through a ptrace interface then let's leave offFOLL_FORCE.
Ptrace and /proc/self/mem both use FOLL_FORCE. I think ptrace will always need it or something like it for debugging.
To jog your memory, this series doesn't change what uses FOLL_FORCE. It just sets the shadow stack rules to be the same as read-only memory. So even though shadow stack memory is sort of writable, it's a bit more locked down and FOLL_FORCE is required to write to it with GUP.
If we just remove FOLL_FORCE from /proc/self/mem, something will probably break right? How do we do this? Some sort of opt-in?
Re: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 15 lines · Rick P
Quoted 10 lines · Kees Cook
Quoted 6 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc.
But yea ptrace poke will still need to useFOLL_FORCEand be able to
write through shadow stacks.I'd prefer to avoid adding more
FOLL_FORCEif we can. If gdb can do stack manipulations through a ptrace interface then let's leave offFOLL_FORCE.Ptrace and
/proc/self/memboth useFOLL_FORCE. I think ptrace will always need it or something like it for debugging.
Quoted 7 lines
To jog your memory, this series doesn't change what uses
FOLL_FORCE. It just sets the shadow stack rules to be the same as read-only memory. So even though shadow stack memory is sort of writable, it's a bit more locked down andFOLL_FORCEis required to write to it with GUP.If we just remove
FOLL_FORCEfrom/proc/self/mem, something will probably break right? How do we do this? Some sort of opt-in?
I don't think removing that is an option. It's another debug interface that has been allowing such access for ever ...
Blocking /proc/self/mem access completely for selected processes might be the better alternative.
Signature · 3 lines
Thanks, David / dhildenb
Kees Cook
maintainerRe: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 21 lines · David Hildenbrand
Quoted 20 lines · Rick P
Quoted 10 lines · Kees Cook
Quoted 6 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc.
But yea ptrace poke will still need to useFOLL_FORCEand be able to
write through shadow stacks.I'd prefer to avoid adding more
FOLL_FORCEif we can. If gdb can do stack manipulations through a ptrace interface then let's leave offFOLL_FORCE.Ptrace and
/proc/self/memboth useFOLL_FORCE. I think ptrace will always need it or something like it for debugging.To jog your memory, this series doesn't change what uses
FOLL_FORCE. It just sets the shadow stack rules to be the same as read-only memory. So even though shadow stack memory is sort of writable, it's a bit more locked down andFOLL_FORCEis required to write to it with GUP.
Quoted 7 lines
Quoted 2 lines
If we just remove
FOLL_FORCEfrom/proc/self/mem, something will probably break right? How do we do this? Some sort of opt-in?I don't think removing that is an option. It's another debug interface that has been allowing such access for ever ...
Blocking
/proc/self/memaccess completely for selected processes might be the better alternative.
Yeah, this would be nice. Kind of like being undumpable or no_new_privs.
Signature · 1 line
Kees Cook
RE: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
In reply to Edgecombe, Rick PQuoted 17 lines · David Hildenbrand
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
Isn't it possible to overwrite GOT pointers using the same vector? So I think it's merely reflecting the status quo.
There was some debate on this.
/proc/self/memcan currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite executable code?Quoted 6 lines
The question is, if there is reasonable debugging reason to keep it. I
assume if a debugger would adjust the ordinary stack, it would have to adjust the shadow stack as well (oh my ...). So it sounds reasonable to have it in theory at least ... not sure when debugger would support that, but maybe they already do.
Quoted 4 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc. But yea ptrace poke will still need to use
FOLL_FORCEand be able to write through shadow stacks.
Our patches for GDB use /proc/PID/mem to read/write shadow stack memory. However, I think it should be possible to change this to ptrace but GDB normally uses /proc/PID/mem to read/write target memory.
Regards,
Christina
Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
RE: [PATCH v5 23/39] mm: Don't allow write GUPs to shadow stack memory
Quoted 22 lines
Quoted 17 lines · David Hildenbrand
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
Isn't it possible to overwrite GOT pointers using the same vector? So I think it's merely reflecting the status quo.
There was some debate on this.
/proc/self/memcan currently write through read-only memory which protects executable code. So should shadow stack get separate rules? Is ROP a worry when you can overwrite executable code?Quoted 6 lines
The question is, if there is reasonable debugging reason to keep it. I
assume if a debugger would adjust the ordinary stack, it would have to adjust the shadow stack as well (oh my ...). So it sounds reasonable to have it in theory at least ... not sure when debugger would support that, but maybe they already do.Quoted 4 lines
GDB support for shadow stack is queued up for whenever the kernel interface settles. I believe it just uses ptrace, and not this proc. But yea ptrace poke will still need to use
FOLL_FORCEand be able to write through shadow stacks.
Quoted 7 lines
Our patches for GDB use
/proc/PID/memto read/write shadow stack memory.
However, I think it should be possible to change this to ptrace but GDB normally uses/proc/PID/memto read/write target memory.Regards,
Christina
I just noticed that GDBSERVER actually uses ptrace, so our patches currently use both: ptrace and proc/PID/mem to read/write shadow stack memory.
Regards,
Christina
Intel Deutschland GmbH
Registered Address: Am Campeon 10, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de <http://www.intel.de>
Managing Directors: Christin Eisenschmid, Sharon Heck, Tiffany Doon Silva Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
PATCH v5 24/39 x86/mm: Introduce MAP_ABOVE4G
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly.
One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a
general protection fault which will in turn trigger a segfault if it
tries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by.
So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though.
If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall.
Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's own randomization in the bottom up case.
For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G both are potentially valid, so both are used. In the bottomup search path, the default behavior is already consistent with MAP_ABOVE4G since mmap base should be above 4GB.
Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB. So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode with shadow stack enabled would usually segfault anyway. This is already pretty decent guard rails. But the addition of MAP_ABOVE4G is some small complexity spent to make it make it more complete.
---
v5:
- New patch
arch/x86/include/uapi/asm/mman.h | 1 + arch/x86/kernel/sys_x86_64.c | 6 +++++- include/linux/mman.h | 4 ++++ 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/mman.h +1 -0
@@ -3,6 +3,7 @@#define _ASM_X86_MMAN_H#define MAP_32BIT 0x40/* only give out 32bit addresses */ +#define MAP_ABOVE4G 0x80/* only map above 4GB */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS #define arch_calc_vm_prot_bits (prot, key) ( \
arch/x86/kernel/sys_x86_64.c +5 -1
@@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.flags =VM_UNMAPPED_AREA_TOPDOWN ; info.length = len; - info.low_limit =PAGE_SIZE ; +if (!in_32bit_syscall () && (flags &MAP_ABOVE4G )) + info.low_limit =0x100000000 ; +else + info.low_limit =PAGE_SIZE ; + info.high_limit =get_mmap_base (0 );/*
include/linux/mman.h +4 -0
@@ -15,6 +15,9 @@#ifndef MAP_32BIT #define MAP_32BIT 0#endif +#ifndef MAP_ABOVE4G +#define MAP_ABOVE4G 0 +#endif #ifndef MAP_HUGE_2MB #define MAP_HUGE_2MB 0#endif
@@ -50,6 +53,7 @@ | MAP_STACK \ |MAP_HUGETLB \ |MAP_32BIT \ + |MAP_ABOVE4G \ |MAP_HUGE_2MB \ | MAP_HUGE_1GB)
Signature · 1 line
2.17.1
PATCH v5 25/39 mm: Warn on shadow stack memory in wrong vma
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured.
In order to check if a pte is shadow stack in core mm code, add default implementations for pte_shstk() and pmd_shstk().
---
v5:
- Fix typo in commit log
v3:
- New patch
arch/x86/include/asm/pgtable.h | 2 ++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 2 ++ mm/memory.c | 2 ++ 4 files changed, 20 insertions(+)
arch/x86/include/asm/pgtable.h +2 -0
@@ -129,6 +129,7 @@ static inline bool pte_dirty(pte_t pte)return pte_flags (pte) & _PAGE_DIRTY_BITS; } +#define pte_shstk pte_shstkstatic inline bool pte_shstk (pte_t pte) {if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK ))
@@ -147,6 +148,7 @@ static inline bool pmd_dirty(pmd_t pmd)return pmd_flags (pmd) & _PAGE_DIRTY_BITS; } +#define pmd_shstk pmd_shstkstatic inline bool pmd_shstk (pmd_t pmd) {if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK ))
include/linux/pgtable.h +14 -0
@@ -539,6 +539,20 @@ static inline pte_t pte_mkwrite_shstk(pte_t pte) }#endif +#ifndef pte_shstk +static inline bool pte_shstk (pte_t pte) +{ +return false; +} +#endif + +#ifndef pmd_shstk +static inline bool pmd_shstk (pmd_t pte) +{ +return false; +} +#endif +#ifndef pmd_mkwrite_shstkstatic inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) {
mm/huge_memory.c +2 -0
@@ -1700,6 +1700,8 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,*/ orig_pmd =pmdp_huge_get_and_clear_full (vma, addr, pmd, tlb->fullmm); +VM_WARN_ON_ONCE (!(vma->vm_flags &VM_SHADOW_STACK ) && +pmd_shstk (orig_pmd));tlb_remove_pmd_tlb_entry (tlb, pmd, addr);if (vma_is_special_huge (vma)) {if (arch_needs_pgtable_deposit ())
mm/memory.c +2 -0
@@ -1381,6 +1381,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,continue ; ptent =ptep_get_and_clear_full (mm, addr, pte, tlb->fullmm); +VM_WARN_ON_ONCE (!(vma->vm_flags &VM_SHADOW_STACK ) && +pte_shstk (ptent));tlb_remove_tlb_entry (tlb, pte, addr);zap_install_uffd_wp_if_needed (vma, addr, pte, details, ptent);
Signature · 1 line
2.17.1
Re: [PATCH v5 25/39] mm: Warn on shadow stack memory in wrong vma
Quoted 16 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured.
Quoted 6 lines
In order to check if a pte is shadow stack in core mm code, add default implementations for
pte_shstk()andpmd_shstk().Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 26/39 x86: Introduce userspace API for shadow stack
In reply to Rick EdgecombeFrom: "Kirill A. Shutemov" <[email protected]>
Add three new arch_prctl() handles:
- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.
- ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error
The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().
This is preparation patch. It does not implement any features.
---
v4:
- Remove references to CET and replace with shadow stack (Peterz)
v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM
v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)
arch/x86/include/asm/processor.h | 6 +++++ arch/x86/include/asm/shstk.h | 21 +++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 6 +++++ arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/process_64.c | 7 ++++- arch/x86/kernel/shstk.c | 44 +++++++++++++++++++++++++++++++ 6 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/processor.h +6 -0
@@ -28,6 +28,7 @@ struct vm86;#include <asm/unwind_hints.h> #include <asm/vmxfeatures.h> #include <asm/vdso/processor.h> +#include <asm/shstk.h> #include <linux/personality.h> #include <linux/cache.h>
@@ -475,6 +476,11 @@ struct thread_struct {*/ u32 pkru; +#ifdef CONFIG_X86_USER_SHADOW_STACK +unsigned long features; +unsigned long features_locked; +#endif +/* Floating point and extended processor state */ struct fpu fpu;/*
arch/x86/include/asm/shstk.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SHSTK_H +#define _ASM_X86_SHSTK_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> + +struct task_struct ; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +long shstk_prctl (struct task_struct *task,int option,unsigned long features); +void reset_thread_features (void ); +#else +static inline long shstk_prctl (struct task_struct *task,int option, +unsigned long features) {return -EINVAL ; } +static inline void reset_thread_features (void ) {} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_SHSTK_H */
arch/x86/include/uapi/asm/prctl.h +6 -0
@@ -20,4 +20,10 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +/* Don't use 0x3001-0x3004 because of old glibcs */ + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +2 -0
@@ -146,6 +146,8 @@ obj-$(CONFIG_CALL_THUNKS) += callthunks.o obj-$(CONFIG_X86_CET) += cet.o +obj-$(CONFIG_X86_USER_SHADOW_STACK) += shstk.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/process_64.c +6 -1
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,load_gs_index (__USER_DS); } +reset_thread_features (); +loadsegment (fs,0 );loadsegment (es, _ds);loadsegment (ds, _ds);
@@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_MAP_VDSO_64 :return prctl_map_vdso (&vdso_image_64, arg2);#endif - +case ARCH_SHSTK_ENABLE : +case ARCH_SHSTK_DISABLE : +case ARCH_SHSTK_LOCK : +return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;break ;
arch/x86/kernel/shstk.c new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 26/39] x86: Introduce userspace API for shadow stack
Quoted 10 lines · Rick Edgecombe
From: "Kirill A. Shutemov" <[email protected]>
Add three new
arch_prctl()handles:- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.
-
ARCH_SHSTK_LOCKprevents future disabling or enabling of the specified feature. Returns 0 on success or an error
Quoted 8 lines
The features are handled per-thread and inherited over fork(2)/clone(2), but reset on
exec().This is preparation patch. It does not implement any features.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 27/39 x86/shstk: Add user-mode shadow stack support
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).
Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.
Do not support IA32 emulation or x32.
---
v5:
- Switch to EOPNOTSUPP
- Use MAP_ABOVE4G
- Move set_clr_bits_msrl() to patch where it is first used
v4:
- Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz)
v3:
- Use define for set_clr_bits_msrl() (Kees)
- Make some functions static (Kees)
- Change feature_foo() to features_foo() (Kees)
- Centralize shadow stack size rlimit checks (Kees)
- Disable x32 support
v2:
- Get rid of unnessary shstk->base checks
- Don't support IA32 emulation
arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 146 ++++++++++++++++++++++++++++++ 4 files changed, 158 insertions(+)
arch/x86/include/asm/processor.h +2 -0
@@ -479,6 +479,8 @@ struct thread_struct {#ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features;unsigned long features_locked; + +struct thread_shstk shstk;#endif /* Floating point and extended processor state */
arch/x86/include/asm/shstk.h +7 -0
@@ -8,12 +8,19 @@struct task_struct ;#ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { +u64 base; +u64 size; +}; +long shstk_prctl (struct task_struct *task,int option,unsigned long features);void reset_thread_features (void ); +void shstk_free (struct task_struct *p);#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long features) {return -EINVAL ; }static inline void reset_thread_features (void ) {} +static inline void shstk_free (struct task_struct *p) {}#endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */
arch/x86/include/uapi/asm/prctl.h +3 -0
@@ -26,4 +26,7 @@#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/shstk.c +146 -0
160 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 27/39] x86/shstk: Add user-mode shadow stack support
Quoted 10 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with
VM_SHADOW_STACKflag and has a fixed size ofmin(RLIMIT_STACK, 4GB).Keep the task's shadow stack address and size in
thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.
Quoted 5 lines
Do not support IA32 emulation or x32.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 28/39 x86/shstk: Handle thread shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways.
With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.
For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.
For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.
Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.
For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().
Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.
Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent.
During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner.
---
v3:
- Fix update_fpu_shstk() stub (Mike Rapoport)
- Fix chunks around alloc_shstk() in wrong patch (Kees)
- Fix stack_size/flags swap (Kees)
- Use centalized stack size logic (Kees)
v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx)
v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.
Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)
arch/x86/include/asm/fpu/sched.h | 3 +- arch/x86/include/asm/mmu_context.h | 2 ++ arch/x86/include/asm/shstk.h | 7 +++++ arch/x86/kernel/fpu/core.c | 41 +++++++++++++++++++++++++++- arch/x86/kernel/process.c | 18 +++++++++++- arch/x86/kernel/shstk.c | 44 ++++++++++++++++++++++++++++-- 6 files changed, 110 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/sched.h +2 -1
@@ -11,7 +11,8 @@extern void save_fpregs_to_fpstate (struct fpu *fpu);extern void fpu__drop (struct fpu *fpu); -extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal); +extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal, +unsigned long shstk_addr);extern void fpu_flush_thread (void );/*
arch/x86/include/asm/mmu_context.h +2 -0
@@ -147,6 +147,8 @@ do { \#else #define deactivate_mm (tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + shstk_free(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0)
arch/x86/include/asm/shstk.h +7 -0
@@ -15,11 +15,18 @@ struct thread_shstk {long shstk_prctl (struct task_struct *task,int option,unsigned long features);void reset_thread_features (void ); +int shstk_alloc_thread_stack (struct task_struct *p,unsigned long clone_flags, +unsigned long stack_size, +unsigned long *shstk_addr);void shstk_free (struct task_struct *p);#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long features) {return -EINVAL ; }static inline void reset_thread_features (void ) {} +static inline int shstk_alloc_thread_stack (struct task_struct *p, +unsigned long clone_flags, +unsigned long stack_size, +unsigned long *shstk_addr) {return 0 ; }static inline void shstk_free (struct task_struct *p) {}#endif /* CONFIG_X86_USER_SHADOW_STACK */
arch/x86/kernel/fpu/core.c +40 -1
54 lines not on this page · raw message
arch/x86/kernel/process.c +17 -1
@@ -48,6 +48,7 @@#include <asm/frame.h> #include <asm/unwind.h> #include <asm/tdx.h> +#include <asm/shstk.h> #include "process.h"
@@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t); +shstk_free (tsk);fpu__drop (fpu); }
@@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)struct inactive_task_frame *frame;struct fork_frame *fork_frame;struct pt_regs *childregs; +unsigned long shstk_addr =0 ;int ret =0 ; childregs =task_pt_regs (p);
@@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) frame->flags =X86_EFLAGS_FIXED ;#endif -fpu_clone (p, clone_flags, args->fn); +/* Allocate a new shadow stack for pthread if needed */ + ret =shstk_alloc_thread_stack (p, clone_flags, args->stack_size, + &shstk_addr); +if (ret) +return ret; + +fpu_clone (p, clone_flags, args->fn, shstk_addr);/* Kernel thread ? */ if (unlikely (p->flags &PF_KTHREAD )) {
@@ -220,6 +229,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)if (!ret &&unlikely (test_tsk_thread_flag (current,TIF_IO_BITMAP )))io_bitmap_share (p); +/* +* If copy_thread() if failing, don't leak the shadow stack possibly +* allocated in shstk_alloc_thread_stack() above. +*/ +if (ret) +shstk_free (p); +return ret; }
arch/x86/kernel/shstk.c +42 -2
62 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 29/39 x86/shstk: Introduce routines modifying shstk
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.
A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.
Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.
In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.
To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.
---
v5:
- Fix typo in commit log
v3:
- Drop shstk_check_rstor_token()
- Fail put_shstk_data() if bit 63 is set in the data (Kees)
- Add comment in create_rstor_token() (Kees)
- Pull in create_rstor_token() changes from future patch (Kees)
v2:
- Add data helpers for writing to shadow stack.
v1:
- Use xsave helpers.
arch/x86/include/asm/special_insns.h | 13 +++++ arch/x86/kernel/shstk.c | 73 ++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
@@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static inline int write_user_shstk_64 (u64 __user *addr,u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); +return 0 ; +fail: +return -EFAULT ; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ +#define nop () asm volatile ("nop")static inline void serialize (void )
arch/x86/kernel/shstk.c +73 -0
91 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v5 29/39] x86/shstk: Introduce routines modifying shstk
Quoted 26 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.
A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.
Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.
In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.
To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.
Quoted 3 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 30/39 x86/shstk: Handle signals for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.
One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.
The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.
For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.
So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.
In sigreturn, verify SSP is stored in the data format and pop the shadow stack.
---
v3:
- Drop shstk_setup_rstor_token() (Kees)
- Drop x32 signal support, since x32 support is dropped
v2:
- Switch to new shstk signal format
v1:
- Use xsave helpers.
- Expand commit log.
Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.
arch/x86/include/asm/shstk.h | 5 ++ arch/x86/kernel/shstk.c | 98 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_64.c | 6 +++ 4 files changed, 110 insertions(+)
arch/x86/include/asm/shstk.h +5 -0
@@ -6,6 +6,7 @@#include <linux/types.h> struct task_struct ; +struct ksignal ;#ifdef CONFIG_X86_USER_SHADOW_STACK struct thread_shstk {
@@ -19,6 +20,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,unsigned long stack_size, unsigned long *shstk_addr);void shstk_free (struct task_struct *p); +int setup_signal_shadow_stack (struct ksignal *ksig); +int restore_signal_shadow_stack (void );#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long features) {return -EINVAL ; }
@@ -28,6 +31,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long stack_size, unsigned long *shstk_addr) { return 0; }static inline void shstk_free (struct task_struct *p) {} +static inline int setup_signal_shadow_stack (struct ksignal *ksig) {return 0 ; } +static inline int restore_signal_shadow_stack (void ) {return 0 ; }#endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */
arch/x86/kernel/shstk.c +98 -0
104 lines not on this page · raw message
arch/x86/kernel/signal.c +1 -0
@@ -40,6 +40,7 @@#include <asm/syscall.h> #include <asm/sigframe.h> #include <asm/signal.h> +#include <asm/shstk.h> static inline int is_ia32_compat_frame (struct ksignal *ksig) {
arch/x86/kernel/signal_64.c +6 -0
@@ -175,6 +175,9 @@ int x64_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) frame =get_sigframe (ksig, regs,sizeof (struct rt_sigframe ), &fp); uc_flags =frame_uc_flags (regs); +if (setup_signal_shadow_stack (ksig)) +return -EFAULT ; +if (!user_access_begin (frame,sizeof (*frame)))return -EFAULT ;
@@ -260,6 +263,9 @@ SYSCALL_DEFINE0(rt_sigreturn)if (!restore_sigcontext (regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; +if (restore_signal_shadow_stack ()) + goto badframe; +if (restore_altstack (&frame->uc.uc_stack)) goto badframe;
Signature · 1 line
2.17.1
PATCH v5 31/39 x86/shstk: Introduce map_shadow_stack syscall
In reply to Rick EdgecombeWhen operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks.
Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable.
Previously, a new PROT_SHADOW_STACK was attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window.
The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token.
First, a new madvise() flag was explored, which could operate on the PROT_SHADOW_STACK memory. This had a couple downsides:
1. Extra checks were needed in mprotect() to prevent writable memory from ever becoming PROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the new madvise() to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to.
3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior.
So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.
The following example demonstrates how to create a new shadow stack with map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);
---
v5:
- Fix addr/mapped_addr (Kees)
- Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this)
- Return error for addresses below 4G
v3:
- Change syscall common -> 64 (Kees)
- Use bit shift notation instead of 0x1 for uapi header (Kees)
- Call do_mmap() with MAP_FIXED_NOREPLACE (Kees)
- Block unsupported flags (Kees)
- Require size >= 8 to set token (Kees)
v2:
- Change syscall to take address like mmap() for CRIU's usage
v1:
- New patch (replaces PROT_SHADOW_STACK).
arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/uapi/asm/mman.h | 3 ++ arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++---- include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 2 +- kernel/sys_ni.c | 1 + 6 files changed, 58 insertions(+), 9 deletions(-)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 64 map_shadow_stack sys_map_shadow_stack # # Due to a historical design error, certain syscalls are numbered differently
arch/x86/include/uapi/asm/mman.h +3 -0
@@ -13,6 +13,9 @@ ((key) & 0x8 ? VM_PKEY_BIT3 : 0))#endif +/* Flags for map_shadow_stack(2) */ +#define SHADOW_STACK_SET_TOKEN (1ULL << 0)/* Set up a restore token in the shadow stack */ +#include <asm-generic/mman.h> #endif /* _ASM_X86_MMAN_H */
arch/x86/kernel/shstk.c +51 -8
96 lines not on this page · raw message
include/linux/syscalls.h +1 -0
@@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags);asmlinkage long sys_set_mempolicy_home_node (unsigned long start,unsigned long len,unsigned long home_node,unsigned long flags); +asmlinkage long sys_map_shadow_stack (unsigned long addr,unsigned long size,unsigned int flags);/* * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +1 -1
@@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)#undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452/* * 32 bit systems traditionally used different
kernel/sys_ni.c +1 -0
@@ -381,6 +381,7 @@ COND_SYSCALL(vm86old);COND_SYSCALL (modify_ldt);COND_SYSCALL (vm86);COND_SYSCALL (kexec_file_load); +COND_SYSCALL (map_shadow_stack);/* s390 */ COND_SYSCALL (s390_pci_mmio_read);
Signature · 1 line
2.17.1
Re: [PATCH v5 31/39] x86/shstk: Introduce map_shadow_stack syscall
Quoted 45 lines · Rick Edgecombe
When operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks.
Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable.
Previously, a new
PROT_SHADOW_STACKwas attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window.The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token.
First, a new
madvise()flag was explored, which could operate on thePROT_SHADOW_STACKmemory. This had a couple downsides:
1. Extra checks were needed inmprotect()to prevent writable memory from ever becomingPROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the newmadvise()to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to.
3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior.So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new
map_shadow_stacksyscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.
Quoted 7 lines
The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk =map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 32/39 x86/shstk: Support WRSS for userspace
In reply to Rick EdgecombeFor the current shadow stack implementation, shadow stacks contents can't easily be provisioned with arbitrary data. This property helps apps protect themselves better, but also restricts any potential apps that may want to do exotic things at the expense of a little security.
The x86 shadow stack feature introduces a new instruction, WRSS, which can be enabled to write directly to shadow stack permissioned memory from userspace. Allow it to get enabled via the prctl interface.
Only enable the userspace WRSS instruction, which allows writes to userspace shadow stacks from userspace. Do not allow it to be enabled independently of shadow stack, as HW does not support using WRSS when shadow stack is disabled.
From a fault handler perspective, WRSS will behave very similar to WRUSS, which is treated like a user access from a #PF err code perspective.
---
v5:
- Switch to EOPNOTSUPP
- Move set_clr_bits_msrl() to patch where it is first used
- Commit log formatting
v3:
- Make wrss_control() static
- Fix verbiage in commit log (Kees)
v2:
- Add some commit log verbiage from (Dave Hansen)
v1:
- New patch.
arch/x86/include/asm/msr.h | 11 +++++++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/shstk.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 42 insertions(+), 1 deletion(-)
arch/x86/include/asm/msr.h +11 -0
@@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs);int msr_set_bit (u32 msr,u8 bit);int msr_clear_bit (u32 msr,u8 bit); +/* Helper that can never get accidentally un-inlined. */ +#define set_clr_bits_msrl (msr, set, clear) do { \ + u64 __val, __new_val; \ + \ + rdmsrl(msr, __val); \ + __new_val = (__val & ~(clear)) | (set); \ + \ + if (__new_val != __val) \ + wrmsrl(msr, __new_val); \ +} while (0) +#ifdef CONFIG_SMP int rdmsr_on_cpu (unsigned int cpu,u32 msr_no,u32 *l,u32 *h);int wrmsr_on_cpu (unsigned int cpu,u32 msr_no,u32 l,u32 h);
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -28,5 +28,6 @@/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1)#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/shstk.c +30 -1
@@ -364,6 +364,35 @@ void shstk_free(struct task_struct *tsk)unmap_shadow_stack (shstk->base, shstk->size); } +static int wrss_control (bool enable) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return -EOPNOTSUPP ; + +/* +* Only enable wrss if shadow stack is enabled. If shadow stack is not +* enabled, wrss will already be disabled, so don't bother clearing it +* when disabling. +*/ +if (!features_enabled (ARCH_SHSTK_SHSTK )) +return -EPERM ; + +/* Already enabled/disabled? */ +if (features_enabled (ARCH_SHSTK_WRSS ) == enable) +return 0 ; + +fpregs_lock_and_load (); +if (enable) { +set_clr_bits_msrl (MSR_IA32_U_CET ,CET_WRSS_EN ,0 ); +features_set (ARCH_SHSTK_WRSS ); + }else { +set_clr_bits_msrl (MSR_IA32_U_CET ,0 ,CET_WRSS_EN ); +features_clr (ARCH_SHSTK_WRSS ); + } +fpregs_unlock (); + +return 0 ; +}static int shstk_disable (void ) {
@@ -381,7 +410,7 @@ static int shstk_disable(void)fpregs_unlock ();shstk_free (current); -features_clr (ARCH_SHSTK_SHSTK ); +features_clr (ARCH_SHSTK_SHSTK |ARCH_SHSTK_WRSS );return 0 ; }
Signature · 1 line
2.17.1
Re: [PATCH v5 32/39] x86/shstk: Support WRSS for userspace
Quoted 14 lines · Rick Edgecombe
For the current shadow stack implementation, shadow stacks contents can't easily be provisioned with arbitrary data. This property helps apps protect themselves better, but also restricts any potential apps that may want to do exotic things at the expense of a little security.
The x86 shadow stack feature introduces a new instruction, WRSS, which can be enabled to write directly to shadow stack permissioned memory from userspace. Allow it to get enabled via the prctl interface.
Only enable the userspace WRSS instruction, which allows writes to userspace shadow stacks from userspace. Do not allow it to be enabled independently of shadow stack, as HW does not support using WRSS when shadow stack is disabled.
Quoted 6 lines
From a fault handler perspective, WRSS will behave very similar to WRUSS, which is treated like a user access from a #PF err code perspective.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 33/39 x86: Expose thread features in /proc/$PID/status
In reply to Rick EdgecombeApplications and loaders can have logic to decide whether to enable shadow stack. They usually don't report whether shadow stack has been enabled or not, so there is no way to verify whether an application actually is protected by shadow stack.
Add two lines in /proc/$PID/status to report enabled and locked features.
Since, this involves referring to arch specific defines in asm/prctl.h, implement an arch breakout to emit the feature lines.
---
v4:
- Remove "CET" references
v3:
- Move to /proc/pid/status (Kees)
v2:
- New patch
arch/x86/kernel/cpu/proc.c | 23 +++++++++++++++++++++++ fs/proc/array.c | 6 ++++++ include/linux/proc_fs.h | 2 ++ 3 files changed, 31 insertions(+)
arch/x86/kernel/cpu/proc.c +23 -0
@@ -4,6 +4,8 @@#include <linux/string.h> #include <linux/seq_file.h> #include <linux/cpufreq.h> +#include <asm/prctl.h> +#include <linux/proc_fs.h> #include "cpu.h"
@@ -175,3 +177,24 @@ const struct seq_operations cpuinfo_op = { .stop = c_stop, .show = show_cpuinfo, }; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +static void dump_x86_features (struct seq_file *m,unsigned long features) +{ +if (features &ARCH_SHSTK_SHSTK ) +seq_puts (m,"shstk " ); +if (features &ARCH_SHSTK_WRSS ) +seq_puts (m,"wrss " ); +} + +void arch_proc_pid_thread_features (struct seq_file *m,struct task_struct *task) +{ +seq_puts (m,"x86_Thread_features:\t" ); +dump_x86_features (m, task->thread.features); +seq_putc (m,'\n' ); + +seq_puts (m,"x86_Thread_features_locked:\t" ); +dump_x86_features (m, task->thread.features_locked); +seq_putc (m,'\n' ); +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */
fs/proc/array.c +6 -0
@@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm)seq_printf (m,"THP_enabled:\t%d\n" , thp_enabled); } +__weak void arch_proc_pid_thread_features (struct seq_file *m, +struct task_struct *task) +{ +} +int proc_pid_status (struct seq_file *m,struct pid_namespace *ns,struct pid *pid,struct task_struct *task) {
@@ -451,6 +456,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,task_cpus_allowed (m, task);cpuset_task_status_allowed (m, task);task_context_switch_counts (m, task); +arch_proc_pid_thread_features (m, task);return 0 ; }
include/linux/proc_fs.h +2 -0
@@ -158,6 +158,8 @@ int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task);#endif /* CONFIG_PROC_PID_ARCH_STATUS */ +void arch_proc_pid_thread_features (struct seq_file *m,struct task_struct *task); +#else /* CONFIG_PROC_FS */ static inline void proc_root_init (void )
Signature · 1 line
2.17.1
PATCH v5 34/39 x86/shstk: Wire in shadow stack interface
In reply to Rick EdgecombeThe kernel now has the main shadow stack functionality to support applications. Wire in the WRSS and shadow stack enable/disable functions into the existing shadow stack API skeleton.
---
v4:
- Remove "CET" references
v2:
- Split from other patches
arch/x86/kernel/shstk.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/shstk.c +8 -0
@@ -465,9 +465,17 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features)return -EINVAL ;if (option ==ARCH_SHSTK_DISABLE ) { +if (features &ARCH_SHSTK_WRSS ) +return wrss_control (false); +if (features &ARCH_SHSTK_SHSTK ) +return shstk_disable ();return -EINVAL ; }/* Handle ARCH_SHSTK_ENABLE */ +if (features &ARCH_SHSTK_SHSTK ) +return shstk_setup (); +if (features &ARCH_SHSTK_WRSS ) +return wrss_control (true);return -EINVAL ; }
Signature · 1 line
2.17.1
PATCH v5 35/39 selftests/x86: Add shadow stack test
In reply to Rick EdgecombeAdd a simple selftest for exercising some shadow stack behavior:
- map_shadow_stack syscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
- mprotect() of shadow stack memory
- Userfaultfd on shadow stack memory
Since this test exercises a recently added syscall manually, it needs to find the automatically created __NR_foo defines. Per the selftest documentation, KHDR_INCLUDES can be used to help the selftest Makefile's find the headers from the kernel source. This way the new selftest can be built inside the kernel source tree without installing the headers to the system. So also add KHDR_INCLUDES as described in the selftest docs, to facilitate this.
---
v5:
- Update 32 bit signal test with new ABI and better asm
v4:
- Add test for 32 bit signal ABI blocking
v3:
- Change "+m" to "=m" in write_shstk() (Andrew Cooper)
- Fix userfaultfd test with transparent huge pages by doing a MADV_DONTNEED, since the token write faults in the while stack with huge pages.
v2:
- Change print statements to more align with other selftests
- Add more tests
- Add KHDR_INCLUDES to Makefile
tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 667 ++++++++++++++++++ 2 files changed, 669 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
tools/testing/selftests/x86/Makefile +2 -2
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx test_shadow_stack # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
@@ -34,7 +34,7 @@ BINARIES_64 := $(TARGETS_C_64BIT_ALL:%=%_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) -CFLAGS := -O2 -g -std=gnu99 -pthread -Wall +CFLAGS := -O2 -g -std=gnu99 -pthread -Wall $(KHDR_INCLUDES) # call32_from_64 in thunks.S uses absolute addresses. ifeq ($(CAN_BUILD_WITH_NOPIE),1)
tools/testing/selftests/x86/test_shadow_stack.c new file +667 -0
667 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v5 36/39 x86/fpu: Add helper for initing features
In reply to Rick EdgecombeIf an xfeature is saved in a buffer, the xfeature's bit will be set in xsave->header.xfeatures. The CPU may opt to not save the xfeature if it is in it's init state. In this case the xfeature buffer address cannot be retrieved with get_xsave_addr().
Future patches will need to handle the case of writing to an xfeature that may not be saved. So provide helpers to init an xfeature in an xsave buffer.
This could of course be done directly by reaching into the xsave buffer, however this would not be robust against future changes to optimize the xsave buffer by compacting it. In that case the xsave buffer would need to be re-arranged as well. So the logic properly belongs encapsulated in a helper where the logic can be unified.
---
v2:
- New patch
arch/x86/kernel/fpu/xstate.c | 58 +++++++++++++++++++++++++++++------- arch/x86/kernel/fpu/xstate.h | 6 ++++ 2 files changed, 53 insertions(+), 11 deletions(-)
arch/x86/kernel/fpu/xstate.c +47 -11
@@ -934,6 +934,24 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr)return (void *)xsave +xfeature_get_offset (xcomp_bv, xfeature_nr); } +static int xsave_buffer_access_checks (int xfeature_nr) +{ +/* +* Do we even *have* xsave state? +*/ +if (!boot_cpu_has (X86_FEATURE_XSAVE )) +return 1 ; + +/* +* We should not ever be requesting features that we +* have not enabled. +*/ +if (WARN_ON_ONCE (!xfeature_enabled (xfeature_nr))) +return 1 ; + +return 0 ; +} +/* * Given the xsave area and a state inside, this function returns the * address of the state.
@@ -954,17 +972,7 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr)*/ void *get_xsave_addr (struct xregs_state *xsave,int xfeature_nr) { -/* -* Do we even *have* xsave state? -*/ -if (!boot_cpu_has (X86_FEATURE_XSAVE )) -return NULL ; - -/* -* We should not ever be requesting features that we -* have not enabled. -*/ -if (WARN_ON_ONCE (!xfeature_enabled (xfeature_nr))) +if (xsave_buffer_access_checks (xfeature_nr))return NULL ;/*
@@ -984,6 +992,34 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr)return __raw_xsave_addr (xsave, xfeature_nr); } +/* +* Given the xsave area and a state inside, this function +* initializes an xfeature in the buffer. +* +* get_xsave_addr() will return NULL if the feature bit is +* not present in the header. This function will make it so +* the xfeature buffer address is ready to be retrieved by +* get_xsave_addr(). +* +* Inputs: +* xstate: the thread's storage area for all FPU data +* xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, +* XFEATURE_SSE, etc...) +* Output: +* 1 if the feature cannot be inited, 0 on success +*/ +int init_xfeature (struct xregs_state *xsave,int xfeature_nr) +{ +if (xsave_buffer_access_checks (xfeature_nr)) +return 1 ; + +/* +* Mark the feature inited. +*/ + xsave->header.xfeatures |=BIT_ULL (xfeature_nr); +return 0 ; +} +#ifdef CONFIG_ARCH_HAS_PKEYS /*
arch/x86/kernel/fpu/xstate.h +6 -0
@@ -54,6 +54,12 @@ extern void fpu__init_cpu_xstate(void);extern void fpu__init_system_xstate (unsigned int legacy_size);extern void *get_xsave_addr (struct xregs_state *xsave,int xfeature_nr); +extern int init_xfeature (struct xregs_state *xsave,int xfeature_nr); + +static inline int xfeature_saved (struct xregs_state *xsave,int xfeature_nr) +{ +return xsave->header.xfeatures &BIT_ULL (xfeature_nr); +}static inline u64 xfeatures_mask_supervisor (void ) {
Signature · 1 line
2.17.1
PATCH v5 37/39 x86: Add PTRACE interface for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Some applications (like GDB) would like to tweak shadow stack state via ptrace. This allows for existing functionality to continue to work for seized shadow stack applications. Provide an regset interface for manipulating the shadow stack pointer (SSP).
There is already ptrace functionality for accessing xstate, but this does not include supervisor xfeatures. So there is not a completely clear place for where to put the shadow stack state. Adding it to the user xfeatures regset would complicate that code, as it currently shares logic with signals which should not have supervisor features.
Don't add a general supervisor xfeature regset like the user one, because it is better to maintain flexibility for other supervisor xfeatures to define their own interface. For example, an xfeature may decide not to expose all of it's state to userspace, as is actually the case for shadow stack ptrace functionality. A lot of enum values remain to be used, so just put it in dedicated shadow stack regset.
The only downside to not having a generic supervisor xfeature regset, is that apps need to be enlightened of any new supervisor xfeature exposed this way (i.e. they can't try to have generic save/restore logic). But maybe that is a good thing, because they have to think through each new xfeature instead of encountering issues when new a new supervisor xfeature was added.
By adding a shadow stack regset, it also has the effect of including the shadow stack state in a core dump, which could be useful for debugging.
The shadow stack specific xstate includes the SSP, and the shadow stack and WRSS enablement status. Enabling shadow stack or wrss in the kernel involves more than just flipping the bit. The kernel is made aware that it has to do extra things when cloning or handling signals. That logic is triggered off of separate feature enablement state kept in the task struct. So the flipping on HW shadow stack enforcement without notifying the kernel to change its behavior would severely limit what an application could do without crashing, and the results would depend on kernel internal implementation details. There is also no known use for controlling this state via prtace today. So only expose the SSP, which is something that userspace already has indirect control over.
---
v5:
- Check shadow stack enablement status for tracee (rppt)
- Fix typo in comment
v4:
- Make shadow stack only. Reduce to only supporting SSP register, and remove CET references (peterz)
- Add comment to not use 0x203, because binutils already looks for it in coredumps. (Christina Schimpe)
v3:
- Drop dependence on thread.shstk.size, and use thread.features bits
- Drop 32 bit support
v2:
- Check alignment on ssp.
- Block IBT bits.
- Handle init states instead of returning error.
- Add verbose commit log justifying the design.
arch/x86/include/asm/fpu/regset.h | 7 +-- arch/x86/kernel/fpu/regset.c | 87 +++++++++++++++++++++++++++++++ arch/x86/kernel/ptrace.c | 12 +++++ include/uapi/linux/elf.h | 2 + 4 files changed, 105 insertions(+), 3 deletions(-)
arch/x86/include/asm/fpu/regset.h +4 -3
@@ -7,11 +7,12 @@#include <linux/regset.h> -extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active; +extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active, + ssp_active;extern user_regset_get2_fn fpregs_get, xfpregs_get, fpregs_soft_get, - xstateregs_get; + xstateregs_get, ssp_get;extern user_regset_set_fn fpregs_set, xfpregs_set, fpregs_soft_set, - xstateregs_set; + xstateregs_set, ssp_set;/* * xstateregs_active == regset_fpregs_active. Please refer to the comment
arch/x86/kernel/fpu/regset.c +87 -0
99 lines not on this page · raw message
arch/x86/kernel/ptrace.c +12 -0
@@ -58,6 +58,7 @@ enum x86_regset_64 {REGSET64_FP ,REGSET64_IOPERM ,REGSET64_XSTATE , + REGSET64_SSP, };#define REGSET_GENERAL \
@@ -1267,6 +1268,17 @@ static struct user_regset x86_64_regsets[] __ro_after_init = { .active = ioperm_active, .regset_get = ioperm_get }, +#ifdef CONFIG_X86_USER_SHADOW_STACK + [REGSET64_SSP] = { + .core_note_type = NT_X86_SHSTK, + .n = 1, + .size = sizeof(u64), + .align = sizeof(u64), + .active = ssp_active, + .regset_get = ssp_get, + .set = ssp_set + }, +#endif }; static const struct user_regset_view user_x86_64_view = {
include/uapi/linux/elf.h +2 -0
@@ -406,6 +406,8 @@ typedef struct elf64_shdr {#define NT_386_TLS 0x200/* i386 TLS slots (struct user_desc) */ #define NT_386_IOPERM 0x201/* x86 io permission bitmap (1=deny) */ #define NT_X86_XSTATE 0x202/* x86 extended state using xsave */ +/* Old binutils treats 0x203 as a CET state */ +#define NT_X86_SHSTK 0x204/* x86 SHSTK state */ #define NT_S390_HIGH_GPRS 0x300/* s390 upper register halves */ #define NT_S390_TIMER 0x301/* s390 timer register */ #define NT_S390_TODCMP 0x302/* s390 TOD clock comparator register */
Signature · 1 line
2.17.1
Re: [PATCH v5 37/39] x86: Add PTRACE interface for shadow stack
Quoted 42 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
Some applications (like GDB) would like to tweak shadow stack state via ptrace. This allows for existing functionality to continue to work for seized shadow stack applications. Provide an regset interface for manipulating the shadow stack pointer (SSP).
There is already ptrace functionality for accessing xstate, but this does not include supervisor xfeatures. So there is not a completely clear place for where to put the shadow stack state. Adding it to the user xfeatures regset would complicate that code, as it currently shares logic with signals which should not have supervisor features.
Don't add a general supervisor xfeature regset like the user one, because it is better to maintain flexibility for other supervisor xfeatures to define their own interface. For example, an xfeature may decide not to expose all of it's state to userspace, as is actually the case for shadow stack ptrace functionality. A lot of enum values remain to be used, so just put it in dedicated shadow stack regset.
The only downside to not having a generic supervisor xfeature regset, is that apps need to be enlightened of any new supervisor xfeature exposed this way (i.e. they can't try to have generic save/restore logic). But maybe that is a good thing, because they have to think through each new xfeature instead of encountering issues when new a new supervisor xfeature was added.
By adding a shadow stack regset, it also has the effect of including the shadow stack state in a core dump, which could be useful for debugging.
The shadow stack specific xstate includes the SSP, and the shadow stack and WRSS enablement status. Enabling shadow stack or wrss in the kernel involves more than just flipping the bit. The kernel is made aware that it has to do extra things when cloning or handling signals. That logic is triggered off of separate feature enablement state kept in the task struct. So the flipping on HW shadow stack enforcement without notifying the kernel to change its behavior would severely limit what an application could do without crashing, and the results would depend on kernel internal implementation details. There is also no known use for controlling this state via prtace today. So only expose the SSP, which is something that userspace already has indirect control over.
Quoted 4 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
PATCH v5 38/39 x86/shstk: Add ARCH_SHSTK_UNLOCK
In reply to Rick EdgecombeFrom: Mike Rapoport <[email protected]>
Userspace loaders may lock features before a CRIU restore operation has the chance to set them to whatever state is required by the process being restored. Allow a way for CRIU to unlock features. Add it as an arch_prctl() like the other shadow stack operations, but restrict it being called by the ptrace arch_pctl() interface.
---
v4:
- Add to docs that it is ptrace only.
- Remove "CET" references
v3:
- Depend on CONFIG_CHECKPOINT_RESTORE (Kees)
Documentation/x86/shstk.rst | 4 ++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 9 +++++++-- 4 files changed, 13 insertions(+), 2 deletions(-)
Documentation/x86/shstk.rst +4 -0
@@ -73,6 +73,10 @@ arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) are ignored. The mask is ORed with the existing value. So any feature bits set here cannot be enabled or disabled afterwards. +arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) + Unlock features. 'features' is a mask of all features to unlock. All + bits set are processed, unset bits are ignored. Only works via ptrace. + The return values are as follows. On success, return 0. On error, errno can be::
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -25,6 +25,7 @@#define ARCH_SHSTK_ENABLE 0x5001#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0)
arch/x86/kernel/process_64.c +1 -0
@@ -835,6 +835,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_SHSTK_ENABLE :case ARCH_SHSTK_DISABLE :case ARCH_SHSTK_LOCK : +case ARCH_SHSTK_UNLOCK :return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;
arch/x86/kernel/shstk.c +7 -2
@@ -452,9 +452,14 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features)return 0 ; } -/* Don't allow via ptrace */ -if (task != current) +/* Only allow via ptrace */ +if (task != current) { +if (option ==ARCH_SHSTK_UNLOCK &&IS_ENABLED (CONFIG_CHECKPOINT_RESTORE )) { + task->thread.features_locked &= ~features; +return 0 ; + }return -EINVAL ; + }/* Do not allow to change locked features */ if (features & task->thread.features_locked)
Signature · 1 line
2.17.1
PATCH v5 39/39 x86/shstk: Add ARCH_SHSTK_STATUS
In reply to Rick EdgecombeCRIU and GDB need to get the current shadow stack and WRSS enablement status. This information is already available via /proc/pid/status, but this is inconvenient for CRIU because it involves parsing the text output in an area of the code where this is difficult. Provide a status arch_prctl(), ARCH_SHSTK_STATUS for retrieving the status. Have arg2 be a userspace address, and make the new arch_prctl simply copy the features out to userspace.
---
v5:
- Fix typo in commit log
v4:
- New patch
Documentation/x86/shstk.rst | 6 ++++++ arch/x86/include/asm/shstk.h | 4 ++-- arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 8 +++++++- 5 files changed, 17 insertions(+), 3 deletions(-)
Documentation/x86/shstk.rst +6 -0
@@ -77,6 +77,11 @@ arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) Unlock features. 'features' is a mask of all features to unlock. All bits set are processed, unset bits are ignored. Only works via ptrace. +arch_prctl(ARCH_SHSTK_STATUS, unsigned long addr) + Copy the currently enabled features to the address passed in addr. The + features are described using the bits passed into the others in + 'features'. + The return values are as follows. On success, return 0. On error, errno can be::
@@ -84,6 +89,7 @@ be:: -ENOTSUPP if the feature is not supported by the hardware or kernel. -EINVAL arguments (non existing feature, etc) + -EFAULT if could not copy information back to userspace The feature's bits supported are::
arch/x86/include/asm/shstk.h +2 -2
@@ -14,7 +14,7 @@ struct thread_shstk {u64 size; }; -long shstk_prctl (struct task_struct *task,int option,unsigned long features); +long shstk_prctl (struct task_struct *task,int option,unsigned long arg2);void reset_thread_features (void ); int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size,
@@ -24,7 +24,7 @@ int setup_signal_shadow_stack(struct ksignal *ksig);int restore_signal_shadow_stack (void );#else static inline long shstk_prctl (struct task_struct *task,int option, -unsigned long features) {return -EINVAL ; } +unsigned long arg2) {return -EINVAL ; }static inline void reset_thread_features (void ) {} static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -26,6 +26,7 @@#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003#define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0)
arch/x86/kernel/process_64.c +1 -0
@@ -836,6 +836,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_SHSTK_DISABLE :case ARCH_SHSTK_LOCK :case ARCH_SHSTK_UNLOCK : +case ARCH_SHSTK_STATUS :return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;
arch/x86/kernel/shstk.c +7 -1
@@ -445,8 +445,14 @@ SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsireturn alloc_shstk (addr, aligned_size, size, set_tok); } -long shstk_prctl (struct task_struct *task,int option,unsigned long features) +long shstk_prctl (struct task_struct *task,int option,unsigned long arg2) { +unsigned long features = arg2; + +if (option ==ARCH_SHSTK_STATUS ) { +return put_user (task->thread.features, (unsigned long __user *)arg2); + } +if (option ==ARCH_SHSTK_LOCK ) { task->thread.features_locked |= features;return 0 ;
Signature · 1 line
2.17.1
Re: [PATCH v5 39/39] x86/shstk: Add ARCH_SHSTK_STATUS
Quoted 11 lines · Rick Edgecombe
CRIU and GDB need to get the current shadow stack and WRSS enablement status. This information is already available via
/proc/pid/status, but this is inconvenient for CRIU because it involves parsing the text output in an area of the code where this is difficult. Provide a statusarch_prctl(),ARCH_SHSTK_STATUSfor retrieving the status. Have arg2 be a userspace address, and make the newarch_prctlsimply copy the features out to userspace.Tested-by Pengfei Xu <[email protected]>Suggested-by Mike Rapoport <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
Andrew Morton
maintainerQuoted 1 line · On Thu, 19 Jan 2023 13:22:38 -0800 Rick Edgecombe <[email protected]>…
SHSTK
Sounds like me trying to swear in Russian while drunk.
Is there any chance of s/shstk/shadow_stack/g?
Quoted 8 lines · Andrew Morton
Quoted 1 line · On Thu, 19 Jan 2023 13:22:38 -0800 Rick Edgecombe <…
SHSTK
Sounds like me trying to swear in Russian while drunk.
Is there any chance of s/shstk/shadow_stack/g?
I'm fine with the name change. I think shstk got debated and picked early in the history of the series before I got involved. "shstk" is nice and short, but it's not completely clear what it is unless you already know about shadow stack. So there is a tradeoff of clarity and line length/wrapping. Does anyone else have any strong opinions?
Quoted 8 lines · Rick P
Quoted 7 lines · Andrew Morton
Quoted 1 line · On Thu, 19 Jan 2023 13:22:38 -0800 Rick Edgecombe <…
SHSTK
Sounds like me trying to swear in Russian while drunk.
Quoted 7 lines
Quoted 1 line
Is there any chance of s/shstk/shadow_stack/g?
I'm fine with the name change. I think shstk got debated and picked early in the history of the series before I got involved. "shstk" is nice and short, but it's not completely clear what it is unless you already know about shadow stack. So there is a tradeoff of clarity and line length/wrapping. Does anyone else have any strong opinions?
I prefer SHSTK because it specifically means x86's hardware shadow stack from CET. Lots of things can (and have) implemented things called "shadow stack".
Signature · 1 line
Kees Cook
Quoted 19 lines · Kees Cook
Quoted 10 lines · Rick P
Quoted 8 lines · Andrew Morton
Quoted 1 line · On Thu, 19 Jan 2023 13:22:38 -0800 Rick Edgecombe <…
SHSTK
Sounds like me trying to swear in Russian while drunk.
Is there any chance of s/shstk/shadow_stack/g?
Quoted 7 lines
I'm fine with the name change. I think shstk got debated and picked early in the history of the series before I got involved. "shstk" is
nice and short, but it's not completely clear what it is unless you already know about shadow stack. So there is a tradeoff of clarity and
line length/wrapping. Does anyone else have any strong opinions?
Quoted 4 lines
I prefer SHSTK because it specifically means x86's hardware shadow stack from CET. Lots of things can (and have) implemented things called
"shadow stack".
This makes sense to, especially if we can hide it more from the core-mm code per David Hildebrand's suggestion. I guess I'll leave it for now unless anyone else has a stronger opinion.
Quoted 2 lines · Rick Edgecombe
I left tested-by tags in place per discussion with testers. Testers, please retest.
Re-tested on my AMD system (Dell PowerEdge R6515 w/ EPYC 7713) and it looks like everything is still working properly.
The selftests seem to run cleanly:
[INFO] new_ssp = 7ff19be0dff8, *new_ssp = 7ff19be0e001
[INFO] changing ssp from 7ff19c7f1ff0 to 7ff19be0dff8
[INFO] ssp is now 7ff19be0e000
[OK] Shadow stack pivot
[OK] Shadow stack faults
[INFO] Corrupting shadow stack
[INFO] Generated shadow stack violation successfully
[OK] Shadow stack violation test
[INFO] Gup read -> shstk access success
[INFO] Gup write -> shstk access success
[INFO] Violation from normal write
[INFO] Gup read -> write access success
[INFO] Violation from normal write
[INFO] Gup write -> write access success
[INFO] Cow gup write -> write access success
[OK] Shadow gup test
[INFO] Violation from shstk access
[OK] mprotect() test
[OK] Userfaultfd test
[OK] 32 bit test
Additionally, I could see the control protection messages in dmesg when running the shstk violation test from here:
https://gitlab.com/cet-software/cet-smoke-test
ld-linux-x86-64[99764] control protection ip:401139 sp:7fff025507d8 ssp:7f186e017fd8 error:1(near ret) in shstk1[401000+1000]
Quoted 6 lines · Rick Edgecombe
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.
Quoted 141 lines
Previous version [1].
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/Kirill A. Shutemov (1):
x86: Introduce userspace API for shadow stackMike Rapoport (1):
x86/shstk: AddARCH_SHSTK_UNLOCKRick Edgecombe (14):
x86/fpu: Add helper for modifying xstate x86/mm: Introduce_PAGE_COW
x86/mm: Start actually marking_PAGE_COW
mm: Handle faultless write upgrades for shstk
mm: Don't allow write GUPs to shadow stack memory x86/mm: IntroduceMAP_ABOVE4G
mm: Warn on shadow stack memory in wrong vma x86/shstk: Introducemap_shadow_stacksyscall x86/shstk: Support WRSS for userspace
x86: Expose thread features in/proc/$PID/statusx86/shstk: Wire in shadow stack interface selftests/x86: Add shadow stack test x86/fpu: Add helper for initing features x86/shstk: AddARCH_SHSTK_STATUSYu-cheng Yu (23):
Documentation/x86: Add CET shadow stack description x86/shstk: AddKconfigoption for shadow stack x86/cpufeatures: Add CPU feature flags for shadow stacks x86/cpufeatures: Enable CET CR4 bit for shadow stack x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
x86: Add user control-protection fault handler x86/mm: Remove_PAGE_DIRTYfrom kernel RO pages x86/mm: Movepmd_write(),pud_write()up in the file x86/mm: Updatepte_modifyfor_PAGE_COW
x86/mm: Updateptep_set_wrprotect()andpmdp_set_wrprotect()for
transition from_PAGE_DIRTYto_PAGE_COW
mm: MoveVM_UFFD_MINOR_BITfrom 37 to 38
mm: IntroduceVM_SHADOW_STACKfor shadow stack memory x86/mm: Check shadow stack page fault errors
x86/mm: Updatemaybe_mkwrite()for shadow stack
mm: Fixup places that callpte_mkwrite()directly
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting
mm: Re-introducevm_flagstodo_mmap()
x86/shstk: Add user-mode shadow stack support x86/shstk: Handle thread shadow stack
x86/shstk: Introduce routines modifying shstk x86/shstk: Handle signals for shadow stack
x86: Add PTRACE interface for shadow stackDocumentation/filesystems/proc.rst | 1 + Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 176 +++++ arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/Kconfig | 24 + arch/x86/Kconfig.assembler | 5 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/disabled-features.h | 16 +- arch/x86/include/asm/fpu/api.h | 9 + arch/x86/include/asm/fpu/regset.h | 7 +- arch/x86/include/asm/fpu/sched.h | 3 +- arch/x86/include/asm/fpu/types.h | 16 +- arch/x86/include/asm/fpu/xstate.h | 6 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/mmu_context.h | 2 + arch/x86/include/asm/msr.h | 11 + arch/x86/include/asm/pgtable.h | 338 ++++++++- arch/x86/include/asm/pgtable_types.h | 65 +- arch/x86/include/asm/processor.h | 8 + arch/x86/include/asm/shstk.h | 40 ++ arch/x86/include/asm/special_insns.h | 13 + arch/x86/include/asm/tlbflush.h | 3 +- arch/x86/include/asm/trap_pf.h | 2 + arch/x86/include/asm/traps.h | 12 + arch/x86/include/uapi/asm/mman.h | 4 + arch/x86/include/uapi/asm/prctl.h | 12 + arch/x86/kernel/Makefile | 4 + arch/x86/kernel/cet.c | 152 ++++ arch/x86/kernel/cpu/common.c | 35 +- arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/proc.c | 23 + arch/x86/kernel/fpu/core.c | 59 +- arch/x86/kernel/fpu/regset.c | 87 +++ arch/x86/kernel/fpu/xstate.c | 148 ++-- arch/x86/kernel/fpu/xstate.h | 6 + arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/process.c | 18 +- arch/x86/kernel/process_64.c | 9 +- arch/x86/kernel/ptrace.c | 12 + arch/x86/kernel/shstk.c | 492 +++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 8 +- arch/x86/kernel/sys_x86_64.c | 6 +- arch/x86/kernel/traps.c | 87 --- arch/x86/mm/fault.c | 38 + arch/x86/mm/pat/set_memory.c | 2 +- arch/x86/mm/pgtable.c | 6 + arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- fs/aio.c | 2 +- fs/proc/array.c | 6 + fs/proc/task_mmu.c | 3 + include/linux/mm.h | 59 +- include/linux/mman.h | 4 + include/linux/pgtable.h | 35 + include/linux/proc_fs.h | 2 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/siginfo.h | 3 +- include/uapi/asm-generic/unistd.h | 2 +- include/uapi/linux/elf.h | 2 + ipc/shm.c | 2 +- kernel/sys_ni.c | 1 + mm/gup.c | 2 +- mm/huge_memory.c | 12 +- mm/memory.c | 7 +- mm/migrate_device.c | 4 +- mm/mmap.c | 12 +- mm/nommu.c | 4 +- mm/userfaultfd.c | 10 +- mm/util.c | 2 +- tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 667 ++++++++++++++++++ 78 files changed, 2578 insertions(+), 259 deletions(-) create mode 100644 Documentation/x86/shstk.rst create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/cet.c create mode 100644 arch/x86/kernel/shstk.c create mode 100644 tools/testing/selftests/x86/test_shadow_stack.cSignature · 1 line
2.17.1