Shadow stacks for userspace
Subsystems (44)
v6 cover Shadow stacks for userspace
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.
The main use case for shadow stack is providing protection against return oriented programming attacks. It works by maintaining a secondary (shadow) stack using a special memory type that has protections against modification. When executing a CALL instruction, the processor pushes the return address to both the normal stack and to the special permission shadow stack. Upon RET, the processor pops the shadow stack copy and compares it to the normal stack copy. For more details, see the coverletter from v1 [0].
The main changes in this version are the MM suggestions by David Hildenbrand to have pte_mkwrite() take a vma, and rename _PAGE_COW.
The former is split over three patches:
mm: Introduce pte_mkwrite_kernel()
s390/mm: Introduce pmd_mkwrite_kernel()
mm: Make pte_mkwrite() take a VMA
With these changes, and an adjustment to "mm: Warn on shadow stack memory in wrong vma", references to "shstk" are now only in x86 arch code, hopefully addressing Andrew Morton's concerns. There are still a couple VM_SHADOW_STACK references, which seems to be in keeping with the treatment of other VM_HIGH_ARCH flags. If other shadow stack implementations end up with identical logic, it can easily be refactored at that point.
There was also some more feedback from Boris which was incorporated.
I left tested-by tags in place per discussion with testers. Testers, please retest.
Previous version [1].
Thanks,
Rick
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
Kirill A. Shutemov (1):
x86: Introduce userspace API for shadow stack
Mike Rapoport (1):
x86/shstk: Add ARCH_SHSTK_UNLOCK
Rick Edgecombe (19):
x86/fpu: Add helper for modifying xstate
x86: Move control protection handler to separate file
mm: Introduce pte_mkwrite_kernel()
s390/mm: Introduce pmd_mkwrite_kernel()
mm: Make pte_mkwrite() take a VMA
x86/mm: Introduce _PAGE_SAVED_DIRTY
x86/mm: Start actually marking _PAGE_SAVED_DIRTY x86/mm: Teach pte_mkwrite() about stack memory
mm: Don't allow write GUPs to shadow stack memory x86/mm: Introduce MAP_ABOVE4G
mm: Warn on shadow stack memory in wrong vma x86/mm: Warn if create Write=0,Dirty=1 with raw prot x86/shstk: Introduce map_shadow_stack syscall x86/shstk: Support WRSS for userspace
x86: Expose thread features in /proc/$PID/status x86/shstk: Wire in shadow stack interface selftests/x86: Add shadow stack test x86/fpu: Add helper for initing features x86/shstk: Add ARCH_SHSTK_STATUS
Yu-cheng Yu (20):
Documentation/x86: Add CET shadow stack description x86/shstk: Add Kconfig option for shadow stack x86/cpufeatures: Add CPU feature flags for shadow stacks x86/cpufeatures: Enable CET CR4 bit for shadow stack x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states x86/shstk: Add user control-protection fault handler x86/mm: Remove _PAGE_DIRTY from kernel RO pages x86/mm: Move pmd_write(), pud_write() up in the file x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY
mm: Move VM_UFFD_MINOR_BIT from 37 to 38
mm: Introduce VM_SHADOW_STACK for shadow stack memory x86/mm: Check shadow stack page fault errors
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting
mm: Re-introduce vm_flags to do_mmap()
x86/shstk: Add user-mode shadow stack support x86/shstk: Handle thread shadow stack
x86/shstk: Introduce routines modifying shstk x86/shstk: Handle signals for shadow stack
x86: Add PTRACE interface for shadow stack
Documentation/filesystems/proc.rst | 1 + Documentation/mm/arch_pgtable_helpers.rst | 9 +- Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 176 +++++ arch/alpha/include/asm/pgtable.h | 6 +- arch/arc/include/asm/hugepage.h | 2 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 7 +- arch/arm/include/asm/pgtable-3level.h | 7 +- arch/arm/include/asm/pgtable.h | 2 +- arch/arm/kernel/signal.c | 2 +- arch/arm64/include/asm/pgtable.h | 9 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 +- arch/csky/include/asm/pgtable.h | 2 +- arch/hexagon/include/asm/pgtable.h | 2 +- arch/ia64/include/asm/pgtable.h | 2 +- arch/loongarch/include/asm/pgtable.h | 4 +- arch/m68k/include/asm/mcf_pgtable.h | 2 +- arch/m68k/include/asm/motorola_pgtable.h | 6 +- arch/m68k/include/asm/sun3_pgtable.h | 6 +- arch/microblaze/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +- arch/nios2/include/asm/pgtable.h | 2 +- arch/openrisc/include/asm/pgtable.h | 2 +- arch/parisc/include/asm/pgtable.h | 6 +- arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 4 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +- arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +- arch/riscv/include/asm/pgtable.h | 6 +- arch/s390/include/asm/hugetlb.h | 4 +- arch/s390/include/asm/pgtable.h | 14 +- arch/s390/mm/pageattr.c | 4 +- arch/sh/include/asm/pgtable_32.h | 10 +- arch/sparc/include/asm/pgtable_32.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/um/include/asm/pgtable.h | 2 +- arch/x86/Kconfig | 24 + arch/x86/Kconfig.assembler | 5 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/cpufeatures.h | 2 + arch/x86/include/asm/disabled-features.h | 16 +- arch/x86/include/asm/fpu/api.h | 9 + arch/x86/include/asm/fpu/regset.h | 7 +- arch/x86/include/asm/fpu/sched.h | 3 +- arch/x86/include/asm/fpu/types.h | 16 +- arch/x86/include/asm/fpu/xstate.h | 6 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/mmu_context.h | 2 + arch/x86/include/asm/msr.h | 11 + arch/x86/include/asm/pgtable.h | 322 ++++++++- arch/x86/include/asm/pgtable_types.h | 71 +- arch/x86/include/asm/processor.h | 8 + arch/x86/include/asm/shstk.h | 40 ++ arch/x86/include/asm/special_insns.h | 13 + arch/x86/include/asm/tlbflush.h | 3 +- arch/x86/include/asm/trap_pf.h | 2 + arch/x86/include/asm/traps.h | 12 + arch/x86/include/uapi/asm/mman.h | 4 + arch/x86/include/uapi/asm/prctl.h | 12 + arch/x86/kernel/Makefile | 4 + arch/x86/kernel/cet.c | 152 ++++ arch/x86/kernel/cpu/common.c | 35 +- arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/proc.c | 23 + arch/x86/kernel/fpu/core.c | 59 +- arch/x86/kernel/fpu/regset.c | 86 +++ arch/x86/kernel/fpu/xstate.c | 148 ++-- arch/x86/kernel/fpu/xstate.h | 6 + arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/process.c | 18 +- arch/x86/kernel/process_64.c | 9 +- arch/x86/kernel/ptrace.c | 12 + arch/x86/kernel/shstk.c | 491 +++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 8 +- arch/x86/kernel/sys_x86_64.c | 6 +- arch/x86/kernel/traps.c | 87 --- arch/x86/mm/fault.c | 38 + arch/x86/mm/pat/set_memory.c | 4 +- arch/x86/mm/pgtable.c | 38 + arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/mmu_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- arch/xtensa/include/asm/pgtable.h | 2 +- fs/aio.c | 2 +- fs/proc/array.c | 6 + fs/proc/task_mmu.c | 3 + include/asm-generic/hugetlb.h | 4 +- include/linux/mm.h | 46 +- include/linux/mman.h | 4 + include/linux/pgtable.h | 14 + include/linux/proc_fs.h | 2 + include/linux/syscalls.h | 1 + include/uapi/asm-generic/siginfo.h | 3 +- include/uapi/asm-generic/unistd.h | 2 +- include/uapi/linux/elf.h | 2 + ipc/shm.c | 2 +- kernel/sys_ni.c | 1 + mm/debug_vm_pgtable.c | 16 +- mm/gup.c | 2 +- mm/huge_memory.c | 7 +- mm/hugetlb.c | 4 +- mm/memory.c | 5 +- mm/migrate_device.c | 2 +- mm/mmap.c | 12 +- mm/mprotect.c | 2 +- mm/nommu.c | 4 +- mm/userfaultfd.c | 2 +- mm/util.c | 2 +- tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 676 ++++++++++++++++++ 117 files changed, 2671 insertions(+), 324 deletions(-) create mode 100644 Documentation/x86/shstk.rst create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/cet.c create mode 100644 arch/x86/kernel/shstk.c create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
Signature · 1 line
2.17.1
PATCH v6 1/41 Documentation/x86: Add CET shadow stack description
From: Yu-cheng Yu <[email protected]>
Introduce a new document on Control-flow Enforcement Technology (CET).
---
v5:
- Literal format tweaks (Bagas Sanjaya)
- Update EOPNOTSUPP text due to unification after comment from (Kees)
- Update 32 bit signal support with new behavior
- Remove capitalization on shadow stack (Boris)
- Fix typo
v4:
- Drop clearcpuid piece (Boris)
- Add some info about 32 bit
v3:
- Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper)
- Clarify which arch_prctl's can take multiple bits. (Kees)
- Describe ASLR characteristics of thread shadow stacks. (Kees)
- Add exec section. (Andrew Cooper)
- Fix some capitalization (Bagas Sanjaya)
- Update new location of enablement status proc.
- Add info about new user_shstk software capability.
- Add more info about what the kernel pushes to the shadow stack on signal.
v2:
- Updated to new arch_prctl() API
- Add bit about new proc status
--- Documentation/x86/index.rst | 1 + Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 Documentation/x86/shstk.rst
Documentation/x86/index.rst +1 -0
@@ -22,6 +22,7 @@ x86-specific Documentation mtrr pat intel-hfi + shstk iommu intel_txt amd-memory-encryption
Documentation/x86/shstk.rst new file +166 -0
166 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 2/41 x86/shstk: Add Kconfig option for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection.
Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create CONFIG_CET to signify that at least one CET feature is configured.
---
v5:
- Remove capitalization of shadow stack (Boris)
v3:
- Add X86_CET (Kees)
- Add back WRUSS dependency (Kees)
- Fix verbiage (Dave)
- Change from promt to bool (Kirill)
- Add more to commit log
v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)
Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.
--- arch/x86/Kconfig | 24 ++++++++++++++++++++++++ arch/x86/Kconfig.assembler | 5 +++++ 2 files changed, 29 insertions(+)
arch/x86/Kconfig +24 -0
@@ -1851,6 +1851,11 @@ config CC_HAS_IBT (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \ $(as-instr,endbr64) +config X86_CET + def_bool n + help + CET features configured (Shadow stack or IBT) + config X86_KERNEL_IBT prompt "Indirect Branch Tracking" def_bool y
@@ -1858,6 +1863,7 @@ config X86_KERNEL_IBT # https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f depends on !LD_IS_LLD || LLD_VERSION >= 140000 select OBJTOOL + select X86_CET help Build the kernel with support for Indirect Branch Tracking, a hardware support course-grain forward-edge Control Flow Integrity
@@ -1952,6 +1958,24 @@ config X86_SGX If unsure, say N. +config X86_USER_SHADOW_STACK + bool "X86 userspace shadow stack" + depends on AS_WRUSS + depends on X86_64 + select ARCH_USES_HIGH_VMA_FLAGS + select X86_CET + help + Shadow stack protection is a hardware feature that detects function + return address corruption. This helps mitigate ROP attacks. + Applications must be enabled to use it, and old userspace does not + get protection "for free". + + CPUs supporting shadow stacks were first released in 2020. + + See Documentation/x86/shstk.rst for more information. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/Kconfig.assembler +5 -0
@@ -19,3 +19,8 @@ config AS_TPAUSE def_bool $(as-instr,tpause %ecx) help Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7 + +config AS_WRUSS + def_bool $(as-instr,wrussq %rax$(comma)(%rbx)) + help + Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.17.1
PATCH v6 3/41 x86/cpufeatures: Add CPU feature flags for shadow stacks
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).
To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.
The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches.
---
v5:
- Drop "shstk" from cpuinfo (Boris)
- Remove capitalization on shadow stack (Boris)
v3:
- Add user specific shadow stack cpu cap (Andrew Cooper)
- Drop reviewed-bys from Boris and Kees due to the above change.
v2:
- Remove IBT reference in commit log (Kees)
- Describe xsaves dependency using text from (Dave)
v1:
- Remove IBT, can be added in a follow on IBT series.
--- arch/x86/include/asm/cpufeatures.h | 2 ++ arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +2 -0
@@ -309,6 +309,7 @@#define X86_FEATURE_MSR_TSX_CTRL (11*32+20)/* "" MSR IA32_TSX_CTRL (Intel) implemented */ #define X86_FEATURE_SMBA (11*32+21)/* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22)/* "" Bandwidth Monitoring Event Configuration */ +#define X86_FEATURE_USER_SHSTK (11*32+23)/* Shadow stack support for user mode applications */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4)/* AVX VNNI instructions */
@@ -375,6 +376,7 @@#define X86_FEATURE_OSPKE (16*32+ 4)/* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5)/* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6)/* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7)/* "" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8)/* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9)/* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10)/* Carry-Less Multiplication Double Quadword */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -99,6 +99,12 @@# define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31))#endif +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define DISABLE_USER_SHSTK 0 +#else +#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -114,7 +120,7 @@#define DISABLED_MASK9 (DISABLE_SGX)#define DISABLED_MASK10 0#define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ - DISABLE_CALL_DEPTH_TRACKING) + DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK)#define DISABLED_MASK12 0#define DISABLED_MASK13 0#define DISABLED_MASK14 0
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
@@ -81,6 +81,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, {} };
Signature · 1 line
2.17.1
PATCH v6 4/41 x86/cpufeatures: Enable CET CR4 bit for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs.
Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.
Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.
---
v5:
- Remove #ifdeffery (Boris)
v4:
- Add back dedicated command line disable: "nousershstk" (Boris)
v3:
- Remove stay new line (Boris)
- Simplify commit log (Andrew Cooper)
v2:
- In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support.
- Clear MSR_IA32_U_CET as well. (PeterZ)
--- arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/common.c +27 -8
@@ -599,27 +599,43 @@ __noendbr void ibt_restore(u64 save) static __always_inline void setup_cet(struct cpuinfo_x86 *c) { -u64 msr =CET_ENDBR_EN ; +bool user_shstk, kernel_ibt; -if (!HAS_KERNEL_IBT || - !cpu_feature_enabled (X86_FEATURE_IBT )) +if (!IS_ENABLED (CONFIG_X86_CET ))return ; -wrmsrl (MSR_IA32_S_CET , msr); + kernel_ibt =HAS_KERNEL_IBT &&cpu_feature_enabled (X86_FEATURE_IBT ); + user_shstk =cpu_feature_enabled (X86_FEATURE_SHSTK ) && +IS_ENABLED (CONFIG_X86_USER_SHADOW_STACK ); + +if (!kernel_ibt && !user_shstk) +return ; + +if (user_shstk) +set_cpu_cap (c,X86_FEATURE_USER_SHSTK ); + +if (kernel_ibt) +wrmsrl (MSR_IA32_S_CET ,CET_ENDBR_EN ); +else +wrmsrl (MSR_IA32_S_CET ,0 ); +cr4_set_bits (X86_CR4_CET ); -if (!ibt_selftest ()) { +if (kernel_ibt && !ibt_selftest ()) {pr_err ("IBT selftest: Failed!\n" );wrmsrl (MSR_IA32_S_CET ,0 );setup_clear_cpu_cap (X86_FEATURE_IBT ); -return ; } } __noendbr void cet_disable(void) { -if (cpu_feature_enabled (X86_FEATURE_IBT )) -wrmsrl (MSR_IA32_S_CET ,0 ); +if (!(cpu_feature_enabled (X86_FEATURE_IBT ) || +cpu_feature_enabled (X86_FEATURE_SHSTK ))) +return ; + +wrmsrl (MSR_IA32_S_CET ,0 ); +wrmsrl (MSR_IA32_U_CET ,0 ); }/*
@@ -1476,6 +1492,9 @@ static void __init cpu_parse_early_param(void)if (cmdline_find_option_bool (boot_command_line,"noxsaves" ))setup_clear_cpu_cap (X86_FEATURE_XSAVES ); +if (cmdline_find_option_bool (boot_command_line,"nousershstk" )) +setup_clear_cpu_cap (X86_FEATURE_USER_SHSTK ); + arglen =cmdline_find_option (boot_command_line,"clearcpuid" , arg,sizeof (arg));if (arglen <=0 )return ;
Signature · 1 line
2.17.1
PATCH v6 5/41 x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operation
The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size.
Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required.
Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable.
While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks.
Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET.
To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches.
---
v5:
- Move comments from end of lines in cet_user_state struct (Boris)
v3:
- Add missing "is" in commit log (Boris)
- Change to case statement for struct size checking (Boris)
- Adjust commas on xfeature_names (Kees, Boris)
v2:
- Change name to XFEATURE_CET_KERNEL_UNUSED (peterz)
KVM refresh:
- Reword commit log using some verbiage posted by Dave Hansen
- Remove unlikely to be used supervisor cet xsave struct
- Clarify that supervisor cet state is not saved by xsave
- Remove unused supervisor MSRs
--- arch/x86/include/asm/fpu/types.h | 16 +++++- arch/x86/include/asm/fpu/xstate.h | 6 ++- arch/x86/kernel/fpu/xstate.c | 90 +++++++++++++++---------------- 3 files changed, 61 insertions(+), 51 deletions(-)
arch/x86/include/asm/fpu/types.h +14 -2
@@ -115,8 +115,8 @@ enum xfeature {XFEATURE_PT_UNIMPLEMENTED_SO_FAR ,XFEATURE_PKRU ,XFEATURE_PASID , -XFEATURE_RSRVD_COMP_11 , -XFEATURE_RSRVD_COMP_12 , +XFEATURE_CET_USER , +XFEATURE_CET_KERNEL_UNUSED ,XFEATURE_RSRVD_COMP_13 ,XFEATURE_RSRVD_COMP_14 , XFEATURE_LBR,
@@ -138,6 +138,8 @@ enum xfeature {#define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)#define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU)#define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL_UNUSED)#define XFEATURE_MASK_LBR (1 << XFEATURE_LBR)#define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG)#define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA)
@@ -252,6 +254,16 @@ struct pkru_state {u32 pad; } __packed; +/* +* State component 11 is Control-flow Enforcement user states +*/ +struct cet_user_state { +/* user control-flow settings */ +u64 user_cet; +/* user shadow stack pointer */ +u64 user_ssp; +}; +/* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth).
arch/x86/include/asm/fpu/xstate.h +4 -2
@@ -50,7 +50,8 @@#define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA/* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER)/* * A supervisor state component may not always contain valuable information,
@@ -77,7 +78,8 @@* Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL)/* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
arch/x86/kernel/fpu/xstate.c +43 -47
137 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 6/41 x86/fpu: Add helper for modifying xstate
In reply to Rick EdgecombeJust like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.
When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.
The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.
Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.
---
v6:
- Drop "but appear to work" (Boris)
v5:
- Fix spelling error (Boris)
- Don't export fpregs_lock_and_load() (Boris)
v3:
- Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees)
- Elaborate in comment about helper. (Dave)
v2:
- Drop optimization of writing directly the buffer, and change API accordingly.
- fpregs_lock_and_load() suggested by tglx
- Some commit log verbiage from dhansen
--- arch/x86/include/asm/fpu/api.h | 9 +++++++++ arch/x86/kernel/fpu/core.c | 18 ++++++++++++++++++ 2 files changed, 27 insertions(+)
arch/x86/include/asm/fpu/api.h +9 -0
@@ -82,6 +82,15 @@ static inline void fpregs_unlock(void)preempt_enable (); } +/* +* FPU state gets lazily restored before returning to userspace. So when in the +* kernel, the valid FPU state may be kept in the buffer. This function will force +* restore all the fpu state to the registers early if needed, and lock them from +* being automatically saved/restored. Then FPU state can be modified safely in the +* registers, before unlocking with fpregs_unlock(). +*/ +void fpregs_lock_and_load (void ); +#ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent (void );#else
arch/x86/kernel/fpu/core.c +18 -0
@@ -753,6 +753,24 @@ void switch_fpu_return(void) }EXPORT_SYMBOL_GPL (switch_fpu_return); +void fpregs_lock_and_load (void ) +{ +/* +* fpregs_lock() only disables preemption (mostly). So modifying state +* in an interrupt could screw up some in progress fpregs operation. +* Warn about it. +*/ +WARN_ON_ONCE (!irq_fpu_usable ()); +WARN_ON_ONCE (current->flags &PF_KTHREAD ); + +fpregs_lock (); + +fpregs_assert_state_consistent (); + +if (test_thread_flag (TIF_NEED_FPU_LOAD )) +fpregs_restore_userregs (); +} +#ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this
Signature · 1 line
2.17.1
PATCH v6 7/41 x86: Move control protection handler to separate file
In reply to Rick EdgecombeToday the control protection handler is defined in traps.c and used only for the kernel IBT feature. To reduce ifdeffery, move it to it's own file. In future patches, functionality will be added to make this handler also handle user shadow stack faults. So name the file cet.c.
No functional change.
---
v6:
- Split move to cet.c and shadow stack enhancements to fault handler to separate files. (Kees)
--- arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/cet.c | 76 ++++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/traps.c | 75 --------------------------------------- 3 files changed, 78 insertions(+), 75 deletions(-) create mode 100644 arch/x86/kernel/cet.c
arch/x86/kernel/Makefile +2 -0
@@ -144,6 +144,8 @@ obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_CALL_THUNKS) += callthunks.o +obj-$(CONFIG_X86_CET) += cet.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/cet.c new file +76 -0
76 lines not on this page · raw message
arch/x86/kernel/traps.c +0 -75
81 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 8/41 x86/shstk: Add user control-protection fault handler
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.
There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file cet.c, which is compiled in the case of either of the two CET features supported in the
kernel: kernel IBT or user mode shadow stack. Move some static inline functions from traps.c into a header so they can be used in cet.c.
Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it.
Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable.
The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler.
---
v6:
- Split into separate patches (Kees)
- Change to "x86/shstk" in commit log (Boris)
v5:
- Move to separate file to advoid ifdeffery (Boris)
- Improvements to commit log (Boris)
- Rename control_protection_err (Boris)
- Move comment from end of line in IBT fault handler (Boris)
v3:
- Shorten user/kernel #CP handler function names (peterz)
- Restore CP_ENDBR check to kernel handler (peterz)
- Utilize CONFIG_X86_CET (Kees)
- Unify "unexpected" warnings (Andrew Cooper)
- Use 2d array for error code chars (Andrew Cooper)
- Add comment about why to read SSP MSR before enabling interrupts
v2:
- Integrate with kernel IBT fault handler
- Update printed messages. (Dave)
- Remove array_index_nospec() usage. (Dave)
- Remove IBT messages. (Dave)
- Add enclave error code bit processing it case it can get triggered somehow.
- Add extra "unknown" in control_protection_err.
--- arch/arm/kernel/signal.c | 2 +- arch/arm64/kernel/signal.c | 2 +- arch/arm64/kernel/signal32.c | 2 +- arch/sparc/kernel/signal32.c | 2 +- arch/sparc/kernel/signal_64.c | 2 +- arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/idtentry.h | 2 +- arch/x86/include/asm/traps.h | 12 +++ arch/x86/kernel/cet.c | 94 +++++++++++++++++++++--- arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/signal_32.c | 2 +- arch/x86/kernel/signal_64.c | 2 +- arch/x86/kernel/traps.c | 12 --- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm.S | 2 +- include/uapi/asm-generic/siginfo.h | 3 +- 16 files changed, 117 insertions(+), 34 deletions(-)
arch/arm/kernel/signal.c +1 -1
@@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/arm64/kernel/signal.c +1 -1
@@ -1176,7 +1176,7 @@ void __init minsigstksz_setup(void)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/arm64/kernel/signal32.c +1 -1
@@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/sparc/kernel/signal32.c +1 -1
@@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/sparc/kernel/signal_64.c +1 -1
@@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/include/asm/disabled-features.h +7 -1
@@ -105,6 +105,12 @@#define DISABLE_USER_SHSTK (1 << (X86_FEATURE_USER_SHSTK & 31))#endif +#ifdef CONFIG_X86_KERNEL_IBT +#define DISABLE_IBT 0 +#else +#define DISABLE_IBT (1 << (X86_FEATURE_IBT & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -128,7 +134,7 @@#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ DISABLE_ENQCMD)#define DISABLED_MASK17 0 -#define DISABLED_MASK18 0 +#define DISABLED_MASK18 (DISABLE_IBT)#define DISABLED_MASK19 0#define DISABLED_MASK20 0#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 21)
arch/x86/include/asm/idtentry.h +1 -1
@@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF, xenpv_exc_double_fault);#endif /* #CP */ -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_CP , exc_control_protection);#endif
arch/x86/include/asm/traps.h +12 -0
@@ -47,4 +47,16 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs, struct stack_info *info);#endif +static inline void cond_local_irq_enable (struct pt_regs *regs) +{ +if (regs->flags &X86_EFLAGS_IF ) +local_irq_enable (); +} + +static inline void cond_local_irq_disable (struct pt_regs *regs) +{ +if (regs->flags &X86_EFLAGS_IF ) +local_irq_disable (); +} +#endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/cet.c +85 -9
113 lines not on this page · raw message
arch/x86/kernel/idt.c +1 -1
@@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = { ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE),#endif -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET INTG(X86_TRAP_CP, asm_exc_control_protection),#endif
arch/x86/kernel/signal_32.c +1 -1
@@ -402,7 +402,7 @@ int ia32_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/kernel/signal_64.c +1 -1
@@ -403,7 +403,7 @@ void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact)*/ static_assert (NSIGILL ==11 );static_assert (NSIGFPE ==15 ); -static_assert (NSIGSEGV ==9 ); +static_assert (NSIGSEGV ==10 );static_assert (NSIGBUS ==5 );static_assert (NSIGTRAP ==6 );static_assert (NSIGCHLD ==6 );
arch/x86/kernel/traps.c +0 -12
@@ -77,18 +77,6 @@DECLARE_BITMAP (system_vectors,NR_VECTORS ); -static inline void cond_local_irq_enable (struct pt_regs *regs) -{ -if (regs->flags &X86_EFLAGS_IF ) -local_irq_enable (); -} - -static inline void cond_local_irq_disable (struct pt_regs *regs) -{ -if (regs->flags &X86_EFLAGS_IF ) -local_irq_disable (); -} - __always_inline int is_valid_bugaddr(unsigned long addr) {if (addr <TASK_SIZE_MAX )
arch/x86/xen/enlighten_pv.c +1 -1
@@ -640,7 +640,7 @@ static struct trap_array_entry trap_array[] = {TRAP_ENTRY (exc_coprocessor_error, false ),TRAP_ENTRY (exc_alignment_check, false ),TRAP_ENTRY (exc_simd_coprocessor_error, false ), -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET TRAP_ENTRY(exc_control_protection, false ), #endif };
arch/x86/xen/xen-asm.S +1 -1
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check -#ifdef CONFIG_X86_KERNEL_IBT +#ifdef CONFIG_X86_CET xen_pv_trap asm_exc_control_protection #endif #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +2 -1
@@ -242,7 +242,8 @@ typedef struct siginfo {#define SEGV_ADIPERR 7/* Precise MCD exception */ #define SEGV_MTEAERR 8/* Asynchronous ARM MTE error */ #define SEGV_MTESERR 9/* Synchronous ARM MTE exception */ -#define NSIGSEGV 9 +#define SEGV_CPERR 10/* Control protection fault */ +#define NSIGSEGV 10/* * SIGBUS si_codes
Signature · 1 line
2.17.1
PATCH v6 9/41 x86/mm: Remove _PAGE_DIRTY from kernel RO pages
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages.
In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory.
When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs.
---
v6:
- Also remove dirty from newly added set_memory_rox()
v5:
- Spelling and grammer in commit log (Boris)
v3:
- Update commit log (Andrew Cooper, Peterz)
v2:
- Normalize PTE bit descriptions between patches
--- arch/x86/include/asm/pgtable_types.h | 6 +++--- arch/x86/mm/pat/set_memory.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
@@ -192,10 +192,10 @@ enum page_cache_mode {#define _KERNPG_TABLE (__PP|__RW| 0|___A| 0|___D| 0| 0| _ENC)#define _PAGE_TABLE_NOENC (__PP|__RW|_USR|___A| 0|___D| 0| 0)#define _PAGE_TABLE (__PP|__RW|_USR|___A| 0|___D| 0| 0| _ENC) -#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX|___D| 0|___G) -#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0|___D| 0|___G) +#define __PAGE_KERNEL_RO (__PP| 0| 0|___A|__NX| 0| 0|___G) +#define __PAGE_KERNEL_ROX (__PP| 0| 0|___A| 0| 0| 0|___G)#define __PAGE_KERNEL_NOCACHE (__PP|__RW| 0|___A|__NX|___D| 0|___G| __NC) -#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX|___D| 0|___G) +#define __PAGE_KERNEL_VVAR (__PP| 0|_USR|___A|__NX| 0| 0|___G)#define __PAGE_KERNEL_LARGE (__PP|__RW| 0|___A|__NX|___D|_PSE|___G)#define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW| 0|___A| 0|___D|_PSE|___G)#define __PAGE_KERNEL_WP (__PP|__RW| 0|___A|__NX|___D| 0|___G| __WP)
arch/x86/mm/pat/set_memory.c +2 -2
@@ -2073,12 +2073,12 @@ int set_memory_nx(unsigned long addr, int numpages)int set_memory_ro (unsigned long addr,int numpages) { -return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW),0 ); +return change_page_attr_clear (&addr, numpages,__pgprot (_PAGE_RW | _PAGE_DIRTY),0 ); }int set_memory_rox (unsigned long addr,int numpages) { -pgprot_t clr =__pgprot (_PAGE_RW); +pgprot_t clr =__pgprot (_PAGE_RW | _PAGE_DIRTY);if (__supported_pte_mask & _PAGE_NX) clr.pgprot |= _PAGE_NX;
Signature · 1 line
2.17.1
PATCH v6 10/41 x86/mm: Move pmd_write(), pud_write() up in the file
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
To prepare the introduction of _PAGE_SAVED_DIRTY, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes.
--- arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
arch/x86/include/asm/pgtable.h +12 -12
@@ -160,6 +160,18 @@ static inline int pte_write(pte_t pte)return pte_flags (pte) & _PAGE_RW; } +#define pmd_write pmd_write +static inline int pmd_write (pmd_t pmd) +{ +return pmd_flags (pmd) & _PAGE_RW; +} + +#define pud_write pud_write +static inline int pud_write (pud_t pud) +{ +return pud_flags (pud) & _PAGE_RW; +} +static inline int pte_huge (pte_t pte) {return pte_flags (pte) & _PAGE_PSE;
@@ -1120,12 +1132,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); -#define pmd_write pmd_write -static inline int pmd_write (pmd_t pmd) -{ -return pmd_flags (pmd) & _PAGE_RW; -} -#define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEARstatic inline pmd_t pmdp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp)
@@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); } -#define pud_write pud_write -static inline int pud_write (pud_t pud) -{ -return pud_flags (pud) & _PAGE_RW; -} -#ifndef pmdp_establish#define pmdp_establish pmdp_establish static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
Signature · 1 line
2.17.1
PATCH v6 11/41 mm: Introduce pte_mkwrite_kernel()
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for pte_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pte_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pte_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pte_mkwrite_kernel() and switch the kernel users over to it. Have pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pte_mkwrite() take a VMA.
Only do this for architectures that need it because they call pte_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst.
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/[email protected]/#t
v6:
- New patch
--- arch/arm64/include/asm/pgtable.h | 7 ++++++- arch/arm64/mm/trans_pgd.c | 4 ++-- arch/s390/include/asm/pgtable.h | 7 ++++++- arch/s390/mm/pageattr.c | 2 +- arch/x86/include/asm/pgtable.h | 7 ++++++- arch/x86/xen/mmu_pv.c | 2 +- 6 files changed, 22 insertions(+), 7 deletions(-)
arch/arm64/include/asm/pgtable.h +6 -1
@@ -180,13 +180,18 @@ static inline pmd_t set_pmd_bit(pmd_t pmd, pgprot_t prot)return pmd; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite_kernel (pte_t pte) { pte =set_pte_bit (pte,__pgprot (PTE_WRITE )); pte =clear_pte_bit (pte,__pgprot (PTE_RDONLY ));return pte; } +static inline pte_t pte_mkwrite (pte_t pte) +{ +return pte_mkwrite_kernel (pte); +} +static inline pte_t pte_mkclean (pte_t pte) { pte =clear_pte_bit (pte,__pgprot (PTE_DIRTY ));
arch/arm64/mm/trans_pgd.c +2 -2
@@ -40,7 +40,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)* read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ -set_pte (dst_ptep,pte_mkwrite (pte)); +set_pte (dst_ptep,pte_mkwrite_kernel (pte)); }else if (debug_pagealloc_enabled () && !pte_none (pte)) {/* * debug_pagealloc will removed the PTE_VALID bit if
@@ -53,7 +53,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)*/ BUG_ON (!pfn_valid (pte_pfn (pte))); -set_pte (dst_ptep,pte_mkpresent (pte_mkwrite (pte))); +set_pte (dst_ptep,pte_mkpresent (pte_mkwrite_kernel (pte))); } }
arch/s390/include/asm/pgtable.h +6 -1
@@ -991,7 +991,7 @@ static inline pte_t pte_wrprotect(pte_t pte)return set_pte_bit (pte,__pgprot (_PAGE_PROTECT)); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite_kernel (pte_t pte) { pte =set_pte_bit (pte,__pgprot (_PAGE_WRITE));if (pte_val (pte) & _PAGE_DIRTY)
@@ -999,6 +999,11 @@ static inline pte_t pte_mkwrite(pte_t pte)return pte; } +static inline pte_t pte_mkwrite (pte_t pte) +{ +return pte_mkwrite_kernel (pte); +} +static inline pte_t pte_mkclean (pte_t pte) { pte =clear_pte_bit (pte,__pgprot (_PAGE_DIRTY));
arch/s390/mm/pageattr.c +1 -1
@@ -96,7 +96,7 @@ static int walk_pte_level(pmd_t *pmdp, unsigned long addr, unsigned long end,if (flags &SET_MEMORY_RO ) new =pte_wrprotect (new);else if (flags &SET_MEMORY_RW ) - new =pte_mkwrite (pte_mkdirty (new)); + new =pte_mkwrite_kernel (pte_mkdirty (new));if (flags &SET_MEMORY_NX ) new =set_pte_bit (new,__pgprot (_PAGE_NOEXEC));else if (flags &SET_MEMORY_X )
arch/x86/include/asm/pgtable.h +6 -1
@@ -364,11 +364,16 @@ static inline pte_t pte_mkyoung(pte_t pte)return pte_set_flags (pte, _PAGE_ACCESSED); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite_kernel (pte_t pte) {return pte_set_flags (pte, _PAGE_RW); } +static inline pte_t pte_mkwrite (pte_t pte) +{ +return pte_mkwrite_kernel (pte); +} +static inline pte_t pte_mkhuge (pte_t pte) {return pte_set_flags (pte, _PAGE_PSE);
arch/x86/xen/mmu_pv.c +1 -1
@@ -150,7 +150,7 @@ void make_lowmem_page_readwrite(void *vaddr)if (pte ==NULL )return ;/* vaddr missing */ - ptev =pte_mkwrite (*pte); + ptev =pte_mkwrite_kernel (*pte);if (HYPERVISOR_update_va_mapping (address, ptev,0 ))BUG ();
Signature · 1 line
2.17.1
Re: [PATCH v6 11/41] mm: Introduce pte_mkwrite_kernel()
Quoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pte_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpte_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepte_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpte_mkwrite_kernel()and switch the kernel users over to it. Havepte_mkwrite()andpte_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepte_mkwrite()take a VMA.Only do this for architectures that need it because they call
pte_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.
Quoted 9 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
I think it's a little weird that it's the only PTE helper taking a vma, but it does seem like the right approach.
Signature · 1 line
Kees Cook
Re: [PATCH v6 11/41] mm: Introduce pte_mkwrite_kernel()
Quoted 30 lines · Kees Cook
Quoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pte_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpte_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepte_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpte_mkwrite_kernel()and switch the kernel users over to it. Havepte_mkwrite()andpte_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepte_mkwrite()take a VMA.Only do this for architectures that need it because they call
pte_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.Quoted 9 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Quoted 2 lines
I think it's a little weird that it's the only PTE helper taking a vma, but it does seem like the right approach.
Right. We could pass the vm flags instead, but not sure if that really improves the situation. So unless someone has a better idea, this LGTM.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 11/41] mm: Introduce pte_mkwrite_kernel()
In reply to Rick EdgecombeQuoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pte_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpte_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepte_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpte_mkwrite_kernel()and switch the kernel users over to it. Havepte_mkwrite()andpte_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepte_mkwrite()take a VMA.Only do this for architectures that need it because they call
pte_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.
Quoted 10 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Do we also have to care about pmd_mkwrite() ?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 11/41] mm: Introduce pte_mkwrite_kernel()
In reply to Rick EdgecombeQuoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pte_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpte_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepte_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpte_mkwrite_kernel()and switch the kernel users over to it. Havepte_mkwrite()andpte_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepte_mkwrite()take a VMA.Only do this for architectures that need it because they call
pte_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.
Quoted 10 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
PATCH v6 12/41 s390/mm: Introduce pmd_mkwrite_kernel()
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for pmd_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pmd_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pmd_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pmd_mkwrite_kernel() and switch the kernel users over to it. Have pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pmd_mkwrite() take a VMA.
Only do this for architectures that need it because they call pmd_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst.
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/[email protected]/#t
v6:
- New patch
--- arch/s390/include/asm/pgtable.h | 7 ++++++- arch/s390/mm/pageattr.c | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-)
arch/s390/include/asm/pgtable.h +6 -1
@@ -1425,7 +1425,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)return set_pmd_bit (pmd,__pgprot (_SEGMENT_ENTRY_PROTECT)); } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite_kernel (pmd_t pmd) { pmd =set_pmd_bit (pmd,__pgprot (_SEGMENT_ENTRY_WRITE));if (pmd_val (pmd) & _SEGMENT_ENTRY_DIRTY)
@@ -1433,6 +1433,11 @@ static inline pmd_t pmd_mkwrite(pmd_t pmd)return pmd; } +static inline pmd_t pmd_mkwrite (pmd_t pmd) +{ +return pmd_mkwrite_kernel (pmd); +} +static inline pmd_t pmd_mkclean (pmd_t pmd) { pmd =clear_pmd_bit (pmd,__pgprot (_SEGMENT_ENTRY_DIRTY));
arch/s390/mm/pageattr.c +1 -1
@@ -146,7 +146,7 @@ static void modify_pmd_page(pmd_t *pmdp, unsigned long addr,if (flags &SET_MEMORY_RO ) new =pmd_wrprotect (new);else if (flags &SET_MEMORY_RW ) - new =pmd_mkwrite (pmd_mkdirty (new)); + new =pmd_mkwrite_kernel (pmd_mkdirty (new));if (flags &SET_MEMORY_NX ) new =set_pmd_bit (new,__pgprot (_SEGMENT_ENTRY_NOEXEC));else if (flags &SET_MEMORY_X )
Signature · 1 line
2.17.1
Re: [PATCH v6 12/41] s390/mm: Introduce pmd_mkwrite_kernel()
Quoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pmd_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpmd_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepmd_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpmd_mkwrite_kernel()and switch the kernel users over to it. Havepmd_mkwrite()andpmd_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepmd_mkwrite()take a VMA.Only do this for architectures that need it because they call
pmd_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.
Quoted 7 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Yup, 1:1 refactor.
Signature · 1 line
Kees Cook
Re: [PATCH v6 12/41] s390/mm: Introduce pmd_mkwrite_kernel()
In reply to Rick EdgecombeQuoted 19 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pmd_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpmd_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepmd_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpmd_mkwrite_kernel()and switch the kernel users over to it. Havepmd_mkwrite()andpmd_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepmd_mkwrite()take a VMA.Only do this for architectures that need it because they call
pmd_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.
Quoted 8 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Heh, that answers my question to patch #11
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 12/41] s390/mm: Introduce pmd_mkwrite_kernel()
In reply to Rick EdgecombeQuoted 26 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these changes is to allow for
pmd_mkwrite()to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convertpmd_mkwrite()to take a VMA in order to facilitate this, however there are places in the kernel wherepmd_mkwrite()is called outside of the context of a VMA. These are for kernel memory. So create a new variant calledpmd_mkwrite_kernel()and switch the kernel users over to it. Havepmd_mkwrite()andpmd_mkwrite_kernel()be the same for now. Future patches will introduce changes to makepmd_mkwrite()take a VMA.Only do this for architectures that need it because they call
pmd_mkwrite()in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it inarch_pgtable_helpers.rst.Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
...
Quoted 4 lines
--- arch/s390/include/asm/pgtable.h | 7 ++++++- arch/s390/mm/pageattr.c | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-)
Re: [PATCH v6 12/41] s390/mm: Introduce pmd_mkwrite_kernel()
Quoted 1 line · Heiko Carstens
Acked-by Heiko Carstens <[email protected]>
Thanks!
PATCH v6 13/41 mm: Make pte_mkwrite() take a VMA
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call pte_mkwrite().
In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So make pte_mkwrite() take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.
Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().
No functional change.
---
Hi Non-x86 Arch’s,
x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.
Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.
Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/[email protected]/#t
v6:
- New patch
--- Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++--- arch/alpha/include/asm/pgtable.h | 6 +++++- arch/arc/include/asm/hugepage.h | 2 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++- arch/arm/include/asm/pgtable-3level.h | 7 ++++++- arch/arm/include/asm/pgtable.h | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/csky/include/asm/pgtable.h | 2 +- arch/hexagon/include/asm/pgtable.h | 2 +- arch/ia64/include/asm/pgtable.h | 2 +- arch/loongarch/include/asm/pgtable.h | 4 ++-- arch/m68k/include/asm/mcf_pgtable.h | 2 +- arch/m68k/include/asm/motorola_pgtable.h | 6 +++++- arch/m68k/include/asm/sun3_pgtable.h | 6 +++++- arch/microblaze/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +++--- arch/nios2/include/asm/pgtable.h | 2 +- arch/openrisc/include/asm/pgtable.h | 2 +- arch/parisc/include/asm/pgtable.h | 6 +++++- arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++-- arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +- arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +- arch/riscv/include/asm/pgtable.h | 6 +++--- arch/s390/include/asm/hugetlb.h | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/sh/include/asm/pgtable_32.h | 10 ++++++++-- arch/sparc/include/asm/pgtable_32.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 +++--- arch/um/include/asm/pgtable.h | 2 +- arch/x86/include/asm/pgtable.h | 6 ++++-- arch/xtensa/include/asm/pgtable.h | 2 +- include/asm-generic/hugetlb.h | 4 ++-- include/linux/mm.h | 2 +- mm/debug_vm_pgtable.c | 16 ++++++++-------- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 4 ++-- mm/memory.c | 4 ++-- mm/migrate_device.c | 2 +- mm/mprotect.c | 2 +- mm/userfaultfd.c | 2 +- 42 files changed, 106 insertions(+), 69 deletions(-)
Documentation/mm/arch_pgtable_helpers.rst +6 -3
@@ -48,7 +48,8 @@ PTE Page Table Helpers +---------------------------+--------------------------------------------------+ | pte_mkclean | Creates a clean PTE | +---------------------------+--------------------------------------------------+ -| pte_mkwrite | Creates a writable PTE | +| pte_mkwrite | Creates a writable PTE of the type specified by | +| | the VMA. | +---------------------------+--------------------------------------------------+ | pte_wrprotect | Creates a write protected PTE | +---------------------------+--------------------------------------------------+
@@ -120,7 +121,8 @@ PMD Page Table Helpers +---------------------------+--------------------------------------------------+ | pmd_mkclean | Creates a clean PMD | +---------------------------+--------------------------------------------------+ -| pmd_mkwrite | Creates a writable PMD | +| pmd_mkwrite | Creates a writable PMD of the type specified by | +| | the VMA. | +---------------------------+--------------------------------------------------+ | pmd_wrprotect | Creates a write protected PMD | +---------------------------+--------------------------------------------------+
@@ -224,7 +226,8 @@ HugeTLB Page Table Helpers +---------------------------+--------------------------------------------------+ | huge_pte_mkdirty | Creates a dirty HugeTLB | +---------------------------+--------------------------------------------------+ -| huge_pte_mkwrite | Creates a writable HugeTLB | +| huge_pte_mkwrite | Creates a writable HugeTLB of the type specified | +| | by the VMA. | +---------------------------+--------------------------------------------------+ | huge_pte_wrprotect | Creates a write protected HugeTLB | +---------------------------+--------------------------------------------------+
arch/alpha/include/asm/pgtable.h +5 -1
@@ -253,9 +253,13 @@ extern inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED;extern inline pte_t pte_wrprotect (pte_t pte) {pte_val (pte) |= _PAGE_FOW;return pte; }extern inline pte_t pte_mkclean (pte_t pte) {pte_val (pte) &= ~(__DIRTY_BITS);return pte; }extern inline pte_t pte_mkold (pte_t pte) {pte_val (pte) &= ~(__ACCESS_BITS);return pte; } -extern inline pte_t pte_mkwrite (pte_t pte) {pte_val (pte) &= ~_PAGE_FOW;return pte; }extern inline pte_t pte_mkdirty (pte_t pte) {pte_val (pte) |= __DIRTY_BITS;return pte; }extern inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |= __ACCESS_BITS;return pte; } +extern inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +pte_val (pte) &= ~_PAGE_FOW; +return pte; +}/* * The smp_rmb() in the following functions are required to order the load of
arch/arc/include/asm/hugepage.h +1 -1
@@ -21,7 +21,7 @@ static inline pmd_t pte_pmd(pte_t pte) }#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd))) -#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite (pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))#define pmd_mkdirty (pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd)))#define pmd_mkyoung (pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/arc/include/asm/pgtable-bits-arcv2.h +6 -1
@@ -84,7 +84,6 @@ PTE_BIT_FUNC(mknotpresent, &= ~(_PAGE_PRESENT)); PTE_BIT_FUNC(wrprotect, &= ~(_PAGE_WRITE)); -PTE_BIT_FUNC(mkwrite, |= (_PAGE_WRITE)); PTE_BIT_FUNC(mkclean, &= ~(_PAGE_DIRTY)); PTE_BIT_FUNC(mkdirty, |= (_PAGE_DIRTY)); PTE_BIT_FUNC(mkold, &= ~(_PAGE_ACCESSED));
@@ -92,6 +91,12 @@ PTE_BIT_FUNC(mkyoung, |= (_PAGE_ACCESSED)); PTE_BIT_FUNC(mkspecial, |= (_PAGE_SPECIAL)); PTE_BIT_FUNC(mkhuge, |= (_PAGE_HW_SZ)); +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +pte_val (pte) |= (_PAGE_WRITE); +return pte; +} +static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) {return __pte ((pte_val (pte) & _PAGE_CHG_MASK) |pgprot_val (newprot));
arch/arm/include/asm/pgtable-3level.h +6 -1
@@ -199,11 +199,16 @@ static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; } PMD_BIT_FUNC(wrprotect, |= L_PMD_SECT_RDONLY); PMD_BIT_FUNC(mkold, &= ~PMD_SECT_AF); -PMD_BIT_FUNC(mkwrite, &= ~L_PMD_SECT_RDONLY); PMD_BIT_FUNC(mkdirty, |= L_PMD_SECT_DIRTY); PMD_BIT_FUNC(mkclean, &= ~L_PMD_SECT_DIRTY); PMD_BIT_FUNC(mkyoung, |= PMD_SECT_AF); +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) +{ +pmd_val (pmd) |=L_PMD_SECT_RDONLY ; +return pmd; +} +#define pmd_mkhuge (pmd) (__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))#define pmd_pfn (pmd) (((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
arch/arm/include/asm/pgtable.h +1 -1
@@ -227,7 +227,7 @@ static inline pte_t pte_wrprotect(pte_t pte)return set_pte_bit (pte,__pgprot (L_PTE_RDONLY )); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return clear_pte_bit (pte,__pgprot (L_PTE_RDONLY )); }
arch/arm64/include/asm/pgtable.h +2 -2
@@ -187,7 +187,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return pte_mkwrite_kernel (pte); }
@@ -489,7 +489,7 @@ static inline int pmd_trans_huge(pmd_t pmd)#define pmd_cont (pmd) pte_cont(pmd_pte(pmd))#define pmd_wrprotect (pmd) pte_pmd(pte_wrprotect(pmd_pte(pmd)))#define pmd_mkold (pmd) pte_pmd(pte_mkold(pmd_pte(pmd))) -#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite (pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))#define pmd_mkclean (pmd) pte_pmd(pte_mkclean(pmd_pte(pmd)))#define pmd_mkdirty (pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd)))#define pmd_mkyoung (pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/csky/include/asm/pgtable.h +1 -1
@@ -176,7 +176,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;if (pte_val (pte) & _PAGE_MODIFIED)
arch/hexagon/include/asm/pgtable.h +1 -1
@@ -297,7 +297,7 @@ static inline pte_t pte_wrprotect(pte_t pte) }/* pte_mkwrite - mark page as writable */ -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;return pte;
arch/ia64/include/asm/pgtable.h +1 -1
@@ -265,7 +265,7 @@ ia64_phys_addr_valid (unsigned long addr)* access rights: */ #define pte_wrprotect (pte) (__pte(pte_val(pte) & ~_PAGE_AR_RW)) -#define pte_mkwrite (pte) (__pte(pte_val(pte) | _PAGE_AR_RW)) +#define pte_mkwrite (pte, vma) (__pte(pte_val(pte) | _PAGE_AR_RW))#define pte_mkold (pte) (__pte(pte_val(pte) & ~_PAGE_A))#define pte_mkyoung (pte) (__pte(pte_val(pte) | _PAGE_A))#define pte_mkclean (pte) (__pte(pte_val(pte) & ~_PAGE_D))
arch/loongarch/include/asm/pgtable.h +2 -2
@@ -360,7 +360,7 @@ static inline pte_t pte_mkdirty(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;if (pte_val (pte) & _PAGE_MODIFIED)
@@ -460,7 +460,7 @@ static inline int pmd_write(pmd_t pmd)return !!(pmd_val (pmd) & _PAGE_WRITE); } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {pmd_val (pmd) |= _PAGE_WRITE;if (pmd_val (pmd) & _PAGE_MODIFIED)
arch/m68k/include/asm/mcf_pgtable.h +1 -1
@@ -208,7 +208,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |=CF_PAGE_WRITABLE ;return pte;
arch/m68k/include/asm/motorola_pgtable.h +5 -1
@@ -152,7 +152,6 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & _PAGE_ACCESSED;static inline pte_t pte_wrprotect (pte_t pte) {pte_val (pte) |= _PAGE_RONLY;return pte; }static inline pte_t pte_mkclean (pte_t pte) {pte_val (pte) &= ~_PAGE_DIRTY;return pte; }static inline pte_t pte_mkold (pte_t pte) {pte_val (pte) &= ~_PAGE_ACCESSED;return pte; } -static inline pte_t pte_mkwrite (pte_t pte) {pte_val (pte) &= ~_PAGE_RONLY;return pte; }static inline pte_t pte_mkdirty (pte_t pte) {pte_val (pte) |= _PAGE_DIRTY;return pte; }static inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |= _PAGE_ACCESSED;return pte; }static inline pte_t pte_mknocache (pte_t pte)
@@ -165,6 +164,11 @@ static inline pte_t pte_mkcache(pte_t pte)pte_val (pte) = (pte_val (pte) & _CACHEMASK040) | m68k_supervisor_cachemode;return pte; } +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +pte_val (pte) &= ~_PAGE_RONLY; +return pte; +}#define swapper_pg_dir kernel_pg_dirextern pgd_t kernel_pg_dir[128 ];
arch/m68k/include/asm/sun3_pgtable.h +5 -1
@@ -140,10 +140,14 @@ static inline int pte_young(pte_t pte) { return pte_val(pte) & SUN3_PAGE_ACCESSstatic inline pte_t pte_wrprotect (pte_t pte) {pte_val (pte) &= ~SUN3_PAGE_WRITEABLE ;return pte; }static inline pte_t pte_mkclean (pte_t pte) {pte_val (pte) &= ~SUN3_PAGE_MODIFIED ;return pte; }static inline pte_t pte_mkold (pte_t pte) {pte_val (pte) &= ~SUN3_PAGE_ACCESSED ;return pte; } -static inline pte_t pte_mkwrite (pte_t pte) {pte_val (pte) |=SUN3_PAGE_WRITEABLE ;return pte; }static inline pte_t pte_mkdirty (pte_t pte) {pte_val (pte) |=SUN3_PAGE_MODIFIED ;return pte; }static inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |=SUN3_PAGE_ACCESSED ;return pte; }static inline pte_t pte_mknocache (pte_t pte) {pte_val (pte) |=SUN3_PAGE_NOCACHE ;return pte; } +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +pte_val (pte) |=SUN3_PAGE_WRITEABLE ; +return pte; +}// use this version when caches work... //static inline pte_t pte_mkcache(pte_t pte) { pte_val(pte) &= SUN3_PAGE_NOCACHE; return pte; } // until then, use:
arch/microblaze/include/asm/pgtable.h +1 -1
@@ -263,7 +263,7 @@ static inline pte_t pte_mkread(pte_t pte) \ {pte_val (pte) |= _PAGE_USER;return pte; }static inline pte_t pte_mkexec (pte_t pte) \ {pte_val (pte) |= _PAGE_USER | _PAGE_EXEC;return pte; } -static inline pte_t pte_mkwrite (pte_t pte) \ +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) \ {pte_val (pte) |= _PAGE_RW;return pte; }static inline pte_t pte_mkdirty (pte_t pte) \ {pte_val (pte) |= _PAGE_DIRTY;return pte; }
arch/mips/include/asm/pgtable.h +3 -3
@@ -309,7 +309,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { pte.pte_low |= _PAGE_WRITE;if (pte.pte_low & _PAGE_MODIFIED) {
@@ -364,7 +364,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;if (pte_val (pte) & _PAGE_MODIFIED)
@@ -591,7 +591,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)return pmd; } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {pmd_val (pmd) |= _PAGE_WRITE;if (pmd_val (pmd) & _PAGE_MODIFIED)
arch/nios2/include/asm/pgtable.h +1 -1
@@ -129,7 +129,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;return pte;
arch/openrisc/include/asm/pgtable.h +1 -1
@@ -247,7 +247,7 @@ static inline pte_t pte_mkold(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITE;return pte;
arch/parisc/include/asm/pgtable.h +5 -1
@@ -328,8 +328,12 @@ static inline pte_t pte_mkold(pte_t pte) { pte_val(pte) &= ~_PAGE_ACCESSED; retustatic inline pte_t pte_wrprotect (pte_t pte) {pte_val (pte) &= ~_PAGE_WRITE;return pte; }static inline pte_t pte_mkdirty (pte_t pte) {pte_val (pte) |= _PAGE_DIRTY;return pte; }static inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |= _PAGE_ACCESSED;return pte; } -static inline pte_t pte_mkwrite (pte_t pte) {pte_val (pte) |= _PAGE_WRITE;return pte; }static inline pte_t pte_mkspecial (pte_t pte) {pte_val (pte) |= _PAGE_SPECIAL;return pte; } +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +pte_val (pte) |= _PAGE_WRITE; +return pte; +}/* * Huge pte definitions.
arch/powerpc/include/asm/book3s/32/pgtable.h +1 -1
@@ -471,7 +471,7 @@ static inline pte_t pte_mkpte(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) | _PAGE_RW); }
arch/powerpc/include/asm/book3s/64/pgtable.h +2 -2
@@ -600,7 +600,7 @@ static inline pte_t pte_mkexec(pte_t pte)return __pte_raw (pte_raw (pte) |cpu_to_be64 (_PAGE_EXEC)); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {/* * write implies read, hence set both
@@ -1072,7 +1072,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)#define pmd_mkdirty (pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd)))#define pmd_mkclean (pmd) pte_pmd(pte_mkclean(pmd_pte(pmd)))#define pmd_mkyoung (pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd))) -#define pmd_mkwrite (pmd) pte_pmd(pte_mkwrite(pmd_pte(pmd))) +#define pmd_mkwrite (pmd, vma) pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY #define pmd_soft_dirty (pmd) pte_soft_dirty(pmd_pte(pmd))
arch/powerpc/include/asm/nohash/32/pgtable.h +1 -1
@@ -171,7 +171,7 @@ void unmap_kernel_page(unsigned long va); do { pte_update(mm, addr, ptep, ~0, 0, 0); } while (0)#ifndef pte_mkwrite -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) | _PAGE_RW); }
arch/powerpc/include/asm/nohash/32/pte-8xx.h +1 -1
@@ -101,7 +101,7 @@ static inline int pte_write(pte_t pte)#define pte_write pte_write -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) & ~_PAGE_RO); }
arch/powerpc/include/asm/nohash/64/pgtable.h +1 -1
@@ -85,7 +85,7 @@#ifndef __ASSEMBLY__/* pte_clear moved to later in this file */ -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) | _PAGE_RW); }
arch/riscv/include/asm/pgtable.h +3 -3
@@ -338,7 +338,7 @@ static inline pte_t pte_wrprotect(pte_t pte)/* static inline pte_t pte_mkread(pte_t pte) */ -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) | _PAGE_WRITE); }
@@ -624,9 +624,9 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)return pte_pmd (pte_mkyoung (pmd_pte (pmd))); } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) { -return pte_pmd (pte_mkwrite (pmd_pte (pmd))); +return pte_pmd (pte_mkwrite (pmd_pte (pmd), vma)); }static inline pmd_t pmd_wrprotect (pmd_t pmd)
arch/s390/include/asm/hugetlb.h +2 -2
@@ -102,9 +102,9 @@ static inline int huge_pte_dirty(pte_t pte)return pte_dirty (pte); } -static inline pte_t huge_pte_mkwrite (pte_t pte) +static inline pte_t huge_pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { -return pte_mkwrite (pte); +return pte_mkwrite (pte, vma); }static inline pte_t huge_pte_mkdirty (pte_t pte)
arch/s390/include/asm/pgtable.h +2 -2
@@ -999,7 +999,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return pte_mkwrite_kernel (pte); }
@@ -1433,7 +1433,7 @@ static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd)return pmd; } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {return pmd_mkwrite_kernel (pmd); }
arch/sh/include/asm/pgtable_32.h +8 -2
@@ -351,6 +351,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte)#define PTE_BIT_FUNC (h,fn,op) \ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; } +#define PTE_BIT_FUNC_VMA (h,fn,op) \ +static inline pte_t pte_##fn(pte_t pte, struct vm_area_struct *vma) \ +{ \ + pte.pte_##h op; \ + return pte; \ +}#ifdef CONFIG_X2TLB /*
@@ -359,11 +365,11 @@ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; }* kernel permissions), we attempt to couple them a bit more sanely here. */ PTE_BIT_FUNC(high, wrprotect, &= ~(_PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE)); -PTE_BIT_FUNC(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE); +PTE_BIT_FUNC_VMA(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE); PTE_BIT_FUNC(high, mkhuge, |= _PAGE_SZHUGE);#else PTE_BIT_FUNC(low, wrprotect, &= ~_PAGE_RW); -PTE_BIT_FUNC(low, mkwrite, |= _PAGE_RW); +PTE_BIT_FUNC_VMA(low, mkwrite, |= _PAGE_RW); PTE_BIT_FUNC(low, mkhuge, |= _PAGE_SZHUGE);#endif
arch/sparc/include/asm/pgtable_32.h +1 -1
@@ -241,7 +241,7 @@ static inline pte_t pte_mkold(pte_t pte)return __pte (pte_val (pte) & ~SRMMU_REF ); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return __pte (pte_val (pte) |SRMMU_WRITE ); }
arch/sparc/include/asm/pgtable_64.h +3 -3
@@ -463,7 +463,7 @@ static inline pte_t pte_mkclean(pte_t pte)return __pte (val); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {unsigned long val =pte_val (pte), mask;
@@ -753,11 +753,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)return __pmd (pte_val (pte)); } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {pte_t pte =__pte (pmd_val (pmd)); - pte =pte_mkwrite (pte); + pte =pte_mkwrite (pte, vma);return __pmd (pte_val (pte)); }
arch/um/include/asm/pgtable.h +1 -1
@@ -204,7 +204,7 @@ static inline pte_t pte_mkyoung(pte_t pte)return (pte); } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {if (unlikely (pte_get_bits (pte, _PAGE_RW)))return pte;
arch/x86/include/asm/pgtable.h +4 -2
@@ -369,7 +369,9 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)return pte_set_flags (pte, _PAGE_RW); } -static inline pte_t pte_mkwrite (pte_t pte) +struct vm_area_struct ; + +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {return pte_mkwrite_kernel (pte); }
@@ -470,7 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)return pmd_set_flags (pmd, _PAGE_ACCESSED); } -static inline pmd_t pmd_mkwrite (pmd_t pmd) +static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {return pmd_set_flags (pmd, _PAGE_RW); }
arch/xtensa/include/asm/pgtable.h +1 -1
@@ -258,7 +258,7 @@ static inline pte_t pte_mkdirty(pte_t pte) {pte_val (pte) |= _PAGE_DIRTY;return pte; }static inline pte_t pte_mkyoung (pte_t pte) {pte_val (pte) |= _PAGE_ACCESSED;return pte; } -static inline pte_t pte_mkwrite (pte_t pte) +static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) {pte_val (pte) |= _PAGE_WRITABLE;return pte; }#define pgprot_noncached (prot) \
include/asm-generic/hugetlb.h +2 -2
@@ -20,9 +20,9 @@ static inline unsigned long huge_pte_dirty(pte_t pte)return pte_dirty (pte); } -static inline pte_t huge_pte_mkwrite (pte_t pte) +static inline pte_t huge_pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { -return pte_mkwrite (pte); +return pte_mkwrite (pte, vma); }static inline pte_t huge_pte_mkdirty (pte_t pte)
include/linux/mm.h +1 -1
@@ -1102,7 +1102,7 @@ void free_compound_page(struct page *page);static inline pte_t maybe_mkwrite (pte_t pte,struct vm_area_struct *vma) {if (likely (vma->vm_flags &VM_WRITE )) - pte =pte_mkwrite (pte); + pte =pte_mkwrite (pte, vma);return pte; }
mm/debug_vm_pgtable.c +8 -8
@@ -107,10 +107,10 @@ static void __init pte_basic_tests(struct pgtable_debug_args *args, int idx)WARN_ON (!pte_same (pte, pte));WARN_ON (!pte_young (pte_mkyoung (pte_mkold (pte))));WARN_ON (!pte_dirty (pte_mkdirty (pte_mkclean (pte)))); -WARN_ON (!pte_write (pte_mkwrite (pte_wrprotect (pte)))); +WARN_ON (!pte_write (pte_mkwrite (pte_wrprotect (pte), args->vma)));WARN_ON (pte_young (pte_mkold (pte_mkyoung (pte))));WARN_ON (pte_dirty (pte_mkclean (pte_mkdirty (pte)))); -WARN_ON (pte_write (pte_wrprotect (pte_mkwrite (pte)))); +WARN_ON (pte_write (pte_wrprotect (pte_mkwrite (pte, args->vma))));WARN_ON (pte_dirty (pte_wrprotect (pte_mkclean (pte))));WARN_ON (!pte_dirty (pte_wrprotect (pte_mkdirty (pte)))); }
@@ -151,7 +151,7 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) pte =pte_mkclean (pte);set_pte_at (args->mm, args->vaddr, args->ptep, pte);flush_dcache_page (page); - pte =pte_mkwrite (pte); + pte =pte_mkwrite (pte, args->vma); pte =pte_mkdirty (pte);ptep_set_access_flags (args->vma, args->vaddr, args->ptep, pte,1 ); pte =ptep_get (args->ptep);
@@ -197,10 +197,10 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)WARN_ON (!pmd_same (pmd, pmd));WARN_ON (!pmd_young (pmd_mkyoung (pmd_mkold (pmd))));WARN_ON (!pmd_dirty (pmd_mkdirty (pmd_mkclean (pmd)))); -WARN_ON (!pmd_write (pmd_mkwrite (pmd_wrprotect (pmd)))); +WARN_ON (!pmd_write (pmd_mkwrite (pmd_wrprotect (pmd), args->vma)));WARN_ON (pmd_young (pmd_mkold (pmd_mkyoung (pmd))));WARN_ON (pmd_dirty (pmd_mkclean (pmd_mkdirty (pmd)))); -WARN_ON (pmd_write (pmd_wrprotect (pmd_mkwrite (pmd)))); +WARN_ON (pmd_write (pmd_wrprotect (pmd_mkwrite (pmd, args->vma))));WARN_ON (pmd_dirty (pmd_wrprotect (pmd_mkclean (pmd))));WARN_ON (!pmd_dirty (pmd_wrprotect (pmd_mkdirty (pmd))));/*
@@ -251,7 +251,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args) pmd =pmd_mkclean (pmd);set_pmd_at (args->mm, vaddr, args->pmdp, pmd);flush_dcache_page (page); - pmd =pmd_mkwrite (pmd); + pmd =pmd_mkwrite (pmd, args->vma); pmd =pmd_mkdirty (pmd);pmdp_set_access_flags (args->vma, vaddr, args->pmdp, pmd,1 ); pmd =READ_ONCE (*args->pmdp);
@@ -903,8 +903,8 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args) pte =mk_huge_pte (page, args->page_prot);WARN_ON (!huge_pte_dirty (huge_pte_mkdirty (pte))); -WARN_ON (!huge_pte_write (huge_pte_mkwrite (huge_pte_wrprotect (pte)))); -WARN_ON (huge_pte_write (huge_pte_wrprotect (huge_pte_mkwrite (pte)))); +WARN_ON (!huge_pte_write (huge_pte_mkwrite (huge_pte_wrprotect (pte), args->vma))); +WARN_ON (huge_pte_write (huge_pte_wrprotect (huge_pte_mkwrite (pte, args->vma))));#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB pte =pfn_pte (args->fixed_pmd_pfn, args->page_prot);
mm/huge_memory.c +3 -3
@@ -554,7 +554,7 @@ __setup("transparent_hugepage=", setup_transparent_hugepage);pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) {if (likely (vma->vm_flags &VM_WRITE )) - pmd =pmd_mkwrite (pmd); + pmd =pmd_mkwrite (pmd, vma);return pmd; }
@@ -1587,7 +1587,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) pmd =pmd_modify (oldpmd, vma->vm_page_prot); pmd =pmd_mkyoung (pmd);if (writable) - pmd =pmd_mkwrite (pmd); + pmd =pmd_mkwrite (pmd, vma);set_pmd_at (vma->vm_mm, haddr, vmf->pmd, pmd);update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);spin_unlock (vmf->ptl);
@@ -1935,7 +1935,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,/* See change_pte_range(). */ if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pmd_write (entry) &&can_change_pmd_writable (vma, addr, entry)) - entry =pmd_mkwrite (entry); + entry =pmd_mkwrite (entry, vma); ret =HPAGE_PMD_NR ;set_pmd_at (mm, addr, pmd, entry);
mm/hugetlb.c +2 -2
@@ -4898,7 +4898,7 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,if (writable) { entry =huge_pte_mkwrite (huge_pte_mkdirty (mk_huge_pte (page, - vma->vm_page_prot))); + vma->vm_page_prot)), vma); }else { entry =huge_pte_wrprotect (mk_huge_pte (page, vma->vm_page_prot));
@@ -4914,7 +4914,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma, {pte_t entry; - entry =huge_pte_mkwrite (huge_pte_mkdirty (huge_ptep_get (ptep))); + entry =huge_pte_mkwrite (huge_pte_mkdirty (huge_ptep_get (ptep)), vma);if (huge_ptep_set_access_flags (vma, address, ptep, entry,1 ))update_mmu_cache (vma, address, ptep); }
mm/memory.c +2 -2
@@ -4089,7 +4089,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) entry =mk_pte (page, vma->vm_page_prot); entry =pte_sw_mkyoung (entry);if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); + entry =pte_mkwrite (pte_mkdirty (entry), vma); vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);
@@ -4777,7 +4777,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) pte =pte_modify (old_pte, vma->vm_page_prot); pte =pte_mkyoung (pte);if (writable) - pte =pte_mkwrite (pte); + pte =pte_mkwrite (pte, vma);ptep_modify_prot_commit (vma, vmf->address, vmf->pte, old_pte, pte);update_mmu_cache (vma, vmf->address, vmf->pte);pte_unmap_unlock (vmf->pte, vmf->ptl);
mm/migrate_device.c +1 -1
@@ -646,7 +646,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } entry =mk_pte (page, vma->vm_page_prot);if (vma->vm_flags &VM_WRITE ) - entry =pte_mkwrite (pte_mkdirty (entry)); + entry =pte_mkwrite (pte_mkdirty (entry), vma); } ptep =pte_offset_map_lock (mm, pmdp, addr, &ptl);
mm/mprotect.c +1 -1
@@ -200,7 +200,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb,if ((cp_flags &MM_CP_TRY_CHANGE_WRITABLE ) && !pte_write (ptent) &&can_change_pte_writable (vma, addr, ptent)) - ptent =pte_mkwrite (ptent); + ptent =pte_mkwrite (ptent, vma);ptep_modify_prot_commit (vma, addr, pte, oldpte, ptent);if (pte_needs_flush (oldpte, ptent))
mm/userfaultfd.c +1 -1
@@ -85,7 +85,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, }if (writable) - _dst_pte =pte_mkwrite (_dst_pte); + _dst_pte =pte_mkwrite (_dst_pte, dst_vma);else /* * We need this to make sure write bit removed; as mk_pte()
Signature · 1 line
2.17.1
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
Quoted 39 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call
pte_mkwrite().In addition to
VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So makepte_mkwrite()take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.Apply the same changes for
pmd_mkwrite()andhuge_pte_mkwrite().No functional change.
Cc Michal Simek <[email protected]>Cc Dinh Nguyen <[email protected]>
Quoted 8 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
I'm not an arch maintainer, but it looks like a correct tree-wide refactor.
Signature · 1 line
Kees Cook
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
In reply to Rick EdgecombeQuoted 39 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call
pte_mkwrite().In addition to
VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So makepte_mkwrite()take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.Apply the same changes for
pmd_mkwrite()andhuge_pte_mkwrite().No functional change.
Cc Michal Simek <[email protected]>Cc Dinh Nguyen <[email protected]>
Quoted 8 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
I'm not an arch maintainer, but it looks like a correct tree-wide refactor.
Signature · 1 line
Kees Cook
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
In reply to Rick EdgecombeQuoted 4 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
...
Quoted 16 lines
---
Hi Non-x86 Arch’s,x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite()to take a VMA, so awareness of shadow stack memory can be moved into x86 code.Since
pXX_mkwrite()is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details insidearch/x86.
...
Quoted 17 lines
--- Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++--- arch/alpha/include/asm/pgtable.h | 6 +++++- arch/arc/include/asm/hugepage.h | 2 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++- arch/arm/include/asm/pgtable-3level.h | 7 ++++++- arch/arm/include/asm/pgtable.h | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/csky/include/asm/pgtable.h | 2 +- arch/hexagon/include/asm/pgtable.h | 2 +- arch/ia64/include/asm/pgtable.h | 2 +- arch/loongarch/include/asm/pgtable.h | 4 ++-- arch/m68k/include/asm/mcf_pgtable.h | 2 +- arch/m68k/include/asm/motorola_pgtable.h | 6 +++++- arch/m68k/include/asm/sun3_pgtable.h | 6 +++++- arch/microblaze/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +++---
Quoted 8 lines
arch/nios2/include/asm/pgtable.h | 2 +- arch/openrisc/include/asm/pgtable.h | 2 +- arch/parisc/include/asm/pgtable.h | 6 +++++- arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++-- arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +- arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +-
Looks like you discovered the joys of ppc's at-least 5 different MMU implementations, sorry :)
cheers
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
Quoted 1 line · Michael Ellerman
Thanks!
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
In reply to Rick EdgecombeQuoted 109 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call
pte_mkwrite().In addition to
VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So makepte_mkwrite()take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.Apply the same changes for
pmd_mkwrite()andhuge_pte_mkwrite().No functional change.
Cc Michal Simek <[email protected]>Cc Dinh Nguyen <[email protected]>Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
Hi Non-x86 Arch’s,x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change
pXX_mkwrite()to take a VMA, so awareness of shadow stack memory can be moved into x86 code.Since
pXX_mkwrite()is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details insidearch/x86.Testing was just 0-day build testing.
Hopefully that is enough context. Thanks!
[0] https://lore.kernel.org/lkml/[email protected]/#t
v6:
- New patch--- Documentation/mm/arch_pgtable_helpers.rst | 9 ++++++--- arch/alpha/include/asm/pgtable.h | 6 +++++- arch/arc/include/asm/hugepage.h | 2 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 7 ++++++- arch/arm/include/asm/pgtable-3level.h | 7 ++++++- arch/arm/include/asm/pgtable.h | 2 +- arch/arm64/include/asm/pgtable.h | 4 ++-- arch/csky/include/asm/pgtable.h | 2 +- arch/hexagon/include/asm/pgtable.h | 2 +- arch/ia64/include/asm/pgtable.h | 2 +- arch/loongarch/include/asm/pgtable.h | 4 ++-- arch/m68k/include/asm/mcf_pgtable.h | 2 +- arch/m68k/include/asm/motorola_pgtable.h | 6 +++++- arch/m68k/include/asm/sun3_pgtable.h | 6 +++++- arch/microblaze/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +++--- arch/nios2/include/asm/pgtable.h | 2 +- arch/openrisc/include/asm/pgtable.h | 2 +- arch/parisc/include/asm/pgtable.h | 6 +++++- arch/powerpc/include/asm/book3s/32/pgtable.h | 2 +- arch/powerpc/include/asm/book3s/64/pgtable.h | 4 ++-- arch/powerpc/include/asm/nohash/32/pgtable.h | 2 +- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 2 +- arch/powerpc/include/asm/nohash/64/pgtable.h | 2 +- arch/riscv/include/asm/pgtable.h | 6 +++--- arch/s390/include/asm/hugetlb.h | 4 ++-- arch/s390/include/asm/pgtable.h | 4 ++-- arch/sh/include/asm/pgtable_32.h | 10 ++++++++-- arch/sparc/include/asm/pgtable_32.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 +++--- arch/um/include/asm/pgtable.h | 2 +- arch/x86/include/asm/pgtable.h | 6 ++++-- arch/xtensa/include/asm/pgtable.h | 2 +- include/asm-generic/hugetlb.h | 4 ++-- include/linux/mm.h | 2 +-
Quoted 8 lines
mm/debug_vm_pgtable.c | 16 ++++++++-------- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 4 ++-- mm/memory.c | 4 ++-- mm/migrate_device.c | 2 +- mm/mprotect.c | 2 +- mm/userfaultfd.c | 2 +- 42 files changed, 106 insertions(+), 69 deletions(-)
That looks painful but IMHO worth it :)
Signature · 9 lines
Thanks, David / dhildenb _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
Quoted 3 lines · David Hildenbrand
That looks painful but IMHO worth it :)
Thanks. Yes it was not the most fun, but I agree - worth it.
Re: [PATCH v6 13/41] mm: Make pte_mkwrite() take a VMA
In reply to Rick EdgecombeQuoted 39 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call
pte_mkwrite().In addition to
VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So makepte_mkwrite()take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.Apply the same changes for
pmd_mkwrite()andhuge_pte_mkwrite().No functional change.
Cc Michal Simek <[email protected]>Cc Dinh Nguyen <[email protected]>
Quoted 9 lines
Tested-by Pengfei Xu <[email protected]>Suggested-by David Hildenbrand <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
PATCH v6 14/41 x86/mm: Introduce _PAGE_SAVED_DIRTY
In reply to Rick EdgecombeSome OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_SAVED_DIRTY in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW via fork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the SavedDirty bit. This happens in copy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.
(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support set Dirty=1.
There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.
Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to start creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are in place.
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK
v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)
v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability
v3:
- Add comment around _PAGE_TABLE in response to comment from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)
--- arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++--- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 138 insertions(+), 9 deletions(-)
arch/x86/include/asm/pgtable.h +79 -0
@@ -301,6 +301,45 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +/* +* COW and other write protection operations can result in Dirty=1,Write=0 +* PTEs. But in the case of X86_FEATURE_USER_SHSTK, the software SavedDirty bit +* is used, since the Dirty=1,Write=0 will result in the memory being treated as +* shadow stack by the HW. So when creating dirty, write-protected memory, a +* software bit is used _PAGE_BIT_SAVED_DIRTY. The following functions +* pte_mksaveddirty() and pte_clear_saveddirty() take a conventional dirty, +* write-protected PTE (Write=0,Dirty=1) and transition it to the shadow stack +* compatible version. (Write=0,SavedDirty=1). +*/ +static inline pte_t pte_mksaveddirty (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_SAVED_DIRTY); +} + +static inline pte_t pte_clear_saveddirty (pte_t pte) +{ +/* +* _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK kernels, +* since the HW dirty bit can be used without creating shadow stack +* memory. See the _PAGE_SAVED_DIRTY definition for more details. +*/ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + +/* +* PTE is getting copied-on-write, so it will be dirtied +* if writable, or made shadow stack if shadow stack and +* being copied on access. Set the dirty bit for both +* cases. +*/ + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {
@@ -420,6 +459,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_mksaveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_clear_saveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd) {
@@ -491,6 +550,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_mksaveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_clear_saveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_SAVED_DIRTY); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);
arch/x86/include/asm/pgtable_types.h +57 -8
97 lines not on this page · raw message
arch/x86/include/asm/tlbflush.h +2 -1
@@ -273,7 +273,8 @@ static inline bool pte_flags_need_flush(unsigned long oldflags,const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT | _PAGE_ACCESSED;const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 | - _PAGE_SOFTW3 | _PAGE_SOFTW4; + _PAGE_SOFTW3 | _PAGE_SOFTW4 | + _PAGE_SAVED_DIRTY;const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT | _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT | _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 |
Signature · 1 line
2.17.1
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 13 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.
The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.
Quoted 7 lines
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined
_PAGE_SAVED_DIRTYin place of the hardware_PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:
I would simplify (see below) and not repeat what the patch contains as comments already that detailed.
Quoted 184 lines
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW viafork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the SavedDirty bit. This happens incopy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, andget_user_pages(FOLL_FORCE)needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens atfork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support setDirty=1.There are six bits left available to software in the 64-bit PTE after consuming a bit for
_PAGE_SAVED_DIRTY. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.Implement only the infrastructure for
_PAGE_SAVED_DIRTY. Changes to start creating_PAGE_SAVED_DIRTYPTEs will follow once other pieces are in place.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Reviewed-by Kees Cook <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v6:
- Rename_PAGE_COWto_PAGE_SAVED_DIRTY(David Hildenbrand)
- Add_PAGE_SAVED_DIRTYto_PAGE_CHG_MASKv5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)v4:
- Teachpte_flags_need_flush()about_PAGE_COWbit
- Break apart patch for better bisectabilityv3:
- Add comment around_PAGE_TABLEin response to comment from (Andrew Cooper)
- Check for PSE inpmd_shstk(Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)--- arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++--- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 138 insertions(+), 9 deletions(-)arch/x86/include/asm/pgtable.h +79
-0@@ -301,6 +301,45 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +/* +* COW and other write protection operations can result in Dirty=1,Write=0 +* PTEs. But in the case of X86_FEATURE_USER_SHSTK, the software SavedDirty bit +* is used, since the Dirty=1,Write=0 will result in the memory being treated as +* shadow stack by the HW. So when creating dirty, write-protected memory, a +* software bit is used _PAGE_BIT_SAVED_DIRTY. The following functions +* pte_mksaveddirty() and pte_clear_saveddirty() take a conventional dirty, +* write-protected PTE (Write=0,Dirty=1) and transition it to the shadow stack +* compatible version. (Write=0,SavedDirty=1). +*/ +static inline pte_t pte_mksaveddirty (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_SAVED_DIRTY); +} + +static inline pte_t pte_clear_saveddirty (pte_t pte) +{ +/* +* _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK kernels, +* since the HW dirty bit can be used without creating shadow stack +* memory. See the _PAGE_SAVED_DIRTY definition for more details. +*/ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + +/* +* PTE is getting copied-on-write, so it will be dirtied +* if writable, or made shadow stack if shadow stack and +* being copied on access. Set the dirty bit for both +* cases. +*/ + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {@@ -420,6 +459,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_mksaveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_clear_saveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd) {@@ -491,6 +550,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_mksaveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_clear_saveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_SAVED_DIRTY); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags (pud, _PAGE_ACCESSED);arch/x86/include/asm/pgtable_types.h +2
-1@@ -21,7 +21,8 @@#define _PAGE_BIT_SOFTW2 10/* " */ #define _PAGE_BIT_SOFTW3 11/* " */ #define _PAGE_BIT_PAT_LARGE 12/* On 2MB or 1GB pages */ -#define _PAGE_BIT_SOFTW4 58/* available for programmer */ +#define _PAGE_BIT_SOFTW4 57/* available for programmer */ +#define _PAGE_BIT_SOFTW5 58/* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59/* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60/* Protection Keys, bit 2/4 */ #define _PAGE_BIT_PKEY_BIT2 61/* Protection Keys, bit 3/4 */
arch/x86/include/asm/pgtable_types.h line 42
Quoted 8 lines
arch/x86/include/asm/pgtable_types.h +5
-0@@ -34,6 +35,15 @@#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3/* software dirty tracking */ #define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4 +/* +* Indicates a Saved Dirty bit page. +*/ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5/* copy-on-write */
Nope, not "copy-on-write" :) It's more like "dirty bit when the hw-dirty bit cannot be used". Maybe simply drop the comment.
Quoted 8 lines
arch/x86/include/asm/pgtable_types.h +4
-0+#else +#define _PAGE_BIT_SAVED_DIRTY 0 +#endif +/* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */ #define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL
arch/x86/include/asm/pgtable_types.h line 133
Quoted 7 lines
arch/x86/include/asm/pgtable_types.h +4
-0@@ -117,6 +127,40 @@#define _PAGE_SOFTW4 (_AT(pteval_t, 0))#endif +/* +* The hardware requires shadow stack to be read-only and Dirty. +* _PAGE_SAVED_DIRTY is a software-only bit used to separate copy-on-write +* PTEs from shadow stack PTEs:
I'd suggest phrasing this differently. COW is just one scenario where this can happen. Also, I don't think that the description of "separation" is correct.
Something like the following maybe?
"
However, there are valid cases where the kernel might create read-only PTEs that are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit, to avoid creating a wrong "shadow stack" PTEs. Such PTEs have (Write=0,SavedDirty=1,Dirty=0) set.
Note that on processors without shadow stack support, the _PAGE_SAVED_DIRTY remains unused. "
The I would simply drop below (which is also too COW-specific I think).
Quoted 22 lines
arch/x86/include/asm/pgtable_types.h +22
-0+* +* (Write=0,SavedDirty=1,Dirty=0): +* - A modified, copy-on-write (COW) page. Previously when a typical +* anonymous writable mapping was made COW via fork(), the kernel would +* mark it Write=0,Dirty=1. Now it will instead use the Cow bit. This +* happens in copy_present_pte(). +* - A R/O page that has been COW'ed. The user page is in a R/O VMA, +* and get_user_pages(FOLL_FORCE) needs a writable copy. The page fault +* handler creates a copy of the page and sets the new copy's PTE as +* Write=0 and SavedDirty=1. +* - A shared shadow stack PTE. When a shadow stack page is being shared +* among processes (this happens at fork()), its PTE is made Dirty=0, so +* the next shadow stack access causes a fault, and the page is +* duplicated and Dirty=1 is set again. This is the COW equivalent for +* shadow stack pages, even though it's copy-on-access rather than +* copy-on-write. +* +* (Write=0,SavedDirty=0,Dirty=1): +* - A shadow stack PTE. +* - A Cow PTE created when a processor without shadow stack support set +* Dirty=1. +*/
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 34 lines · David Hildenbrand
Quoted 21 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.Quoted 11 lines
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
in places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined_PAGE_SAVED_DIRTYin place of the hardware_PAGE_DIRTY.
In other words, whenever Linux needs to create Write=0,Dirty=1, it instead
creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:
Quoted 3 lines
I would simplify (see below) and not repeat what the patch contains as
comments already that detailed.
This verbiage has had quite a bit of x86 maintainer attention already. I hear what you are saying, but I'm a bit hesitant to take style suggestions at this point for fear of the situation where people ask for changes back and forth across different versions. Unless any x86 maintainers want to chime in again? More responses below.
Quoted 222 lines
Quoted 222 lines
(Write=0,SavedDirty=1,Dirty=0):
- A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW viafork(), the kernel
would
mark it Write=0,Dirty=1. Now it will instead use the SavedDirty
bit.
This happens incopy_present_pte().
- A R/O page that has been COW'ed. The user page is in a R/O VMA, andget_user_pages(FOLL_FORCE)needs a writable copy. The page
fault
handler creates a copy of the page and sets the new copy's PTE
as
Write=0 and SavedDirty=1.
- A shared shadow stack PTE. When a shadow stack page is being
shared
among processes (this happens atfork()), its PTE is made
Dirty=0, so
the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent
for
shadow stack pages, even though it's copy-on-access rather than copy-on-write.(Write=0,SavedDirty=0,Dirty=1):
- A shadow stack PTE.
- A Cow PTE created when a processor without shadow stack support
setDirty=1.There are six bits left available to software in the 64-bit PTE after
consuming a bit for_PAGE_SAVED_DIRTY. No space is consumed in 32- bit
kernels because shadow stacks are not enabled there.Implement only the infrastructure for
_PAGE_SAVED_DIRTY. Changes to start
creating_PAGE_SAVED_DIRTYPTEs will follow once other pieces are in place.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Reviewed-by Kees Cook <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v6:
- Rename_PAGE_COWto_PAGE_SAVED_DIRTY(David Hildenbrand)
- Add_PAGE_SAVED_DIRTYto_PAGE_CHG_MASKv5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)v4:
- Teachpte_flags_need_flush()about_PAGE_COWbit
- Break apart patch for better bisectabilityv3:
- Add comment around_PAGE_TABLEin response to comment from (Andrew Cooper)
- Check for PSE inpmd_shstk(Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples
have
multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86
thing
(peterz)
- Fix swapping of Write/Dirty (PeterZ)--- arch/x86/include/asm/pgtable.h | 79 ++++++++++++++++++++++++++++ arch/x86/include/asm/pgtable_types.h | 65 ++++++++++++++++++++--- arch/x86/include/asm/tlbflush.h | 3 +- 3 files changed, 138 insertions(+), 9 deletions(-)arch/x86/include/asm/pgtable.h +2
-0@@ -301,6 +301,45 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)return native_make_pte (v & ~clear); } +/* +* COW and other write protection operations can result in Dirty=1,Write=0
+ * PTEs. But in the case ofX86_FEATURE_USER_SHSTK, the software SavedDirty bit
+ * is used, since the Dirty=1,Write=0 will result in the memory being treated as
+ * shadow stack by the HW. So when creating dirty, write-protected memory, a
+ * software bit is used_PAGE_BIT_SAVED_DIRTY. The following functions
+ *pte_mksaveddirty()andpte_clear_saveddirty()take a conventional dirty,
+ * write-protected PTE (Write=0,Dirty=1) and transition it to the shadow stack
+ * compatible version. (Write=0,SavedDirty=1).
+ */arch/x86/include/asm/pgtable.h +13
-0+static inline pte_t pte_mksaveddirty (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + + pte =pte_clear_flags (pte, _PAGE_DIRTY); +return pte_set_flags (pte, _PAGE_SAVED_DIRTY); +} + +static inline pte_t pte_clear_saveddirty (pte_t pte) +{ +/* +* _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK kernels,
arch/x86/include/asm/pgtable.h +1
-0+* since the HW dirty bit can be used without creating shadow stack
arch/x86/include/asm/pgtable.h +1
-0+* memory. See the _PAGE_SAVED_DIRTY definition for more details.
arch/x86/include/asm/pgtable.h +34
-0+*/ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pte; + +/* +* PTE is getting copied-on-write, so it will be dirtied +* if writable, or made shadow stack if shadow stack and +* being copied on access. Set the dirty bit for both +* cases. +*/ + pte =pte_set_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pte_uffd_wp (pte_t pte) {@@ -420,6 +459,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)return native_make_pmd (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_mksaveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_set_flags (pmd, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pmd_t pmd_clear_saveddirty (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pmd; + + pmd =pmd_set_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_SAVED_DIRTY); +} +#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP static inline int pmd_uffd_wp (pmd_t pmd){arch/x86/include/asm/pgtable.h +20
-0@@ -491,6 +550,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)return native_make_pud (v & ~clear); } +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_mksaveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_clear_flags (pud, _PAGE_DIRTY); +return pud_set_flags (pud, _PAGE_SAVED_DIRTY); +} + +/* See comments above pte_mksaveddirty() */ +static inline pud_t pud_clear_saveddirty (pud_t pud) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return pud; + + pud =pud_set_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_SAVED_DIRTY); +} +static inline pud_t pud_mkold (pud_t pud) {return pud_clear_flags(pud, _PAGE_ACCESSED);arch/x86/include/asm/pgtable_types.h +2
-1@@ -21,7 +21,8 @@#define _PAGE_BIT_SOFTW2 10/* " */ #define _PAGE_BIT_SOFTW3 11/* " */ #define _PAGE_BIT_PAT_LARGE 12/* On 2MB or 1GB pages */ -#define _PAGE_BIT_SOFTW4 58/* available for programmer */ +#define _PAGE_BIT_SOFTW4 57/* available for programmer */ +#define _PAGE_BIT_SOFTW5 58/* available for programmer */ #define _PAGE_BIT_PKEY_BIT0 59/* Protection Keys, bit 1/4 */ #define _PAGE_BIT_PKEY_BIT1 60/* Protection Keys, bit 2/4 */
#define_PAGE_BIT_PKEY_BIT261 /* Protection Keys, bit 3/4
*/arch/x86/include/asm/pgtable_types.h +0
-0@@ -34,6 +35,15 @@#define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3/* software dirty tracking */
#define_PAGE_BIT_DEVMAP_PAGE_BIT_SOFTW4
Quoted 10 lines
Quoted 6 lines
arch/x86/include/asm/pgtable_types.h +5
-0+/* +* Indicates a Saved Dirty bit page. +*/ +#ifdef CONFIG_X86_USER_SHADOW_STACK +#define _PAGE_BIT_SAVED_DIRTY _PAGE_BIT_SOFTW5/* copy-on-write */
Nope, not "copy-on-write" :) It's more like "dirty bit when the hw- dirty
bit cannot be used". Maybe simply drop the comment.
Oops, I missed this when I scrubbed _PAGE_COW. Thanks. Will fix.
Quoted 36 lines
Quoted 16 lines
arch/x86/include/asm/pgtable_types.h +4
-0+#else +#define _PAGE_BIT_SAVED_DIRTY 0 +#endif +/* If _PAGE_BIT_PRESENT is clear, we use these: */ /* - if the user mapped it with PROT_NONE; pte_present gives true */
#define_PAGE_BIT_PROTNONE_PAGE_BIT_GLOBALarch/x86/include/asm/pgtable_types.h +3
-0@@ -117,6 +127,40 @@#define _PAGE_SOFTW4 (_AT(pteval_t, 0))#endif +/* +* The hardware requires shadow stack to be read-only and Dirty. +* _PAGE_SAVED_DIRTY is a software-only bit used to separate copy- Quoted 2 lines
on-write
+ * PTEs from shadow stack PTEs:I'd suggest phrasing this differently. COW is just one scenario where
this can happen. Also, I don't think that the description of "separation" is correct.Something like the following maybe?
"
However, there are valid cases where the kernel might create read- only
PTEs that are dirty (e.g.,fork(),mprotect(), uffd-wp(), soft-dirty tracking). In this case, the_PAGE_SAVED_DIRTYbit is used instead of
the HW-dirty bit, to avoid creating a wrong "shadow stack" PTEs. Such
PTEs have (Write=0,SavedDirty=1,Dirty=0) set.
Quoted 6 lines
Note that on processors without shadow stack support, the
_PAGE_SAVED_DIRTYremains unused.
"The I would simply drop below (which is also too COW-specific I think).
COW is the main situation where shadow stacks become read-only. So, as an example it is nice in that COW covers all the scenarios discussed. Again, do any x86 maintainers want to weigh in here?
Quoted 36 lines
Quoted 34 lines
+ *
+ * (Write=0,SavedDirty=1,Dirty=0):
+ * - A modified, copy-on-write (COW) page. Previously when a typical
+ * anonymous writable mapping was made COW viafork(), the
kernel would
+ * mark it Write=0,Dirty=1. Now it will instead use the Cow
bit. This
+ * happens incopy_present_pte().
+ * - A R/O page that has been COW'ed. The user page is in a R/O VMA,
+ * andget_user_pages(FOLL_FORCE)needs a writable copy. The
page fault
+ * handler creates a copy of the page and sets the new copy's
PTE as
+ * Write=0 and SavedDirty=1.
+ * - A shared shadow stack PTE. When a shadow stack page is being shared
+ * among processes (this happens atfork()), its PTE is made
Dirty=0, so
+ * the next shadow stack access causes a fault, and the page is
+ * duplicated and Dirty=1 is set again. This is the COW
equivalent for
+ * shadow stack pages, even though it's copy-on-access rather
than
+ * copy-on-write.
+ *
+ * (Write=0,SavedDirty=0,Dirty=1):
+ * - A shadow stack PTE.
+ * - A Cow PTE created when a processor without shadow stack support set
+ * Dirty=1.
+ */
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 39 lines · Rick P
Quoted 34 lines · David Hildenbrand
Quoted 21 lines · Rick Edgecombe
Some OSes have a greater dependence on software available bits in PTEs than
Linux. That left the hardware architects looking for a way to represent a
new memory type (shadow stack) within the existing bits. They chose to
repurpose a lightly-used state: Write=0,Dirty=1. So in order to support
shadow stack memory, Linux should avoid creating memory with this PTE bit
combination unless it intends for it to be shadow stack.The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate
a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and*
generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which
supports shadow stacks will no longer exhibit this oddity.Quoted 11 lines
So that leaves Write=0,Dirty=1 PTEs created in software. To achieve this,
in places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined_PAGE_SAVED_DIRTYin place of the hardware_PAGE_DIRTY.
In other words, whenever Linux needs to create Write=0,Dirty=1, it instead
creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1. Further differentiated by VMA flags, these PTE bit combinations would be set as follows for various types of memory:Quoted 3 lines
I would simplify (see below) and not repeat what the patch contains as
comments already that detailed.
Quoted 5 lines
This verbiage has had quite a bit of x86 maintainer attention already. I hear what you are saying, but I'm a bit hesitant to take style suggestions at this point for fear of the situation where people ask for changes back and forth across different versions. Unless any x86 maintainers want to chime in again? More responses below.
Sure, for my taste this is (1) too repetitive (2) too verbose (3) to specialized. But whatever x86 maintainers prefer.
[...]
Quoted 14 lines
Quoted 14 lines
"
However, there are valid cases where the kernel might create read- only
PTEs that are dirty (e.g.,fork(),mprotect(), uffd-wp(), soft-dirty tracking). In this case, the_PAGE_SAVED_DIRTYbit is used instead of
the HW-dirty bit, to avoid creating a wrong "shadow stack" PTEs. Such
PTEs have (Write=0,SavedDirty=1,Dirty=0) set.Note that on processors without shadow stack support, the
_PAGE_SAVED_DIRTYremains unused.
"
Quoted 6 lines
Quoted 2 lines
The I would simply drop below (which is also too COW-specific I think).
COW is the main situation where shadow stacks become read-only. So, as an example it is nice in that COW covers all the scenarios discussed. Again, do any x86 maintainers want to weigh in here?
Again, I'd not specialize on COW in all patches to much (IMHO, it creates more confusion than it actually helps for understanding what's happening) and just call it a read-only PTE that is dirty. Simple as that. And it's easy to see why that's problematic, because read-only PTEs that are dirty would be identified as shadow stack PTEs, which we want to work around.
Again, just my 2 cents. I'm not an x86 maintainer ;)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 10 lines · David Hildenbrand
Again, I'd not specialize on COW in all patches to much (IMHO, it creates more confusion than it actually helps for understanding what's
happening) and just call it a read-only PTE that is dirty. Simple as that. And it's easy to see why that's problematic, because read-only PTEs that are dirty would be identified as shadow stack PTEs, which we
want to work around.Again, just my 2 cents. I'm not an x86 maintainer ;)
Right, I see the point. Let's see if they have any opinion. There is a bit of a historical reason for the focus on COW. As you well know the dirty bit used to be important for that case. But I think it's still not a terrible example. It covers some typical cases, but yes we don't want to mislead the reader that it is a Cow only scenario.
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
In reply to David HildenbrandOn 2/21/23 00:38, David Hildenbrand wrote:> Sure, for my taste this is
(1) too repetitive (2) too verbose (3) to
Quoted 1 line
specialized. But whatever x86 maintainers prefer.
At this point, I'm not going to be too nitpicky. I personally think we need to get _something_ merged. We can then nitpick it to death once its in the tree.
So I prefer whatever will move the set along. ;)
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 11 lines · Dave Hansen
On 2/21/23 00:38, David Hildenbrand wrote:> Sure, for my taste this is
(1) too repetitive (2) too verbose (3) toQuoted 1 line
specialized. But whatever x86 maintainers prefer.
At this point, I'm not going to be too nitpicky. I personally think we
need to get _something_ merged. We can then nitpick it to death once its in the tree.So I prefer whatever will move the set along. ;)
Ok, David's general suggestion across these x86/mm patches is to make things less COW specific. Sounds like you don't have a problem with that. I'll just do that and hope I don't stir up any additional concerns. Thanks all.
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
In reply to Dave HansenQuoted 7 lines · Dave Hansen
On 2/21/23 00:38, David Hildenbrand wrote:> Sure, for my taste this is
(1) too repetitive (2) too verbose (3) toQuoted 1 line
specialized. But whatever x86 maintainers prefer.
At this point, I'm not going to be too nitpicky. I personally think we need to get _something_ merged. We can then nitpick it to death once its in the tree.
Yes, but ... do we have to rush right now?
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
Also, I think concise patch descriptions and comments are not necessarily nitpicking like "please rename that variable".
Quoted 2 lines
So I prefer whatever will move the set along. ;)
If the plan is to merge it in the next merge window (which I suspect, but I might be wrong), I suggest including it in -next fairly soonish, and in the meantime, polish the remaining bits.
Knowing the plan would be good ;)
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 2 lines · David Hildenbrand
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
No way. It's 6.4 material at the earliest.
I'm just saying to Rick not to worry _too_ much about earlier feedback from me if folks have more recent review feedback.
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 8 lines · Dave Hansen
Quoted 2 lines · David Hildenbrand
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
No way. It's 6.4 material at the earliest.
I'm just saying to Rick not to worry _too_ much about earlier feedback from me if folks have more recent review feedback.
Great. So I hope we can get this into -next soon and that we'll only get non-earth-shattering feedback so this can land in 6.4.
Signature · 3 lines
Thanks, David / dhildenb
Kees Cook
maintainerRe: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 11 lines · David Hildenbrand
Quoted 8 lines · Dave Hansen
Quoted 2 lines · David Hildenbrand
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
No way. It's 6.4 material at the earliest.
I'm just saying to Rick not to worry _too_ much about earlier feedback from me if folks have more recent review feedback.
Great. So I hope we can get this into -next soon and that we'll only get non-earth-shattering feedback so this can land in 6.4.
Yes please. Who's going to take it? :)
Signature · 1 line
Kees Cook
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 11 lines · Kees Cook
Quoted 9 lines · David Hildenbrand
Quoted 7 lines · Dave Hansen
Quoted 2 lines · David Hildenbrand
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
No way. It's 6.4 material at the earliest.
I'm just saying to Rick not to worry _too_ much about earlier feedback from me if folks have more recent review feedback.
Great. So I hope we can get this into -next soon and that we'll only get non-earth-shattering feedback so this can land in 6.4.
Yes please. Who's going to take it? 😄
I'm more than happy to queue it in x86/mm. I'll plan to queue Rick's next version that he posts and then we can do any fixes or minor changes on top of that.
Re: [PATCH v6 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY
Quoted 8 lines · Dave Hansen
Quoted 7 lines · Kees Cook
Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Dave Hansen
Quoted 2 lines · David Hildenbrand
This series wasn't in -next and we're in the merge window. Is the plan to still include it into this merge window?
No way. It's 6.4 material at the earliest.
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
I'm just saying to Rick not to worry _too_ much about earlier feedback from me if folks have more recent review feedback.
Great. So I hope we can get this into -next soon and that we'll only get non-earth-shattering feedback so this can land in 6.4.
Yes please. Who's going to take it? 😄
I'm more than happy to queue it in x86/mm. I'll plan to queue Rick's next version that he posts and then we can do any fixes or minor changes on top of that.
That sounds lovely; thank you! :)
Signature · 1 line
Kees Cook
PATCH v6 15/41 x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,SavedDirty=1.
When a PTE goes from Write=1,Dirty=1 to Write=0,SavedDirty=1, it could become a transient shadow stack PTE in two cases:
1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary.
2. When _PAGE_DIRTY is replaced with _PAGE_SAVED_DIRTY non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.
In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature.
Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy.
Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.
---
v6:
- Fix comment and log to update for _PAGE_COW being replaced with _PAGE_SAVED_DIRTY.
v5:
- Commit log verbiage and formatting (Boris)
- Remove capitalization on shadow stack (Boris)
- Fix i386 warning on recent clang
v3:
- Remove unnecessary #ifdef (Dave Hansen)
v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)
--- arch/x86/include/asm/pgtable.h | 35 ++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
arch/x86/include/asm/pgtable.h +35 -0
@@ -1192,6 +1192,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +#ifdef CONFIG_X86_USER_SHADOW_STACK +/* +* Avoid accidentally creating shadow stack PTEs +* (Write=0,Dirty=1). Use cmpxchg() to prevent races with +* the hardware setting Dirty=1. +*/ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) { +pte_t old_pte, new_pte; + + old_pte =READ_ONCE (*ptep); +do { + new_pte =pte_wrprotect (old_pte); + }while (!try_cmpxchg (&ptep->pte, &old_pte.pte, new_pte.pte)); + +return ; + } +#endif clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); }
@@ -1244,6 +1261,24 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +#ifdef CONFIG_X86_USER_SHADOW_STACK +/* +* Avoid accidentally creating shadow stack PTEs +* (Write=0,Dirty=1). Use cmpxchg() to prevent races with +* the hardware setting Dirty=1. +*/ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) { +pmd_t old_pmd, new_pmd; + + old_pmd =READ_ONCE (*pmdp); +do { + new_pmd =pmd_wrprotect (old_pmd); + }while (!try_cmpxchg (&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd)); + +return ; + } +#endif +clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); }
Signature · 1 line
2.17.1
PATCH v6 16/41 x86/mm: Start actually marking _PAGE_SAVED_DIRTY
In reply to Rick EdgecombeThe recently introduced _PAGE_SAVED_DIRTY should be used instead of the HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply the same changes to pmd and pud.
For pte_modify() this is a bit trickier. It takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one.
Modify it to also move _PAGE_DIRTY to _PAGE_SAVED_DIRTY. To avoid creating Write=0,Dirty=1 PTEs, pte_modify() needs to avoid:
1. Marking Write=0 PTEs Dirty=1
2. Marking Dirty=1 PTEs Write=0
The first case cannot happen as the existing behavior of pte_modify() is to filter out any Dirty bit passed in newprot. Handle the second case by shifting _PAGE_DIRTY=1 to _PAGE_SAVED_DIRTY=1 if the PTE was write protected by the pte_modify() call. Apply the same changes to pmd_modify().
---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Open code _PAGE_SAVED_DIRTY part in pte_modify() (Boris)
- Change the logic so the open coded part is not too ugly
- Merge pte_modify() patch with this one because of the above
v4:
- Break part patch for better bisectability
--- arch/x86/include/asm/pgtable.h | 168 ++++++++++++++++++++++++++++----- 1 file changed, 145 insertions(+), 23 deletions(-)
arch/x86/include/asm/pgtable.h +145 -23
@@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags;* The following only work if pte_present() is true. * Undefined behaviour if not.. */ -static inline int pte_dirty (pte_t pte) +static inline bool pte_dirty (pte_t pte) { -return pte_flags (pte) & _PAGE_DIRTY; +return pte_flags (pte) & _PAGE_DIRTY_BITS; +} + +static inline bool pte_shstk (pte_t pte) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return false; + +return (pte_flags (pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY; }static inline int pte_young (pte_t pte)
@@ -134,9 +142,18 @@ static inline int pte_young(pte_t pte)return pte_flags (pte) & _PAGE_ACCESSED; } -static inline int pmd_dirty (pmd_t pmd) +static inline bool pmd_dirty (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_DIRTY; +return pmd_flags (pmd) & _PAGE_DIRTY_BITS; +} + +static inline bool pmd_shstk (pmd_t pmd) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return false; + +return (pmd_flags (pmd) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); }#define pmd_young pmd_young
@@ -145,9 +162,9 @@ static inline int pmd_young(pmd_t pmd)return pmd_flags (pmd) & _PAGE_ACCESSED; } -static inline int pud_dirty (pud_t pud) +static inline bool pud_dirty (pud_t pud) { -return pud_flags (pud) & _PAGE_DIRTY; +return pud_flags (pud) & _PAGE_DIRTY_BITS; }static inline int pud_young (pud_t pud)
@@ -157,13 +174,21 @@ static inline int pud_young(pud_t pud)static inline int pte_write (pte_t pte) { -return pte_flags (pte) & _PAGE_RW; +/* +* Shadow stack pages are logically writable, but do not have +* _PAGE_RW. Check for them separately from _PAGE_RW itself. +*/ +return (pte_flags (pte) & _PAGE_RW) ||pte_shstk (pte); }#define pmd_write pmd_writestatic inline int pmd_write (pmd_t pmd) { -return pmd_flags (pmd) & _PAGE_RW; +/* +* Shadow stack pages are logically writable, but do not have +* _PAGE_RW. Check for them separately from _PAGE_RW itself. +*/ +return (pmd_flags (pmd) & _PAGE_RW) ||pmd_shstk (pmd); }#define pud_write pud_write
@@ -375,7 +400,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)static inline pte_t pte_mkclean (pte_t pte) { -return pte_clear_flags (pte, _PAGE_DIRTY); +return pte_clear_flags (pte, _PAGE_DIRTY_BITS); }static inline pte_t pte_mkold (pte_t pte)
@@ -385,7 +410,16 @@ static inline pte_t pte_mkold(pte_t pte)static inline pte_t pte_wrprotect (pte_t pte) { -return pte_clear_flags (pte, _PAGE_RW); + pte =pte_clear_flags (pte, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PTE (Write=0,Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pte_dirty (pte)) + pte =pte_mksaveddirty (pte); +return pte; }static inline pte_t pte_mkexec (pte_t pte)
@@ -395,7 +429,19 @@ static inline pte_t pte_mkexec(pte_t pte)static inline pte_t pte_mkdirty (pte_t pte) { -return pte_set_flags (pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pteval_t dirty = _PAGE_DIRTY; + +/* Avoid creating Dirty=1,Write=0 PTEs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pte_write (pte)) + dirty = _PAGE_SAVED_DIRTY; + +return pte_set_flags (pte, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pte_t pte_mkwrite_shstk (pte_t pte) +{ +/* pte_clear_saveddirty() also sets Dirty=1 */ +return pte_clear_saveddirty (pte); }static inline pte_t pte_mkyoung (pte_t pte)
@@ -412,7 +458,12 @@ struct vm_area_struct;static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) { -return pte_mkwrite_kernel (pte); + pte =pte_mkwrite_kernel (pte); + +if (pte_dirty (pte)) + pte =pte_clear_saveddirty (pte); + +return pte; }static inline pte_t pte_mkhuge (pte_t pte)
@@ -503,17 +554,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd)static inline pmd_t pmd_mkclean (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_DIRTY); +return pmd_clear_flags (pmd, _PAGE_DIRTY_BITS); }static inline pmd_t pmd_wrprotect (pmd_t pmd) { -return pmd_clear_flags (pmd, _PAGE_RW); + pmd =pmd_clear_flags (pmd, _PAGE_RW); +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PMD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pmd_dirty (pmd)) + pmd =pmd_mksaveddirty (pmd); +return pmd; }static inline pmd_t pmd_mkdirty (pmd_t pmd) { -return pmd_set_flags (pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pmdval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PMDs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pmd_write (pmd)) + dirty = _PAGE_SAVED_DIRTY; + +return pmd_set_flags (pmd, dirty | _PAGE_SOFT_DIRTY); +} + +static inline pmd_t pmd_mkwrite_shstk (pmd_t pmd) +{ +return pmd_clear_saveddirty (pmd); }static inline pmd_t pmd_mkdevmap (pmd_t pmd)
@@ -533,7 +603,12 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) { -return pmd_set_flags (pmd, _PAGE_RW); + pmd =pmd_set_flags (pmd, _PAGE_RW); + +if (pmd_dirty (pmd)) + pmd =pmd_clear_saveddirty (pmd); + +return pmd; }static inline pud_t pud_set_flags (pud_t pud,pudval_t set)
@@ -577,17 +652,32 @@ static inline pud_t pud_mkold(pud_t pud)static inline pud_t pud_mkclean (pud_t pud) { -return pud_clear_flags (pud, _PAGE_DIRTY); +return pud_clear_flags (pud, _PAGE_DIRTY_BITS); }static inline pud_t pud_wrprotect (pud_t pud) { -return pud_clear_flags (pud, _PAGE_RW); + pud =pud_clear_flags (pud, _PAGE_RW); + +/* +* Blindly clearing _PAGE_RW might accidentally create +* a shadow stack PUD (RW=0, Dirty=1). Move the hardware +* dirty value to the software bit. +*/ +if (pud_dirty (pud)) + pud =pud_mksaveddirty (pud); +return pud; }static inline pud_t pud_mkdirty (pud_t pud) { -return pud_set_flags (pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY); +pudval_t dirty = _PAGE_DIRTY; + +/* Avoid creating (HW)Dirty=1, Write=0 PUDs */ +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && !pud_write (pud)) + dirty = _PAGE_SAVED_DIRTY; + +return pud_set_flags (pud, dirty | _PAGE_SOFT_DIRTY); }static inline pud_t pud_mkdevmap (pud_t pud)
@@ -607,7 +697,11 @@ static inline pud_t pud_mkyoung(pud_t pud)static inline pud_t pud_mkwrite (pud_t pud) { -return pud_set_flags (pud, _PAGE_RW); + pud =pud_set_flags (pud, _PAGE_RW); + +if (pud_dirty (pud)) + pud =pud_clear_saveddirty (pud); +return pud; }#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
@@ -724,6 +818,8 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask);static inline pte_t pte_modify (pte_t pte,pgprot_t newprot) {pteval_t val =pte_val (pte), oldval = val; +bool wr_protected; +pte_t pte_result;/* * Chop off the NX bit (if present), and add the NX portion of
@@ -732,17 +828,43 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) val &= _PAGE_CHG_MASK; val |=check_pgprot (newprot) & ~_PAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PTE_PFN_MASK ); -return __pte (val); + + pte_result =__pte (val); + +/* +* Do the saveddirty fixup if the PTE was just write protected and +* it's dirty. +*/ + wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW); +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && wr_protected && + (val & _PAGE_DIRTY)) + pte_result =pte_mksaveddirty (pte_result); + +return pte_result; }static inline pmd_t pmd_modify (pmd_t pmd,pgprot_t newprot) {pmdval_t val =pmd_val (pmd), oldval = val; +bool wr_protected; +pmd_t pmd_result; - val &= _HPAGE_CHG_MASK; + val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY); val |=check_pgprot (newprot) & ~_HPAGE_CHG_MASK; val =flip_protnone_guard (oldval, val,PHYSICAL_PMD_PAGE_MASK ); -return __pmd (val); + + pmd_result =__pmd (val); + +/* +* Do the saveddirty fixup if the PMD was just write protected and +* it's dirty. +*/ + wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW); +if (cpu_feature_enabled (X86_FEATURE_USER_SHSTK ) && wr_protected && + (val & _PAGE_DIRTY)) + pmd_result =pmd_mksaveddirty (pmd_result); + +return pmd_result; }/*
Signature · 1 line
2.17.1
PATCH v6 17/41 mm: Move VM_UFFD_MINOR_BIT from 37 to 38
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Future patches will introduce a new VM flag VM_SHADOW_STACK that will be VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake of order, make all VM_HIGH_ARCH_BITs stay together by moving VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be introduced as 37.
--- include/linux/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
@@ -366,7 +366,7 @@ extern unsigned int kobjsize(const void *objp);#endif #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR -# define VM_UFFD_MINOR_BIT 37 +# define VM_UFFD_MINOR_BIT 38# define VM_UFFD_MINOR BIT(VM_UFFD_MINOR_BIT)/* UFFD minor faults */ #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ # define VM_UFFD_MINOR VM_NONE
Signature · 1 line
2.17.1
PATCH v6 18/41 mm: Introduce VM_SHADOW_STACK for shadow stack memory
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.
---
v6:
- Add comment about VM_SHADOW_STACK not being allowed with VM_SHARED (David Hildenbrand)
v3:
- Drop arch specific change in arch_vma_name(). The memory can show as anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags() (Boris)
--- Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+)
Documentation/filesystems/proc.rst +1 -0
@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic will
fs/proc/task_mmu.c +3 -0
@@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui",#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif };size_t i;
include/linux/mm.h +8 -0
@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp);#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_USER_SHADOW_STACK +# define VM_SHADOW_STACK VM_HIGH_ARCH_5/* Should not be set with VM_SHARED */ +#else +# define VM_SHADOW_STACK VM_NONE +#endif +#if defined(CONFIG_X86 )# define VM_PAT VM_ARCH_1/* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC )
Signature · 1 line
2.17.1
Re: [PATCH v6 18/41] mm: Introduce VM_SHADOW_STACK for shadow stack memory
Quoted 11 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
A shadow stack PTE must be read-only and have
_PAGE_DIRTYset. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. IntroduceVM_SHADOW_STACKto track shadow stack VMAs.
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory, such as x86 Control-flow Enforcement Technology (CET). Let's add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware."
Quoted 9 lines
Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>
Quoted 4 lines
---
v6:
- Add comment aboutVM_SHADOW_STACKnot being allowed withVM_SHARED(David Hildenbrand)
Might want to add some more meat to the patch description why that is the case.
Quoted 57 lines
v3:
- Drop arch specific change inarch_vma_name(). The memory can show as anonymous (Kirill)
- ChangeCONFIG_ARCH_HAS_SHADOW_STACKtoCONFIG_X86_USER_SHADOW_STACKinshow_smap_vma_flags()(Boris)--- Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+)Documentation/filesystems/proc.rst +1
-0@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associated mnemonic willfs/proc/task_mmu.c +3
-0@@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui",#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif };size_t i;include/linux/mm.h +2
-0@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS
include/linux/mm.h extern unsigned int kobjsize(const void *objp); · line 340
Quoted 4 lines
include/linux/mm.h +1
-0@@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp);#endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_X86_USER_SHADOW_STACK
Should we abstract this to CONFIG_ARCH_USER_SHADOW_STACK, seeing that other architectures might similarly need it?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 18/41] mm: Introduce VM_SHADOW_STACK for shadow stack memory
Quoted 17 lines · David Hildenbrand
Quoted 9 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Quoted 6 lines
A shadow stack PTE must be read-only and have
_PAGE_DIRTYset. However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These
two cases are handled differently for page faults. IntroduceVM_SHADOW_STACKto track shadow stack VMAs.
Quoted 7 lines
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory, such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware."
Ah yea, that top blurb was added to all the non-x86 arch patches after some feedback from Andrew Morton. He had said basically (in some more colorful language) that the changelogs (at the time) were written assuming the reader knows what a shadow stack is.
So it might be worth keeping a little more info in the log?
Quoted 16 lines
Quoted 10 lines
Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Reviewed-by Kirill A. Shutemov <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>Quoted 5 lines
---
v6:
- Add comment aboutVM_SHADOW_STACKnot being allowed withVM_SHARED
(David Hildenbrand)
Quoted 3 lines
Might want to add some more meat to the patch description why that is
the case.
Sure.
Quoted 72 lines
Quoted 72 lines
v3:
- Drop arch specific change inarch_vma_name(). The memory can
show as
anonymous (Kirill)
- ChangeCONFIG_ARCH_HAS_SHADOW_STACKtoCONFIG_X86_USER_SHADOW_STACK
inshow_smap_vma_flags()(Boris)--- Documentation/filesystems/proc.rst | 1 + fs/proc/task_mmu.c | 3 +++ include/linux/mm.h | 8 ++++++++ 3 files changed, 12 insertions(+)Documentation/filesystems/proc.rst +1
-0@@ -564,6 +564,7 @@ encoded manner. The codes are the following: mt arm64 MTE allocation tags are enabled um userfaultfd missing tracking uw userfaultfd wr-protect tracking + ss shadow stack page == ======================================= Note that there is no guarantee that every flag and associatedmnemonic will
fs/proc/task_mmu.c +3
-0@@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR [ilog2(VM_UFFD_MINOR)] = "ui",#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ +#ifdef CONFIG_X86_USER_SHADOW_STACK + [ilog2(VM_SHADOW_STACK)] = "ss", +#endif };size_t i;include/linux/mm.h +0
-0@@ -315,11 +315,13 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64- bit architectures */
#defineVM_HIGH_ARCH_BIT_335 /* bit only usable on 64-
bit architectures */
#defineVM_HIGH_ARCH_BIT_436 /* bit only usable on 64-
bit architectures */
+#defineVM_HIGH_ARCH_BIT_537 /* bit only usable on 64-bit
architectures */
#defineVM_HIGH_ARCH_0BIT(VM_HIGH_ARCH_BIT_0)
#defineVM_HIGH_ARCH_1BIT(VM_HIGH_ARCH_BIT_1)
#defineVM_HIGH_ARCH_2BIT(VM_HIGH_ARCH_BIT_2)
#defineVM_HIGH_ARCH_3BIT(VM_HIGH_ARCH_BIT_3)
#defineVM_HIGH_ARCH_4BIT(VM_HIGH_ARCH_BIT_4)include/linux/mm.h +1
-0+#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ #ifdef CONFIG_ARCH_HAS_PKEYS @@ -335,6 +337,12 @@ extern unsigned int kobjsize(const void *objp);#endif #endif /* CONFIG_ARCH_HAS_PKEYS */
Quoted 6 lines
include/linux/mm.h extern unsigned int kobjsize(const void · line 341
Quoted 1 line
include/linux/mm.h +1
-0+#ifdef CONFIG_X86_USER_SHADOW_STACK Should we abstract this to
CONFIG_ARCH_USER_SHADOW_STACK, seeing that
other architectures might similarly need it?
There was an ARCH_HAS_SHADOW_STACK but it got removed following this discussion:
https://lore.kernel.org/lkml/[email protected]/
Now we have this new RFC for riscv as potentially a second implementation. But it is still very early, and I'm not sure anyone knows exactly what the similarities will be in a mature version. So I think it would be better to refactor in an ARCH_HAS_SHADOW_STACK later (and similar abstractions) once that series is more mature and we have an idea of what pieces will be shared. I don't have a problem in principle with an ARCH config, just don't think we should do it yet.
Quoted 1 line
David Hildenbrand
Acked-byRe: [PATCH v6 18/41] mm: Introduce VM_SHADOW_STACK for shadow stack memory
Quoted 26 lines · Rick P
Quoted 17 lines · David Hildenbrand
Quoted 9 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Quoted 6 lines
A shadow stack PTE must be read-only and have
_PAGE_DIRTYset. However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These
two cases are handled differently for page faults. IntroduceVM_SHADOW_STACKto track shadow stack VMAs.Quoted 7 lines
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory, such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware."
Quoted 4 lines
Ah yea, that top blurb was added to all the non-x86 arch patches after some feedback from Andrew Morton. He had said basically (in some more colorful language) that the changelogs (at the time) were written assuming the reader knows what a shadow stack is.
Okay. It's a bit repetitive, though.
Ideally, we'd just explain it in the cover letter in detail and Andrews's script would include the cover letter in the first commit. IIRC, that's what usually happens.
Quoted 2 lines
So it might be worth keeping a little more info in the log?
Copying the same paragraph into each commit is IMHO a bit repetitive. But these are just my 2 cents.
[...]
Quoted 9 lines
Quoted 3 lines
Should we abstract this to
CONFIG_ARCH_USER_SHADOW_STACK, seeing that
other architectures might similarly need it?There was an
ARCH_HAS_SHADOW_STACKbut it got removed following this discussion:
Quoted 7 lines
Now we have this new RFC for riscv as potentially a second implementation. But it is still very early, and I'm not sure anyone knows exactly what the similarities will be in a mature version. So I think it would be better to refactor in an
ARCH_HAS_SHADOW_STACKlater (and similar abstractions) once that series is more mature and we have an idea of what pieces will be shared. I don't have a problem in principle with an ARCH config, just don't think we should do it yet.
Okay, easy to factor out later.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 18/41] mm: Introduce VM_SHADOW_STACK for shadow stack memory
Quoted 60 lines · David Hildenbrand
Quoted 26 lines · Rick P
Quoted 17 lines · David Hildenbrand
Quoted 9 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Quoted 6 lines
A shadow stack PTE must be read-only and have
_PAGE_DIRTYset. However,
read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These
two cases are handled differently for page faults. IntroduceVM_SHADOW_STACKto track shadow stack VMAs.Quoted 7 lines
I suggest simplifying and abstracting that description.
"New hardware extensions implement support for shadow stack memory, such
as x86 Control-flow Enforcement Technology (CET). Let's add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware."Quoted 4 lines
Ah yea, that top blurb was added to all the non-x86 arch patches after some feedback from Andrew Morton. He had said basically (in some more colorful language) that the changelogs (at the time) were written assuming the reader knows what a shadow stack is.
Okay. It's a bit repetitive, though.
Ideally, we'd just explain it in the cover letter in detail and Andrews's script would include the cover letter in the first commit. IIRC, that's what usually happens.
Quoted 2 lines
So it might be worth keeping a little more info in the log?
Copying the same paragraph into each commit is IMHO a bit repetitive. But these are just my 2 cents.
[...]
Quoted 9 lines
Quoted 3 lines
Should we abstract this to
CONFIG_ARCH_USER_SHADOW_STACK, seeing that
other architectures might similarly need it?There was an
ARCH_HAS_SHADOW_STACKbut it got removed following this discussion:Quoted 7 lines
Now we have this new RFC for riscv as potentially a second implementation. But it is still very early, and I'm not sure anyone knows exactly what the similarities will be in a mature version. So I think it would be better to refactor in an
ARCH_HAS_SHADOW_STACKlater (and similar abstractions) once that series is more mature and we have an idea of what pieces will be shared. I don't have a problem in principle with an ARCH config, just don't think we should do it yet.
Quoted 1 line
Okay, easy to factor out later.
I would be more than happy if this config name would've been abstracted out and arches can choose to implement. It's a bit sad that it was generic earlier and was later changed due to lack of support from other architectures. Now there are three architectures who either already support shadow stack (x86), announced the support (aarch64) or are planning to support (riscv).
However given patch reduction I will get due to pte_mkwrite refactor, I am in favor of future refactor for config.
Quoted 8 lines
Signature · 3 lines
Thanks, David / dhildenb
PATCH v6 19/41 x86/mm: Check shadow stack page fault errors
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.
The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.
For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes.
Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.
---
v6:
- Update comment due to rename of Cow bit to SavedDirty
v5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)
v4:
- Further improve comment talking about FAULT_FLAG_WRITE (Peterz)
v3:
- Improve comment talking about using FAULT_FLAG_WRITE (Peterz)
--- arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)
arch/x86/include/asm/trap_pf.h +2 -0
@@ -11,6 +11,7 @@* bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access +* bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code {
@@ -20,6 +21,7 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , +X86_PF_SHSTK =1 <<6 , X86_PF_SGX = 1 << 15, };
arch/x86/mm/fault.c +38 -0
@@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; +/* +* Shadow stack accesses (PF_SHSTK=1) are only permitted to +* shadow stack VMAs. All other accesses result in an error. +*/ +if (error_code &X86_PF_SHSTK ) { +if (unlikely (!(vma->vm_flags &VM_SHADOW_STACK ))) +return 1 ; +if (unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; +return 0 ; + } +if (error_code &X86_PF_WRITE ) {/* write, present and write, not present: */ +if (unlikely (vma->vm_flags &VM_SHADOW_STACK )) +return 1 ;if (unlikely (!(vma->vm_flags &VM_WRITE )))return 1 ;return 0 ;
@@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* +* When a page becomes COW it changes from a shadow stack permission +* page (Write=0,Dirty=1) to (Write=0,Dirty=0,SavedDirty=1), which is simply +* read-only to the CPU. When shadow stack is enabled, a RET would +* normally pop the shadow stack by reading it with a "shadow stack +* read" access. However, in the COW case the shadow stack memory does +* not have shadow stack permissions, it is read-only. So it will +* generate a fault. +* +* For conventionally writable pages, a read can be serviced with a +* read only PTE, and COW would not have to happen. But for shadow +* stack, there isn't the concept of read-only shadow stack memory. +* If it is shadow stack permission, it can be modified via CALL and +* RET instructions. So COW needs to happen before any memory can be +* mapped with shadow stack permissions. +* +* Shadow stack accesses (read or write) need to be serviced with +* shadow stack permission memory, so in the case of a shadow stack +* read access, treat it as a WRITE fault so both COW will happen and +* the write fault path will tickle maybe_mkwrite() and map the memory +* shadow stack. +*/ +if (error_code &X86_PF_SHSTK ) + flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_WRITE ) flags |=FAULT_FLAG_WRITE ;if (error_code &X86_PF_INSTR )
Signature · 1 line
2.17.1
Re: [PATCH v6 19/41] x86/mm: Check shadow stack page fault errors
Quoted 127 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).
Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.
The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.
This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set
FAULT_FLAG_WRITEfor shadow stack accesses, even if the access was a shadow stack read.For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes.
Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Reviewed-by Kees Cook <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v6:
- Update comment due to rename of Cow bit to SavedDirtyv5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)v4:
- Further improve comment talking aboutFAULT_FLAG_WRITE(Peterz)v3:
- Improve comment talking about usingFAULT_FLAG_WRITE(Peterz)--- arch/x86/include/asm/trap_pf.h | 2 ++ arch/x86/mm/fault.c | 38 ++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+)arch/x86/include/asm/trap_pf.h +2
-0@@ -11,6 +11,7 @@* bit 3 == 1: use of reserved bit detected * bit 4 == 1: fault was an instruction fetch * bit 5 == 1: protection keys block access +* bit 6 == 1: shadow stack access fault * bit 15 == 1: SGX MMU page-fault */ enum x86_pf_error_code {@@ -20,6 +21,7 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , +X86_PF_SHSTK =1 <<6 , X86_PF_SGX = 1 << 15, };arch/x86/mm/fault.c +26
-0@@ -1138,8 +1138,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma) (error_code & X86_PF_INSTR), foreign))return 1 ; +/* +* Shadow stack accesses (PF_SHSTK=1) are only permitted to +* shadow stack VMAs. All other accesses result in an error. +*/ +if (error_code &X86_PF_SHSTK ) { +if (unlikely (!(vma->vm_flags &VM_SHADOW_STACK ))) +return 1 ; +if (unlikely (!(vma->vm_flags &VM_WRITE ))) +return 1 ; +return 0 ; + } +if (error_code &X86_PF_WRITE ) {/* write, present and write, not present: */ +if (unlikely (vma->vm_flags &VM_SHADOW_STACK )) +return 1 ;if (unlikely (!(vma->vm_flags &VM_WRITE )))return 1 ;return 0 ;@@ -1331,6 +1345,30 @@ void do_user_addr_fault(struct pt_regs *regs,perf_sw_event (PERF_COUNT_SW_PAGE_FAULTS ,1 , regs, address); +/* +* When a page becomes COW it changes from a shadow stack permission +* page (Write=0,Dirty=1) to (Write=0,Dirty=0,SavedDirty=1), which is simply +* read-only to the CPU. When shadow stack is enabled, a RET would +* normally pop the shadow stack by reading it with a "shadow stack +* read" access. However, in the COW case the shadow stack memory does +* not have shadow stack permissions, it is read-only. So it will +* generate a fault. +* +* For conventionally writable pages, a read can be serviced with a +* read only PTE, and COW would not have to happen. But for shadow +* stack, there isn't the concept of read-only shadow stack memory.
arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1369
Quoted 10 lines
arch/x86/mm/fault.c +10
-0+* If it is shadow stack permission, it can be modified via CALL and +* RET instructions. So COW needs to happen before any memory can be +* mapped with shadow stack permissions. +* +* Shadow stack accesses (read or write) need to be serviced with +* shadow stack permission memory, so in the case of a shadow stack +* read access, treat it as a WRITE fault so both COW will happen and +* the write fault path will tickle maybe_mkwrite() and map the memory +* shadow stack. +*/
Again, I suggest dropping all details about COW from this comment and from the patch description. It's just one such case that can happen.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 19/41] x86/mm: Check shadow stack page fault errors
Quoted 39 lines · David Hildenbrand
Quoted 36 lines
(quoted diff) +2
-0+ /* + * When a page becomes COW it changes from a shadow stackpermission
+ * page (Write=0,Dirty=1) to (Write=0,Dirty=0,SavedDirty=1),
which is simply
+ * read-only to the CPU. When shadow stack is enabled, a RET
would
+ * normally pop the shadow stack by reading it with a "shadow
stack
+ * read" access. However, in the COW case the shadow stack
memory does
+ * not have shadow stack permissions, it is read-only. So it
willarch/x86/mm/fault.c +3
-0+* generate a fault. +* +* For conventionally writable pages, a read can be serviced with a
+ * read only PTE, and COW would not have to happen. But for
shadow
+ * stack, there isn't the concept of read-only shadow stack
memory.
+ * If it is shadow stack permission, it can be modified via
CALL and
+ * RET instructions. So COW needs to happen before any memory
can bearch/x86/mm/fault.c +3
-0+* mapped with shadow stack permissions. +* +* Shadow stack accesses (read or write) need to be serviced with
+ * shadow stack permission memory, so in the case of a shadow
stack
+ * read access, treat it as a WRITE fault so both COW will
happen and
+ * the write fault path will ticklemaybe_mkwrite()and map
the memoryarch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1369
Quoted 2 lines
arch/x86/mm/fault.c +2
-0+* shadow stack. +*/
Quoted 3 lines
Again, I suggest dropping all details about COW from this comment and
from the patch description. It's just one such case that can happen.
Hi David,
I was just trying to edit this one to drop COW details, but I think in this case, one of the major reasons for the code *is* actually COW. We are not working around the whole inadvertent shadow stack memory piece here, but something else: Making sure shadow stack memory is faulted in and doing COW if required to make this possible. I came up with this, does it seem better?
/*
* For conventionally writable pages, a read can be serviced with a *
read only PTE. But for shadow stack, there isn't a concept of
* read-
only shadow stack memory. If it a PTE has the shadow stack
*
permission, it can be modified via CALL and RET instructions. So
* core
MM needs to fault in a writable PTE and do things it already
* does for
write faults.
*
* Shadow stack accesses (read or write) need to be
serviced with
* shadow stack permission memory, so in the case of a
shadow stack
* read access, treat it as a WRITE fault so both any
required COW will
* happen and the write fault path will ticklemaybe_mkwrite() and map
* the memory shadow stack.
*/
Thanks,
Rick
Re: [PATCH v6 19/41] x86/mm: Check shadow stack page fault errors
Quoted 40 lines · Rick P
Quoted 39 lines · David Hildenbrand
Quoted 38 lines
(quoted diff) +2
-0+ /* + * When a page becomes COW it changes from a shadow stackpermission
+ * page (Write=0,Dirty=1) to (Write=0,Dirty=0,SavedDirty=1),
which is simply
+ * read-only to the CPU. When shadow stack is enabled, a RET
would
+ * normally pop the shadow stack by reading it with a "shadow
stack
+ * read" access. However, in the COW case the shadow stack
memory does
+ * not have shadow stack permissions, it is read-only. So it
willarch/x86/mm/fault.c +3
-0+* generate a fault. +* +* For conventionally writable pages, a read can be serviced with a
+ * read only PTE, and COW would not have to happen. But for
shadow
+ * stack, there isn't the concept of read-only shadow stack
memory.
+ * If it is shadow stack permission, it can be modified via
CALL and
+ * RET instructions. So COW needs to happen before any memory
can bearch/x86/mm/fault.c +3
-0+* mapped with shadow stack permissions. +* +* Shadow stack accesses (read or write) need to be serviced with
+ * shadow stack permission memory, so in the case of a shadow
stack
+ * read access, treat it as a WRITE fault so both COW will
happen and
+ * the write fault path will ticklemaybe_mkwrite()and map
the memoryarch/x86/mm/fault.c +2
-0+* shadow stack. +*/
Quoted 5 lines
Quoted 3 lines
Again, I suggest dropping all details about COW from this comment and
from the patch description. It's just one such case that can happen.Hi David,
Hi Rick,
Quoted 7 lines
I was just trying to edit this one to drop COW details, but I think in this case, one of the major reasons for the code *is* actually COW. We are not working around the whole inadvertent shadow stack memory piece here, but something else: Making sure shadow stack memory is faulted in and doing COW if required to make this possible. I came up with this, does it seem better?
Regarding the fault handling I completely agree. We have to treat a read like a write event. And as read-only shadow stack PTEs don't exist, we have to tell the MM to create a writable one for us.
Quoted 17 lines
/*
* For conventionally writable pages, a read can be serviced with a *
read only PTE. But for shadow stack, there isn't a concept of
* read-
only shadow stack memory. If it a PTE has the shadow stack
*
permission, it can be modified via CALL and RET instructions. So
* core
MM needs to fault in a writable PTE and do things it already
* does for
write faults.
*
* Shadow stack accesses (read or write) need to be
serviced with
Quoted 8 lines
* shadow stack permission memory, so in the case of a
shadow stack
* read access, treat it as a WRITE fault so both any
required COW will
* happen and the write fault path will ticklemaybe_mkwrite()and map
* the memory shadow stack.
*/
That sounds good! I'd rewrite the last part slightly.
"
Shadow stack accesses (read or write) need to be serviced with
shadow stack permission memory, which always include write permissions. So in the case of a shadow stack read access, treat it as a WRITE fault. This will make sure that MM will prepare everything (e.g., break COW) such that maybe_mkwrite() can create a proper shadow stack PTE. "
Signature · 3 lines
Thanks, David / dhildenb
PATCH v6 20/41 x86/mm: Teach pte_mkwrite() about stack memory
In reply to Rick EdgecombeIf a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So when it is made writable with pte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now that pte_mkwrite() takes a VMA, and places where shadow stack memory might be created pass one, pte_mkwrite() can know when it should do this.
So make pte_mkwrite() create shadow stack memory when the VMA has the VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite().
This requires referencing VM_SHADOW_STACK in these functions, which are currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK is located) can't be pulled in without causing problems for files that reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c, where they can safely reference VM_SHADOW_STACK.
---
v6:
- New patch
--- arch/x86/include/asm/pgtable.h | 20 ++------------------ arch/x86/mm/pgtable.c | 26 ++++++++++++++++++++++++++ 2 files changed, 28 insertions(+), 18 deletions(-)
arch/x86/include/asm/pgtable.h +2 -18
@@ -456,15 +456,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)struct vm_area_struct ; -static inline pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) -{ - pte =pte_mkwrite_kernel (pte); - -if (pte_dirty (pte)) - pte =pte_clear_saveddirty (pte); - -return pte; -} +pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma);static inline pte_t pte_mkhuge (pte_t pte) {
@@ -601,15 +593,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)return pmd_set_flags (pmd, _PAGE_ACCESSED); } -static inline pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) -{ - pmd =pmd_set_flags (pmd, _PAGE_RW); - -if (pmd_dirty (pmd)) - pmd =pmd_clear_saveddirty (pmd); - -return pmd; -} +pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma);static inline pud_t pud_set_flags (pud_t pud,pudval_t set) {
arch/x86/mm/pgtable.c +26 -0
@@ -880,3 +880,29 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#endif /* CONFIG_X86_64 */ #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + +pte_t pte_mkwrite (pte_t pte,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHADOW_STACK ) +return pte_mkwrite_shstk (pte); + + pte =pte_mkwrite_kernel (pte); + +if (pte_dirty (pte)) + pte =pte_clear_saveddirty (pte); + +return pte; +} + +pmd_t pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_SHADOW_STACK ) +return pmd_mkwrite_shstk (pmd); + + pmd =pmd_set_flags (pmd, _PAGE_RW); + +if (pmd_dirty (pmd)) + pmd =pmd_clear_saveddirty (pmd); + +return pmd; +}
Signature · 1 line
2.17.1
Re: [PATCH v6 20/41] x86/mm: Teach pte_mkwrite() about stack memory
Quoted 9 lines · Rick Edgecombe
If a VMA has the
VM_SHADOW_STACKflag, it is shadow stack memory. So when it is made writable withpte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now thatpte_mkwrite()takes a VMA, and places where shadow stack memory might be created pass one,pte_mkwrite()can know when it should do this.So make
pte_mkwrite()create shadow stack memory when the VMA has theVM_SHADOW_STACKflag. Do the same thing forpmd_mkwrite().
Quoted 8 lines
This requires referencing
VM_SHADOW_STACKin these functions, which are currently defined inpgtable.h, howevermm.h(whereVM_SHADOW_STACKis located) can't be pulled in without causing problems for files that referencepgtable.h. So also move pte/pmd_mkwrite() intopgtable.c, where they can safely referenceVM_SHADOW_STACK.Tested-by Pengfei Xu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Is there any realistic performance impact from making these not inline now?
Signature · 1 line
Kees Cook
Re: [PATCH v6 20/41] x86/mm: Teach pte_mkwrite() about stack memory
Quoted 24 lines · Kees Cook
Quoted 23 lines · Rick Edgecombe
If a VMA has the
VM_SHADOW_STACKflag, it is shadow stack memory. So
when it is made writable withpte_mkwrite(), it should create shadow
stack memory, not conventionally writable memory. Now thatpte_mkwrite()
takes a VMA, and places where shadow stack memory might be created pass
one,pte_mkwrite()can know when it should do this.So make
pte_mkwrite()create shadow stack memory when the VMA has theVM_SHADOW_STACKflag. Do the same thing forpmd_mkwrite().This requires referencing
VM_SHADOW_STACKin these functions, which are
currently defined inpgtable.h, howevermm.h(whereVM_SHADOW_STACKis
located) can't be pulled in without causing problems for files that referencepgtable.h. So also move pte/pmd_mkwrite() intopgtable.c, where
they can safely referenceVM_SHADOW_STACK.
Quoted 6 lines
Quoted 2 lines
Tested-by Pengfei Xu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Is there any realistic performance impact from making these not inline
now?
Hmm, I can't say definitively. I would think in write protecting operations, the big cost would not be the PTE setters. For mapping things read-only from the beginning (user text, etc), I'm not sure. I guess it gives the compiler less flexibility, but also gives it the option to have one copy and so less text size overall for the kernel.
Are there any specific microbenchmarks we could run?
Re: [PATCH v6 20/41] x86/mm: Teach pte_mkwrite() about stack memory
In reply to Rick EdgecombeQuoted 9 lines · Rick Edgecombe
If a VMA has the
VM_SHADOW_STACKflag, it is shadow stack memory. So when it is made writable withpte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now thatpte_mkwrite()takes a VMA, and places where shadow stack memory might be created pass one,pte_mkwrite()can know when it should do this.So make
pte_mkwrite()create shadow stack memory when the VMA has theVM_SHADOW_STACKflag. Do the same thing forpmd_mkwrite().
Quoted 9 lines
This requires referencing
VM_SHADOW_STACKin these functions, which are currently defined inpgtable.h, howevermm.h(whereVM_SHADOW_STACKis located) can't be pulled in without causing problems for files that referencepgtable.h. So also move pte/pmd_mkwrite() intopgtable.c, where they can safely referenceVM_SHADOW_STACK.Tested-by Pengfei Xu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
PATCH v6 21/41 mm: Add guard pages around a shadow stack.
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks.
Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.
The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:
addq $0x80, %rsp
However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:
READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stackREAD_ONCE(ssp-8); // read+discard last popped stack element
The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.
This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work.
---
v5:
- Fix typo in commit log
v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)
v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)
Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.
--- include/linux/mm.h | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-)
include/linux/mm.h +26 -5
@@ -2980,15 +2980,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)return mtree_load (&mm->mm_mt, addr); } +static inline unsigned long stack_guard_start_gap (struct vm_area_struct *vma) +{ +if (vma->vm_flags &VM_GROWSDOWN ) +return stack_guard_gap; + +/* +* Shadow stack pointer is moved by CALL, RET, and INCSSPQ. +* INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB +* and touches the first and the last element in the range, which +* triggers a page fault if the range is not in a shadow stack. +* Because of this, creating 4-KB guard pages around a shadow +* stack prevents these instructions from going beyond. +* +* Creation of VM_SHADOW_STACK is tightly controlled, so a vma +* can't be both VM_GROWSDOWN and VM_SHADOW_STACK +*/ +if (vma->vm_flags &VM_SHADOW_STACK ) +return PAGE_SIZE ; + +return 0 ; +} +static inline unsigned long vm_start_gap (struct vm_area_struct *vma) { +unsigned long gap =stack_guard_start_gap (vma);unsigned long vm_start = vma->vm_start; -if (vma->vm_flags &VM_GROWSDOWN ) { - vm_start -= stack_guard_gap; -if (vm_start > vma->vm_start) - vm_start =0 ; - } + vm_start -= gap; +if (vm_start > vma->vm_start) + vm_start =0 ;return vm_start; }
Signature · 1 line
2.17.1
PATCH v6 22/41 mm/mmap: Add shadow stack pages to memory accounting
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Account shadow stack pages to stack memory.
---
v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill)
v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.
Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.
Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().
--- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+)
mm/mmap.c +2 -0
@@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages; else if (is_data_mapping(flags)) mm->data_vm += npages; }
Signature · 1 line
2.17.1
Re: [PATCH v6 22/41] mm/mmap: Add shadow stack pages to memory accounting
Quoted 40 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Account shadow stack pages to stack memory.
Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v3:
- Remove unneededVM_SHADOW_STACKcheck inaccountable_mapping()(Kirill)v2:
- Removeis_shadow_stack_mapping()and just change it to directly bitwise andVM_SHADOW_STACK.Yu-cheng v26:
- Remove redundant #ifdefCONFIG_MMU.Yu-cheng v25:
- Remove #ifdefCONFIG_ARCH_HAS_SHADOW_STACKforis_shadow_stack_mapping().--- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+)mm/mmap.c +0
-0
mm/mmap.c void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) · line 3294
Quoted 5 lines
mm/mmap.c +2
-0@@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages;
Why not modify is_stack_mapping() ?
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 22/41] mm/mmap: Add shadow stack pages to memory accounting
Quoted 40 lines · David Hildenbrand
Quoted 39 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Account shadow stack pages to stack memory.
Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v3:
- Remove unneededVM_SHADOW_STACKcheck inaccountable_mapping()(Kirill)v2:
- Removeis_shadow_stack_mapping()and just change it to
directly bitwise
andVM_SHADOW_STACK.Yu-cheng v26:
- Remove redundant #ifdefCONFIG_MMU.Yu-cheng v25:
- Remove #ifdefCONFIG_ARCH_HAS_SHADOW_STACKforis_shadow_stack_mapping().--- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 13 lines
mm/mmap.c void vm_stat_account(struct mm_struct *mm, · line 3295
Quoted 11 lines
mm/mmap.c +2
-0@@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages;Why not modify
is_stack_mapping()?
It kind of sticks out a little in this conditional, but is_stack_mapping() has this comment:
/*
* Stack area - automatically grows in one direction *
* VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
* do_mmap() forbids all other combinations.
*/
Shadow stack don't grow, so it doesn't quite fit. There used to be an is_shadow_stack_mapping(), but it was removed because all that was needed (for the time being) was the simple bitwise AND:
Re: [PATCH v6 22/41] mm/mmap: Add shadow stack pages to memory accounting
Quoted 64 lines · Rick P
Quoted 40 lines · David Hildenbrand
Quoted 39 lines · Rick Edgecombe
From: Yu-cheng Yu <[email protected]>
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.Account shadow stack pages to stack memory.
Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Co-developed-by Rick Edgecombe <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>Cc Kees Cook <[email protected]>---
v3:
- Remove unneededVM_SHADOW_STACKcheck inaccountable_mapping()(Kirill)v2:
- Removeis_shadow_stack_mapping()and just change it to
directly bitwise
andVM_SHADOW_STACK.Yu-cheng v26:
- Remove redundant #ifdefCONFIG_MMU.Yu-cheng v25:
- Remove #ifdefCONFIG_ARCH_HAS_SHADOW_STACKforis_shadow_stack_mapping().--- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+)Quoted 13 lines
mm/mmap.c void vm_stat_account(struct mm_struct *mm, · line 3295
Quoted 11 lines
mm/mmap.c +2
-0@@ -3290,6 +3290,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages) mm->exec_vm += npages; else if (is_stack_mapping(flags)) mm->stack_vm += npages; + else if (flags & VM_SHADOW_STACK) + mm->stack_vm += npages;Why not modify
is_stack_mapping()?It kind of sticks out a little in this conditional, but
is_stack_mapping()has this comment:
/*
* Stack area - automatically grows in one direction *
*VM_GROWSUP/VM_GROWSDOWNVMAs are always private anonymous:
*do_mmap()forbids all other combinations.
*/
Quoted 5 lines
Shadow stack don't grow, so it doesn't quite fit. There used to be an
is_shadow_stack_mapping(), but it was removed because all that was needed (for the time being) was the simple bitwise AND:
As there is only a single user of is_stack_mapping(), I'd simply have adjusted the doc of is_stack_mapping() to include shadow stacks.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 22/41] mm/mmap: Add shadow stack pages to memory accounting
Quoted 11 lines · David Hildenbrand
Quoted 11 lines
Quoted 1 line
Why not modify
is_stack_mapping()?It kind of sticks out a little in this conditional, but
is_stack_mapping()has this comment:
/*
* Stack area - automatically grows in one direction *
*VM_GROWSUP/VM_GROWSDOWNVMAs are always private anonymous:
*do_mmap()forbids all other combinations.
*/
Quoted 6 lines
Quoted 6 lines
Shadow stack don't grow, so it doesn't quite fit. There used to be an
is_shadow_stack_mapping(), but it was removed because all that was needed (for the time being) was the simple bitwise AND:
https://lore.kernel.org/lkml/[email protected]/
Quoted 4 lines
As there is only a single user of
is_stack_mapping(), I'd simply have
adjusted the doc ofis_stack_mapping()to include shadow stacks.
Ok, I'll update the comment and add it there.
PATCH v6 23/41 mm: Re-introduce vm_flags to do_mmap()
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by:
commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()").
There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap().
--- fs/aio.c | 2 +- include/linux/mm.h | 3 ++- ipc/shm.c | 2 +- mm/mmap.c | 10 +++++----- mm/nommu.c | 4 ++-- mm/util.c | 2 +- 6 files changed, 12 insertions(+), 11 deletions(-)
fs/aio.c +1 -1
@@ -554,7 +554,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->mmap_base =do_mmap (ctx->aio_ring_file,0 , ctx->mmap_size,PROT_READ |PROT_WRITE , -MAP_SHARED ,0 , &unused,NULL ); +MAP_SHARED ,0 ,0 , &unused,NULL );mmap_write_unlock (mm);if (IS_ERR ((void *)ctx->mmap_base)) { ctx->mmap_size =0 ;
include/linux/mm.h +2 -1
@@ -2890,7 +2890,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf);extern unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot,unsigned long flags, -unsigned long pgoff,unsigned long *populate,struct list_head *uf); +vm_flags_t vm_flags,unsigned long pgoff,unsigned long *populate, +struct list_head *uf);extern int do_mas_munmap (struct ma_state *mas,struct mm_struct *mm,unsigned long start,size_t len,struct list_head *uf,bool downgrade);
ipc/shm.c +1 -1
@@ -1662,7 +1662,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg, goto invalid; } - addr =do_mmap (file, addr, size, prot, flags,0 , &populate,NULL ); + addr =do_mmap (file, addr, size, prot, flags,0 ,0 , &populate,NULL ); *raddr = addr; err =0 ;if (IS_ERR_VALUE (addr))
mm/mmap.c +5 -5
@@ -1238,11 +1238,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,*/ unsigned long do_mmap (struct file *file,unsigned long addr,unsigned long len,unsigned long prot, -unsigned long flags,unsigned long pgoff, -unsigned long *populate,struct list_head *uf) +unsigned long flags,vm_flags_t vm_flags, +unsigned long pgoff,unsigned long *populate, +struct list_head *uf) {struct mm_struct *mm = current->mm; -vm_flags_t vm_flags;int pkey =0 ;validate_mm (mm);
@@ -1303,7 +1303,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,* to. we assume access permissions have been handled by the open * of the memory object, so we don't do any here. */ - vm_flags =calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | + vm_flags |=calc_vm_prot_bits (prot, pkey) |calc_vm_flag_bits (flags) | mm->def_flags |VM_MAYREAD |VM_MAYWRITE |VM_MAYEXEC ;if (flags &MAP_LOCKED )
@@ -2877,7 +2877,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size, file =get_file (vma->vm_file); ret =do_mmap (vma->vm_file, start, size, - prot, flags, pgoff, &populate,NULL ); + prot, flags,0 , pgoff, &populate,NULL );fput (file); out:mmap_write_unlock (mm);
mm/nommu.c +2 -2
@@ -1042,6 +1042,7 @@ unsigned long do_mmap(struct file *file,unsigned long len, unsigned long prot, unsigned long flags, + vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate, struct list_head *uf)
@@ -1049,7 +1050,6 @@ unsigned long do_mmap(struct file *file,struct vm_area_struct *vma;struct vm_region *region;struct rb_node *rb; -vm_flags_t vm_flags;unsigned long capabilities, result;int ret;MA_STATE (mas, ¤t->mm->mm_mt,0 ,0 );
@@ -1069,7 +1069,7 @@ unsigned long do_mmap(struct file *file,/* we've determined that we can make the mapping, now translate what we * now know into VMA flags */ - vm_flags =determine_vm_flags (file, prot, flags, capabilities); + vm_flags |=determine_vm_flags (file, prot, flags, capabilities);/* we're going to need to record the mapping */
mm/util.c +1 -1
@@ -517,7 +517,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,if (!ret) {if (mmap_write_lock_killable (mm))return -EINTR ; - ret =do_mmap (file, addr, len, prot, flag, pgoff, &populate, + ret =do_mmap (file, addr, len, prot, flag,0 , pgoff, &populate, &uf);mmap_write_unlock (mm);userfaultfd_unmap_complete (mm, &uf);
Signature · 1 line
2.17.1
PATCH v6 24/41 mm: Don't allow write GUPs to shadow stack memory
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.
Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.
---
v3:
- Add comment in __pte_access_permitted() (Dave)
- Remove unneeded shadow stack specific check in
__pte_access_permitted() (Jann)
--- arch/x86/include/asm/pgtable.h | 5 +++++ mm/gup.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +5 -0
@@ -1641,6 +1641,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write) {unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER; +/* +* Write=0,Dirty=1 PTEs are shadow stack, which the kernel +* shouldn't generally allow access to, but since they +* are already Write=0, the below logic covers both cases. +*/ if (write) need_pte_bits |= _PAGE_RW;
mm/gup.c +1 -1
@@ -982,7 +982,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)return -EFAULT ;if (write) { -if (!(vm_flags &VM_WRITE )) { +if (!(vm_flags &VM_WRITE ) || (vm_flags &VM_SHADOW_STACK )) {if (!(gup_flags &FOLL_FORCE ))return -EFAULT ;/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
Signature · 1 line
2.17.1
Re: [PATCH v6 24/41] mm: Don't allow write GUPs to shadow stack memory
Quoted 34 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via
get_user_pages(, FOLL_WRITE)operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.Still allow
FOLL_FORCEto write through shadow stack protections, as it does for read-only protections.Reviewed-by Kees Cook <[email protected]>Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v3:
- Add comment in__pte_access_permitted()(Dave)
- Remove unneeded shadow stack specific check in__pte_access_permitted() (Jann) --- arch/x86/include/asm/pgtable.h | 5 +++++ mm/gup.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)arch/x86/include/asm/pgtable.h +0
-0
arch/x86/include/asm/pgtable.h static inline bool __pte_access_permitted(unsigned long pteval, bool write) · line 1650
Quoted 10 lines
arch/x86/include/asm/pgtable.h +5
-0@@ -1641,6 +1641,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write) {unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER; +/* +* Write=0,Dirty=1 PTEs are shadow stack, which the kernel +* shouldn't generally allow access to, but since they +* are already Write=0, the below logic covers both cases. +*/ if (write) need_pte_bits |= _PAGE_RW;
So, GUP fast will always fail when writing ...
Quoted 6 lines
mm/gup.c +0
-0
mm/gup.c static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) · line 988
Quoted 8 lines
mm/gup.c +1
-1@@ -982,7 +982,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)return -EFAULT ;if (write) { -if (!(vm_flags &VM_WRITE )) { +if (!(vm_flags &VM_WRITE ) || (vm_flags &VM_SHADOW_STACK )) {if (!(gup_flags &FOLL_FORCE ))return -EFAULT ;/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
and ordinary GUP without FOLL_FORCE.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 24/41] mm: Don't allow write GUPs to shadow stack memory
Quoted 3 lines · David Hildenbrand
and ordinary GUP without
FOLL_FORCE.
Thanks! And for the other acks.
PATCH v6 25/41 x86/mm: Introduce MAP_ABOVE4G
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly.
One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a
general protection fault which will in turn trigger a segfault if it
tries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by.
So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though.
If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall.
Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's own randomization in the bottom up case.
For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G both are potentially valid, so both are used. In the bottomup search path, the default behavior is already consistent with MAP_ABOVE4G since mmap base should be above 4GB.
Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB. So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode with shadow stack enabled would usually segfault anyway. This is already pretty decent guard rails. But the addition of MAP_ABOVE4G is some small complexity spent to make it make it more complete.
---
v5:
- New patch
--- arch/x86/include/uapi/asm/mman.h | 1 + arch/x86/kernel/sys_x86_64.c | 6 +++++- include/linux/mman.h | 4 ++++ 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/mman.h +1 -0
@@ -3,6 +3,7 @@#define _ASM_X86_MMAN_H#define MAP_32BIT 0x40/* only give out 32bit addresses */ +#define MAP_ABOVE4G 0x80/* only map above 4GB */ #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS #define arch_calc_vm_prot_bits (prot, key) ( \
arch/x86/kernel/sys_x86_64.c +5 -1
@@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.flags =VM_UNMAPPED_AREA_TOPDOWN ; info.length = len; - info.low_limit =PAGE_SIZE ; +if (!in_32bit_syscall () && (flags &MAP_ABOVE4G )) + info.low_limit =0x100000000 ; +else + info.low_limit =PAGE_SIZE ; + info.high_limit =get_mmap_base (0 );/*
include/linux/mman.h +4 -0
@@ -15,6 +15,9 @@#ifndef MAP_32BIT #define MAP_32BIT 0#endif +#ifndef MAP_ABOVE4G +#define MAP_ABOVE4G 0 +#endif #ifndef MAP_HUGE_2MB #define MAP_HUGE_2MB 0#endif
@@ -50,6 +53,7 @@ | MAP_STACK \ |MAP_HUGETLB \ |MAP_32BIT \ + |MAP_ABOVE4G \ |MAP_HUGE_2MB \ | MAP_HUGE_1GB)
Signature · 1 line
2.17.1
Re: [PATCH v6 25/41] x86/mm: Introduce MAP_ABOVE4G
Quoted 24 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly.
One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals.
On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a
general protection fault which will in turn trigger a segfault if ittries to transition to 32 bit mode with shadow stack enabled.
This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by.
Quoted 8 lines
So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce
MAP_ABOVE4G. The behavior is pretty much likeMAP_32BIT, except that it has the opposite address range. The are a few differences though.If both
MAP_32BITandMAP_ABOVE4Gare provided, the kernel will use theMAP_ABOVE4Gbehavior. LikeMAP_32BIT,MAP_ABOVE4Gis ignored in a 32 bit syscall.
Should the interface refuse to accept both set instead?
Signature · 1 line
Kees Cook
Re: [PATCH v6 25/41] x86/mm: Introduce MAP_ABOVE4G
Quoted 48 lines · Kees Cook
Quoted 47 lines · Rick Edgecombe
The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which require some core mm changes to function properly.One of the properties is that the shadow stack pointer (SSP), which is a
CPU register that points to the shadow stack like the stack pointer points
to the stack, can't be pointing outside of the 32 bit address space when
the CPU is executing in 32 bit mode. It is desirable to prevent executing
in 32 bit mode when shadow stack is enabled because the kernel can't easily
support 32 bit signals.On x86 it is possible to transition to 32 bit mode without any special
interaction with the kernel, by doing a "far call" to a 32 bit segment.
So the shadow stack implementation can use this address space behavior
as a feature, by enforcing that shadow stack memory is always crated
outside of the 32 bit address space. This way userspace will trigger ageneral protection fault which will in turn trigger a segfault ifit
tries to transition to 32 bit mode with shadow stack enabled.This provides a clean error generating border for the user if they try
attempt to do 32 bit mode shadow stack, rather than leave the kernel in a
half working state for userspace to be surprised by.So to allow future shadow stack enabling patches to map shadow stacks
out of the 32 bit address space, introduceMAP_ABOVE4G. The behavior
is pretty much likeMAP_32BIT, except that it has the opposite address
range. The are a few differences though.
Quoted 7 lines
Quoted 5 lines
If both
MAP_32BITandMAP_ABOVE4Gare provided, the kernel will use theMAP_ABOVE4Gbehavior. LikeMAP_32BIT,MAP_ABOVE4Gis ignored in a 32 bit
syscall.Should the interface refuse to accept both set instead?
I guess that might be less surprising. But I think to do this would either require adding logic to core mm or a new arch breakout. I actually kind of wish there was an easy way to keep this flag from being used from userspace and just be a kernel only thing. It is only used internally in this series and there isn't any know use for userspace.
Quoted 2 lines
Reviewed-by Kees Cook <[email protected]>
PATCH v6 26/41 mm: Warn on shadow stack memory in wrong vma
In reply to Rick EdgecombeThe x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.
One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured.
In order to check if a pte is shadow stack in core mm code, add two arch breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack specific code to be kept in arch/x86.
---
v6:
- Add arch breakout to remove shstk from core MM code.
v5:
- Fix typo in commit log
v3:
- New patch
--- arch/x86/include/asm/pgtable.h | 6 ++++++ arch/x86/mm/pgtable.c | 12 ++++++++++++ include/linux/pgtable.h | 14 ++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 34 insertions(+)
arch/x86/include/asm/pgtable.h +6 -0
@@ -1687,6 +1687,12 @@ static inline bool arch_has_hw_pte_young(void)return true; } +#define arch_check_zapped_pte arch_check_zapped_pte +void arch_check_zapped_pte (struct vm_area_struct *vma,pte_t pte); + +#define arch_check_zapped_pmd arch_check_zapped_pmd +void arch_check_zapped_pmd (struct vm_area_struct *vma,pmd_t pmd); +#ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_youngstatic inline bool arch_has_hw_nonleaf_pmd_young (void )
arch/x86/mm/pgtable.c +12 -0
@@ -906,3 +906,15 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)return pmd; } + +void arch_check_zapped_pte (struct vm_area_struct *vma,pte_t pte) +{ +VM_WARN_ON_ONCE (!(vma->vm_flags &VM_SHADOW_STACK ) && +pte_shstk (pte)); +} + +void arch_check_zapped_pmd (struct vm_area_struct *vma,pmd_t pmd) +{ +VM_WARN_ON_ONCE (!(vma->vm_flags &VM_SHADOW_STACK ) && +pmd_shstk (pmd)); +}
include/linux/pgtable.h +14 -0
@@ -291,6 +291,20 @@ static inline bool arch_has_hw_pte_young(void) }#endif +#ifndef arch_check_zapped_pte +static inline void arch_check_zapped_pte (struct vm_area_struct *vma, +pte_t pte) +{ +} +#endif + +#ifndef arch_check_zapped_pmd +static inline void arch_check_zapped_pmd (struct vm_area_struct *vma, +pmd_t pmd) +{ +} +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
mm/huge_memory.c +1 -0
@@ -1696,6 +1696,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,*/ orig_pmd =pmdp_huge_get_and_clear_full (vma, addr, pmd, tlb->fullmm); +arch_check_zapped_pmd (vma, orig_pmd);tlb_remove_pmd_tlb_entry (tlb, pmd, addr);if (vma_is_special_huge (vma)) {if (arch_needs_pgtable_deposit ())
mm/memory.c +1 -0
@@ -1377,6 +1377,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,continue ; ptent =ptep_get_and_clear_full (mm, addr, pte, tlb->fullmm); +arch_check_zapped_pte (vma, ptent);tlb_remove_tlb_entry (tlb, pte, addr);zap_install_uffd_wp_if_needed (vma, addr, pte, details, ptent);
Signature · 1 line
2.17.1
PATCH v6 27/41 x86/mm: Warn if create Write=0,Dirty=1 with raw prot
In reply to Rick EdgecombeWhen user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected.
However, when a PTE is created by a raw prot using mk_pte(), mk_pte() can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift.
The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning to make sure no new mk_pte() start doing this.
---
v6:
- New patch (Note, this has already been a useful warning, it caught the newly added set_memory_rox() doing this)
--- arch/x86/include/asm/pgtable.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +9 -1
@@ -1032,7 +1032,15 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)* (Currently stuck as a macro because of indirect forward reference * to linux/mm.h:page_to_nid()) */ -#define mk_pte (page, pgprot) pfn_pte(page_to_pfn(page), (pgprot)) +#define mk_pte (page, pgprot) \ +({ \ + pgprot_t __pgprot = pgprot; \ + \ + WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && \ + (pgprot_val(__pgprot) & (_PAGE_DIRTY | _PAGE_RW)) == \ + _PAGE_DIRTY); \ + pfn_pte(page_to_pfn(page), __pgprot); \ +})static inline int pmd_bad (pmd_t pmd) {
Signature · 1 line
2.17.1
Re: [PATCH v6 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot
Quoted 37 lines · Rick Edgecombe
When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside
pte_wrprotect()because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected.However, when a PTE is created by a raw prot using
mk_pte(),mk_pte()can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift.The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the
pte_mkfoo()helpers. Add a warning to make sure no newmk_pte()start doing this.Tested-by Pengfei Xu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v6:
- New patch (Note, this has already been a useful warning, it caught the newly addedset_memory_rox()doing this)--- arch/x86/include/asm/pgtable.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)arch/x86/include/asm/pgtable.h +0
-0@@ -1032,7 +1032,15 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)* (Currently stuck as a macro because of indirect forward reference * to linux/mm.h:page_to_nid()) */
arch/x86/include/asm/pgtable.h static inline unsigned long pmd_page_vaddr(pmd_t pmd) · line 1043
Quoted 10 lines
arch/x86/include/asm/pgtable.h +9
-1-#define mk_pte (page, pgprot) pfn_pte(page_to_pfn(page), (pgprot)) +#define mk_pte (page, pgprot) \ +({ \ + pgprot_t __pgprot = pgprot; \ + \ + WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && \ + (pgprot_val(__pgprot) & (_PAGE_DIRTY | _PAGE_RW)) == \ + _PAGE_DIRTY); \ + pfn_pte(page_to_pfn(page), __pgprot); \ +})
This only warns? Should it also enforce the state?
Signature · 1 line
Kees Cook
Re: [PATCH v6 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot
Quoted 24 lines · Kees Cook
Quoted 24 lines
arch/x86/include/asm/pgtable.h +0
-0@@ -1032,7 +1032,15 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) * (Currently stuck as a macro because of indirect forwardreference
* to linux/mm.h:page_to_nid())
*/
-#definemk_pte(page, pgprot)pfn_pte(page_to_pfn(page),
(pgprot))
+#define mk_pte(page,
pgprot) \arch/x86/include/asm/pgtable.h +2
-0+({ \ + pgprot_t __pgprot =pgprot; \
arch/x86/include/asm/pgtable.h +2
-0+ \ + WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_USER_SHSTK)&& \
+ (pgprot_val(__pgprot)& (_PAGE_DIRTY|_PAGE_RW))
Quoted 8 lines
Quoted 6 lines
== \
+_PAGE_DIRTY); \
+ pfn_pte(page_to_pfn(page),__pgprot); \
+})This only warns? Should it also enforce the state?
Hmm, you mean something like forcing Dirty=0 if Write=0?
The thing we are worried about here is some new x86 code that creates Write=0,Dirty=1 PTEs directly because the developer is unaware or forgot about shadow stack. The issue the warning actually caught was kernel memory being marked Write=0,Dirty=1, which today is more about consistency than any functional issue. But if some future hypothetical code was creating a userspace PTE like this, and depending on the memory being read-only, then the enforcement would be useful and potentially save the day.
The downside is that it adds tricky logic into a low level helper that shouldn't be required unless strange and wrong new code is added in the future. And then it is still only useful if the warning doesn't catch the issue in testing. And then there would be some slight risk that the Dirty bit was expected to be there in some PTE without shadow stack exposure, and a functional bug would be introduced.
I'm waffling here. I could be convinced either way. Hopefully that helps characterize the dilemma at least.
PATCH v6 28/41 x86: Introduce userspace API for shadow stack
In reply to Rick EdgecombeFrom: "Kirill A. Shutemov" <[email protected]>
Add three new arch_prctl() handles:
- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.
- ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error
The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().
This is preparation patch. It does not implement any features.
---
v4:
- Remove references to CET and replace with shadow stack (Peterz)
v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM
v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)
--- arch/x86/include/asm/processor.h | 6 +++++ arch/x86/include/asm/shstk.h | 21 +++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 6 +++++ arch/x86/kernel/Makefile | 2 ++ arch/x86/kernel/process_64.c | 7 ++++- arch/x86/kernel/shstk.c | 44 +++++++++++++++++++++++++++++++ 6 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/shstk.h create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/processor.h +6 -0
@@ -28,6 +28,7 @@ struct vm86;#include <asm/unwind_hints.h> #include <asm/vmxfeatures.h> #include <asm/vdso/processor.h> +#include <asm/shstk.h> #include <linux/personality.h> #include <linux/cache.h>
@@ -475,6 +476,11 @@ struct thread_struct {*/ u32 pkru; +#ifdef CONFIG_X86_USER_SHADOW_STACK +unsigned long features; +unsigned long features_locked; +#endif +/* Floating point and extended processor state */ struct fpu fpu;/*
arch/x86/include/asm/shstk.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_SHSTK_H +#define _ASM_X86_SHSTK_H + +#ifndef __ASSEMBLY__ +#include <linux/types.h> + +struct task_struct ; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +long shstk_prctl (struct task_struct *task,int option,unsigned long features); +void reset_thread_features (void ); +#else +static inline long shstk_prctl (struct task_struct *task,int option, +unsigned long arg2) {return -EINVAL ; } +static inline void reset_thread_features (void ) {} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ + +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_X86_SHSTK_H */
arch/x86/include/uapi/asm/prctl.h +6 -0
@@ -20,4 +20,10 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +/* Don't use 0x3001-0x3004 because of old glibcs */ + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +2 -0
@@ -146,6 +146,8 @@ obj-$(CONFIG_CALL_THUNKS) += callthunks.o obj-$(CONFIG_X86_CET) += cet.o +obj-$(CONFIG_X86_USER_SHADOW_STACK) += shstk.o + ### # 64 bit specific files ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/process_64.c +6 -1
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,load_gs_index (__USER_DS); } +reset_thread_features (); +loadsegment (fs,0 );loadsegment (es, _ds);loadsegment (ds, _ds);
@@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_MAP_VDSO_64 :return prctl_map_vdso (&vdso_image_64, arg2);#endif - +case ARCH_SHSTK_ENABLE : +case ARCH_SHSTK_DISABLE : +case ARCH_SHSTK_LOCK : +return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;break ;
arch/x86/kernel/shstk.c new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v6 28/41] x86: Introduce userspace API for shadow stack
arch/x86/include/uapi/asm/prctl.h line 23
Quoted 9 lines · Rick Edgecombe
arch/x86/include/uapi/asm/prctl.h +1
-0@@ -20,4 +20,10 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003 +/* Don't use 0x3001-0x3004 because of old glibcs */
So where is this all new interface to userspace programs documented? Do we have an agreement with all the involved parties that this is how we're going to support shadow stacks and that this is what userspace should do?
I'd like to avoid one more fiasco with glibc etc here...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v6 28/41] x86: Introduce userspace API for shadow stack
Quoted 10 lines · Borislav Petkov
Quoted 9 lines · Rick Edgecombe
arch/x86/include/uapi/asm/prctl.h +0
-0@@ -20,4 +20,10 @@#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003
Quoted 3 lines
arch/x86/include/uapi/asm/prctl.h line 23
Quoted 1 line
arch/x86/include/uapi/asm/prctl.h +1
-0+/* Don't use 0x3001-0x3004 because of old glibcs */ So where is this all new interface to userspace programs documented?
In the first patch:
https://lore.kernel.org/lkml/[email protected]/
Then some more documentation is added about ARCH_SHSTK_UNLOCK and ARCH_SHSTK_STATUS (which are for CRIU) in those patches.
Quoted 6 lines
Do
we have an agreement with all the involved parties that this is how we're going to support shadow stacks and that this is what userspace should do?I'd like to avoid one more fiasco with glibc etc here...
There are glibc patches prepared by HJ to use the new interface and it's my understanding that he has discussed the changes with the other glibc folks. Those glibc patches are used for testing these kernel patches, but will not get upstream until the kernel patches to avoid repeating the past problems. So I think it's as prepared as it can be.
One future thing that might come up... Glibc has this mode called "permissive mode". When glibc is configured this way (compile time or env var), it is supposed to disable shadow stack when dlopen()ing any DSO that doesn't have the shadow stack elf header bit. The problem is, it doesn't really work as intended. It only turns off shadow stack for the calling thread, leaving the other threads free to call into the DSO with shadow stack enabled. It's not clear yet if they are going to be able to fix it in userspace. So this prompted some thinking of if there could be an additional kernel mode to help glibc in this scenario. But for non-permissive mode, glibc is queued up to use the interface as is.
Borislav Petkov
maintainerRe: [PATCH v6 28/41] x86: Introduce userspace API for shadow stack
Quoted 6 lines · Rick P
In the first patch:
https://lore.kernel.org/lkml/[email protected]/
Then some more documentation is added about
ARCH_SHSTK_UNLOCKandARCH_SHSTK_STATUS(which are for CRIU) in those patches.
Right, I was thinking more about ARCH_PRCTL(2), the man page.
But you can send that to the manpages folks later. I.e., it should be nearly impossible to be missed. :)
Quoted 5 lines
There are glibc patches prepared by HJ to use the new interface and it's my understanding that he has discussed the changes with the other glibc folks. Those glibc patches are used for testing these kernel patches, but will not get upstream until the kernel patches to avoid repeating the past problems. So I think it's as prepared as it can be.
Good.
Quoted 4 lines
One future thing that might come up... Glibc has this mode called "permissive mode". When glibc is configured this way (compile time or env var), it is supposed to disable shadow stack when dlopen()ing any DSO that doesn't have the shadow stack elf header bit.
Maybe I don't understand all the possible use cases but if I were interested in using shadow stack, then I'd enable it for all objects. And if I want permissive, I'd disable it for all. A mixed thing sounds like a mixed can of worms waiting to be opened.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v6 28/41] x86: Introduce userspace API for shadow stack
Quoted 4 lines · Borislav Petkov
Quoted 3 lines · Rick P
In the first patch:
https://lore.kernel.org/lkml/[email protected]/
Quoted 8 lines
Quoted 3 lines
Then some more documentation is added about
ARCH_SHSTK_UNLOCKandARCH_SHSTK_STATUS(which are for CRIU) in those patches.Right, I was thinking more about
ARCH_PRCTL(2), the man page.But you can send that to the manpages folks later. I.e., it should be nearly impossible to be missed. :)
Sure I can add something. Somewhere I have a man page for map_shadow_stack written up as well.
Quoted 19 lines
Quoted 9 lines
There are glibc patches prepared by HJ to use the new interface and it's my understanding that he has discussed the changes with the other
glibc folks. Those glibc patches are used for testing these kernel patches, but will not get upstream until the kernel patches to avoid
repeating the past problems. So I think it's as prepared as it can be.Good.
Quoted 6 lines
One future thing that might come up... Glibc has this mode called "permissive mode". When glibc is configured this way (compile time or
env var), it is supposed to disable shadow stack when dlopen()ing any
DSO that doesn't have the shadow stack elf header bit.
Quoted 2 lines
Maybe I don't understand all the possible use cases but if I were interested in using shadow stack, then I'd enable it for all objects.
Enabling for all objects is the ideal, but in practice distros don't have that.
Quoted 3 lines
And if I want permissive, I'd disable it for all. A mixed thing sounds
like a mixed can of worms waiting to be opened.
It is definitely a can of worms.
PATCH v6 29/41 x86/shstk: Add user-mode shadow stack support
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).
Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.
Do not support IA32 emulation or x32.
---
v5:
- Switch to EOPNOTSUPP
- Use MAP_ABOVE4G
- Move set_clr_bits_msrl() to patch where it is first used
v4:
- Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz)
v3:
- Use define for set_clr_bits_msrl() (Kees)
- Make some functions static (Kees)
- Change feature_foo() to features_foo() (Kees)
- Centralize shadow stack size rlimit checks (Kees)
- Disable x32 support
v2:
- Get rid of unnecessary shstk->base checks
- Don't support IA32 emulation
--- arch/x86/include/asm/processor.h | 2 + arch/x86/include/asm/shstk.h | 7 ++ arch/x86/include/uapi/asm/prctl.h | 3 + arch/x86/kernel/shstk.c | 145 ++++++++++++++++++++++++++++++ 4 files changed, 157 insertions(+)
arch/x86/include/asm/processor.h +2 -0
@@ -479,6 +479,8 @@ struct thread_struct {#ifdef CONFIG_X86_USER_SHADOW_STACK unsigned long features;unsigned long features_locked; + +struct thread_shstk shstk;#endif /* Floating point and extended processor state */
arch/x86/include/asm/shstk.h +7 -0
@@ -8,12 +8,19 @@struct task_struct ;#ifdef CONFIG_X86_USER_SHADOW_STACK +struct thread_shstk { +u64 base; +u64 size; +}; +long shstk_prctl (struct task_struct *task,int option,unsigned long features);void reset_thread_features (void ); +void shstk_free (struct task_struct *p);#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long arg2) {return -EINVAL ; }static inline void reset_thread_features (void ) {} +static inline void shstk_free (struct task_struct *p) {}#endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */
arch/x86/include/uapi/asm/prctl.h +3 -0
@@ -26,4 +26,7 @@#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003 +/* ARCH_SHSTK_ features bits */ +#define ARCH_SHSTK_SHSTK (1ULL << 0) +#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/shstk.c +145 -0
159 lines not on this page · raw message
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v6 29/41] x86/shstk: Add user-mode shadow stack support
Quoted 1 line · Rick Edgecombe
Do not support IA32 emulation or x32.
Because? Simplicity?
No one cares about 32-bit?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v6 29/41] x86/shstk: Add user-mode shadow stack support
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Rick Edgecombe
Do not support IA32 emulation or x32.
Because? Simplicity?
No one cares about 32-bit?
Yea a little of both. Originally shadow stack 32 bit emulation was supported for 64 bit kernels, but then Andy Lutomirski asked for the signal ABI to flexible enough to support alt shadow stacks in the future. This involved stuffing things on the shadow stack that didn't work well for 32 bit. Resolving this wasn't exhaustively explored, but there weren't any obvious things that jumped out.
Then there was the question of how much 32 bit CET apps running on 64 bit kernels would get used. Since shadow stack needs a re-compile this would only be for newly build 32 bit binaries, not old legacy binaries that seems to be the thing often using legacy emulation. So it was kind of not expected to be used much or at all, so any kind of complications tipped the scales toward dropping it. PeterZ brought up WINE running 32 bit Windows apps, but apparently Windows doesn't support 32 bit CET either. And then there is that we can always add it later if a big use shows up.
I'll add a little more info in the commit log about this.
Borislav Petkov
maintainerRe: [PATCH v6 29/41] x86/shstk: Add user-mode shadow stack support
Quoted 1 line · Rick P
I'll add a little more info in the commit log about this.
Thanks.
And I can't do anything but agree with the train of thought you guys are having - 32-bit is a dead horse and we should stop trying to revive it.
:-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v6 30/41 x86/shstk: Handle thread shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways.
With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.
For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.
For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.
Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.
For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().
Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.
Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent.
During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner.
---
v3:
- Fix update_fpu_shstk() stub (Mike Rapoport)
- Fix chunks around alloc_shstk() in wrong patch (Kees)
- Fix stack_size/flags swap (Kees)
- Use centralized stack size logic (Kees)
v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx)
v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.
Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)
--- arch/x86/include/asm/fpu/sched.h | 3 ++- arch/x86/include/asm/mmu_context.h | 2 ++ arch/x86/include/asm/shstk.h | 7 +++++ arch/x86/kernel/fpu/core.c | 41 +++++++++++++++++++++++++++- arch/x86/kernel/process.c | 18 ++++++++++++- arch/x86/kernel/shstk.c | 43 ++++++++++++++++++++++++++++-- 6 files changed, 109 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/sched.h +2 -1
@@ -11,7 +11,8 @@extern void save_fpregs_to_fpstate (struct fpu *fpu);extern void fpu__drop (struct fpu *fpu); -extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal); +extern int fpu_clone (struct task_struct *dst,unsigned long clone_flags,bool minimal, +unsigned long shstk_addr);extern void fpu_flush_thread (void );/*
arch/x86/include/asm/mmu_context.h +2 -0
@@ -147,6 +147,8 @@ do { \#else #define deactivate_mm (tsk, mm) \ do { \ + if (!tsk->vfork_done) \ + shstk_free(tsk); \ load_gs_index(0); \ loadsegment(fs, 0); \ } while (0)
arch/x86/include/asm/shstk.h +7 -0
@@ -15,11 +15,18 @@ struct thread_shstk {long shstk_prctl (struct task_struct *task,int option,unsigned long features);void reset_thread_features (void ); +int shstk_alloc_thread_stack (struct task_struct *p,unsigned long clone_flags, +unsigned long stack_size, +unsigned long *shstk_addr);void shstk_free (struct task_struct *p);#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long arg2) {return -EINVAL ; }static inline void reset_thread_features (void ) {} +static inline int shstk_alloc_thread_stack (struct task_struct *p, +unsigned long clone_flags, +unsigned long stack_size, +unsigned long *shstk_addr) {return 0 ; }static inline void shstk_free (struct task_struct *p) {}#endif /* CONFIG_X86_USER_SHADOW_STACK */
arch/x86/kernel/fpu/core.c +40 -1
54 lines not on this page · raw message
arch/x86/kernel/process.c +17 -1
@@ -48,6 +48,7 @@#include <asm/frame.h> #include <asm/unwind.h> #include <asm/tdx.h> +#include <asm/shstk.h> #include "process.h"
@@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk)free_vm86 (t); +shstk_free (tsk);fpu__drop (fpu); }
@@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)struct inactive_task_frame *frame;struct fork_frame *fork_frame;struct pt_regs *childregs; +unsigned long shstk_addr =0 ;int ret =0 ; childregs =task_pt_regs (p);
@@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) frame->flags =X86_EFLAGS_FIXED ;#endif -fpu_clone (p, clone_flags, args->fn); +/* Allocate a new shadow stack for pthread if needed */ + ret =shstk_alloc_thread_stack (p, clone_flags, args->stack_size, + &shstk_addr); +if (ret) +return ret; + +fpu_clone (p, clone_flags, args->fn, shstk_addr);/* Kernel thread ? */ if (unlikely (p->flags &PF_KTHREAD )) {
@@ -220,6 +229,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)if (!ret &&unlikely (test_tsk_thread_flag (current,TIF_IO_BITMAP )))io_bitmap_share (p); +/* +* If copy_thread() if failing, don't leak the shadow stack possibly +* allocated in shstk_alloc_thread_stack() above. +*/ +if (ret) +shstk_free (p); +return ret; }
arch/x86/kernel/shstk.c +41 -2
61 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 31/41 x86/shstk: Introduce routines modifying shstk
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.
A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.
Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.
In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.
To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.
---
v5:
- Fix typo in commit log
v3:
- Drop shstk_check_rstor_token()
- Fail put_shstk_data() if bit 63 is set in the data (Kees)
- Add comment in create_rstor_token() (Kees)
- Pull in create_rstor_token() changes from future patch (Kees)
v2:
- Add data helpers for writing to shadow stack.
v1:
- Use xsave helpers.
--- arch/x86/include/asm/special_insns.h | 13 +++++ arch/x86/kernel/shstk.c | 73 ++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
@@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p) : [pax] "a" (p)); } +#ifdef CONFIG_X86_USER_SHADOW_STACK +static inline int write_user_shstk_64 (u64 __user *addr,u64 val) +{ + asm_volatile_goto("1: wrussq %[val], (%[addr])\n" + _ASM_EXTABLE(1b, %l[fail]) + :: [addr] "r" (addr), [val] "r" (val) + :: fail); +return 0 ; +fail: +return -EFAULT ; +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */ +#define nop () asm volatile ("nop")static inline void serialize (void )
arch/x86/kernel/shstk.c +73 -0
91 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v6 32/41 x86/shstk: Handle signals for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.
One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.
The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.
For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.
So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.
In sigreturn, verify SSP is stored in the data format and pop the shadow stack.
---
v3:
- Drop shstk_setup_rstor_token() (Kees)
- Drop x32 signal support, since x32 support is dropped
v2:
- Switch to new shstk signal format
v1:
- Use xsave helpers.
- Expand commit log.
Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.
--- arch/x86/include/asm/shstk.h | 5 ++ arch/x86/kernel/shstk.c | 98 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/signal.c | 1 + arch/x86/kernel/signal_64.c | 6 +++ 4 files changed, 110 insertions(+)
arch/x86/include/asm/shstk.h +5 -0
@@ -6,6 +6,7 @@#include <linux/types.h> struct task_struct ; +struct ksignal ;#ifdef CONFIG_X86_USER_SHADOW_STACK struct thread_shstk {
@@ -19,6 +20,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,unsigned long stack_size, unsigned long *shstk_addr);void shstk_free (struct task_struct *p); +int setup_signal_shadow_stack (struct ksignal *ksig); +int restore_signal_shadow_stack (void );#else static inline long shstk_prctl (struct task_struct *task,int option,unsigned long arg2) {return -EINVAL ; }
@@ -28,6 +31,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p, unsigned long stack_size, unsigned long *shstk_addr) { return 0; }static inline void shstk_free (struct task_struct *p) {} +static inline int setup_signal_shadow_stack (struct ksignal *ksig) {return 0 ; } +static inline int restore_signal_shadow_stack (void ) {return 0 ; }#endif /* CONFIG_X86_USER_SHADOW_STACK */ #endif /* __ASSEMBLY__ */
arch/x86/kernel/shstk.c +98 -0
104 lines not on this page · raw message
arch/x86/kernel/signal.c +1 -0
@@ -40,6 +40,7 @@#include <asm/syscall.h> #include <asm/sigframe.h> #include <asm/signal.h> +#include <asm/shstk.h> static inline int is_ia32_compat_frame (struct ksignal *ksig) {
arch/x86/kernel/signal_64.c +6 -0
@@ -175,6 +175,9 @@ int x64_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs) frame =get_sigframe (ksig, regs,sizeof (struct rt_sigframe ), &fp); uc_flags =frame_uc_flags (regs); +if (setup_signal_shadow_stack (ksig)) +return -EFAULT ; +if (!user_access_begin (frame,sizeof (*frame)))return -EFAULT ;
@@ -260,6 +263,9 @@ SYSCALL_DEFINE0(rt_sigreturn)if (!restore_sigcontext (regs, &frame->uc.uc_mcontext, uc_flags)) goto badframe; +if (restore_signal_shadow_stack ()) + goto badframe; +if (restore_altstack (&frame->uc.uc_stack)) goto badframe;
Signature · 1 line
2.17.1
PATCH v6 33/41 x86/shstk: Introduce map_shadow_stack syscall
In reply to Rick EdgecombeWhen operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks.
Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable.
Previously, a new PROT_SHADOW_STACK was attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window.
The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token.
First, a new madvise() flag was explored, which could operate on the PROT_SHADOW_STACK memory. This had a couple downsides:
1. Extra checks were needed in mprotect() to prevent writable memory from ever becoming PROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the new madvise() to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to.
3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior.
So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.
The following example demonstrates how to create a new shadow stack with map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);
---
v5:
- Fix addr/mapped_addr (Kees)
- Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this)
- Return error for addresses below 4G
v3:
- Change syscall common -> 64 (Kees)
- Use bit shift notation instead of 0x1 for uapi header (Kees)
- Call do_mmap() with MAP_FIXED_NOREPLACE (Kees)
- Block unsupported flags (Kees)
- Require size >= 8 to set token (Kees)
v2:
- Change syscall to take address like mmap() for CRIU's usage
v1:
- New patch (replaces PROT_SHADOW_STACK).
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/uapi/asm/mman.h | 3 ++ arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++---- include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 2 +- kernel/sys_ni.c | 1 + 6 files changed, 58 insertions(+), 9 deletions(-)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -372,6 +372,7 @@ 448 common process_mrelease sys_process_mrelease 449 common futex_waitv sys_futex_waitv 450 common set_mempolicy_home_node sys_set_mempolicy_home_node +451 64 map_shadow_stack sys_map_shadow_stack # # Due to a historical design error, certain syscalls are numbered differently
arch/x86/include/uapi/asm/mman.h +3 -0
@@ -13,6 +13,9 @@ ((key) & 0x8 ? VM_PKEY_BIT3 : 0))#endif +/* Flags for map_shadow_stack(2) */ +#define SHADOW_STACK_SET_TOKEN (1ULL << 0)/* Set up a restore token in the shadow stack */ +#include <asm-generic/mman.h> #endif /* _ASM_X86_MMAN_H */
arch/x86/kernel/shstk.c +51 -8
96 lines not on this page · raw message
include/linux/syscalls.h +1 -0
@@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags);asmlinkage long sys_set_mempolicy_home_node (unsigned long start,unsigned long len,unsigned long home_node,unsigned long flags); +asmlinkage long sys_map_shadow_stack (unsigned long addr,unsigned long size,unsigned int flags);/* * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +1 -1
@@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)#undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452/* * 32 bit systems traditionally used different
kernel/sys_ni.c +1 -0
@@ -381,6 +381,7 @@ COND_SYSCALL(vm86old);COND_SYSCALL (modify_ldt);COND_SYSCALL (vm86);COND_SYSCALL (kexec_file_load); +COND_SYSCALL (map_shadow_stack);/* s390 */ COND_SYSCALL (s390_pci_mmio_read);
Signature · 1 line
2.17.1
Re: [PATCH v6 33/41] x86/shstk: Introduce map_shadow_stack syscall
Quoted 36 lines · Rick Edgecombe
When operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks.
Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable.
Previously, a new
PROT_SHADOW_STACKwas attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window.The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token.
First, a new
madvise()flag was explored, which could operate on thePROT_SHADOW_STACKmemory. This had a couple downsides:
1. Extra checks were needed inmprotect()to prevent writable memory from ever becomingPROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the newmadvise()to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to.
3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior.
Quoted 8 lines
So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new
map_shadow_stacksyscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.
Was following ever attempted?
void *shstk = mmap(0, size, PROT_SHADOWSTACK, ...);
- limit PROT_SHADOWSTACK protection flag to only mmap (and thus mprotect can't convert memory from shadow stack to non-shadow stack type or vice versa)
- limit PROT_SHADOWSTACK protection flag to anonymous memory only.
- top level mmap handler to put a token at the base using WRUSS if prot == PROT_SHADOWSTACK
You essentially would get shadow stack manufacturing with existing (single) syscall.
Acting a bit selfish here, this allows other architectures as well to re-use this and do their own implementation of mapping and placing the token at the base.
Quoted 209 lines
The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk =map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Reviewed-by Kees Cook <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
v5:
- Fix addr/mapped_addr (Kees)
- Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this)
- Return error for addresses below 4Gv3:
- Change syscall common -> 64 (Kees)
- Use bit shift notation instead of0x1for uapi header (Kees)
- Calldo_mmap()withMAP_FIXED_NOREPLACE(Kees)
- Block unsupported flags (Kees)
- Require size >= 8 to set token (Kees)v2:
- Change syscall to take address likemmap()for CRIU's usagev1:
- New patch (replacesPROT_SHADOW_STACK).
---arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/uapi/asm/mman.h | 3 ++ arch/x86/kernel/shstk.c | 59 ++++++++++++++++++++++---- include/linux/syscalls.h | 1 + include/uapi/asm-generic/unistd.h | 2 +- kernel/sys_ni.c | 1 + 6 files changed, 58 insertions(+), 9 deletions(-)arch/x86/entry/syscalls/syscall_64.tbl +0
-0@@ -372,6 +372,7 @@448 common
process_mreleasesys_process_mrelease
449 commonfutex_waitvsys_futex_waitv
450 commonset_mempolicy_home_nodesys_set_mempolicy_home_node
+451 64map_shadow_stacksys_map_shadow_stack#
# Due to a historical design error, certain syscalls are numbered differentlyarch/x86/include/uapi/asm/mman.h +3
-0@@ -13,6 +13,9 @@ ((key) & 0x8 ? VM_PKEY_BIT3 : 0))#endif +/* Flags for map_shadow_stack(2) */ +#define SHADOW_STACK_SET_TOKEN (1ULL << 0)/* Set up a restore token in the shadow stack */ +#include <asm-generic/mman.h> #endif /* _ASM_X86_MMAN_H */ arch/x86/kernel/shstk.c +51
-8@@ -17,6 +17,7 @@#include <linux/compat.h> #include <linux/sizes.h> #include <linux/user.h> +#include <linux/syscalls.h> #include <asm/msr.h> #include <asm/fpu/xstate.h> #include <asm/fpu/types.h> @@ -71,19 +72,31 @@ static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)return 0 ; } -static unsigned long alloc_shstk (unsigned long size) +static unsigned long alloc_shstk (unsigned long addr,unsigned long size, +unsigned long token_offset,bool set_res_tok) {int flags =MAP_ANONYMOUS |MAP_PRIVATE |MAP_ABOVE4G ;struct mm_struct *mm = current->mm; -unsigned long addr, unused; +unsigned long mapped_addr, unused; -mmap_write_lock (mm); - addr =do_mmap (NULL ,0 , size,PROT_READ , flags, -VM_SHADOW_STACK |VM_WRITE ,0 , &unused,NULL ); +if (addr) + flags |=MAP_FIXED_NOREPLACE ; +mmap_write_lock (mm); + mapped_addr =do_mmap (NULL , addr, size,PROT_READ , flags, +VM_SHADOW_STACK |VM_WRITE ,0 , &unused,NULL );mmap_write_unlock (mm); -return addr; +if (!set_res_tok ||IS_ERR_VALUE (mapped_addr)) + goto out; + +if (create_rstor_token (mapped_addr + token_offset,NULL )) { +vm_munmap (mapped_addr, size); +return -EINVAL ; + } + +out: +return mapped_addr; }static unsigned long adjust_shstk_size (unsigned long size)@@ -134,7 +147,7 @@ static int shstk_setup(void)return -EOPNOTSUPP ; size =adjust_shstk_size (0 ); - addr =alloc_shstk (size); + addr =alloc_shstk (0 , size,0 , false);if (IS_ERR_VALUE (addr))return PTR_ERR ((void *)addr);@@ -178,7 +191,7 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,return 0 ; size =adjust_shstk_size (stack_size); - addr =alloc_shstk (size); + addr =alloc_shstk (0 , size,0 , false);if (IS_ERR_VALUE (addr))return PTR_ERR ((void *)addr);@@ -371,6 +384,36 @@ static int shstk_disable(void)return 0 ; } +SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags) +{ +bool set_tok = flags &SHADOW_STACK_SET_TOKEN ; +unsigned long aligned_size; + +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return -EOPNOTSUPP ; + +if (flags & ~SHADOW_STACK_SET_TOKEN ) +return -EINVAL ; + +/* If there isn't space for a token */ +if (set_tok && size <8 ) +return -EINVAL ; + +if (addr && addr <=0xFFFFFFFF ) +return -EINVAL ; + +/* +* An overflow would result in attempting to write the restore token +* to the wrong location. Not catastrophic, but just return the right +* error code and block it. +*/ + aligned_size =PAGE_ALIGN (size); +if (aligned_size < size) +return -EOVERFLOW ; + +return alloc_shstk (addr, aligned_size, size, set_tok); +} +long shstk_prctl (struct task_struct *task,int option,unsigned long features) {if (option ==ARCH_SHSTK_LOCK ) {include/linux/syscalls.h +1
-0@@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags);asmlinkage long sys_set_mempolicy_home_node (unsigned long start,unsigned long len,unsigned long home_node,unsigned long flags); +asmlinkage long sys_map_shadow_stack (unsigned long addr,unsigned long size,unsigned int flags);/* * Architecture-specific system calls include/uapi/asm-generic/unistd.h +1
-1@@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv) __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)#undef __NR_syscalls -#define __NR_syscalls 451 +#define __NR_syscalls 452/* * 32 bit systems traditionally used different kernel/sys_ni.c +1
-0@@ -381,6 +381,7 @@ COND_SYSCALL(vm86old);COND_SYSCALL (modify_ldt);COND_SYSCALL (vm86);COND_SYSCALL (kexec_file_load); +COND_SYSCALL (map_shadow_stack);/* s390 */ COND_SYSCALL (s390_pci_mmio_read);Signature · 1 line
2.17.1
Re: [PATCH v6 33/41] x86/shstk: Introduce map_shadow_stack syscall
Quoted 82 lines · Deepak Gupta
Quoted 55 lines · Rick Edgecombe
When operating with shadow stacks enabled, the kernel will automatically
allocate shadow stacks for new threads, however in some cases userspace
will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and
pivoting to userspace managed stacks.Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup
with a restore token so that userspace can pivot to them via the RSTORSSP
instruction. But, the security design of shadow stack's is that they
should not be written to except in limited circumstances. This presents a
problem for userspace, as to how userspace can provision this special
data, without allowing for the shadow stack to be generally writable.Previously, a new
PROT_SHADOW_STACKwas attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was
found to not be secure enough, as other thread's could write to the shadow stack during the writable window.The kernel can use a special instruction, WRUSS, to write directly to
userspace shadow stacks. So the solution can be that memory can be mapped
as shadow stack permissions from the beginning (never generally writable
in userspace), and the kernel itself can write the restore token.First, a new
madvise()flag was explored, which could operate on thePROT_SHADOW_STACKmemory. This had a couple downsides:
1. Extra checks were needed inmprotect()to prevent writable memory from
ever becomingPROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the newmadvise()to prevent
restore tokens being written into the middle of pre-used shadow
stacks.
It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had
never been
written to.
3. It stood out from the rest of the madvise flags, as more of direct
action than a hint at future desired behavior.Quoted 14 lines
So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a newmap_shadow_stacksyscall to allow userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.Was following ever attempted?
void *shstk =
mmap(0, size, PROT_SHADOWSTACK, ...);
- limitPROT_SHADOWSTACKprotection flag to only mmap (and thus mprotect can't
convert memory from shadow stack to non-shadow stack type or vice
versa)
- limitPROT_SHADOWSTACKprotection flag to anonymous memory only.
- top level mmap handler to put a token at the base using WRUSS if prot ==PROT_SHADOWSTACK
Quoted 6 lines
You essentially would get shadow stack manufacturing with existing (single) syscall.
Acting a bit selfish here, this allows other architectures as well to re-use this and
do their own implementation of mapping and placing the token at the base.
Yes, I looked at it. You end up with a pile of checks and hooks added to mmap() and various other places as you outline. We also now have the MAP_ABOVE4G limitation for x86 shadow stack that would need checking for too. It's not exactly a clean fit. Then, callers would have to pass special x86 flags in anyway.
It doesn't seem like the complexity of the checks is worth saving the tiny syscall. Is there some reason why riscv can't use the same syscall stub? It doesn't need to live forever in x86 code. Not sure what the savings are for riscv of the mmap+checks approach are either...
I did wonder if there could be some sort of more general syscall for mapping and provisioning special security-type memory. But we probably need a few more non-shadow stack examples to get an idea of what that would look like.
Re: [PATCH v6 33/41] x86/shstk: Introduce map_shadow_stack syscall
Quoted 91 lines · Rick P
Quoted 82 lines · Deepak Gupta
Quoted 55 lines · Rick Edgecombe
When operating with shadow stacks enabled, the kernel will automatically
allocate shadow stacks for new threads, however in some cases userspace
will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and
pivoting to userspace managed stacks.Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup
with a restore token so that userspace can pivot to them via the RSTORSSP
instruction. But, the security design of shadow stack's is that they
should not be written to except in limited circumstances. This presents a
problem for userspace, as to how userspace can provision this special
data, without allowing for the shadow stack to be generally writable.Previously, a new
PROT_SHADOW_STACKwas attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was
found to not be secure enough, as other thread's could write to the shadow stack during the writable window.The kernel can use a special instruction, WRUSS, to write directly to
userspace shadow stacks. So the solution can be that memory can be mapped
as shadow stack permissions from the beginning (never generally writable
in userspace), and the kernel itself can write the restore token.First, a new
madvise()flag was explored, which could operate on thePROT_SHADOW_STACKmemory. This had a couple downsides:
1. Extra checks were needed inmprotect()to prevent writable memory from
ever becomingPROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the newmadvise()to prevent
restore tokens being written into the middle of pre-used shadow
stacks.
It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had
never been
written to.
3. It stood out from the rest of the madvise flags, as more of direct
action than a hint at future desired behavior.Quoted 14 lines
So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a newmap_shadow_stacksyscall to allow userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.Was following ever attempted?
void *shstk =
mmap(0, size, PROT_SHADOWSTACK, ...);
- limitPROT_SHADOWSTACKprotection flag to only mmap (and thus mprotect can't
convert memory from shadow stack to non-shadow stack type or vice
versa)
- limitPROT_SHADOWSTACKprotection flag to anonymous memory only.
- top level mmap handler to put a token at the base using WRUSS if prot ==PROT_SHADOWSTACKQuoted 6 lines
You essentially would get shadow stack manufacturing with existing (single) syscall.
Acting a bit selfish here, this allows other architectures as well to re-use this and
do their own implementation of mapping and placing the token at the base.
Quoted 5 lines
Yes, I looked at it. You end up with a pile of checks and hooks added to
mmap()and various other places as you outline. We also now have theMAP_ABOVE4Glimitation for x86 shadow stack that would need checking for too. It's not exactly a clean fit. Then, callers would have to pass special x86 flags in anyway.
riscv has mechanisms using which a 32bit app can run on 64bit kernel. So technically if there are 32bit and 64bit code in address space, MAP_ABOVE4G could be useful.
Although I am not sure (or aware of) if there are such requirement from app/developers yet (to guarantee address mapping above 4G)
But I see this as orthogonal to memory protection flags.
Quoted 5 lines
It doesn't seem like the complexity of the checks is worth saving the tiny syscall. Is there some reason why riscv can't use the same syscall stub? It doesn't need to live forever in x86 code. Not sure what the savings are for riscv of the mmap+checks approach are either...
I don't see a lot of extra complexity here.
If mprotect and friends don't know about PROT_SHADOWSTACK, they'll just fail by default (which is desired)
It's only mmap that needs to be enlightened. And it can just pass VMA_SHADOW_STACK to do_mmap if input is PROT_SHADOWSTACK.
Adding a syscall just for mapping shadow stack is weird when it can be solved with existing system calls.
As you say in your response below, it would be good to have such a syscall which serve larger purposes (e.g. provisioning special security-type memory)
arm64's memory tagging is one such example. Not exactly security-type memory (but eventual application is security for this feature) . It adds extra meaning to virtual addresses (i.e. an address has tags). arm64 went about using a protection flag PROT_MTE instead of a special system call.
Being said that since this patch has gone through multiple revisions and I am new to the party. If others dont have issues on this special system call,
I think it's fine then. In case of riscv I can choose to use this mechanism or go via arm's route to define PROT_SHADOWSTACK which is arch specific.
Quoted 5 lines
I did wonder if there could be some sort of more general syscall for mapping and provisioning special security-type memory. But we probably need a few more non-shadow stack examples to get an idea of what that would look like.
As I mentioned memory tagging and thus PROT_MTE is already such a use case which uses mmap/mprotect protection flags to designate special meaning to a virtual address.
Re: [PATCH v6 33/41] x86/shstk: Introduce map_shadow_stack syscall
Quoted 8 lines · Deepak Gupta
Quoted 7 lines
It doesn't seem like the complexity of the checks is worth saving the
tiny syscall. Is there some reason why riscv can't use the same syscall
stub? It doesn't need to live forever in x86 code. Not sure what the
savings are for riscv of the mmap+checks approach are either...
Quoted 4 lines
I don't see a lot of extra complexity here.
Ifmprotectand friends don't know aboutPROT_SHADOWSTACK, they'll
just fail by default (which is desired)
To me, the options look like: cram it into an existing syscall or create one that does exactly what is needed.
To replace the new syscall with mmap(), with the existing MM implementation, I think you would need to add to mmap:
1. Limit PROT_SHADOW_STACK to anonymous, private memory.
2. Limit PROT_SHADOW_STACK to MAP_ABOVE4G (or create a MAP_SHADOW_STACK that does both). MAP_ABOVE4G protects from using shadow stack in 32 bit mode, after some ABI issues were raised. So it is supposed to be enforced.
3. Add additional logic for MAP_ABOVE4G to work with MAP_FIXED as is required by CRIU.
4. Add another MAP_ flag to specify whether to write the token (AFAIK the first flag that would do something like that. So then likely have debates about whether it fits into the flags). Sort out the behavior of trying to write the token to a non-PROT_SHADOW_STACK mmap call.
5. Add arch breakout for mmap to call into the token writing.
I think you are right that no mprotect() changes would be needed with the current shadow stack MM implementation (it wasn't the case for the original PROT_SHADOW_STACK). But I'm still not sure if it is simpler then the syscall.
I do wonder a little bit about trying to remove some of these limitations of the existing shadow stack MM implementation, which could make an mmap based interface a little better fit. Like if someday shadow stack memory added support for all the options that mmap supports. But I'm not sure if that would just result in more complexity in other places (MM, signals) that would barely get used. Like, I'm not sure how shadow stack permissioned mmap()ed files should work. You would have to require writable files to map them as shadow stack, but then that is not as locked down as we have today with the anonymous memory. And a "shadow stack" file permission would seem a bit overboard.
Then probably some more dirty bit issues with mmaped files. I'm not fully sure. That one was definitely an issue when PROT_SHADOW_STACK was dropped, but David Hildenbrand has now removed at least some of the issues it hit.
So the optimum solution might depend on if we add more shadow stack MM support later. But it is always possible to add mmap support later too.
Quoted 11 lines
It's only
mmapthat needs to be enlightened. And it can just passVMA_SHADOW_STACKtodo_mmapif input isPROT_SHADOWSTACK.Adding a syscall just for mapping shadow stack is weird when it can be
solved with existing system calls.
As you say in your response below, it would be good to have such a syscall which serve larger purposes (e.g. provisioning special security-type memory)
Quoted 6 lines
arm64's memory tagging is one such example. Not exactly security-type memory (but eventual application is security for this feature) . It adds extra meaning to virtual addresses (i.e. an address has tags).
arm64 went about using a protection flagPROT_MTEinstead of a special system call.
It looks like that memory can be written with a special instruction? And so it doesn't need to be provisioned by the kernel, as is the case here.
Quoted 7 lines
Being said that since this patch has gone through multiple revisions and I am new to the party. If others dont have issues on this special system call,
I think it's fine then. In case of riscv I can choose to use this mechanism or go via arm's route to definePROT_SHADOWSTACKwhich is arch specific.
Ok, sounds good. The advice I got from maintainers after the many attempts to cram it into the existing interfaces was: don't be afraid to add a syscall. And sure enough, when MAP_ABOVE4G came along it continued to make things simpler.
PATCH v6 34/41 x86/shstk: Support WRSS for userspace
In reply to Rick EdgecombeFor the current shadow stack implementation, shadow stacks contents can't easily be provisioned with arbitrary data. This property helps apps protect themselves better, but also restricts any potential apps that may want to do exotic things at the expense of a little security.
The x86 shadow stack feature introduces a new instruction, WRSS, which can be enabled to write directly to shadow stack permissioned memory from userspace. Allow it to get enabled via the prctl interface.
Only enable the userspace WRSS instruction, which allows writes to userspace shadow stacks from userspace. Do not allow it to be enabled independently of shadow stack, as HW does not support using WRSS when shadow stack is disabled.
From a fault handler perspective, WRSS will behave very similar to WRUSS, which is treated like a user access from a #PF err code perspective.
---
v6:
- Make set_clr_bits_msrl() avoid side affects in 'msr'
v5:
- Switch to EOPNOTSUPP
- Move set_clr_bits_msrl() to patch where it is first used
- Commit log formatting
v3:
- Make wrss_control() static
- Fix verbiage in commit log (Kees)
v2:
- Add some commit log verbiage from (Dave Hansen)
v1:
- New patch.
--- arch/x86/include/asm/msr.h | 11 +++++++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/shstk.c | 32 ++++++++++++++++++++++++++++++- 3 files changed, 43 insertions(+), 1 deletion(-)
arch/x86/include/asm/msr.h +11 -0
@@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs);int msr_set_bit (u32 msr,u8 bit);int msr_clear_bit (u32 msr,u8 bit); +/* Helper that can never get accidentally un-inlined. */ +#define set_clr_bits_msrl (msr, set, clear) do { \ + u64 __val, __new_val, __msr = msr; \ + \ + rdmsrl(__msr, __val); \ + __new_val = (__val & ~(clear)) | (set); \ + \ + if (__new_val != __val) \ + wrmsrl(__msr, __new_val); \ +} while (0) +#ifdef CONFIG_SMP int rdmsr_on_cpu (unsigned int cpu,u32 msr_no,u32 *l,u32 *h);int wrmsr_on_cpu (unsigned int cpu,u32 msr_no,u32 l,u32 h);
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -28,5 +28,6 @@/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1)#endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/shstk.c +31 -1
@@ -363,6 +363,36 @@ void shstk_free(struct task_struct *tsk)unmap_shadow_stack (shstk->base, shstk->size); } +static int wrss_control (bool enable) +{ +if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK )) +return -EOPNOTSUPP ; + +/* +* Only enable wrss if shadow stack is enabled. If shadow stack is not +* enabled, wrss will already be disabled, so don't bother clearing it +* when disabling. +*/ +if (!features_enabled (ARCH_SHSTK_SHSTK )) +return -EPERM ; + +/* Already enabled/disabled? */ +if (features_enabled (ARCH_SHSTK_WRSS ) == enable) +return 0 ; + +fpregs_lock_and_load (); +if (enable) { +set_clr_bits_msrl (MSR_IA32_U_CET ,CET_WRSS_EN ,0 ); +features_set (ARCH_SHSTK_WRSS ); + }else { +set_clr_bits_msrl (MSR_IA32_U_CET ,0 ,CET_WRSS_EN ); +features_clr (ARCH_SHSTK_WRSS ); + } +fpregs_unlock (); + +return 0 ; +} +static int shstk_disable (void ) {if (!cpu_feature_enabled (X86_FEATURE_USER_SHSTK ))
@@ -379,7 +409,7 @@ static int shstk_disable(void)fpregs_unlock ();shstk_free (current); -features_clr (ARCH_SHSTK_SHSTK ); +features_clr (ARCH_SHSTK_SHSTK |ARCH_SHSTK_WRSS );return 0 ; }
Signature · 1 line
2.17.1
PATCH v6 35/41 x86: Expose thread features in /proc/$PID/status
In reply to Rick EdgecombeApplications and loaders can have logic to decide whether to enable shadow stack. They usually don't report whether shadow stack has been enabled or not, so there is no way to verify whether an application actually is protected by shadow stack.
Add two lines in /proc/$PID/status to report enabled and locked features.
Since, this involves referring to arch specific defines in asm/prctl.h, implement an arch breakout to emit the feature lines.
---
v4:
- Remove "CET" references
v3:
- Move to /proc/pid/status (Kees)
v2:
- New patch
--- arch/x86/kernel/cpu/proc.c | 23 +++++++++++++++++++++++ fs/proc/array.c | 6 ++++++ include/linux/proc_fs.h | 2 ++ 3 files changed, 31 insertions(+)
arch/x86/kernel/cpu/proc.c +23 -0
@@ -4,6 +4,8 @@#include <linux/string.h> #include <linux/seq_file.h> #include <linux/cpufreq.h> +#include <asm/prctl.h> +#include <linux/proc_fs.h> #include "cpu.h"
@@ -175,3 +177,24 @@ const struct seq_operations cpuinfo_op = { .stop = c_stop, .show = show_cpuinfo, }; + +#ifdef CONFIG_X86_USER_SHADOW_STACK +static void dump_x86_features (struct seq_file *m,unsigned long features) +{ +if (features &ARCH_SHSTK_SHSTK ) +seq_puts (m,"shstk " ); +if (features &ARCH_SHSTK_WRSS ) +seq_puts (m,"wrss " ); +} + +void arch_proc_pid_thread_features (struct seq_file *m,struct task_struct *task) +{ +seq_puts (m,"x86_Thread_features:\t" ); +dump_x86_features (m, task->thread.features); +seq_putc (m,'\n' ); + +seq_puts (m,"x86_Thread_features_locked:\t" ); +dump_x86_features (m, task->thread.features_locked); +seq_putc (m,'\n' ); +} +#endif /* CONFIG_X86_USER_SHADOW_STACK */
fs/proc/array.c +6 -0
@@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm)seq_printf (m,"THP_enabled:\t%d\n" , thp_enabled); } +__weak void arch_proc_pid_thread_features (struct seq_file *m, +struct task_struct *task) +{ +} +int proc_pid_status (struct seq_file *m,struct pid_namespace *ns,struct pid *pid,struct task_struct *task) {
@@ -451,6 +456,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns,task_cpus_allowed (m, task);cpuset_task_status_allowed (m, task);task_context_switch_counts (m, task); +arch_proc_pid_thread_features (m, task);return 0 ; }
include/linux/proc_fs.h +2 -0
@@ -158,6 +158,8 @@ int proc_pid_arch_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task);#endif /* CONFIG_PROC_PID_ARCH_STATUS */ +void arch_proc_pid_thread_features (struct seq_file *m,struct task_struct *task); +#else /* CONFIG_PROC_FS */ static inline void proc_root_init (void )
Signature · 1 line
2.17.1
PATCH v6 36/41 x86/shstk: Wire in shadow stack interface
In reply to Rick EdgecombeThe kernel now has the main shadow stack functionality to support applications. Wire in the WRSS and shadow stack enable/disable functions into the existing shadow stack API skeleton.
---
v4:
- Remove "CET" references
v2:
- Split from other patches
--- arch/x86/kernel/shstk.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/shstk.c +8 -0
@@ -464,9 +464,17 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features)return -EINVAL ;if (option ==ARCH_SHSTK_DISABLE ) { +if (features &ARCH_SHSTK_WRSS ) +return wrss_control (false); +if (features &ARCH_SHSTK_SHSTK ) +return shstk_disable ();return -EINVAL ; }/* Handle ARCH_SHSTK_ENABLE */ +if (features &ARCH_SHSTK_SHSTK ) +return shstk_setup (); +if (features &ARCH_SHSTK_WRSS ) +return wrss_control (true);return -EINVAL ; }
Signature · 1 line
2.17.1
PATCH v6 37/41 selftests/x86: Add shadow stack test
In reply to Rick EdgecombeAdd a simple selftest for exercising some shadow stack behavior:
- map_shadow_stack syscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
- mprotect() of shadow stack memory
- Userfaultfd on shadow stack memory
Since this test exercises a recently added syscall manually, it needs to find the automatically created __NR_foo defines. Per the selftest documentation, KHDR_INCLUDES can be used to help the selftest Makefile's find the headers from the kernel source. This way the new selftest can be built inside the kernel source tree without installing the headers to the system. So also add KHDR_INCLUDES as described in the selftest docs, to facilitate this.
---
v6:
- Tweak mprotect test
- Code style tweaks
v5:
- Update 32 bit signal test with new ABI and better asm
v4:
- Add test for 32 bit signal ABI blocking
v3:
- Change "+m" to "=m" in write_shstk() (Andrew Cooper)
- Fix userfaultfd test with transparent huge pages by doing a MADV_DONTNEED, since the token write faults in the while stack with huge pages.
--- tools/testing/selftests/x86/Makefile | 4 +- .../testing/selftests/x86/test_shadow_stack.c | 676 ++++++++++++++++++ 2 files changed, 678 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
tools/testing/selftests/x86/Makefile +2 -2
@@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx test_shadow_stack # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
@@ -34,7 +34,7 @@ BINARIES_64 := $(TARGETS_C_64BIT_ALL:%=%_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) -CFLAGS := -O2 -g -std=gnu99 -pthread -Wall +CFLAGS := -O2 -g -std=gnu99 -pthread -Wall $(KHDR_INCLUDES) # call32_from_64 in thunks.S uses absolute addresses. ifeq ($(CAN_BUILD_WITH_NOPIE),1)
tools/testing/selftests/x86/test_shadow_stack.c new file +676 -0
676 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v6 37/41] selftests/x86: Add shadow stack test
Quoted 16 lines · Rick Edgecombe
Add a simple selftest for exercising some shadow stack behavior:
-map_shadow_stacksyscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
-mprotect()of shadow stack memory
- Userfaultfd on shadow stack memorySince this test exercises a recently added syscall manually, it needs to find the automatically created
__NR_foodefines. Per the selftest documentation,KHDR_INCLUDEScan be used to help the selftest Makefile's find the headers from the kernel source. This way the new selftest can be built inside the kernel source tree without installing the headers to the system. So also addKHDR_INCLUDESas described in the selftest docs, to facilitate this.
Quoted 5 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
I'll get some test hardware and run this myself too, but overall, ignoring the lack of kselftest_harness.h, it looks good:
Signature · 1 line
Kees Cook
Re: [PATCH v6 37/41] selftests/x86: Add shadow stack test
In reply to Rick EdgecombeQuoted 23 lines · Rick Edgecombe
Add a simple selftest for exercising some shadow stack behavior:
-map_shadow_stacksyscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
-mprotect()of shadow stack memory
- Userfaultfd on shadow stack memorySince this test exercises a recently added syscall manually, it needs to find the automatically created
__NR_foodefines. Per the selftest documentation,KHDR_INCLUDEScan be used to help the selftest Makefile's find the headers from the kernel source. This way the new selftest can be built inside the kernel source tree without installing the headers to the system. So also addKHDR_INCLUDESas described in the selftest docs, to facilitate this.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
[...]
tools/testing/selftests/x86/test_shadow_stack.c line 274
Quoted 7 lines
tools/testing/selftests/x86/test_shadow_stack.c new file +7
-0+bool gup_write (void *ptr) +{ +unsigned long val; + +lseek (fd, (unsigned long )ptr,SEEK_SET ); +if (write (fd, &val,sizeof (val)) <0 ) +return 1 ;
/proc/self/mem is for debug/ptrace access (FOLL_FORCE). I think you might also want to add tests for ordinary GUP, checking that we fail to obtain a write pin -- and call these tests "gup_ptrace_read" / "gup_ptrace_write"
An simple approach would be to trigger a read()/write() on a file opened via O_DIRECT, using the shadow stack as buffer. While the write() [reading from the page] is expected to work, a read() [writing to the page] has to fail.
Signature · 3 lines
Thanks, David / dhildenb
Re: [PATCH v6 37/41] selftests/x86: Add shadow stack test
Quoted 34 lines · David Hildenbrand
Quoted 29 lines · Rick Edgecombe
Add a simple selftest for exercising some shadow stack behavior:
-map_shadow_stacksyscall and pivot
- Faulting in shadow stack memory
- Handling shadow stack violations
- GUP of shadow stack memory
-mprotect()of shadow stack memory
- Userfaultfd on shadow stack memorySince this test exercises a recently added syscall manually, it needs
to find the automatically created__NR_foodefines. Per the selftest
documentation,KHDR_INCLUDEScan be used to help the selftest Makefile's
find the headers from the kernel source. This way the new selftest can
be built inside the kernel source tree without installing the headers
to the system. So also addKHDR_INCLUDESas described in the selftest
docs, to facilitate this.Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Co-developed-by Yu-cheng Yu <[email protected]>Signed-off-by Yu-cheng Yu <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
[...]
Quoted 13 lines
tools/testing/selftests/x86/test_shadow_stack.c line 274
Quoted 7 lines
tools/testing/selftests/x86/test_shadow_stack.c new file +7
-0+bool gup_write (void *ptr) +{ +unsigned long val; + +lseek (fd, (unsigned long )ptr,SEEK_SET ); +if (write (fd, &val,sizeof (val)) <0 ) +return 1 ;
/proc/self/memis for debug/ptrace access (FOLL_FORCE). I think you might also want to add tests for ordinary GUP, checking that we fail to
obtain a write pin -- and call these tests "gup_ptrace_read" / "gup_ptrace_write"
Yes, this only tests the FOLL_FORCE case, but it does exercise GUP.
Quoted 7 lines
An simple approach would be to trigger a read()/write() on a file opened
viaO_DIRECT, using the shadow stack as buffer. While thewrite()[reading from the page] is expected to work, aread()[writing to the
page] has to fail.
Hmm, good idea. This would be nice to add.
Borislav Petkov
maintainerRe: [PATCH v6 37/41] selftests/x86: Add shadow stack test
In reply to Rick EdgecombeQuoted 3 lines · Rick Edgecombe
Since this test exercises a recently added syscall manually, it needs to find the automatically created
__NR_foodefines. Per the selftest documentation,KHDR_INCLUDEScan be used to help the selftest Makefile's
Well, why don't you make it easier for the user of this to not have to jump through hoops to get the test built?
IOW, something like the below ontop.
It works if I do
$ make -j<num> test_shadow_stack_64
It would only need to be fixed to work when you do
$ make -j<num>
without arguments as then make does a parallel build.
I guess something like
ifneq ($(filter test_shadow_stack_64, $(MAKECMDGOALS)),) .NOTPARALLEL:
endif
needs to happen but I'm not sure...
Thx.
tools/testing/selftests/x86/Makefile +6 -2
@@ -14,6 +14,10 @@ top_srcdir ?= ../../.. abs_srctree := $(shell cd $(top_srcdir) && pwd) KHDR_INCLUDES := -isystem ${abs_srctree}/usr/include +test_shadow_stack_64: test_shadow_stack.c helpers.h + cd $(top_srcdir) && $(MAKE) headers + $(CC) -m64 -o $@ $(CFLAGS) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) $^ -lrt -ldl + TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \ check_initial_reg_state sigreturn iopl ioperm \ test_vsyscall mov_ss_trap \
@@ -22,7 +26,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx test_shadow_stack + corrupt_xstate_header amx # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall
@@ -38,7 +42,7 @@ BINARIES_64 := $(TARGETS_C_64BIT_ALL:%=%_64) BINARIES_32 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_32)) BINARIES_64 := $(patsubst %,$(OUTPUT)/%,$(BINARIES_64)) -CFLAGS := -O2 -g -std=gnu99 -pthread -Wall $(KHDR_INCLUDES) +CFLAGS := -O2 -g -std=gnu99 -pthread -Wall # call32_from_64 in thunks.S uses absolute addresses. ifeq ($(CAN_BUILD_WITH_NOPIE),1)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v6 37/41] selftests/x86: Add shadow stack test
Quoted 24 lines · Borislav Petkov
Quoted 6 lines · Rick Edgecombe
Since this test exercises a recently added syscall manually, it needs
to find the automatically created__NR_foodefines. Per the selftest
documentation,KHDR_INCLUDEScan be used to help the selftest Makefile'sWell, why don't you make it easier for the user of this to not have to
jump through hoops to get the test built?IOW, something like the below ontop.
It works if I do
$ make -j<num> test_shadow_stack_64It would only need to be fixed to work when you do
$ make -j<num>without arguments as then make does a parallel build.
Quoted 7 lines
I guess something like
ifneq ($(filter
test_shadow_stack_64, $(MAKECMDGOALS)),) .NOTPARALLEL:
endifneeds to happen but I'm not sure...
Ah, I see. I had built the kernel with CONFIG_HEADERS_INSTALL and so this was already done for me.
The proposed Makefile solution seems a bit unusual. What about this less complicated solution to just make this case work?
(quoted diff) +0 -0
b/tools/testing/selftests/x86/test_shadow_stack.c
index 71de3527c67a..02fe1b135ba8 100644
tools/testing/selftests/x86/test_shadow_stack.c +18 -0
@@ -34,6 +34,24 @@#define SS_SIZE 0x200000 +/* +* Define the ABI defines if needed, so people can run the tests +* without building the headers. +*/ +#ifndef __NR_map_shadow_stack +#define __NR_map_shadow_stack 451 +#define SHADOW_STACK_SET_TOKEN (1ULL << 0) + +#define ARCH_SHSTK_ENABLE 0x5001 +#define ARCH_SHSTK_DISABLE 0x5002 +#define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005 + +#define ARCH_SHSTK_SHSTK (1ULL << 0) +#define ARCH_SHSTK_WRSS (1ULL << 1) +#endif +#if (__GNUC__ <8 ) || (__GNUC__ ==8 && __GNUC_MINOR__ <5 )int main (int argc,char *argv[]) {
I was a bit surprised this was even as tricky as the KHDR_INCLUDES part. The other tests seem to be fine only because they use old header definitions that have been there forever. So it seems like the proper way to build the selftests should involve building the headers. So alternatively, why not just always encourage building the headers before running the selftests by warning if ${abs_srctree}/usr/include/linux is not found?
Do either of those seem any better?
Borislav Petkov
maintainerRe: [PATCH v6 37/41] selftests/x86: Add shadow stack test
Quoted 2 lines · Rick P
The proposed
Makefilesolution seems a bit unusual. What about this less complicated solution to just make this case work?
I like simple. :)
Quoted 3 lines
So alternatively, why not just always encourage building the headers before running the selftests by warning if ${abs_srctree}/usr/include/linux is not found?
s/encourage/automate/
Imagine this situation: maintainer says: "please run the selftests". User says, "uh oh, it fails building, I need to figure that out first."
So we should not have users have to figure out stuff if we can code it to happen automatically for the default case.
If they want something special, then they can do all the figuring out they want. :-)
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v6 37/41] selftests/x86: Add shadow stack test
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Rick P
The proposed
Makefilesolution seems a bit unusual. What about this less complicated solution to just make this case work?I like simple. :)
Done!
Quoted 12 lines
Quoted 5 lines
So alternatively, why not just always encourage building the headers
before running the selftests by warning if ${abs_srctree}/usr/include/linux is not found?s/encourage/automate/
Imagine this situation: maintainer says: "please run the selftests". User says, "uh oh, it fails building, I need to figure that out first."
Quoted 6 lines
So we should not have users have to figure out stuff if we can code it
to happen automatically for the default case.If they want something special, then they can do all the figuring out they want. :-)
I guess you also could run into the problem of using old headers. Like say you checkout kernel A, build headers, then check out kernel B and build selftests. You get surprised by using the old headers from A.
Today the selftests don't seem to depend on having a .config or anything. But autobuilding does, so its kind of change to how tied in the selftests are. But maybe it's how it should be for all of them. Hmm.
PATCH v6 38/41 x86/fpu: Add helper for initing features
In reply to Rick EdgecombeIf an xfeature is saved in a buffer, the xfeature's bit will be set in xsave->header.xfeatures. The CPU may opt to not save the xfeature if it is in it's init state. In this case the xfeature buffer address cannot be retrieved with get_xsave_addr().
Future patches will need to handle the case of writing to an xfeature that may not be saved. So provide helpers to init an xfeature in an xsave buffer.
This could of course be done directly by reaching into the xsave buffer, however this would not be robust against future changes to optimize the xsave buffer by compacting it. In that case the xsave buffer would need to be re-arranged as well. So the logic properly belongs encapsulated in a helper where the logic can be unified.
---
v2:
- New patch
--- arch/x86/kernel/fpu/xstate.c | 58 +++++++++++++++++++++++++++++------- arch/x86/kernel/fpu/xstate.h | 6 ++++ 2 files changed, 53 insertions(+), 11 deletions(-)
arch/x86/kernel/fpu/xstate.c +47 -11
@@ -934,6 +934,24 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr)return (void *)xsave +xfeature_get_offset (xcomp_bv, xfeature_nr); } +static int xsave_buffer_access_checks (int xfeature_nr) +{ +/* +* Do we even *have* xsave state? +*/ +if (!boot_cpu_has (X86_FEATURE_XSAVE )) +return 1 ; + +/* +* We should not ever be requesting features that we +* have not enabled. +*/ +if (WARN_ON_ONCE (!xfeature_enabled (xfeature_nr))) +return 1 ; + +return 0 ; +} +/* * Given the xsave area and a state inside, this function returns the * address of the state.
@@ -954,17 +972,7 @@ static void *__raw_xsave_addr(struct xregs_state *xsave, int xfeature_nr)*/ void *get_xsave_addr (struct xregs_state *xsave,int xfeature_nr) { -/* -* Do we even *have* xsave state? -*/ -if (!boot_cpu_has (X86_FEATURE_XSAVE )) -return NULL ; - -/* -* We should not ever be requesting features that we -* have not enabled. -*/ -if (WARN_ON_ONCE (!xfeature_enabled (xfeature_nr))) +if (xsave_buffer_access_checks (xfeature_nr))return NULL ;/*
@@ -984,6 +992,34 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr)return __raw_xsave_addr (xsave, xfeature_nr); } +/* +* Given the xsave area and a state inside, this function +* initializes an xfeature in the buffer. +* +* get_xsave_addr() will return NULL if the feature bit is +* not present in the header. This function will make it so +* the xfeature buffer address is ready to be retrieved by +* get_xsave_addr(). +* +* Inputs: +* xstate: the thread's storage area for all FPU data +* xfeature_nr: state which is defined in xsave.h (e.g. XFEATURE_FP, +* XFEATURE_SSE, etc...) +* Output: +* 1 if the feature cannot be inited, 0 on success +*/ +int init_xfeature (struct xregs_state *xsave,int xfeature_nr) +{ +if (xsave_buffer_access_checks (xfeature_nr)) +return 1 ; + +/* +* Mark the feature inited. +*/ + xsave->header.xfeatures |=BIT_ULL (xfeature_nr); +return 0 ; +} +#ifdef CONFIG_ARCH_HAS_PKEYS /*
arch/x86/kernel/fpu/xstate.h +6 -0
@@ -54,6 +54,12 @@ extern void fpu__init_cpu_xstate(void);extern void fpu__init_system_xstate (unsigned int legacy_size);extern void *get_xsave_addr (struct xregs_state *xsave,int xfeature_nr); +extern int init_xfeature (struct xregs_state *xsave,int xfeature_nr); + +static inline int xfeature_saved (struct xregs_state *xsave,int xfeature_nr) +{ +return xsave->header.xfeatures &BIT_ULL (xfeature_nr); +}static inline u64 xfeatures_mask_supervisor (void ) {
Signature · 1 line
2.17.1
Re: [PATCH v6 38/41] x86/fpu: Add helper for initing features
Quoted 15 lines · Rick Edgecombe
If an xfeature is saved in a buffer, the xfeature's bit will be set in
xsave->header.xfeatures. The CPU may opt to not save the xfeature if it is in it's init state. In this case the xfeature buffer address cannot be retrieved withget_xsave_addr().Future patches will need to handle the case of writing to an xfeature that may not be saved. So provide helpers to init an xfeature in an xsave buffer.
This could of course be done directly by reaching into the xsave buffer, however this would not be robust against future changes to optimize the xsave buffer by compacting it. In that case the xsave buffer would need to be re-arranged as well. So the logic properly belongs encapsulated in a helper where the logic can be unified.
Quoted 3 lines
Tested-by Pengfei Xu <[email protected]>Tested-by John Allen <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>
Signature · 1 line
Kees Cook
PATCH v6 39/41 x86: Add PTRACE interface for shadow stack
In reply to Rick EdgecombeFrom: Yu-cheng Yu <[email protected]>
Some applications (like GDB) would like to tweak shadow stack state via ptrace. This allows for existing functionality to continue to work for seized shadow stack applications. Provide an regset interface for manipulating the shadow stack pointer (SSP).
There is already ptrace functionality for accessing xstate, but this does not include supervisor xfeatures. So there is not a completely clear place for where to put the shadow stack state. Adding it to the user xfeatures regset would complicate that code, as it currently shares logic with signals which should not have supervisor features.
Don't add a general supervisor xfeature regset like the user one, because it is better to maintain flexibility for other supervisor xfeatures to define their own interface. For example, an xfeature may decide not to expose all of it's state to userspace, as is actually the case for shadow stack ptrace functionality. A lot of enum values remain to be used, so just put it in dedicated shadow stack regset.
The only downside to not having a generic supervisor xfeature regset, is that apps need to be enlightened of any new supervisor xfeature exposed this way (i.e. they can't try to have generic save/restore logic). But maybe that is a good thing, because they have to think through each new xfeature instead of encountering issues when new a new supervisor xfeature was added.
By adding a shadow stack regset, it also has the effect of including the shadow stack state in a core dump, which could be useful for debugging.
The shadow stack specific xstate includes the SSP, and the shadow stack and WRSS enablement status. Enabling shadow stack or wrss in the kernel involves more than just flipping the bit. The kernel is made aware that it has to do extra things when cloning or handling signals. That logic is triggered off of separate feature enablement state kept in the task struct. So the flipping on HW shadow stack enforcement without notifying the kernel to change its behavior would severely limit what an application could do without crashing, and the results would depend on kernel internal implementation details. There is also no known use for controlling this state via prtace today. So only expose the SSP, which is something that userspace already has indirect control over.
---
v5:
- Check shadow stack enablement status for tracee (rppt)
- Fix typo in comment
v4:
- Make shadow stack only. Reduce to only supporting SSP register, and remove CET references (peterz)
- Add comment to not use 0x203, because binutils already looks for it in coredumps. (Christina Schimpe)
v3:
- Drop dependence on thread.shstk.size, and use thread.features bits
- Drop 32 bit support
v2:
- Check alignment on ssp.
- Block IBT bits.
- Handle init states instead of returning error.
- Add verbose commit log justifying the design.
--- arch/x86/include/asm/fpu/regset.h | 7 +-- arch/x86/kernel/fpu/regset.c | 86 +++++++++++++++++++++++++++++++ arch/x86/kernel/ptrace.c | 12 +++++ include/uapi/linux/elf.h | 2 + 4 files changed, 104 insertions(+), 3 deletions(-)
arch/x86/include/asm/fpu/regset.h +4 -3
@@ -7,11 +7,12 @@#include <linux/regset.h> -extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active; +extern user_regset_active_fn regset_fpregs_active, regset_xregset_fpregs_active, + ssp_active;extern user_regset_get2_fn fpregs_get, xfpregs_get, fpregs_soft_get, - xstateregs_get; + xstateregs_get, ssp_get;extern user_regset_set_fn fpregs_set, xfpregs_set, fpregs_soft_set, - xstateregs_set; + xstateregs_set, ssp_set;/* * xstateregs_active == regset_fpregs_active. Please refer to the comment
arch/x86/kernel/fpu/regset.c +86 -0
98 lines not on this page · raw message
arch/x86/kernel/ptrace.c +12 -0
@@ -58,6 +58,7 @@ enum x86_regset_64 {REGSET64_FP ,REGSET64_IOPERM ,REGSET64_XSTATE , + REGSET64_SSP, };#define REGSET_GENERAL \
@@ -1267,6 +1268,17 @@ static struct user_regset x86_64_regsets[] __ro_after_init = { .active = ioperm_active, .regset_get = ioperm_get }, +#ifdef CONFIG_X86_USER_SHADOW_STACK + [REGSET64_SSP] = { + .core_note_type = NT_X86_SHSTK, + .n = 1, + .size = sizeof(u64), + .align = sizeof(u64), + .active = ssp_active, + .regset_get = ssp_get, + .set = ssp_set + }, +#endif }; static const struct user_regset_view user_x86_64_view = {
include/uapi/linux/elf.h +2 -0
@@ -406,6 +406,8 @@ typedef struct elf64_shdr {#define NT_386_TLS 0x200/* i386 TLS slots (struct user_desc) */ #define NT_386_IOPERM 0x201/* x86 io permission bitmap (1=deny) */ #define NT_X86_XSTATE 0x202/* x86 extended state using xsave */ +/* Old binutils treats 0x203 as a CET state */ +#define NT_X86_SHSTK 0x204/* x86 SHSTK state */ #define NT_S390_HIGH_GPRS 0x300/* s390 upper register halves */ #define NT_S390_TIMER 0x301/* s390 timer register */ #define NT_S390_TODCMP 0x302/* s390 TOD clock comparator register */
Signature · 1 line
2.17.1
PATCH v6 40/41 x86/shstk: Add ARCH_SHSTK_UNLOCK
In reply to Rick EdgecombeFrom: Mike Rapoport <[email protected]>
Userspace loaders may lock features before a CRIU restore operation has the chance to set them to whatever state is required by the process being restored. Allow a way for CRIU to unlock features. Add it as an arch_prctl() like the other shadow stack operations, but restrict it being called by the ptrace arch_pctl() interface.
---
v4:
- Add to docs that it is ptrace only.
- Remove "CET" references
v3:
- Depend on CONFIG_CHECKPOINT_RESTORE (Kees)
--- Documentation/x86/shstk.rst | 4 ++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 9 +++++++-- 4 files changed, 13 insertions(+), 2 deletions(-)
Documentation/x86/shstk.rst +4 -0
@@ -73,6 +73,10 @@ arch_prctl(ARCH_SHSTK_LOCK, unsigned long features) are ignored. The mask is ORed with the existing value. So any feature bits set here cannot be enabled or disabled afterwards. +arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) + Unlock features. 'features' is a mask of all features to unlock. All + bits set are processed, unset bits are ignored. Only works via ptrace. + The return values are as follows. On success, return 0. On error, errno can be::
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -25,6 +25,7 @@#define ARCH_SHSTK_ENABLE 0x5001#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003 +#define ARCH_SHSTK_UNLOCK 0x5004/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0)
arch/x86/kernel/process_64.c +1 -0
@@ -835,6 +835,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_SHSTK_ENABLE :case ARCH_SHSTK_DISABLE :case ARCH_SHSTK_LOCK : +case ARCH_SHSTK_UNLOCK :return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;
arch/x86/kernel/shstk.c +7 -2
@@ -451,9 +451,14 @@ long shstk_prctl(struct task_struct *task, int option, unsigned long features)return 0 ; } -/* Don't allow via ptrace */ -if (task != current) +/* Only allow via ptrace */ +if (task != current) { +if (option ==ARCH_SHSTK_UNLOCK &&IS_ENABLED (CONFIG_CHECKPOINT_RESTORE )) { + task->thread.features_locked &= ~features; +return 0 ; + }return -EINVAL ; + }/* Do not allow to change locked features */ if (features & task->thread.features_locked)
Signature · 1 line
2.17.1
PATCH v6 41/41 x86/shstk: Add ARCH_SHSTK_STATUS
In reply to Rick EdgecombeCRIU and GDB need to get the current shadow stack and WRSS enablement status. This information is already available via /proc/pid/status, but this is inconvenient for CRIU because it involves parsing the text output in an area of the code where this is difficult. Provide a status arch_prctl(), ARCH_SHSTK_STATUS for retrieving the status. Have arg2 be a userspace address, and make the new arch_prctl simply copy the features out to userspace.
---
v5:
- Fix typo in commit log
v4:
- New patch
--- Documentation/x86/shstk.rst | 6 ++++++ arch/x86/include/asm/shstk.h | 2 +- arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 1 + arch/x86/kernel/shstk.c | 8 +++++++- 5 files changed, 16 insertions(+), 2 deletions(-)
Documentation/x86/shstk.rst +6 -0
@@ -77,6 +77,11 @@ arch_prctl(ARCH_SHSTK_UNLOCK, unsigned long features) Unlock features. 'features' is a mask of all features to unlock. All bits set are processed, unset bits are ignored. Only works via ptrace. +arch_prctl(ARCH_SHSTK_STATUS, unsigned long addr) + Copy the currently enabled features to the address passed in addr. The + features are described using the bits passed into the others in + 'features'. + The return values are as follows. On success, return 0. On error, errno can be::
@@ -84,6 +89,7 @@ be:: -ENOTSUPP if the feature is not supported by the hardware or kernel. -EINVAL arguments (non existing feature, etc) + -EFAULT if could not copy information back to userspace The feature's bits supported are::
arch/x86/include/asm/shstk.h +1 -1
@@ -14,7 +14,7 @@ struct thread_shstk {u64 size; }; -long shstk_prctl (struct task_struct *task,int option,unsigned long features); +long shstk_prctl (struct task_struct *task,int option,unsigned long arg2);void reset_thread_features (void ); int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags, unsigned long stack_size,
arch/x86/include/uapi/asm/prctl.h +1 -0
@@ -26,6 +26,7 @@#define ARCH_SHSTK_DISABLE 0x5002#define ARCH_SHSTK_LOCK 0x5003#define ARCH_SHSTK_UNLOCK 0x5004 +#define ARCH_SHSTK_STATUS 0x5005/* ARCH_SHSTK_ features bits */ #define ARCH_SHSTK_SHSTK (1ULL << 0)
arch/x86/kernel/process_64.c +1 -0
@@ -836,6 +836,7 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)case ARCH_SHSTK_DISABLE :case ARCH_SHSTK_LOCK :case ARCH_SHSTK_UNLOCK : +case ARCH_SHSTK_STATUS :return shstk_prctl (task, option, arg2);default : ret = -EINVAL ;
arch/x86/kernel/shstk.c +7 -1
@@ -444,8 +444,14 @@ SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsireturn alloc_shstk (addr, aligned_size, size, set_tok); } -long shstk_prctl (struct task_struct *task,int option,unsigned long features) +long shstk_prctl (struct task_struct *task,int option,unsigned long arg2) { +unsigned long features = arg2; + +if (option ==ARCH_SHSTK_STATUS ) { +return put_user (task->thread.features, (unsigned long __user *)arg2); + } +if (option ==ARCH_SHSTK_LOCK ) { task->thread.features_locked |= features;return 0 ;
Signature · 1 line
2.17.1
Quoted 4 lines · Rick Edgecombe
This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.
Okay, I've done some bare metal testing, and it all looks happy. The selftest passes, and I can can see the stack address mismatch get detected if I explicitly rewrite the saved function pointer on the stack:
[INFO] Want normal flow
[INFO] Found 0x401890 @ 0x7fff47cf2ef8 [INFO] Normal execution flow [INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8 [INFO] Hijacked execution flow [INFO] Enabling shadow stack [INFO] Want to redirect
[INFO] Found 0x401890 @ 0x7fff47cf2ef8 Segmentation fault (core dumped)
Signature · 1 line
Kees Cook
Quoted 25 lines · Kees Cook
Quoted 7 lines · Rick Edgecombe
This series implements Shadow Stacks for userspace using x86's Control-flow
Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the
shadow stack part of this feature, and just for userspace.Okay, I've done some bare metal testing, and it all looks happy. The selftest passes, and I can can see the stack address mismatch get detected if I explicitly rewrite the saved function pointer on the stack:
[INFO] Want normal flow
[INFO] Found0x401890@0x7fff47cf2ef8
[INFO] Normal execution flow
[INFO] Want to redirect
[INFO] Found0x401890@0x7fff47cf2ef8
[INFO] Hijacked execution flow
[INFO] Enabling shadow stack
[INFO] Want to redirect
[INFO] Found0x401890@0x7fff47cf2ef8
Segmentation fault (core dumped)
Quoted 1 line
Tested-by Kees Cook <[email protected]>
Thanks and for the other tags!
Quoted 6 lines · Rick Edgecombe
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.
For the series
Signature · 2 lines
Sincerely yours, Mike.
Quoted 11 lines · Mike Rapoport
Quoted 9 lines · Rick Edgecombe
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow
Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the
shadow stack part of this feature, and just for userspace.
Quoted 3 lines
For the series
Thanks Mike! Sorry forgot to add it since last time.
Quoted 2 lines · Rick Edgecombe
I left tested-by tags in place per discussion with testers. Testers, please retest.
v6 is still working well on my AMD system (Dell PowerEdge R6515 w/ EPYC 7713).
The selftests run cleanly:
[INFO] new_ssp = 7f53069ffff8, *new_ssp = 7f5306a00001 [INFO] changing ssp from 7f53071ffff0 to 7f53069ffff8 [INFO] ssp is now 7f5306a00000 [OK] Shadow stack pivot [OK] Shadow stack faults [INFO] Corrupting shadow stack [INFO] Generated shadow stack violation successfully [OK] Shadow stack violation test [INFO] Gup read -> shstk access success [INFO] Gup write -> shstk access success [INFO] Violation from normal write [INFO] Gup read -> write access success [INFO] Violation from normal write [INFO] Gup write -> write access success [INFO] Cow gup write -> write access success [OK] Shadow gup test [INFO] Violation from shstk access [OK] mprotect() test [OK] Userfaultfd test [OK] 32 bit test
And I can see the control protection messages in dmesg when running the shstk violation test from here:
https://gitlab.com/cet-software/cet-smoke-test
ld-linux-x86-64[51598] control protection ip:401139 sp:7ffd68b1b7c8 ssp:7fb433578fd8 error:1(near ret) in shstk1[401000+1000]
Hi Rick,
Quoted 2 lines · Rick Edgecombe
Hi,
...
Quoted 4 lines
I left tested-by tags in place per discussion with testers. Testers, please retest.
1. Tested kself-test from user space shstk on ADL-S, TGL-U without Glibc shstk support in CentOS 8 stream OS:
// From the test_shadow_stack code in this patch series:
# ./test_shadow_stack [INFO] new_ssp = 7f014ac2dff8, *new_ssp = 7f014ac2e001 [INFO] changing ssp from 7f014a1ffff0 to 7f014ac2dff8 [INFO] ssp is now 7f014ac2e000 [OK] Shadow stack pivot [OK] Shadow stack faults [INFO] Corrupting shadow stack [INFO] Generated shadow stack violation successfully [OK] Shadow stack violation test [INFO] Gup read -> shstk access success [INFO] Gup write -> shstk access success [INFO] Violation from normal write [INFO] Gup read -> write access success [INFO] Violation from normal write [INFO] Gup write -> write access success [INFO] Cow gup write -> write access success [OK] Shadow gup test [INFO] Violation from shstk access [OK] mprotect() test [OK] Userfaultfd test [OK] 32 bit test
// shstk violation without SHSTK glibc support
// Code link: https://github.com/intel/lkvs/blob/main/cet/shstk_cp.c
# ./shstk_cp [PASS] Enable SHSTK successfully [PASS] Disabling shadow stack successfully [PASS] Re-enable shadow stack successfully [PASS] SHSTK enabled, ssp:7fa3bfe00000 [INFO] do_hack() change address for return: [INFO] Before,ssp:7fa3bfdffff8,*ssp:40133f,rbp:0x7ffc23b5b440,*rbp:7ffc23b5b480,*(rbp+1):40133f [INFO] After, ssp:7fa3bfdffff8,*ssp:40133f,rbp:0x7ffc23b5b440,*rbp:7ffc23b5b480,*(rbp+1):401146 Segmentation fault (core dumped)
Dmesg:
[1117184.518588] shstk_cp[1523882] control protection ip:40122c sp:7ffc23b5b448 ssp:7fa3bfdffff8 error:1(near ret) in shstk_cp[401000+1000]
// shstk ARCH_SHSTK_STATUS read/set test without SHSTK Glibc support // Code link: https://github.com/intel/lkvs/blob/main/cet/shstk_unlock_test.c
# ./shstk_unlock_test [PASS] Parent process enable SHSTK. [PASS] Parent pid:1522040, ssp:0x7f57fc400000 [INFO] pid:1522040, ssp:0x7f57fc3ffff8, *ssp:401799 [PASS] Unlock CET successfully for pid:1522041 [PASS] GET CET REG ret:0, err:0, ssp:7f57fc3ffff8 [PASS] SET CET REG ret:0, err:0, ssp:7f57fc3ffff8 [PASS] SET ssp -1 failed(expected) ret:-1, errno:22 [PASS] GET xstate successfully ret:0 [PASS] SHSTK is enabled in child process [INFO] Child:1522041 origin ssp:0x7f57fc400000 [INFO] Child:1522041, ssp:0x7f57fc400000, bp,0x7ffcf32ba0f0, *bp:401dc0, *(bp+1):7f57fc43ad85 [PASS] Disabling shadow stack succesfully [PASS] SHSTK_STATUS ok, feature:0 is 0, ret:0 [PASS] Child process re-enable ssp [PASS] SHSTK_STATUS ok, feature:1 1st bit is 1, ret:0 [PASS] Child process enabled wrss [PASS] SHSTK_STATUS ok, feature:3 2nd bit is 1, ret:0 [INFO] Child:1522041, ssp:0x7f57fc400000, bp,0x7ffcf32ba0f0, *bp:401dc0, *(bp+1):7f57fc43ad85 [INFO] ssp addr:0x7f57fc400000 is same as ssp_verify:0x7f57fc400000 [PASS] Child process disable shstk successfully. [PASS] Parent process disable shadow stack successfully.
2. Tested fedora37 OS + Hongjiu provided user space SHSTK support Glibc:
// shstk with Glibc support:
// Related Glibc support for Fedora37: http://gnu-4.sc.intel.com/git/?p=hjl/misc.git;a=tree;f=setup/fedora/37;h=63af84a8f28f3d0802f09266e47fb94eb5cdff26;hb=HEAD
# readelf -n shadow_test_fork | head readelf: Warning: Gap in build notes detected from 0x4011d7 to 0x4011e4
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000040 NT_GNU_PROPERTY_TYPE_0
Properties: x86 feature: IBT, SHSTK
...
// shadow_test_fork code is in attached
// gcc -fcf-protection=full -mshstk -O0 -fno-stack-check -fno-stack-protector shadow_test_fork.c -o shadow_test_fork
# ./shadow_test_fork s2 [INFO] s2: stack rbp + 1 [INFO] do_hack() change address for return: [INFO] After change, rbp+1 to hacked:0x401296 Segmentation fault (core dumped)
Dmesg:
[418653.591014] shadow_test_for[16529] control protection ip:401367 sp:7fff6ed0a728 ssp:7f661265bfe0 error:1(near ret) in shadow_test_fork[401000+1000]
All above user space SHSTK tests are passed.
Many thanks Rick and all!
Thanks!
BR.
Pengfei
Quoted 3 lines
Signature · 1 line
2.17.1
// SPDX-License-Identifier: GPL-2.0
/*
* Contributors:
* Pengfei, Xu <[email protected]> * - Test CET shadow stack function, should trigger #CP protection * - Add the print, and show stack address and content before and after
* changed
*/
#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <signal.h>
#include <sched.h>
#include <immintrin.h>
static long hacked(void)
{
printf("[INFO]\tAccess hack function\n");
printf("[FAIL]\tpid=%d Hacked!\n", getpid());
printf("[WARN]\tYou see this line, which means CET shstk #CP failed!\n");
return 1;
}/*
* stack variable y + 1(1 means 8bytes for 64bit, 4bytes for 32bit) is bp,
* and here use bp directly, it's bp hacked not sp hacked, so it should not
* trigger #CP.
*/
static void stack_add1_test(unsigned long changed_bp)
{
unsigned long *func_bp;#ifdef __x86_64__
asm("movq %%rbp,%0" : "=r"(func_bp));
#else
asm("mov %%ebp,%0" : "=r"(func_bp));
#endif
printf("[INFO]\tReal add1 function rbp content:%lx for main rbp.\n",
*func_bp);
*func_bp = changed_bp;
printf("[INFO]\tChange add1 rbp content:%lx, but right main rbp content in it!\n",
*func_bp);
}/* stack base rbp + 1 addr test, which should be hacked and #CP should work */ static unsigned long stack_add2_test(void)
{
unsigned long y;
unsigned long *i, *j;
i = (unsigned long *)_get_ssp();
j = __builtin_frame_address(0);
printf("[INFO]\tdo_hack() change address for return:\n");
printf("[INFO]\tBefore change,y:%lx,&y:%p,j:%p,*j:%lx,*(&j+1):0x%lx, ssp:%p *ssp:0x%lx\n",
y, &y, j, *j, *(j+1), i, *i);
/* j(rbp)+1 is sp address, change rbp+1 to change sp content */
*(j + 1) = (unsigned long)hacked;
printf("[INFO]\tAfter change, rbp+1 to hacked:0x%lx\n", *(j+1));
printf("[INFO]\tAfter hacked &y:%p, *j:0x%lx,*(&j+1):0x%lx\n",
&y, *j, *(j + 1));
/* Debug purpose: it's not related with ret instruction in objdump. */
return y;
}/* stack base y + 3 addr test, which should not be hacked and #CP */ static unsigned long stack_add3_test(void)
{
unsigned long y;
printf("[INFO]\tdo_hack() change address for return:\n");
printf("[INFO]\tBefore change, y:0x%lx, *(&y+2):0x%lx\n", y,
*((unsigned long *)&y + 2));
*((unsigned long *)&y + 3) = (unsigned long)hacked;
printf("[INFO]\tAfter change, *(&y+3) to change:0x%lx\n", (unsigned long)hacked);
printf("[INFO]\tAfter change &y+3:%p,*(&x+2):0x%lx\n",
(unsigned long *)&y + 3, *((unsigned long *)&y + 3));
printf("[INFO]\tAfter changed &y:%p, &y+2:%p,*(&y+2):0x%lx\n",
&y, (unsigned long *)&y + 2, *((unsigned long *)&y + 2));
return y;
}static long stack_long2_test(unsigned long i)
{
unsigned long *p;
printf("[INFO]\tuse rbp + long(+8bytes) size to hack:\n");
/*
* Another way to read rbp
* asm("movq %%rbp,%0" : "=r"(p));
*/
p = __builtin_frame_address(0);
printf("[INFO]\t*(p+1):%lx will be hacked\n", *(p + 1));
*(p + 1) = (unsigned long)hacked;
return 0;
}/* stack base y + 2 change to random value to do shstk violation */ static unsigned long stack_random(unsigned long j)
{
unsigned long y;
unsigned long *p;
y = j;
printf("[INFO]\tSHSTK hack with random value:\n");
#ifdef __x86_64__
asm("movq %%rbp,%0" : "=r"(p));
#else
asm("mov %%ebp,%0" : "=r"(p));#endif
*(p + 1) = j; return y; }
/* stack base y + 2 changed but no return */
static void stack_no_return(void)
{
unsigned long *p;
printf("[INFO]\tSHSTK with void no return function:\n");
#ifdef __x86_64__
asm("movq %%rbp,%0" : "=r"(p));
#else
asm("mov %%ebp,%0" : "=r"(p));#endif
*(p + 1) = (unsigned long)hacked; }
/* buffer overflow change stack base, which should trigger #CP */ static void stack_buf_impact(void)
{
char buffer[20];
int overflow_num = 44;
printf("[INFO]\tbuffer[20]:%x\n", buffer[20]);
memset(buffer, 0, overflow_num);
printf("[INFO]\tbuffer[44]:%x,&buffer[44]:%p\n", buffer[44], &buffer[44]);
printf("[INFO]\tbuffer[20] after overflow:%x\n", buffer[20]);
}/* buffer overflow not change stack base, which should not trigger #CP */ static void stack_buf_no_impact(void)
{
char buf[20];
int overflow_24 = 24, overflow_28 = 28;
printf("[INFO]\tbuf[20]:%x\n", buf[20]);
#ifdef __x86_64__
memset(buf, 0, overflow_28);
#else
memset(buf, 0, overflow_24);
#endif
printf("[INFO]\tbuf[20] after overflow:%x\n", buf[20]);
}/* test hack function */
static int do_hack(void *p)
{
/*
* Ret and then rip will get this value(rbp + 8 bytes in 64 bit OS)
* rbp(8 bytes in 64bit OS)
* *i, *j and so on variable content
*/
unsigned long *i, *j;
i = (unsigned long *)_get_ssp();
j = __builtin_frame_address(0);
printf("[INFO]\tBefore: rbp+8:0x%p content=0x%lx; ssp=0x%p, ssp content=0x%lx\n",
j + 1, *(j + 1), i, *i);
*(j+1) = (unsigned long)hacked;
printf("[INFO]\tAfter: rbp+8:0x%p content=0x%lx; ssp=0x%p, ssp content=0x%lx\n",
j + 1, *(j + 1), i, *i);
return 0;
}/* check shadow stack wo core dump in child pid */ static void stack_wo_core(void)
{
void *s = malloc(0x100000);
if (fork() == 0)
do_hack(s);
}/* test shstk by clone way */
static int stack_clone(void)
{
pid_t cid;
void *child_stack = malloc(0x100000);
if (child_stack == NULL) {
printf("[FAIL]\tmalloc child_stack failed!\n");
return 1;
}
cid = clone(
do_hack, /* function */
child_stack + 0x100000,
SIGCHLD,
0 /*arg*/
);
if (cid == -1) {
printf("[FAIL]\tclone failed!\n");
free(child_stack);
return 1;
}
printf("[INFO]\tparent=%d, child=%d\n", getpid(), cid);
if (waitpid(cid, NULL, 0) == -1) {
printf("[FAIL]\twaitpid() failed!\n");
return 1;
}
printf("[INFO]\tchild exits!\n");
free(child_stack);
return 0;
}/*
* Check shadow stack address and content and
* rbp address and protect address content
*/
static int shadow_stack_check(void)
{
unsigned long y;
unsigned long *bp_a, *ssp_a;
unsigned long long size_bp, size_ssp;
ssp_a = (unsigned long *)_get_ssp();
bp_a = __builtin_frame_address(0);
size_bp = sizeof(*(bp_a + 1));
size_ssp = sizeof(*ssp_a);
printf("[INFO]\t&y=0x%p\n", &y);
printf("[INFO]\tbp=%p,bp+1=%p,*(bp+1):0x%lx(size:%lld) ssp=%p *ssp=0x%lx(size:%lld)\n",
bp_a, bp_a + 1, *(bp_a + 1), size_bp, ssp_a, *ssp_a, size_ssp);
return 0;
}static void usage(void)
{
printf("Usage: [null | s1 | s2 | s3 | sl1 | sr | sn...]\n");
printf(" null: no parm, stack add 2 test, should trigger #CP\n");
printf(" s1: stack add 1 test\n");
printf(" s2: stack add 2 test, should trigger #CP\n");
printf(" s3: stack add 3 test\n");
printf(" sl1: stack with long add 2 test\n");
printf(" sr: stack change to random value\n");
printf(" sn: stack change but no return\n");
printf(" buf1: buffer overflow change stack base\n");
printf(" buf2: buffer overflow not change stack base\n");
printf(" snc: test shadow stack wo core dump\n");
printf(" sc: test shadow stack by clone way\n");
printf(" ssp: check shadow stack addr and content\n");
}int main(int argc, char *argv[])
{
char *parm = "";
unsigned long a = 0, *main_rbp, fake_bp[2];
a = rand();
enum {
e_s1, /* enum stack base, y + 1 */
e_s2, /* enum stack base + 1 addr content change test */ e_s3, /* enum stack base y + 3 */
e_sl1, /* enum stack base with long + 2 */ e_sr, /* enum stack base change to random value */ e_sn, /* enum stack base changed but no return */ e_buf1, /* buffer overflow change stack base */ e_buf2, /* buffer overflow not change stack base */ e_snc, /* shadow stack wo core dump */
e_sc, /* test shstk by stack clone way */ e_ssp /* check shadow stack addr and content */
} option;
#ifdef __x86_64__
asm("movq %%rbp,%0" : "=r"(main_rbp));
#else
asm("mov %%ebp,%0" : "=r"(main_rbp));#endif
/* Use real main rbp address and content to make one fake bp and sp */
fake_bp[0] = *main_rbp;
fake_bp[1] = *(main_rbp + 1);
if (argc == 1) {
usage();
stack_add2_test();
} else {
parm = argv[1];
if (strcmp(argv[1], "s1") == 0)
option = e_s1;
else if (strcmp(argv[1], "s2") == 0)
option = e_s2;
else if (strcmp(argv[1], "s3") == 0)
option = e_s3;
else if (strcmp(argv[1], "sl1") == 0)
option = e_sl1;
else if (strcmp(argv[1], "sr") == 0)
option = e_sr;
else if (strcmp(argv[1], "sn") == 0)
option = e_sn;
else if (strcmp(argv[1], "buf1") == 0)
option = e_buf1;
else if (strcmp(argv[1], "buf2") == 0)
option = e_buf2;
else if (strcmp(argv[1], "snc") == 0)
option = e_snc;
else if (strcmp(argv[1], "sc") == 0)
option = e_sc;
else if (strcmp(argv[1], "ssp") == 0)
option = e_ssp;
else {
usage();
exit(1);
}
}
switch (option) {
case e_s1:
printf("[INFO]\ts1: stack + 1\n");
stack_add1_test((unsigned long)&fake_bp[0]);
break;
case e_s2:
printf("[INFO]\ts2: stack rbp + 1\n");
stack_add2_test();
break;
case e_s3:
printf("[INFO]\ts3: stack + 3\n");
stack_add3_test();
break;
case e_sl1:
printf("[INFO]\tsl1: stack with long + 2, a:0x%lx\n", a);
stack_long2_test(a);
break;
case e_sr:
printf("[INFO]\tsr: stack changed to random value a:0x%lx\n", a);
stack_random(a);
break;
case e_sn:
printf("[INFO]\tsn: stack changed but no return\n");
stack_no_return();
break;
case e_buf1:
printf("buf1: buffer overflow change stack base\n");
stack_buf_impact();
break;
case e_buf2:
printf("[INFO]\tbuf2: buffer overflow not change stack base\n");
stack_buf_no_impact();
break;
case e_snc:
printf("[INFO]\tsnc: test shadow stack wo core dump\n");
stack_wo_core();
break;
case e_sc:
printf("[INFO]\tsc: test shstk by stack clone way\n");
stack_clone();
break;
case e_ssp:
printf("[INFO]\tssp: check shadow stack addr and content\n");
shadow_stack_check();
break;
default:
usage();
exit(1);
}
printf("[RESULTS]\tParent pid=%d is done.\n", getpid());
return 0;
}Borislav Petkov
maintainerQuoted 3 lines · Rick Edgecombe
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow
What is the base commit this applies on?
It ain't v6.2...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteQuoted 9 lines · Borislav Petkov
Quoted 4 lines · Rick Edgecombe
Hi,
This series implements Shadow Stacks for userspace using x86's Control-flow
What is the base commit this applies on?
It ain't v6.2...
It was tip/master the week I sent it out:0a5e985fb1c8