Skip to contentSkip to conversation

Shadow stacks for userspace

PATCH Superseded Rick Edgecombe 13 participants 159 messages Feb 27, 2023 – Mar 20, 2023 Page 1 of 2

Subsystems (44)

v7 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v7 cover Shadow stacks for userspace

Hi,

This series implements Shadow Stacks for userspace using x86's Control-flow Enforcement Technology (CET). CET consists of two related security features:
shadow stacks and indirect branch tracking. This series implements just the shadow stack part of this feature, and just for userspace.

The main use case for shadow stack is providing protection against return oriented programming attacks. It works by maintaining a secondary (shadow) stack using a special memory type that has protections against modification. When executing a CALL instruction, the processor pushes the return address to both the normal stack and to the special permission shadow stack. Upon RET, the processor pops the shadow stack copy and compares it to the normal stack copy. For more details, see the coverletter from v1 [0].

The changes for this version are some more cleanup of comment and commit log verbiage, and small refactor in the memory accounting patch. There was also some feedback from David Hildenbrand about adding GUP tests for the !FOLL_FORCE case. This is currently planned for a fast follow on patch.

Previous version [1].

Thanks,
Rick

[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/

Kirill A. Shutemov (1):
x86: Introduce userspace API for shadow stack

Mike Rapoport (1):
x86/shstk: Add ARCH_SHSTK_UNLOCK

Rick Edgecombe (19):
x86/fpu: Add helper for modifying xstate
x86: Move control protection handler to separate file
mm: Introduce pte_mkwrite_kernel()
s390/mm: Introduce pmd_mkwrite_kernel()
mm: Make pte_mkwrite() take a VMA
x86/mm: Introduce _PAGE_SAVED_DIRTY
x86/mm: Start actually marking _PAGE_SAVED_DIRTY x86/mm: Teach pte_mkwrite() about stack memory
mm: Don't allow write GUPs to shadow stack memory x86/mm: Introduce MAP_ABOVE4G
mm: Warn on shadow stack memory in wrong vma x86/mm: Warn if create Write=0,Dirty=1 with raw prot x86/shstk: Introduce map_shadow_stack syscall x86/shstk: Support WRSS for userspace
x86: Expose thread features in /proc/$PID/status x86/shstk: Wire in shadow stack interface selftests/x86: Add shadow stack test x86/fpu: Add helper for initing features x86/shstk: Add ARCH_SHSTK_STATUS

Yu-cheng Yu (20):
Documentation/x86: Add CET shadow stack description x86/shstk: Add Kconfig option for shadow stack x86/cpufeatures: Add CPU feature flags for shadow stacks x86/cpufeatures: Enable CET CR4 bit for shadow stack x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states x86/shstk: Add user control-protection fault handler x86/mm: Remove _PAGE_DIRTY from kernel RO pages x86/mm: Move pmd_write(), pud_write() up in the file x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY
mm: Move VM_UFFD_MINOR_BIT from 37 to 38
mm: Introduce VM_SHADOW_STACK for shadow stack memory x86/mm: Check shadow stack page fault errors
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting
mm: Re-introduce vm_flags to do_mmap()
x86/shstk: Add user-mode shadow stack support x86/shstk: Handle thread shadow stack
x86/shstk: Introduce routines modifying shstk x86/shstk: Handle signals for shadow stack
x86: Add PTRACE interface for shadow stack

 Documentation/filesystems/proc.rst            |   1 +
 Documentation/mm/arch_pgtable_helpers.rst     |   9 +-
 Documentation/x86/index.rst                   |   1 +
 Documentation/x86/shstk.rst                   | 176 +++++
 arch/alpha/include/asm/pgtable.h              |   6 +-
 arch/arc/include/asm/hugepage.h               |   2 +-
 arch/arc/include/asm/pgtable-bits-arcv2.h     |   7 +-
 arch/arm/include/asm/pgtable-3level.h         |   7 +-
 arch/arm/include/asm/pgtable.h                |   2 +-
 arch/arm/kernel/signal.c                      |   2 +-
 arch/arm64/include/asm/pgtable.h              |   9 +-
 arch/arm64/kernel/signal.c                    |   2 +-
 arch/arm64/kernel/signal32.c                  |   2 +-
 arch/arm64/mm/trans_pgd.c                     |   4 +-
 arch/csky/include/asm/pgtable.h               |   2 +-
 arch/hexagon/include/asm/pgtable.h            |   2 +-
 arch/ia64/include/asm/pgtable.h               |   2 +-
 arch/loongarch/include/asm/pgtable.h          |   4 +-
 arch/m68k/include/asm/mcf_pgtable.h           |   2 +-
 arch/m68k/include/asm/motorola_pgtable.h      |   6 +-
 arch/m68k/include/asm/sun3_pgtable.h          |   6 +-
 arch/microblaze/include/asm/pgtable.h         |   2 +-
 arch/mips/include/asm/pgtable.h               |   6 +-
 arch/nios2/include/asm/pgtable.h              |   2 +-
 arch/openrisc/include/asm/pgtable.h           |   2 +-
 arch/parisc/include/asm/pgtable.h             |   6 +-
 arch/powerpc/include/asm/book3s/32/pgtable.h  |   2 +-
 arch/powerpc/include/asm/book3s/64/pgtable.h  |   4 +-
 arch/powerpc/include/asm/nohash/32/pgtable.h  |   2 +-
 arch/powerpc/include/asm/nohash/32/pte-8xx.h  |   2 +-
 arch/powerpc/include/asm/nohash/64/pgtable.h  |   2 +-
 arch/riscv/include/asm/pgtable.h              |   6 +-
 arch/s390/include/asm/hugetlb.h               |   4 +-
 arch/s390/include/asm/pgtable.h               |  14 +-
 arch/s390/mm/pageattr.c                       |   4 +-
 arch/sh/include/asm/pgtable_32.h              |  10 +-
 arch/sparc/include/asm/pgtable_32.h           |   2 +-
 arch/sparc/include/asm/pgtable_64.h           |   6 +-
 arch/sparc/kernel/signal32.c                  |   2 +-
 arch/sparc/kernel/signal_64.c                 |   2 +-
 arch/um/include/asm/pgtable.h                 |   2 +-
 arch/x86/Kconfig                              |  24 +
 arch/x86/Kconfig.assembler                    |   5 +
 arch/x86/entry/syscalls/syscall_64.tbl        |   1 +
 arch/x86/include/asm/cpufeatures.h            |   2 +
 arch/x86/include/asm/disabled-features.h      |  16 +-
 arch/x86/include/asm/fpu/api.h                |   9 +
 arch/x86/include/asm/fpu/regset.h             |   7 +-
 arch/x86/include/asm/fpu/sched.h              |   3 +-
 arch/x86/include/asm/fpu/types.h              |  16 +-
 arch/x86/include/asm/fpu/xstate.h             |   6 +-
 arch/x86/include/asm/idtentry.h               |   2 +-
 arch/x86/include/asm/mmu_context.h            |   2 +
 arch/x86/include/asm/msr.h                    |  11 +
 arch/x86/include/asm/pgtable.h                | 322 +++++++-
 arch/x86/include/asm/pgtable_types.h          |  56 +-
 arch/x86/include/asm/processor.h              |   8 +
 arch/x86/include/asm/shstk.h                  |  40 +
 arch/x86/include/asm/special_insns.h          |  13 +
 arch/x86/include/asm/tlbflush.h               |   3 +-
 arch/x86/include/asm/trap_pf.h                |   2 +
 arch/x86/include/asm/traps.h                  |  12 +
 arch/x86/include/uapi/asm/mman.h              |   4 +
 arch/x86/include/uapi/asm/prctl.h             |  12 +
 arch/x86/kernel/Makefile                      |   4 +
 arch/x86/kernel/cet.c                         | 152 ++++
 arch/x86/kernel/cpu/common.c                  |  35 +-
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/cpu/proc.c                    |  23 +
 arch/x86/kernel/fpu/core.c                    |  59 +-
 arch/x86/kernel/fpu/regset.c                  |  86 +++
 arch/x86/kernel/fpu/xstate.c                  | 148 ++--
 arch/x86/kernel/fpu/xstate.h                  |   6 +
 arch/x86/kernel/idt.c                         |   2 +-
 arch/x86/kernel/process.c                     |  18 +-
 arch/x86/kernel/process_64.c                  |   9 +-
 arch/x86/kernel/ptrace.c                      |  12 +
 arch/x86/kernel/shstk.c                       | 491 +++++++++++++
 arch/x86/kernel/signal.c                      |   1 +
 arch/x86/kernel/signal_32.c                   |   2 +-
 arch/x86/kernel/signal_64.c                   |   8 +-
 arch/x86/kernel/sys_x86_64.c                  |   6 +-
 arch/x86/kernel/traps.c                       |  87 ---
 arch/x86/mm/fault.c                           |  31 +
 arch/x86/mm/pat/set_memory.c                  |   4 +-
 arch/x86/mm/pgtable.c                         |  38 +
 arch/x86/xen/enlighten_pv.c                   |   2 +-
 arch/x86/xen/mmu_pv.c                         |   2 +-
 arch/x86/xen/xen-asm.S                        |   2 +-
 arch/xtensa/include/asm/pgtable.h             |   2 +-
 fs/aio.c                                      |   2 +-
 fs/proc/array.c                               |   6 +
 fs/proc/task_mmu.c                            |   3 +
 include/asm-generic/hugetlb.h                 |   4 +-
 include/linux/mm.h                            |  46 +-
 include/linux/mman.h                          |   4 +
 include/linux/pgtable.h                       |  14 +
 include/linux/proc_fs.h                       |   2 +
 include/linux/syscalls.h                      |   1 +
 include/uapi/asm-generic/siginfo.h            |   3 +-
 include/uapi/asm-generic/unistd.h             |   2 +-
 include/uapi/linux/elf.h                      |   2 +
 ipc/shm.c                                     |   2 +-
 kernel/sys_ni.c                               |   1 +
 mm/debug_vm_pgtable.c                         |  16 +-
 mm/gup.c                                      |   2 +-
 mm/huge_memory.c                              |   7 +-
 mm/hugetlb.c                                  |   4 +-
 mm/internal.h                                 |   8 +-
 mm/memory.c                                   |   5 +-
 mm/migrate_device.c                           |   2 +-
 mm/mmap.c                                     |  10 +-
 mm/mprotect.c                                 |   2 +-
 mm/nommu.c                                    |   4 +-
 mm/userfaultfd.c                              |   2 +-
 mm/util.c                                     |   2 +-
 tools/testing/selftests/x86/Makefile          |   2 +-
 .../testing/selftests/x86/test_shadow_stack.c | 695 ++++++++++++++++++
 118 files changed, 2669 insertions(+), 327 deletions(-)
 create mode 100644 Documentation/x86/shstk.rst
 create mode 100644 arch/x86/include/asm/shstk.h
 create mode 100644 arch/x86/kernel/cet.c
 create mode 100644 arch/x86/kernel/shstk.c
 create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c
Signature · 1 line
2.17.1
#1 lore raw

PATCH v7 1/41 Documentation/x86: Add CET shadow stack description

From: Yu-cheng Yu <[email protected]>

Introduce a new document on Control-flow Enforcement Technology (CET).

---
v5:
- Literal format tweaks (Bagas Sanjaya)
- Update EOPNOTSUPP text due to unification after comment from (Kees)
- Update 32 bit signal support with new behavior
- Remove capitalization on shadow stack (Boris)
- Fix typo

v4:
- Drop clearcpuid piece (Boris)
- Add some info about 32 bit

v3:
- Clarify kernel IBT is supported by the kernel. (Kees, Andrew Cooper)
- Clarify which arch_prctl's can take multiple bits. (Kees)
- Describe ASLR characteristics of thread shadow stacks. (Kees)
- Add exec section. (Andrew Cooper)
- Fix some capitalization (Bagas Sanjaya)
- Update new location of enablement status proc.
- Add info about new user_shstk software capability.
- Add more info about what the kernel pushes to the shadow stack on signal.

v2:
- Updated to new arch_prctl() API
- Add bit about new proc status

---
 Documentation/x86/index.rst |   1 +
 Documentation/x86/shstk.rst | 166 ++++++++++++++++++++++++++++++++++++
 2 files changed, 167 insertions(+)
 create mode 100644 Documentation/x86/shstk.rst
Documentation/x86/index.rst +1 -0
index c73d133fd37c..8ac64d7de4dc 100644
@@ -22,6 +22,7 @@ x86-specific Documentation
    mtrr
    pat
    intel-hfi
+   shstk
    iommu
    intel_txt
    amd-memory-encryption
Documentation/x86/shstk.rst new file +166 -0

166 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Documentation/x86/shstk.rst line 54

Quoted 13 lines · Rick Edgecombe
Documentation/x86/shstk.rst new file +13 -0
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF note and can be verified
+from readelf/llvm-readelf output::
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications markers directly. Applications
+or loaders must enable CET features using the interface described in section 4.
+Typically this would be done in dynamic loader or static runtime objects, as is
+the case in GLIBC.

Note that this has to be an early decision in libc (ld.so or static exe start code), which will be difficult to hook into system wide security policy settings. (e.g. to force shstk on marked binaries.)

From userspace POV I'd prefer if a static exe did not have to parse its own ELF notes (i.e. kernel enabled shstk based on the marking). But I realize if there is a need for complex shstk enable/disable decision that is better in userspace and if the kernel decision can be overridden then it might as well all be in userspace.

Quoted 9 lines
Documentation/x86/shstk.rst new file +9 -0
+Enabling arch_prctl()'s
+=======================
+
+Elf features should be enabled by the loader using the below arch_prctl's. They
+are only supported in 64 bit user applications.
+
+arch_prctl(ARCH_SHSTK_ENABLE, unsigned long feature)
+    Enable a single feature specified in 'feature'. Can only operate on
+    one feature at a time.

Documentation/x86/shstk.rst line 74

Quoted 10 lines
Documentation/x86/shstk.rst new file +10 -0
+
+arch_prctl(ARCH_SHSTK_DISABLE, unsigned long feature)
+    Disable a single feature specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_SHSTK_LOCK, unsigned long features)
+    Lock in features at their current enabled or disabled status. 'features'
+    is a mask of all features to lock. All bits set are processed, unset bits
+    are ignored. The mask is ORed with the existing value. So any feature bits
+    set here cannot be enabled or disabled afterwards.

The multi-thread behaviour should be documented here: Only the current thread is affected. So an application can only change the setting while single-threaded which is only guaranteed before any user code is executed. Later using the prctl is complicated and most c runtimes would not want to do that (async signalling all threads and prctl from the handler).

In particular these interfaces are not suitable to turn shstk off at dlopen time when an unmarked binary is loaded. Or any other late shstk policy change will not work, so as far as i can see the "permissive" mode in glibc does not work.

Does the main thread have shadow stack allocated before shstk is enabled? is the shadow stack freed when it is disabled? (e.g. what would the instruction reading the SSP do in disabled state?)

Documentation/x86/shstk.rst line 100

Quoted 8 lines
Documentation/x86/shstk.rst new file +8 -0
+Proc Status
+===========
+To check if an application is actually running with shadow stack, the
+user can read the /proc/$PID/status. It will report "wrss" or "shstk"
+depending on what is enabled. The lines look like this::
+
+    x86_Thread_features: shstk wrss
+    x86_Thread_features_locked: shstk wrss

Presumaly /proc/$TID/status and /proc/$PID/task/$TID/status also shows the setting and only valid for the specific thread (not the entire process). So i would note that this for one thread only.

Documentation/x86/shstk.rst line 112

Quoted 11 lines
Documentation/x86/shstk.rst new file +11 -0
+Implementation of the Shadow Stack
+==================================
+
+Shadow Stack Size
+-----------------
+
+A task's shadow stack is allocated from memory to a fixed size of
+MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to
+the maximum size of the normal stack, but capped to 4 GB. However,
+a compat-mode application's address space is smaller, each of its thread's
+shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).

This policy tries to handle all threads with the same shadow stack size logic, which has limitations. I think it should be improved (otherwise some applications will have to turn shstk off):

- RLIMIT_STACK is not an upper bound for the main thread stack size (rlimit can increase/decrease dynamically).
- RLIMIT_STACK only applies to the main thread, so it is not an upper bound for non-main thread stacks.
- i.e. stack size >> startup RLIMIT_STACK is possible and then shadow stack can overflow.
- stack size << startup RLIMIT_STACK is also possible and then VA space is wasted (can lead to OOM with strict memory overcommit).
- clone3 tells the kernel the thread stack size so that should be used instead of RLIMIT_STACK. (clone does not though.)
- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Documentation/x86/shstk.rst line 120

Quoted 7 lines
Documentation/x86/shstk.rst new file +7 -0
+Signal
+------
+
+By default, the main program and its signal handlers use the same shadow
+stack. Because the shadow stack stores only return addresses, a large
+shadow stack covers the condition that both the program stack and the
+signal alternate stack run out.

What does "by default" mean here? Is there a case when the signal handler is not entered with SSP set to the handling thread'd shadow stack?

Documentation/x86/shstk.rst line 128

Quoted 7 lines
Documentation/x86/shstk.rst new file +7 -0
+When a signal happens, the old pre-signal state is pushed on the stack. When
+shadow stack is enabled, the shadow stack specific state is pushed onto the
+shadow stack. Today this is only the old SSP (shadow stack pointer), pushed
+in a special format with bit 63 set. On sigreturn this old SSP token is
+verified and restored by the kernel. The kernel will also push the normal
+restorer address to the shadow stack to help userspace avoid a shadow stack
+violation on the sigreturn path that goes through the restorer.

The kernel pushes on the shadow stack on signal entry so shadow stack overflow cannot be handled. Please document this as non-recoverable failure.

I think it can be made recoverable if signals with alternate stack run on a different shadow stack. And the top of the thread shadow stack is just corrupted instead of pushed in the overflow case. Then longjmp out can be made to work (common in stack overflow handling cases), and reliable crash report from the signal handler works (also common).

Does SSP get stored into the sigcontext struct somewhere?

Quoted 6 lines
Documentation/x86/shstk.rst new file +6 -0
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a
+shadow access triggers a page fault with the shadow stack access bit set

Documentation/x86/shstk.rst line 160

Quoted 10 lines
Documentation/x86/shstk.rst new file +10 -0
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page fault
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow stack
+for the new thread. New shadow stack's behave like mmap() with respect to
+ASLR behavior.

Please document the shadow stack lifetimes here:

I think thread exit unmaps shadow stack and vfork shares shadow stack with parent so exit does not unmap.

I think the map_shadow_stack syscall should be mentioned in this document too.

ABI for initial shadow stack entries:

If one wants to scan the shadow stack how to detect the end (e.g. fast backtrace)? Is it useful to put an invalid value (-1) there? (affects map_shadow_stack syscall too).

thanks.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

#3 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 2 lines · Szabolcs Nagy

...
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

sorry,
ignore this.

#4 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

In reply to Szabolcs Nagy
Quoted 14 lines · Szabolcs Nagy
Quoted 13 lines · Rick Edgecombe
Documentation/x86/shstk.rst new file +4 -0
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF note and can

be verified

Documentation/x86/shstk.rst new file +6 -0
+from readelf/llvm-readelf output::
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications markers directly.

Applications
+or loaders must enable CET features using the interface described

Quoted 8 lines
Quoted 4 lines

in section 4.
+Typically this would be done in dynamic loader or static runtime objects, as is
+the case in GLIBC.

Note that this has to be an early decision in libc (ld.so or static exe start code), which will be difficult to hook into system wide security policy settings. (e.g. to force shstk on marked binaries.)

In the eager enabling (by the kernel) scenario, how is this improved? The loader has to have the option to disable the shadow stack if enabling conditions are not met, so it still has to trust userspace to not do that. Did you have any more specifics on how the policy would work?

Quoted 3 lines

From userspace POV I'd prefer if a static exe did not have to parse its own ELF notes (i.e. kernel enabled shstk based on the marking).

This is actually exactly what happens in the glibc patches. My understand was that it already been discussed amongst glibc folks.

Quoted 3 lines

But I realize if there is a need for complex shstk enable/disable decision that is better in userspace and if the kernel decision can be overridden then it might as well all be in userspace.

A complication with shadow stack in general is that it has to be enabled very early. Otherwise when the program returns from main(), it will get a shadow stack underflow. The old logic in this series would enable shadow stack if the loader had the SHSTK bit (by parsing the header in the kernel). Then later if the conditions were not met to use shadow stack, the loader would call into the kernel again to disable shadow stack.

One problem (there were several with this area) with this eager enabling, was the kernel ended up mapping, briefly using, and then unmapping the shadow stack in the case of a executable not supporting shadow stack. What the glibc patches do today is pretty much the same behavior as before, just with the header parsing moved into userspace. I think letting the component with the most information make the decision leaves open the best opportunity for making it efficient. I wonder if it could be possible for glibc to enable it later than it currently does in the patches and improve the dynamic loader case, but I don't know enough of that code.

Quoted 25 lines
Quoted 25 lines
Documentation/x86/shstk.rst new file +4 -0
+Enabling arch_prctl()'s
+=======================
+
+Elf features should be enabled by the loader using the below

arch_prctl's. They

Documentation/x86/shstk.rst new file +4 -0
+are only supported in 64 bit user applications.
+
+arch_prctl(ARCH_SHSTK_ENABLE, unsigned long feature)
+    Enable a single feature specified in 'feature'. Can only

operate on

Documentation/x86/shstk.rst new file +4 -0
+    one feature at a time.
+
+arch_prctl(ARCH_SHSTK_DISABLE, unsigned long feature)
+    Disable a single feature specified in 'feature'. Can only

operate on

Documentation/x86/shstk.rst new file +4 -0
+    one feature at a time.
+
+arch_prctl(ARCH_SHSTK_LOCK, unsigned long features)
+    Lock in features at their current enabled or disabled status.

'features'
+ is a mask of all features to lock. All bits set are processed,
unset bits
+ are ignored. The mask is ORed with the existing value. So any
feature bits

Quoted 8 lines
Quoted 1 line

+ set here cannot be enabled or disabled afterwards.

The multi-thread behaviour should be documented here: Only the current thread is affected. So an application can only change the setting while single-threaded which is only guaranteed before any user code is executed. Later using the prctl is complicated and most c runtimes would not want to do that (async signalling all threads and prctl from the handler).

It is kind of covered in the fork() docs, but yes there should probably be a reference here too.

Quoted 5 lines

In particular these interfaces are not suitable to turn shstk off at dlopen time when an unmarked binary is loaded. Or any other late shstk policy change will not work, so as far as i can see the "permissive" mode in glibc does not work.

Yes, that is correct. Glibc permissive mode does not fully work. There are some ongoing discussions on how to make it work. Some options don't require kernel changes, and some do. Making it per-thread is complicated for x86 because when shadow stack is off, some of the special shadow stack instructions will cause #UD exception. Glibc (any probably other apps in the future) could be in the middle of executing these instructions when dlopen() was called. So if there was a process wide disable option it would have to be resilient to these #UDs. And even then the code that used them could not be guaranteed to continue to work. For example, if you call the gcc intrinsic _get_ssp() when shadow stack is enabled it could be expected to point to the shadow stack in most cases. If shadow stack gets disabled, rdssp will return 0, in which case reading the shadow stack would segfault. So the all- process disabling solution can't be fully robust when there is any shadow stack specific logic.

The other option discussed was creating trampolines between the linked legacy objects that could know to tell the kernel to disable shadow stack if needed. In this case, shadow stack is disabled for each thread as it calls into the DSO. It's not clear if there can be enough information gleaned from the legacy binaries to know when to generate the trampolines in exotic cases.

A third option might be to have some synchronization between the kernel and userspace around anything using the shadow stack instructions. But there is not much detail filled in there.

So in summary, it's not as simple as making the disable per-process.

Quoted 3 lines

Does the main thread have shadow stack allocated before shstk is enabled?

No.

Quoted 2 lines

is the shadow stack freed when it is disabled? (e.g. what would the instruction reading the SSP do in disabled state?)

Yes.

When shadow stack is disabled rdssp is a NOP, the intrinsic returns NULL.

Quoted 15 lines

Documentation/x86/shstk.rst line 100

Quoted 11 lines
Documentation/x86/shstk.rst new file +3 -0
+Proc Status
+===========
+To check if an application is actually running with shadow stack,

the
+user can read the /proc/$PID/status. It will report "wrss" or "shstk"

Documentation/x86/shstk.rst new file +4 -0
+depending on what is enabled. The lines look like this::
+
+    x86_Thread_features: shstk wrss
+    x86_Thread_features_locked: shstk wrss

Presumaly /proc/$TID/status and /proc/$PID/task/$TID/status also shows the setting and only valid for the specific thread (not the entire process). So i would note that this for one thread only.

Since enabling/disabling is per-thread, and the field is called "x86_Thread_features" I thought it was clear. It's easy to add some more detail though.

Quoted 19 lines
Quoted 12 lines
Documentation/x86/shstk.rst new file +8 -0
+Implementation of the Shadow Stack
+==================================
+
+Shadow Stack Size
+-----------------
+
+A task's shadow stack is allocated from memory to a fixed size of
+MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is

allocated to

(quoted diff) +2 -0
+the maximum size of the normal stack, but capped to 4 GB. However,
+a compat-mode application's address space is smaller, each of its
Quoted 2 lines

thread's
+shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).

This policy tries to handle all threads with the same shadow stack size logic, which has limitations. I think it should be improved (otherwise some applications will have to turn shstk off):

Quoted 10 lines

- RLIMIT_STACK is not an upper bound for the main thread stack size (rlimit can increase/decrease dynamically).
- RLIMIT_STACK only applies to the main thread, so it is not an upper bound for non-main thread stacks.
- i.e. stack size >> startup RLIMIT_STACK is possible and then shadow stack can overflow.
- stack size << startup RLIMIT_STACK is also possible and then VA space is wasted (can lead to OOM with strict memory overcommit).
- clone3 tells the kernel the thread stack size so that should be used instead of RLIMIT_STACK. (clone does not though.)

This actually happens already. I can update the docs.

Quoted 4 lines

- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Hmm, yea. This seems like a good idea, but I don't see why it can't be a follow on. The series is quite big just to get the basics. I have tried to save some of the enhancements (like alt shadow stack) for the future.

Quoted 15 lines
Quoted 11 lines
Documentation/x86/shstk.rst new file +4 -0
+Signal
+------
+
+By default, the main program and its signal handlers use the same

shadow
+stack. Because the shadow stack stores only return addresses, a large
+shadow stack covers the condition that both the program stack and the
+signal alternate stack run out.

What does "by default" mean here? Is there a case when the signal handler
is not entered with SSP set to the handling thread'd shadow stack?

Ah, yea, that could be updated. It is in reference to an alt shadow stack implementation that was held for later.

Quoted 10 lines
Quoted 10 lines

+When a signal happens, the old pre-signal state is pushed on the stack. When
+shadow stack is enabled, the shadow stack specific state is pushed onto the
+shadow stack. Today this is only the old SSP (shadow stack pointer), pushed
+in a special format with bit 63 set. On sigreturn this old SSP token is
+verified and restored by the kernel. The kernel will also push the

Quoted 8 lines
Quoted 4 lines

normal
+restorer address to the shadow stack to help userspace avoid a shadow stack
+violation on the sigreturn path that goes through the restorer.

The kernel pushes on the shadow stack on signal entry so shadow stack overflow cannot be handled. Please document this as non-recoverable failure.

It doesn't hurt to call it out. Please see the below link for future plans to handle this scenario (alt shadow stack).

Quoted 11 lines

I think it can be made recoverable if signals with alternate stack run
on a different shadow stack. And the top of the thread shadow stack is
just corrupted instead of pushed in the overflow case. Then longjmp out
can be made to work (common in stack overflow handling cases), and reliable crash report from the signal handler works (also common).

Does SSP get stored into the sigcontext struct somewhere?

No, it's pushed to the shadow stack only. See the v2 coverletter of the discussion on the design and reasoning:

https://lore.kernel.org/lkml/[email protected]/

Quoted 28 lines
Quoted 21 lines
Documentation/x86/shstk.rst new file +4 -0
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are

required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a
+shadow access triggers a page fault with the shadow stack access bit set

Documentation/x86/shstk.rst new file +3 -0
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and

both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page fault

Documentation/x86/shstk.rst new file +3 -0
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow
Quoted 4 lines

stack
+for the new thread. New shadow stack's behave like mmap() with respect to
+ASLR behavior.

Please document the shadow stack lifetimes here:

Quoted 2 lines

I think thread exit unmaps shadow stack and vfork shares shadow stack with parent so exit does not unmap.

Sure, this can be updated.

Quoted 3 lines

I think the map_shadow_stack syscall should be mentioned in this document too.

There is a man page prepared for this. I plan to update the docs to reference it when it exists and not duplicate the text. There can be a blurb for the time being but it would be short lived.

Quoted 4 lines

If one wants to scan the shadow stack how to detect the end (e.g. fast
backtrace)? Is it useful to put an invalid value (-1) there? (affects map_shadow_stack syscall too).

Interesting idea. I think it's probably not a breaking ABI change if we wanted to add it later.

#5 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 8 lines · Rick Edgecombe
Quoted 4 lines

If one wants to scan the shadow stack how to detect the end (e.g. fast
backtrace)? Is it useful to put an invalid value (-1) there? (affects map_shadow_stack syscall too).

Interesting idea. I think it's probably not a breaking ABI change if we
wanted to add it later.

One complication could be how to handle shadow stacks created outside of thread creation. map_shadow_stack would typically add a token at the end so it could be pivoted to. So then the backtracing algorithm would have to know to skip it or something to find a special start of stack marker.

Alternatively, the thread shadow stacks could get an already used token pushed at the end, to try to match what an in-use map_shadow_stack shadow stack would look like. Then the backtracing algorithm could just look for the same token in both cases. It might get confused in exotic cases and mistake a token in the middle of the stack for the end of the allocation though. Hmm...

#6 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 10 lines · Rick P
Quoted 8 lines · Rick Edgecombe
Quoted 4 lines

If one wants to scan the shadow stack how to detect the end (e.g. fast
backtrace)? Is it useful to put an invalid value (-1) there? (affects map_shadow_stack syscall too).

Interesting idea. I think it's probably not a breaking ABI change if we
wanted to add it later.

Quoted 5 lines

One complication could be how to handle shadow stacks created outside of thread creation. map_shadow_stack would typically add a token at the end so it could be pivoted to. So then the backtracing algorithm would have to know to skip it or something to find a special start of stack marker.

i'd expect the pivot token to disappear once you pivot to it (and a pivot token to appear on the stack you pivoted away from, so you can go back later) otherwise i don't see how swapcontext works.

i'd push an end token and a pivot token on new shadow stacks.

Quoted 6 lines

Alternatively, the thread shadow stacks could get an already used token pushed at the end, to try to match what an in-use map_shadow_stack shadow stack would look like. Then the backtracing algorithm could just look for the same token in both cases. It might get confused in exotic cases and mistake a token in the middle of the stack for the end of the allocation though. Hmm...

a backtracer would search for an end token on an active shadow stack. it should be able to skip other tokens that don't seem to be code addresses. the end token needs to be identifiable and not break security properties. i think it's enough if the backtrace is best effort correct, there can be corner-cases when shadow stack is difficult to interpret, but e.g. a profiler can still make good use of this feature.

#7 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 11 lines · On Thu, 2023-03-02 at 16:34 +0000, [email protected] wrote:
Quoted 10 lines

Alternatively, the thread shadow stacks could get an already used token
pushed at the end, to try to match what an in-use map_shadow_stack shadow stack would look like. Then the backtracing algorithm could just
look for the same token in both cases. It might get confused in exotic
cases and mistake a token in the middle of the stack for the end of the
allocation though. Hmm...

Quoted 7 lines

a backtracer would search for an end token on an active shadow stack. it should be able to skip other tokens that don't seem to be code addresses. the end token needs to be identifiable and not break security properties. i think it's enough if the backtrace is best effort correct, there can be corner-cases when shadow stack is difficult to interpret, but e.g. a profiler can still make good use of this feature.

So just taking a look at this and remembering we used to have an arch_prctl() that returned the thread's shadow stack base and size. Glibc needed it, but we found a way around and dropped it. If we added something like that back, then it could be used for backtracing in the typical thread case and also potentially similar things to what glibc was doing. This also saves ~8 bytes per shadow stack over an end-of- stack marker, so it's a tiny bit better on memory use.

For the end-of-stack-marker solution:
In the case of thread shadow stacks, I'm not seeing any issues testing adding markers at the end. So adding this on top of the existing series for just thread shadow stacks seems lower probability of impact regression wise. Especially if we do it in the near term.

For ucontext/map_shadow_stack, glibc expects a token to be at the size passed in. So we would either have to create a larger allocation (to include the marker) or create a new map_shadow_stack flag to do this (it was expected that there might be new types of initial shadow stack data that the kernel might need to create). It is also possible to pass a non-page aligned size and get zero's at the end of the allocation. In fact glibc does this today in the common case. So that is also an option.

I think I slightly prefer the former arch_prctl() based solution for a few reasons:
- When you need to find the start or end of the shadow stack can you
can just ask for it instead of searching. It can be faster and simpler.
- It saves 8 bytes of memory per shadow stack.

If this turns out to be wrong and we want to do the marker solution much later at some point, the safest option would probably be to create new flags.

But just discussing this with HJ, can you share more on what the usage is? Like which backtracing operation specifically needs the marker? How much does it care about the ucontext case?

#8 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 9 lines · Rick P

I think I slightly prefer the former arch_prctl() based solution for a few reasons:
- When you need to find the start or end of the shadow stack can you
can just ask for it instead of searching. It can be faster and simpler.
- It saves 8 bytes of memory per shadow stack.

If this turns out to be wrong and we want to do the marker solution much later at some point, the safest option would probably be to create new flags.

i see two problems with a get bounds syscall:

- syscall overhead.

- discontinous shadow stack (e.g. alt shadow stack ends with a pointer to the interrupted thread shadow stack, so stack trace can continue there, except you don't know the bounds of that).

Quoted 3 lines

But just discussing this with HJ, can you share more on what the usage is? Like which backtracing operation specifically needs the marker? How much does it care about the ucontext case?

it could be an option for perf or ptracers to sample the stack trace.

in-process collection of stack trace for profiling or crash reporting (e.g. when stack is corrupted) or cross checking stack integrity may use it too.

sometimes parsing /proc/self/smaps maybe enough, but the idea was to enable light-weight backtrace collection in an async-signal-safe way.

syscall overhead in case of frequent stack trace collection can be avoided by caching (in tls) when ssp falls within the thread shadow stack bounds. otherwise caching does not work as the shadow stack may be reused (alt shadow stack or ucontext case).

unfortunately i don't know if syscall overhead is actually a problem (probably not) or if backtrace across signal handlers need to work with alt shadow stack (i guess it should work for crash reporting).

thanks.

#9 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

* szabolcs:

Quoted 4 lines

syscall overhead in case of frequent stack trace collection can be avoided by caching (in tls) when ssp falls within the thread shadow stack bounds. otherwise caching does not work as the shadow stack may be reused (alt shadow stack or ucontext case).

Do we need to perform the system call at each page boundary only? That should reduce overhead to the degree that it should not matter.

Quoted 3 lines

unfortunately i don't know if syscall overhead is actually a problem (probably not) or if backtrace across signal handlers need to work with alt shadow stack (i guess it should work for crash reporting).

Ideally, we would implement the backtrace function (in glibc) as just a shadow stack copy. But this needs to follow the chain of alternate stacks, and it may also need some form of markup for signal handler frames (which need program counter adjustment to reflect that a *non-signal* frame is conceptually nested within the previous instruction, and not the function the return address points to). But I think we can add support for this incrementally.

I assume there is no desire at all on the kernel side that sigaltstack transparently allocates the shadow stack? Because there is no deallocation function today for sigaltstack?

Thanks,
Florian

#10 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 7 lines · Florian Weimer

Ideally, we would implement the backtrace function (in glibc) as just a
shadow stack copy. But this needs to follow the chain of alternate stacks, and it may also need some form of markup for signal handler frames (which need program counter adjustment to reflect that a *non-signal* frame is conceptually nested within the previous instruction, and not the function the return address points to).

In the alt shadow stack case, the shadow stack sigframe will have a special shadow stack frame with a pointer to the shadow stack stack it came from. This may be a thread stack, or some other stack. This writeup in the v2 of the series has more details and analysis on the signal piece:

https://lore.kernel.org/lkml/[email protected]/

So in that design, you should be able to backtrace out of a chain of alt stacks.

Quoted 2 lines

But I
think we can add support for this incrementally.

Yea, I think so too.

Quoted 4 lines

I assume there is no desire at all on the kernel side that sigaltstack
transparently allocates the shadow stack?

It could have some nice benefit for some apps, so I did look into it.

Quoted 2 lines

Because there is no
deallocation function today for sigaltstack?

Yea, this is why we can't do it transparently. There was some discussion up the thread on this.

#11 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 5 lines · Rick P
Quoted 3 lines · Florian Weimer

I assume there is no desire at all on the kernel side that sigaltstack
transparently allocates the shadow stack?

Quoted 7 lines

It could have some nice benefit for some apps, so I did look into it.

Quoted 2 lines

Because there is no
deallocation function today for sigaltstack?

Yea, this is why we can't do it transparently. There was some discussion up the thread on this.

changing/disabling the alt stack is not valid while a handler is executing on it. if we don't allow jumping out and back to an alt stack (swapcontext) then there can be only one alt stack live per thread and change/disable can do the shadow stack free.

if jump back is allowed (linux even makes it race-free with SS_AUTODISARM) then the life-time of alt stack is extended beyond change/disable (jump back to an unregistered alt stack).

to support jump back to an alt stack the requirements are

1) user has to manage an alt shadow stack together with the alt stack (requies user code change, not just libc).

2) kernel has to push a restore token on the thread shadow stack on signal entry (at least in case of alt shadow stack, and deal with corner cases around shadow stack overflow).

#12 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

* szabolcs:

Quoted 9 lines

changing/disabling the alt stack is not valid while a handler is executing on it. if we don't allow jumping out and back to an alt stack (swapcontext) then there can be only one alt stack live per thread and change/disable can do the shadow stack free.

if jump back is allowed (linux even makes it race-free with SS_AUTODISARM) then the life-time of alt stack is extended beyond change/disable (jump back to an unregistered alt stack).

Quoted 8 lines

to support jump back to an alt stack the requirements are

1) user has to manage an alt shadow stack together with the alt stack (requies user code change, not just libc).

2) kernel has to push a restore token on the thread shadow stack on signal entry (at least in case of alt shadow stack, and deal with corner cases around shadow stack overflow).

We need to have a story for stackful coroutine switching as well, not just for sigaltstack. I hope that we can use OpenJDK (Project Loom) and QEMU as guinea pigs. If we have something that works for both, hopefully that covers a broad range of scenarios. Userspace coordination can eventually be handled by glibc; we can deallocate alternate stacks on thread exit fairly easily (at least compared to the current stack 8-).

Thanks,
Florian

#13 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 20 lines · Florian Weimer

* szabolcs:

Quoted 9 lines

changing/disabling the alt stack is not valid while a handler is executing on it. if we don't allow jumping out and back to an alt stack (swapcontext) then there can be only one alt stack live per thread and change/disable can do the shadow stack free.

if jump back is allowed (linux even makes it race-free with SS_AUTODISARM) then the life-time of alt stack is extended beyond change/disable (jump back to an unregistered alt stack).

Quoted 8 lines

to support jump back to an alt stack the requirements are

1) user has to manage an alt shadow stack together with the alt stack (requies user code change, not just libc).

2) kernel has to push a restore token on the thread shadow stack on signal entry (at least in case of alt shadow stack, and deal with corner cases around shadow stack overflow).

Quoted 7 lines

We need to have a story for stackful coroutine switching as well, not just for sigaltstack. I hope that we can use OpenJDK (Project Loom) and QEMU as guinea pigs. If we have something that works for both, hopefully that covers a broad range of scenarios. Userspace coordination can eventually be handled by glibc; we can deallocate alternate stacks on thread exit fairly easily (at least compared to the current stack 8-).

for stackful coroutines we just need a way to

- allocate a shadow stack with a restore token on it.

- switch to a target shadow stack with a restore token on it, while leaving behind a restore token on the old shadow stack.

this is supported via map_shadow_stack syscall and the rstoressp, saveprevssp instruction pair.

otoh there can be many alt shadow stacks per thread alive if we allow jump back (only one of them registered at a time) in fact they can be jumped to even from another thread, so their life-time is not tied to the thread (at least if we allow swapcontext across threads) so i think the libc cannot manage the alt shadow stacks, only user code can in the general case.

and in case a signal runs on an alt shadow stack, the restore token can only be placed by the kernel on the old shadow stack.

thanks.

#14 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

In reply to [email protected]

+Kan for shadow stack perf discussion.

Quoted 38 lines · On Mon, 2023-03-06 at 16:20 +0000, [email protected] wrote:
Quoted 9 lines · Rick P

I think I slightly prefer the former arch_prctl() based solution for a
few reasons:
- When you need to find the start or end of the shadow stack can
you
can just ask for it instead of searching. It can be faster and simpler.
- It saves 8 bytes of memory per shadow stack.

Quoted 4 lines

If this turns out to be wrong and we want to do the marker solution much later at some point, the safest option would probably be to create
new flags.

i see two problems with a get bounds syscall:

- syscall overhead.

- discontinous shadow stack (e.g. alt shadow stack ends with a pointer to the interrupted thread shadow stack, so stack trace can continue there, except you don't know the bounds of that).

Quoted 5 lines

But just discussing this with HJ, can you share more on what the usage
is? Like which backtracing operation specifically needs the marker? How
much does it care about the ucontext case?

it could be an option for perf or ptracers to sample the stack trace.

in-process collection of stack trace for profiling or crash reporting (e.g. when stack is corrupted) or cross checking stack integrity may use it too.

sometimes parsing /proc/self/smaps maybe enough, but the idea was to enable light-weight backtrace collection in an async-signal-safe way.

Quoted 8 lines

syscall overhead in case of frequent stack trace collection can be avoided by caching (in tls) when ssp falls within the thread shadow stack bounds. otherwise caching does not work as the shadow stack may be reused (alt shadow stack or ucontext case).

unfortunately i don't know if syscall overhead is actually a problem (probably not) or if backtrace across signal handlers need to work with alt shadow stack (i guess it should work for crash reporting).

There was a POC done of perf integration. I'm not too knowledgeable on perf, but the patch itself didn't need any new shadow stack bounds ABI. Since it was implemented in the kernel, it could just refer to the kernel's internal data for the thread's shadow stack bounds.

I asked about ucontext (similar to alt shadow stacks in regards to lack of bounds ABI), and apparently perf usually focuses on the thread stacks. Hopefully Kan can lend some more confidence to that assertion.

#15 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 50 lines · Rick P

+Kan for shadow stack perf discussion.

Quoted 38 lines · On Mon, 2023-03-06 at 16:20 +0000, [email protected] wrote:
Quoted 9 lines · Rick P

I think I slightly prefer the former arch_prctl() based solution for a
few reasons:
- When you need to find the start or end of the shadow stack can
you
can just ask for it instead of searching. It can be faster and simpler.
- It saves 8 bytes of memory per shadow stack.

Quoted 4 lines

If this turns out to be wrong and we want to do the marker solution much later at some point, the safest option would probably be to create
new flags.

i see two problems with a get bounds syscall:

- syscall overhead.

- discontinous shadow stack (e.g. alt shadow stack ends with a pointer to the interrupted thread shadow stack, so stack trace can continue there, except you don't know the bounds of that).

Quoted 5 lines

But just discussing this with HJ, can you share more on what the usage
is? Like which backtracing operation specifically needs the marker? How
much does it care about the ucontext case?

it could be an option for perf or ptracers to sample the stack trace.

in-process collection of stack trace for profiling or crash reporting (e.g. when stack is corrupted) or cross checking stack integrity may use it too.

sometimes parsing /proc/self/smaps maybe enough, but the idea was to enable light-weight backtrace collection in an async-signal-safe way.

Quoted 8 lines

syscall overhead in case of frequent stack trace collection can be avoided by caching (in tls) when ssp falls within the thread shadow stack bounds. otherwise caching does not work as the shadow stack may be reused (alt shadow stack or ucontext case).

unfortunately i don't know if syscall overhead is actually a problem (probably not) or if backtrace across signal handlers need to work with alt shadow stack (i guess it should work for crash reporting).

Quoted 8 lines

There was a POC done of perf integration. I'm not too knowledgeable on perf, but the patch itself didn't need any new shadow stack bounds ABI. Since it was implemented in the kernel, it could just refer to the kernel's internal data for the thread's shadow stack bounds.

I asked about ucontext (similar to alt shadow stacks in regards to lack of bounds ABI), and apparently perf usually focuses on the thread stacks. Hopefully Kan can lend some more confidence to that assertion.

The POC perf patch I implemented tries to use the shadow stack to replace the frame pointer to construct a callchain of a user space thread. Yes, it's in the kernel, perf_callchain_user(). I don't think the current X86 perf implementation handle the alt stack either. So the kernel internal data for the thread's shadow stack bounds should be good enough for the perf case.

Thanks,
Kan

#16 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

In reply to Edgecombe, Rick P
Quoted 24 lines · Rick P
Quoted 14 lines · Szabolcs Nagy
Quoted 13 lines · Rick Edgecombe
Documentation/x86/shstk.rst new file +4 -0
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF note and can

be verified

Documentation/x86/shstk.rst new file +6 -0
+from readelf/llvm-readelf output::
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications markers directly.

Applications
+or loaders must enable CET features using the interface described

Quoted 8 lines
Quoted 4 lines

in section 4.
+Typically this would be done in dynamic loader or static runtime objects, as is
+the case in GLIBC.

Note that this has to be an early decision in libc (ld.so or static exe start code), which will be difficult to hook into system wide security policy settings. (e.g. to force shstk on marked binaries.)

Quoted 5 lines

In the eager enabling (by the kernel) scenario, how is this improved? The loader has to have the option to disable the shadow stack if enabling conditions are not met, so it still has to trust userspace to not do that. Did you have any more specifics on how the policy would work?

i guess my issue is that the arch prctls only allow self policing. there is no kernel mechanism to set policy from outside the process that is either inherited or asynchronously set. policy is completely managed by libc (and done very early).

now i understand that async disable does not work (thanks for the explanation), but some control for forced enable/locking inherited across exec could work.

Quoted 5 lines
Quoted 2 lines

From userspace POV I'd prefer if a static exe did not have to parse its own ELF notes (i.e. kernel enabled shstk based on the marking).

This is actually exactly what happens in the glibc patches. My understand was that it already been discussed amongst glibc folks.

there were many glibc patches some of which are committed despite not having an accepted linux abi, so i'm trying to review the linux abi contracts and expect this patch to be authorative, please bear with me.

Quoted 9 lines
Quoted 4 lines

- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Hmm, yea. This seems like a good idea, but I don't see why it can't be a follow on. The series is quite big just to get the basics. I have tried to save some of the enhancements (like alt shadow stack) for the future.

it is actually not obvious how to introduce a limit so it is inherited or reset in a sensible way so i think it is useful to discuss it together with other issues.

Quoted 17 lines
Quoted 3 lines

The kernel pushes on the shadow stack on signal entry so shadow stack overflow cannot be handled. Please document this as non-recoverable failure.

It doesn't hurt to call it out. Please see the below link for future plans to handle this scenario (alt shadow stack).

Quoted 10 lines

I think it can be made recoverable if signals with alternate stack run
on a different shadow stack. And the top of the thread shadow stack is
just corrupted instead of pushed in the overflow case. Then longjmp out
can be made to work (common in stack overflow handling cases), and reliable crash report from the signal handler works (also common).

Quoted 6 lines
Quoted 1 line

Does SSP get stored into the sigcontext struct somewhere?

No, it's pushed to the shadow stack only. See the v2 coverletter of the discussion on the design and reasoning:

https://lore.kernel.org/lkml/[email protected]/

i think this should be part of the initial design as it may be hard to change later.

"sigaltshstk() is separate from sigaltstack(). You can have one without the other, neither or both together. Because the shadow stack specific state is pushed to the shadow stack, the two features don’t need to know about each other."

this means they cannot be changed together atomically.

i'd expect most sigaltstack users to want to be resilient against shadow stack overflow which means non-portable code changes.

i don't see why automatic alt shadow stack allocation would not work (kernel manages it transparently when an alt stack is installed or disabled).

"Since shadow alt stacks are a new feature, longjmp()ing from an alt shadow stack will simply not be supported. If a libc want’s to support this it will need to enable WRSS and write it’s own restore token."

i think longjmp should work without enabling writes to the shadow stack in the libc. this can also affect unwinding across signal handlers (not for c++ but e.g. glibc thread cancellation).

i'd prefer overwriting the shadow stack top entry on overflow to disallowing longjmp out of a shadow stack overflow handler.

Quoted 6 lines
Quoted 2 lines

I think the map_shadow_stack syscall should be mentioned in this document too.

There is a man page prepared for this. I plan to update the docs to reference it when it exists and not duplicate the text. There can be a blurb for the time being but it would be short lived.

i wanted to comment on the syscall because i think it may be better to have a magic mmap MAP_ flag that takes care of everything.

but i can go comment on the specific patch then.

thanks.

#17 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 40 lines · On Thu, 2023-03-02 at 16:14 +0000, [email protected] wrote:
Quoted 30 lines · Rick P
Quoted 20 lines · Szabolcs Nagy
Quoted 19 lines · Rick Edgecombe
Documentation/x86/shstk.rst new file +4 -0
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF note and

can
be verified

Documentation/x86/shstk.rst new file +6 -0
+from readelf/llvm-readelf output::
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications markers

directly.
Applications
+or loaders must enable CET features using the interface described
in section 4.
+Typically this would be done in dynamic loader or static runtime

Quoted 8 lines
Quoted 2 lines

objects, as is
+the case in GLIBC.

Note that this has to be an early decision in libc (ld.so or static
exe start code), which will be difficult to hook into system wide security policy settings. (e.g. to force shstk on marked binaries.)

Quoted 8 lines

In the eager enabling (by the kernel) scenario, how is this improved?
The loader has to have the option to disable the shadow stack if enabling conditions are not met, so it still has to trust userspace to
not do that. Did you have any more specifics on how the policy would
work?

Quoted 8 lines

i guess my issue is that the arch prctls only allow self policing. there is no kernel mechanism to set policy from outside the process that is either inherited or asynchronously set. policy is completely managed by libc (and done very early).

now i understand that async disable does not work (thanks for the explanation), but some control for forced enable/locking inherited across exec could work.

Is the idea that shadow stack would be forced on regardless of if the linked libraries support it? In which case it could be allowed to crash if they do not?

I think the majority of users would prefer the other case where shadow stack is only used if supported, so this sounds like a special case. Rather than lose the flexibility for the typical case, I would think something like this could be an additional enabling mode. glibc could check if shadow stack is already enabled by the kernel using the arch_prctl()s in this case.

We are having to work around the existing broken glibc binaries by not triggering off the elf bits automatically in the kernel, but I suppose if this was a special "I don't care if it crashes" feature, maybe it would be ok. Otherwise we would need to change the elf header bit to exclude the old binaries to even be able to do this, and there was extreme resistance to this idea from the userspace side.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

From userspace POV I'd prefer if a static exe did not have to parse
its own ELF notes (i.e. kernel enabled shstk based on the marking).

Quoted 7 lines
Quoted 2 lines

This is actually exactly what happens in the glibc patches. My understand was that it already been discussed amongst glibc folks.

there were many glibc patches some of which are committed despite not having an accepted linux abi, so i'm trying to review the linux abi contracts and expect this patch to be authorative, please bear with me.

H.J. has some recent ones that work against this kernel series that might interest you. The existing upstream glibc support will not get used due to the enabling interface change to arch_prctl() (this was one of the inspirations of the change actually).

Quoted 13 lines
Quoted 6 lines
Quoted 5 lines

- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Quoted 6 lines

Hmm, yea. This seems like a good idea, but I don't see why it can't be
a follow on. The series is quite big just to get the basics. I have tried to save some of the enhancements (like alt shadow stack) for the
future.

Quoted 4 lines

it is actually not obvious how to introduce a limit so it is inherited
or reset in a sensible way so i think it is useful to discuss it together with other issues.

Looking at this again, I'm not sure why a new rlimit is needed. It seems many of those points were just formulations of that the clone3 stack size was not used, but it actually is and just not documented. If you disagree perhaps you could elaborate on what the requirements are and we can see if it seems tricky to do in a follow up.

Quoted 26 lines
Quoted 26 lines
Quoted 6 lines

The kernel pushes on the shadow stack on signal entry so shadow stack
overflow cannot be handled. Please document this as non- recoverable
failure.

It doesn't hurt to call it out. Please see the below link for future
plans to handle this scenario (alt shadow stack).

Quoted 15 lines

I think it can be made recoverable if signals with alternate stack
run
on a different shadow stack. And the top of the thread shadow stack
is
just corrupted instead of pushed in the overflow case. Then longjmp
out
can be made to work (common in stack overflow handling cases), and
reliable crash report from the signal handler works (also common).

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

Does SSP get stored into the sigcontext struct somewhere?

No, it's pushed to the shadow stack only. See the v2 coverletter of the
discussion on the design and reasoning:

https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines

i think this should be part of the initial design as it may be hard to change later.

This is actually how it came up. Andy Lutomirski said, paraphrasing, "what if we want alt shadow stacks someday, does the signal frame ABI support it?". So I created an ABI that supports it and an initial POC, and said lets hold off on the implementation for the first version and just use the sigframe ABI that will allow it for the future. So the point was to make sure the signal format supported alt shadow stacks to make it easier in the future.

Quoted 7 lines

"sigaltshstk() is separate from sigaltstack(). You can have one without the other, neither or both together. Because the shadow stack specific state is pushed to the shadow stack, the two features don’t need to know about each other."

this means they cannot be changed together atomically.

Not sure why this is needed since they can be used separately. So why tie them together?

Quoted 4 lines

i'd expect most sigaltstack users to want to be resilient against shadow stack overflow which means non-portable code changes.

Portable between architectures? Or between shadow stack vs non-shadow stack?

It does seem like it would not be uncommon for users to want both together, but see below.

Quoted 4 lines

i don't see why automatic alt shadow stack allocation would not work (kernel manages it transparently when an alt stack is installed or disabled).

Ah, I think I see where maybe I can fill you in. Andy Luto had discounted this idea out of hand originally, but I didn't see it at first. sigaltstack lets you set, retrieve, or disable the shadow stack, right... But this doesn't allocate anything, it just sets where the next signal will be handled. This is different than things like threads where there is a new resources being allocated and it makes coming up with logic to guess when to de-allocate the alt shadow stack difficult. You probably already know...

But because of this there can be some modes where the shadow stack is changed while on it. For one example, SS_AUTODISARM will disable the alt shadow stack while switching to it and restore when sigreturning. At which point a new altstack can be set. In the non-shadow stack case this is nice because future signals won't clobber the alt stack if you switch away from it (swapcontext(), etc). But it also means you can "change" the alt stack while on it ("change" sort of, the auto disarm results in the kernel forgetting it temporarily).

I hear where you are coming from with the desire to have it "just work" with existing code, but I think the resulting ABI around the alt shadow stack allocation lifecycle would be way too complicated even if it could be made to work. Hence making a new interface. But also, the idea was that the x86 signal ABI should support handling alt shadow stacks, which is what we have done with this series. If a different interface for configuring it is better than the one from the POC, I'm not seeing a problem jump out. Is there any specific concern about backwards compatibility here?

Quoted 9 lines

"Since shadow alt stacks are a new feature, longjmp()ing from an alt shadow stack will simply not be supported. If a libc want’s to support this it will need to enable WRSS and write it’s own restore token."

i think longjmp should work without enabling writes to the shadow stack in the libc. this can also affect unwinding across signal handlers (not for c++ but e.g. glibc thread cancellation).

glibc today does not support longjmp()ing from a different stack (for example even today after a swapcontext()) when shadow stack is used. If glibc used wrss it could be supported maybe, but otherwise I don't see how the HW can support it.

HJ and I were actually just discussing this the other day. Are you looking at this series with respect to the arm shadow stack feature by any chance? I would love if glibc/tools would document what the shadow stack limitations are. If the all the arch's have the same or similar limitations perhaps this could be one developer guide. For the most part though, the limitations I've encountered are in glibc and the kernel is more the building blocks.

Quoted 12 lines

i'd prefer overwriting the shadow stack top entry on overflow to disallowing longjmp out of a shadow stack overflow handler.

Quoted 7 lines
Quoted 2 lines

I think the map_shadow_stack syscall should be mentioned in this document too.

There is a man page prepared for this. I plan to update the docs to reference it when it exists and not duplicate the text. There can be a
blurb for the time being but it would be short lived.

Quoted 6 lines

i wanted to comment on the syscall because i think it may be better to have a magic mmap MAP_ flag that takes care of everything.

but i can go comment on the specific patch then.

thanks.

A general comment. Not sure if you are aware, but this shadow stack enabling effort is quite old at this point and there have been many discussions on these topics stretching back years. The latest conversation was around getting this series into linux-next soon to get some testing on the MM pieces. I really appreciate getting this ABI feedback as it is always tricky to get right, but at this stage I would hope to be focusing mostly on concrete problems.

I also expect to have some amount of ABI growth going forward with all the normal things that entails. Shadow stack is not special in that it can come fully finalized without the need for the real world usage iterative feedback process. At some point we need to move forward with something, and we have quite a bit of initial changes at this point.

So I would like to minimize the initial implementation unless anyone sees any likely problems with future growth. Can you be clear if you see any concrete problems at this point or are more looking to evaluate the design reasoning? I'm under the assumption there is nothing that would prohibit linux-next testing while any ABI shakedown happens concurrently at least?

Thanks,
Rick

#18 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 3 lines · Rick P

Is the idea that shadow stack would be forced on regardless of if the linked libraries support it? In which case it could be allowed to crash if they do not?

execute a binary
- with shstk enabled and locked (only if marked?).
- with shstk disabled and locked.
could be managed in userspace, but it is libc dependent then.

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Looking at this again, I'm not sure why a new rlimit is needed. It seems many of those points were just formulations of that the clone3 stack size was not used, but it actually is and just not documented. If you disagree perhaps you could elaborate on what the requirements are and we can see if it seems tricky to do in a follow up.

- tiny thread stack and deep signal stack.
(note that this does not really work with glibc because it has implementation internal signals that don't run on alt stack, cannot be masked and don't fit on a tiny thread stack, but with other runtimes this can be a valid use-case, e.g. musl allows tiny thread stacks, < pagesize.)

- thread runtimes with clone (glibc uses clone3 but some dont).

- huge stacks but small call depth (problem if some va limit is hit or memory overcommit is disabled).

Quoted 4 lines
Quoted 4 lines

"sigaltshstk() is separate from sigaltstack(). You can have one without the other, neither or both together. Because the shadow stack specific state is pushed to the shadow stack, the two features don’t need to know about each other."

...

Quoted 13 lines
Quoted 3 lines

i don't see why automatic alt shadow stack allocation would not work (kernel manages it transparently when an alt stack is installed or disabled).

Ah, I think I see where maybe I can fill you in. Andy Luto had discounted this idea out of hand originally, but I didn't see it at first. sigaltstack lets you set, retrieve, or disable the shadow stack, right... But this doesn't allocate anything, it just sets where the next signal will be handled. This is different than things like threads where there is a new resources being allocated and it makes coming up with logic to guess when to de-allocate the alt shadow stack difficult. You probably already know...

Quoted 8 lines

But because of this there can be some modes where the shadow stack is changed while on it. For one example, SS_AUTODISARM will disable the alt shadow stack while switching to it and restore when sigreturning. At which point a new altstack can be set. In the non-shadow stack case this is nice because future signals won't clobber the alt stack if you switch away from it (swapcontext(), etc). But it also means you can "change" the alt stack while on it ("change" sort of, the auto disarm results in the kernel forgetting it temporarily).

the problem with swapcontext is that it may unmask signals that run on the alt stack, which means the code cannot jump back after another signal clobbered the alt stack.

the non-standard SS_AUTODISARM aims to solve this by disabling alt stack settings on signal entry until the handler returns.

so this use case is not about supporting swapcontext out, but about jumping back. however that does not work reliably with this patchset: if swapcontext goes to the thread stack (and not to another stack e.g. used by makecontext), then jump back fails. (and if there is a sigaltshstk installed then even jump out fails.)

assuming
- jump out from alt shadow stack can be made to work.
- alt shadow stack management can be automatic. then this can be improved so jump back works reliably.

Quoted 9 lines

I hear where you are coming from with the desire to have it "just work" with existing code, but I think the resulting ABI around the alt shadow stack allocation lifecycle would be way too complicated even if it could be made to work. Hence making a new interface. But also, the idea was that the x86 signal ABI should support handling alt shadow stacks, which is what we have done with this series. If a different interface for configuring it is better than the one from the POC, I'm not seeing a problem jump out. Is there any specific concern about backwards compatibility here?

sigaltstack syscall behaviour may be hard to change later and currently
- shadow stack overflow cannot be recovered from.
- longjmp out of signal handler fails (with sigaltshstk).
- SS_AUTODISARM does not work (jump back can fail).

Quoted 14 lines
Quoted 8 lines

"Since shadow alt stacks are a new feature, longjmp()ing from an alt shadow stack will simply not be supported. If a libc want’s to support this it will need to enable WRSS and write it’s own restore token."

i think longjmp should work without enabling writes to the shadow stack in the libc. this can also affect unwinding across signal handlers (not for c++ but e.g. glibc thread cancellation).

glibc today does not support longjmp()ing from a different stack (for example even today after a swapcontext()) when shadow stack is used. If glibc used wrss it could be supported maybe, but otherwise I don't see how the HW can support it.

Quoted 7 lines

HJ and I were actually just discussing this the other day. Are you looking at this series with respect to the arm shadow stack feature by any chance? I would love if glibc/tools would document what the shadow stack limitations are. If the all the arch's have the same or similar limitations perhaps this could be one developer guide. For the most part though, the limitations I've encountered are in glibc and the kernel is more the building blocks.

well we hope that shadow stack behaviour and limitations can be similar across targets.

longjmp to different stack should work: it can do the same as setcontext/swapcontext: scan for the pivot token. then only longjmp out of alt shadow stack fails. (this is non-conforming longjmp use, but e.g. qemu relies on it.)

for longjmp out of alt shadow stack, the target shadow stack needs a pivot token, which implies the kernel needs to push that on signal entry, which can overflow. but i suspect that can be handled the same way as stackoverflow on signal entry is handled.

Quoted 14 lines

A general comment. Not sure if you are aware, but this shadow stack enabling effort is quite old at this point and there have been many discussions on these topics stretching back years. The latest conversation was around getting this series into linux-next soon to get some testing on the MM pieces. I really appreciate getting this ABI feedback as it is always tricky to get right, but at this stage I would hope to be focusing mostly on concrete problems.

I also expect to have some amount of ABI growth going forward with all the normal things that entails. Shadow stack is not special in that it can come fully finalized without the need for the real world usage iterative feedback process. At some point we need to move forward with something, and we have quite a bit of initial changes at this point.

Quoted 6 lines

So I would like to minimize the initial implementation unless anyone sees any likely problems with future growth. Can you be clear if you see any concrete problems at this point or are more looking to evaluate the design reasoning? I'm under the assumption there is nothing that would prohibit linux-next testing while any ABI shakedown happens concurrently at least?

understood.

the points that i think are worth raising:

- shadow stack size logic may need to change later. (it can be too big, or too small in practice.)
- shadow stack overflow is not recoverable and the possible fix for that (sigaltshstk) breaks longjmp out of signal handlers.
- jump back after SS_AUTODISARM swapcontext cannot be reliable if alt signal uses thread shadow stack.
- the above two concerns may be mitigated by different sigaltstack behaviour which may be hard to add later.
- end token for backtrace may be useful, if added later it can be hard to check.

thanks.

#19 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 118 lines · On Fri, Mar 3, 2023 at 8:31 AM [email protected] <[email protected]>…
Quoted 3 lines · Rick P

Is the idea that shadow stack would be forced on regardless of if the linked libraries support it? In which case it could be allowed to crash if they do not?

execute a binary
- with shstk enabled and locked (only if marked?).
- with shstk disabled and locked.
could be managed in userspace, but it is libc dependent then.

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

- I think it's better to have a new limit specifically for shadow stack size (which by default can be RLIMIT_STACK) so userspace can adjust it if needed (another reason is that stack size is not always a good indicator of max call depth).

Looking at this again, I'm not sure why a new rlimit is needed. It seems many of those points were just formulations of that the clone3 stack size was not used, but it actually is and just not documented. If you disagree perhaps you could elaborate on what the requirements are and we can see if it seems tricky to do in a follow up.

- tiny thread stack and deep signal stack.
(note that this does not really work with glibc because it has implementation internal signals that don't run on alt stack, cannot be masked and don't fit on a tiny thread stack, but with other runtimes this can be a valid use-case, e.g. musl allows tiny thread stacks, < pagesize.)

- thread runtimes with clone (glibc uses clone3 but some dont).

- huge stacks but small call depth (problem if some va limit is hit or memory overcommit is disabled).

Quoted 4 lines
Quoted 4 lines

"sigaltshstk() is separate from sigaltstack(). You can have one without the other, neither or both together. Because the shadow stack specific state is pushed to the shadow stack, the two features don’t need to know about each other."

...

Quoted 13 lines
Quoted 3 lines

i don't see why automatic alt shadow stack allocation would not work (kernel manages it transparently when an alt stack is installed or disabled).

Ah, I think I see where maybe I can fill you in. Andy Luto had discounted this idea out of hand originally, but I didn't see it at first. sigaltstack lets you set, retrieve, or disable the shadow stack, right... But this doesn't allocate anything, it just sets where the next signal will be handled. This is different than things like threads where there is a new resources being allocated and it makes coming up with logic to guess when to de-allocate the alt shadow stack difficult. You probably already know...

Quoted 8 lines

But because of this there can be some modes where the shadow stack is changed while on it. For one example, SS_AUTODISARM will disable the alt shadow stack while switching to it and restore when sigreturning. At which point a new altstack can be set. In the non-shadow stack case this is nice because future signals won't clobber the alt stack if you switch away from it (swapcontext(), etc). But it also means you can "change" the alt stack while on it ("change" sort of, the auto disarm results in the kernel forgetting it temporarily).

the problem with swapcontext is that it may unmask signals that run on the alt stack, which means the code cannot jump back after another signal clobbered the alt stack.

the non-standard SS_AUTODISARM aims to solve this by disabling alt stack settings on signal entry until the handler returns.

so this use case is not about supporting swapcontext out, but about jumping back. however that does not work reliably with this patchset: if swapcontext goes to the thread stack (and not to another stack e.g. used by makecontext), then jump back fails. (and if there is a sigaltshstk installed then even jump out fails.)

assuming
- jump out from alt shadow stack can be made to work.
- alt shadow stack management can be automatic. then this can be improved so jump back works reliably.

Quoted 9 lines

I hear where you are coming from with the desire to have it "just work" with existing code, but I think the resulting ABI around the alt shadow stack allocation lifecycle would be way too complicated even if it could be made to work. Hence making a new interface. But also, the idea was that the x86 signal ABI should support handling alt shadow stacks, which is what we have done with this series. If a different interface for configuring it is better than the one from the POC, I'm not seeing a problem jump out. Is there any specific concern about backwards compatibility here?

sigaltstack syscall behaviour may be hard to change later and currently
- shadow stack overflow cannot be recovered from.
- longjmp out of signal handler fails (with sigaltshstk).
- SS_AUTODISARM does not work (jump back can fail).

Quoted 14 lines
Quoted 8 lines

"Since shadow alt stacks are a new feature, longjmp()ing from an alt shadow stack will simply not be supported. If a libc want’s to support this it will need to enable WRSS and write it’s own restore token."

i think longjmp should work without enabling writes to the shadow stack in the libc. this can also affect unwinding across signal handlers (not for c++ but e.g. glibc thread cancellation).

glibc today does not support longjmp()ing from a different stack (for example even today after a swapcontext()) when shadow stack is used. If glibc used wrss it could be supported maybe, but otherwise I don't see how the HW can support it.

Quoted 7 lines

HJ and I were actually just discussing this the other day. Are you looking at this series with respect to the arm shadow stack feature by any chance? I would love if glibc/tools would document what the shadow stack limitations are. If the all the arch's have the same or similar limitations perhaps this could be one developer guide. For the most part though, the limitations I've encountered are in glibc and the kernel is more the building blocks.

Quoted 7 lines

well we hope that shadow stack behaviour and limitations can be similar across targets.

longjmp to different stack should work: it can do the same as setcontext/swapcontext: scan for the pivot token. then only longjmp out of alt shadow stack fails. (this is non-conforming longjmp use, but e.g. qemu relies on it.)

Restore token may not be used with longjmp. Unlike setcontext/swapcontext, longjmp is optional. If longjmp isn't called, there will be an extra token on shadow
stack and RET will fail.

Quoted 43 lines

for longjmp out of alt shadow stack, the target shadow stack needs a pivot token, which implies the kernel needs to push that on signal entry, which can overflow. but i suspect that can be handled the same way as stackoverflow on signal entry is handled.

Quoted 14 lines

A general comment. Not sure if you are aware, but this shadow stack enabling effort is quite old at this point and there have been many discussions on these topics stretching back years. The latest conversation was around getting this series into linux-next soon to get some testing on the MM pieces. I really appreciate getting this ABI feedback as it is always tricky to get right, but at this stage I would hope to be focusing mostly on concrete problems.

I also expect to have some amount of ABI growth going forward with all the normal things that entails. Shadow stack is not special in that it can come fully finalized without the need for the real world usage iterative feedback process. At some point we need to move forward with something, and we have quite a bit of initial changes at this point.

Quoted 6 lines

So I would like to minimize the initial implementation unless anyone sees any likely problems with future growth. Can you be clear if you see any concrete problems at this point or are more looking to evaluate the design reasoning? I'm under the assumption there is nothing that would prohibit linux-next testing while any ABI shakedown happens concurrently at least?

understood.

the points that i think are worth raising:

- shadow stack size logic may need to change later. (it can be too big, or too small in practice.)
- shadow stack overflow is not recoverable and the possible fix for that (sigaltshstk) breaks longjmp out of signal handlers.
- jump back after SS_AUTODISARM swapcontext cannot be reliable if alt signal uses thread shadow stack.
- the above two concerns may be mitigated by different sigaltstack behaviour which may be hard to add later.
- end token for backtrace may be useful, if added later it can be hard to check.

thanks.

Signature · 1 line
H.J.
#20 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 11 lines · H.J. Lu
Quoted 4 lines · On Fri, Mar 3, 2023 at 8:31 AM [email protected] <[email protected]>…

longjmp to different stack should work: it can do the same as setcontext/swapcontext: scan for the pivot token. then only longjmp out of alt shadow stack fails. (this is non-conforming longjmp use, but e.g. qemu relies on it.)

Restore token may not be used with longjmp. Unlike setcontext/swapcontext, longjmp is optional. If longjmp isn't called, there will be an extra token on shadow
stack and RET will fail.

what do you mean longjmp is optional?

it can scan the target shadow stack and decide if it's the same as the current one or not and in the latter case there should be a restore token to switch to. then it can INCSSP to reach the target SSP state.

qemu does setjmp, then swapcontext, then longjmp back. swapcontext can change the stack, but leaves a token behind so longjmp can switch back.

#21 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

Quoted 9 lines · On Fri, Mar 3, 2023 at 9:40 AM [email protected] <[email protected]>…
Quoted 7 lines · H.J. Lu
Quoted 4 lines · On Fri, Mar 3, 2023 at 8:31 AM [email protected] <[email protected]>…

longjmp to different stack should work: it can do the same as setcontext/swapcontext: scan for the pivot token. then only longjmp out of alt shadow stack fails. (this is non-conforming longjmp use, but e.g. qemu relies on it.)

Quoted 6 lines
Quoted 4 lines

Restore token may not be used with longjmp. Unlike setcontext/swapcontext, longjmp is optional. If longjmp isn't called, there will be an extra token on shadow
stack and RET will fail.

what do you mean longjmp is optional?

In some cases, longjmp is called to handle an error condition and longjmp won't be called if there is no error.

Quoted 8 lines

it can scan the target shadow stack and decide if it's the same as the current one or not and in the latter case there should be a restore token to switch to. then it can INCSSP to reach the target SSP state.

qemu does setjmp, then swapcontext, then longjmp back. swapcontext can change the stack, but leaves a token behind so longjmp can switch back.

This needs changes to support shadow stack. Replacing setjmp with getcontext and longjmp with setcontext may work for shadow stack.

BTW, there is no testcase in glibc for this usage.

Signature · 1 line
H.J.
#22 lore raw

Re: [PATCH v7 01/41] Documentation/x86: Add CET shadow stack description

In reply to [email protected]
Quoted 4 lines · On Fri, 2023-03-03 at 16:30 +0000, [email protected] wrote:

the points that i think are worth raising:

- shadow stack size logic may need to change later. (it can be too big, or too small in practice.)

Looking at making it more efficient in the future seems great. But since we are not in the position of being able to make shadow stacks completely seamless (see below)

Quoted 7 lines

- shadow stack overflow is not recoverable and the possible fix for that (sigaltshstk) breaks longjmp out of signal handlers.
- jump back after SS_AUTODISARM swapcontext cannot be reliable if alt signal uses thread shadow stack.
- the above two concerns may be mitigated by different sigaltstack behaviour which may be hard to add later.

Are you aware that you can't simply emit a restore token on x86 without first restoring to another restore token? This is why (I'm assuming) glibc uses incssp to implement longjmp instead of just jumping back to the setjmp point with a shadow stack restore. So of course then longjmp can't jump between shadow stacks. So there are sort of two categories of restrictions on binaries that mark the SHSTK elf bit. The first category is that they have to take special steps when switching stacks or jumping around on the stack. Once they handle this, they can work with shadow stack.

The second category is that they can't do certain patterns of jumping around on stacks, regardless of the steps they take. So certain previously allowed software patterns are now impossible, including ones implemented in glibc. (And the exact restrictions on the glibc APIs are not documented and this should be fixed).

If applications will violate either type of these restrictions they should not mark the SHSTK elf bit.

Now that said, there is an exception to these restrictions on x86, which is the WRSS instruction, which can write to the shadow stack. The arch_prctl() interface allows this to be optionally enabled and locked. The v2 signal analysis I pointed earlier, mentions how this might be used by glibc to support more of the currently restricted patterns. Please take a look if you haven't (section "setjmp()/longjmp()"). It also explains why in the non-WRSS scenarios the kernel can't easily help improve the situation.

WRSS opens up writing to the shadow stack, and so a glibc-WRSS mode would be making a security/compatibility tradeoff. I think starting with the more restricted mode was ultimately good in creating a kernel ABI that can support both. If userspace could paper over ABI gaps with WRSS, we might not have realized the issues we did.

Quoted 2 lines

- end token for backtrace may be useful, if added
later it can be hard to check.

Yes this seems like a good idea. Thanks for the suggestion. I'm not sure it can't be added later though. I'll POC it and do some more thinking.

#23 lore raw

PATCH v7 2/41 x86/shstk: Add Kconfig option for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack provides protection for applications against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-shadow stack applications continue to work, but without protection.

Since there is another feature that utilizes CET (Kernel IBT) that will share implementation with shadow stacks, create CONFIG_CET to signify that at least one CET feature is configured.

---
v5:
- Remove capitalization of shadow stack (Boris)

v3:
- Add X86_CET (Kees)
- Add back WRUSS dependency (Kees)
- Fix verbiage (Dave)
- Change from promt to bool (Kirill)
- Add more to commit log

v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)

Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.

---
 arch/x86/Kconfig           | 24 ++++++++++++++++++++++++
 arch/x86/Kconfig.assembler |  5 +++++
 2 files changed, 29 insertions(+)
arch/x86/Kconfig +24 -0
index a825bf031f49..f03791b73f9f 100644
@@ -1851,6 +1851,11 @@ config CC_HAS_IBT
 		  (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \
 		  $(as-instr,endbr64)
 
+config X86_CET
+	def_bool n
+	help
+	  CET features configured (Shadow stack or IBT)
+
 config X86_KERNEL_IBT
 	prompt "Indirect Branch Tracking"
 	def_bool y
@@ -1858,6 +1863,7 @@ config X86_KERNEL_IBT
 	# https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f
 	depends on !LD_IS_LLD || LLD_VERSION >= 140000
 	select OBJTOOL
+	select X86_CET
 	help
 	  Build the kernel with support for Indirect Branch Tracking, a
 	  hardware support course-grain forward-edge Control Flow Integrity
@@ -1952,6 +1958,24 @@ config X86_SGX
 
 	  If unsure, say N.
 
+config X86_USER_SHADOW_STACK
+	bool "X86 userspace shadow stack"
+	depends on AS_WRUSS
+	depends on X86_64
+	select ARCH_USES_HIGH_VMA_FLAGS
+	select X86_CET
+	help
+	  Shadow stack protection is a hardware feature that detects function
+	  return address corruption.  This helps mitigate ROP attacks.
+	  Applications must be enabled to use it, and old userspace does not
+	  get protection "for free".
+
+	  CPUs supporting shadow stacks were first released in 2020.
+
+	  See Documentation/x86/shstk.rst for more information.
+
+	  If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
arch/x86/Kconfig.assembler +5 -0
index b88f784cb02e..8ad41da301e5 100644
@@ -24,3 +24,8 @@ config AS_GFNI
 	def_bool $(as-instr,vgf2p8mulb %xmm0$(comma)%xmm1$(comma)%xmm2)
 	help
 	  Supported by binutils >= 2.30 and LLVM integrated assembler
+
+config AS_WRUSS
+	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
+	help
+	  Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.17.1
#24 lore raw

PATCH v7 3/41 x86/cpufeatures: Add CPU feature flags for shadow stacks

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).

To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.

The shadow stack feature, enumerated by the CPUID bit described above, encompasses both supervisor and userspace support for shadow stack. In near future patches, only userspace shadow stack will be enabled. In expectation of future supervisor shadow stack support, create a software CPU capability to enumerate kernel utilization of userspace shadow stack support. This user shadow stack bit should depend on the HW "shstk" capability and that logic will be implemented in future patches.

---
v5:
- Drop "shstk" from cpuinfo (Boris)
- Remove capitalization on shadow stack (Boris)

v3:
- Add user specific shadow stack cpu cap (Andrew Cooper)
- Drop reviewed-bys from Boris and Kees due to the above change.

v2:
- Remove IBT reference in commit log (Kees)
- Describe xsaves dependency using text from (Dave)

v1:
- Remove IBT, can be added in a follow on IBT series.

---
 arch/x86/include/asm/cpufeatures.h       | 2 ++
 arch/x86/include/asm/disabled-features.h | 8 +++++++-
 arch/x86/kernel/cpu/cpuid-deps.c         | 1 +
 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +2 -0
index 389ea336258f..d01afabcf03e 100644
@@ -309,6 +309,7 @@
 #define X86_FEATURE_MSR_TSX_CTRL	(11*32+20) /* "" MSR IA32_TSX_CTRL (Intel) implemented */
 #define X86_FEATURE_SMBA		(11*32+21) /* "" Slow Memory Bandwidth Allocation */
 #define X86_FEATURE_BMEC		(11*32+22) /* "" Bandwidth Monitoring Event Configuration */
+#define X86_FEATURE_USER_SHSTK		(11*32+23) /* Shadow stack support for user mode applications */
 
 /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
 #define X86_FEATURE_AVX_VNNI		(12*32+ 4) /* AVX VNNI instructions */
@@ -375,6 +376,7 @@
 #define X86_FEATURE_OSPKE		(16*32+ 4) /* OS Protection Keys Enable */
 #define X86_FEATURE_WAITPKG		(16*32+ 5) /* UMONITOR/UMWAIT/TPAUSE Instructions */
 #define X86_FEATURE_AVX512_VBMI2	(16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */
+#define X86_FEATURE_SHSTK		(16*32+ 7) /* "" Shadow stack */
 #define X86_FEATURE_GFNI		(16*32+ 8) /* Galois Field New Instructions */
 #define X86_FEATURE_VAES		(16*32+ 9) /* Vector AES */
 #define X86_FEATURE_VPCLMULQDQ		(16*32+10) /* Carry-Less Multiplication Double Quadword */
arch/x86/include/asm/disabled-features.h +7 -1
index 5dfa4fb76f4b..505f78ddca82 100644
@@ -99,6 +99,12 @@
 # define DISABLE_TDX_GUEST	(1 << (X86_FEATURE_TDX_GUEST & 31))
 #endif
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+#define DISABLE_USER_SHSTK	0
+#else
+#define DISABLE_USER_SHSTK	(1 << (X86_FEATURE_USER_SHSTK & 31))
+#endif
+
 /*
  * Make sure to add features to the correct mask
  */
@@ -114,7 +120,7 @@
 #define DISABLED_MASK9	(DISABLE_SGX)
 #define DISABLED_MASK10	0
 #define DISABLED_MASK11	(DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \
-			 DISABLE_CALL_DEPTH_TRACKING)
+			 DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK)
 #define DISABLED_MASK12	0
 #define DISABLED_MASK13	0
 #define DISABLED_MASK14	0
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
index f6748c8bd647..e462c1d3800a 100644
@@ -81,6 +81,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_XFD,			X86_FEATURE_XSAVES    },
 	{ X86_FEATURE_XFD,			X86_FEATURE_XGETBV1   },
 	{ X86_FEATURE_AMX_TILE,			X86_FEATURE_XFD       },
+	{ X86_FEATURE_SHSTK,			X86_FEATURE_XSAVES    },
 	{}
 };
 
Signature · 1 line
2.17.1
#25 lore raw

PATCH v7 4/41 x86/cpufeatures: Enable CET CR4 bit for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Setting CR4.CET is a prerequisite for utilizing any CET features, most of which also require setting MSRs.

Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However, future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.

Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.

---
v5:
- Remove #ifdeffery (Boris)

v4:
- Add back dedicated command line disable: "nousershstk" (Boris)

v3:
- Remove stay new line (Boris)
- Simplify commit log (Andrew Cooper)

v2:
- In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support.
- Clear MSR_IA32_U_CET as well. (PeterZ)

---
 arch/x86/kernel/cpu/common.c | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/common.c +27 -8
index 8cd4126d8253..cc686e5039be 100644
@@ -600,27 +600,43 @@ __noendbr void ibt_restore(u64 save)
 
 static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 {
-	u64 msr = CET_ENDBR_EN;
+	bool user_shstk, kernel_ibt;
 
-	if (!HAS_KERNEL_IBT ||
-	    !cpu_feature_enabled(X86_FEATURE_IBT))
+	if (!IS_ENABLED(CONFIG_X86_CET))
 		return;
 
-	wrmsrl(MSR_IA32_S_CET, msr);
+	kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT);
+	user_shstk = cpu_feature_enabled(X86_FEATURE_SHSTK) &&
+		     IS_ENABLED(CONFIG_X86_USER_SHADOW_STACK);
+
+	if (!kernel_ibt && !user_shstk)
+		return;
+
+	if (user_shstk)
+		set_cpu_cap(c, X86_FEATURE_USER_SHSTK);
+
+	if (kernel_ibt)
+		wrmsrl(MSR_IA32_S_CET, CET_ENDBR_EN);
+	else
+		wrmsrl(MSR_IA32_S_CET, 0);
+
 	cr4_set_bits(X86_CR4_CET);
 
-	if (!ibt_selftest()) {
+	if (kernel_ibt && !ibt_selftest()) {
 		pr_err("IBT selftest: Failed!\n");
 		wrmsrl(MSR_IA32_S_CET, 0);
 		setup_clear_cpu_cap(X86_FEATURE_IBT);
-		return;
 	}
 }
 
 __noendbr void cet_disable(void)
 {
-	if (cpu_feature_enabled(X86_FEATURE_IBT))
-		wrmsrl(MSR_IA32_S_CET, 0);
+	if (!(cpu_feature_enabled(X86_FEATURE_IBT) ||
+	      cpu_feature_enabled(X86_FEATURE_SHSTK)))
+		return;
+
+	wrmsrl(MSR_IA32_S_CET, 0);
+	wrmsrl(MSR_IA32_U_CET, 0);
 }
 
 /*
@@ -1482,6 +1498,9 @@ static void __init cpu_parse_early_param(void)
 	if (cmdline_find_option_bool(boot_command_line, "noxsaves"))
 		setup_clear_cpu_cap(X86_FEATURE_XSAVES);
 
+	if (cmdline_find_option_bool(boot_command_line, "nousershstk"))
+		setup_clear_cpu_cap(X86_FEATURE_USER_SHSTK);
+
 	arglen = cmdline_find_option(boot_command_line, "clearcpuid", arg, sizeof(arg));
 	if (arglen <= 0)
 		return;
Signature · 1 line
2.17.1
#26 lore raw

PATCH v7 5/41 x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operation

The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size.

Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required.

Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable.

While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks.

Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state is a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET.

To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches.

---
v5:
- Move comments from end of lines in cet_user_state struct (Boris)

v3:
- Add missing "is" in commit log (Boris)
- Change to case statement for struct size checking (Boris)
- Adjust commas on xfeature_names (Kees, Boris)

v2:
- Change name to XFEATURE_CET_KERNEL_UNUSED (peterz)

KVM refresh:
- Reword commit log using some verbiage posted by Dave Hansen
- Remove unlikely to be used supervisor cet xsave struct
- Clarify that supervisor cet state is not saved by xsave
- Remove unused supervisor MSRs

---
 arch/x86/include/asm/fpu/types.h  | 16 +++++-
 arch/x86/include/asm/fpu/xstate.h |  6 ++-
 arch/x86/kernel/fpu/xstate.c      | 90 +++++++++++++++----------------
 3 files changed, 61 insertions(+), 51 deletions(-)
arch/x86/include/asm/fpu/types.h +14 -2
index 7f6d858ff47a..eb810074f1e7 100644
@@ -115,8 +115,8 @@ enum xfeature {
 	XFEATURE_PT_UNIMPLEMENTED_SO_FAR,
 	XFEATURE_PKRU,
 	XFEATURE_PASID,
-	XFEATURE_RSRVD_COMP_11,
-	XFEATURE_RSRVD_COMP_12,
+	XFEATURE_CET_USER,
+	XFEATURE_CET_KERNEL_UNUSED,
 	XFEATURE_RSRVD_COMP_13,
 	XFEATURE_RSRVD_COMP_14,
 	XFEATURE_LBR,
@@ -138,6 +138,8 @@ enum xfeature {
 #define XFEATURE_MASK_PT		(1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)
 #define XFEATURE_MASK_PKRU		(1 << XFEATURE_PKRU)
 #define XFEATURE_MASK_PASID		(1 << XFEATURE_PASID)
+#define XFEATURE_MASK_CET_USER		(1 << XFEATURE_CET_USER)
+#define XFEATURE_MASK_CET_KERNEL	(1 << XFEATURE_CET_KERNEL_UNUSED)
 #define XFEATURE_MASK_LBR		(1 << XFEATURE_LBR)
 #define XFEATURE_MASK_XTILE_CFG		(1 << XFEATURE_XTILE_CFG)
 #define XFEATURE_MASK_XTILE_DATA	(1 << XFEATURE_XTILE_DATA)
@@ -252,6 +254,16 @@ struct pkru_state {
 	u32				pad;
 } __packed;
 
+/*
+ * State component 11 is Control-flow Enforcement user states
+ */
+struct cet_user_state {
+	/* user control-flow settings */
+	u64 user_cet;
+	/* user shadow stack pointer */
+	u64 user_ssp;
+};
+
 /*
  * State component 15: Architectural LBR configuration state.
  * The size of Arch LBR state depends on the number of LBRs (lbr_depth).
arch/x86/include/asm/fpu/xstate.h +4 -2
index cd3dd170e23a..d4427b88ee12 100644
@@ -50,7 +50,8 @@
 #define XFEATURE_MASK_USER_DYNAMIC	XFEATURE_MASK_XTILE_DATA
 
 /* All currently supported supervisor features */
-#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)
+#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \
+					    XFEATURE_MASK_CET_USER)
 
 /*
  * A supervisor state component may not always contain valuable information,
@@ -77,7 +78,8 @@
  * Unsupported supervisor features. When a supervisor feature in this mask is
  * supported in the future, move it to the supported supervisor feature mask.
  */
-#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT)
+#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \
+					      XFEATURE_MASK_CET_KERNEL)
 
 /* All supervisor states including supported and unsupported states. */
 #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
arch/x86/kernel/fpu/xstate.c +43 -47

137 lines not on this page · raw message

Signature · 1 line
2.17.1
#27 lore raw

PATCH v7 6/41 x86/fpu: Add helper for modifying xstate

In reply to Rick Edgecombe

Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.

When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.

The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.

Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.

---
v6:
- Drop "but appear to work" (Boris)

v5:
- Fix spelling error (Boris)
- Don't export fpregs_lock_and_load() (Boris)

v3:
- Rename to fpregs_lock_and_load() to match the unlocking fpregs_unlock(). (Kees)
- Elaborate in comment about helper. (Dave)

v2:
- Drop optimization of writing directly the buffer, and change API accordingly.
- fpregs_lock_and_load() suggested by tglx
- Some commit log verbiage from dhansen

---
 arch/x86/include/asm/fpu/api.h |  9 +++++++++
 arch/x86/kernel/fpu/core.c     | 18 ++++++++++++++++++
 2 files changed, 27 insertions(+)
arch/x86/include/asm/fpu/api.h +9 -0
index 503a577814b2..aadc6893dcaa 100644
@@ -82,6 +82,15 @@ static inline void fpregs_unlock(void)
 		preempt_enable();
 }
 
+/*
+ * FPU state gets lazily restored before returning to userspace. So when in the
+ * kernel, the valid FPU state may be kept in the buffer. This function will force
+ * restore all the fpu state to the registers early if needed, and lock them from
+ * being automatically saved/restored. Then FPU state can be modified safely in the
+ * registers, before unlocking with fpregs_unlock().
+ */
+void fpregs_lock_and_load(void);
+
 #ifdef CONFIG_X86_DEBUG_FPU
 extern void fpregs_assert_state_consistent(void);
 #else
arch/x86/kernel/fpu/core.c +18 -0
index caf33486dc5e..f851558b673f 100644
@@ -753,6 +753,24 @@ void switch_fpu_return(void)
 }
 EXPORT_SYMBOL_GPL(switch_fpu_return);
 
+void fpregs_lock_and_load(void)
+{
+	/*
+	 * fpregs_lock() only disables preemption (mostly). So modifying state
+	 * in an interrupt could screw up some in progress fpregs operation.
+	 * Warn about it.
+	 */
+	WARN_ON_ONCE(!irq_fpu_usable());
+	WARN_ON_ONCE(current->flags & PF_KTHREAD);
+
+	fpregs_lock();
+
+	fpregs_assert_state_consistent();
+
+	if (test_thread_flag(TIF_NEED_FPU_LOAD))
+		fpregs_restore_userregs();
+}
+
 #ifdef CONFIG_X86_DEBUG_FPU
 /*
  * If current FPU state according to its tracking (loaded FPU context on this
Signature · 1 line
2.17.1
#28 lore raw

PATCH v7 7/41 x86: Move control protection handler to separate file

In reply to Rick Edgecombe

Today the control protection handler is defined in traps.c and used only for the kernel IBT feature. To reduce ifdeffery, move it to it's own file. In future patches, functionality will be added to make this handler also handle user shadow stack faults. So name the file cet.c.

No functional change.

---
v6:
- Split move to cet.c and shadow stack enhancements to fault handler to separate files. (Kees)

---
 arch/x86/kernel/Makefile |  2 ++
 arch/x86/kernel/cet.c    | 76 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/traps.c  | 75 ---------------------------------------
 3 files changed, 78 insertions(+), 75 deletions(-)
 create mode 100644 arch/x86/kernel/cet.c
arch/x86/kernel/Makefile +2 -0
index dd61752f4c96..92446f1dedd7 100644
@@ -144,6 +144,8 @@ obj-$(CONFIG_CFI_CLANG)			+= cfi.o
 
 obj-$(CONFIG_CALL_THUNKS)		+= callthunks.o
 
+obj-$(CONFIG_X86_CET)			+= cet.o
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/cet.c new file +76 -0

76 lines not on this page · raw message

arch/x86/kernel/traps.c +0 -75

81 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#29 lore raw

Re: [PATCH v7 07/41] x86: Move control protection handler to separate file

Quoted 1 line · Rick Edgecombe

Subject: Re: [PATCH v7 07/41] x86: Move control protection handler to separate file

x86/traps: ...

but that can be fixed while committing.

Quoted 6 lines

Today the control protection handler is defined in traps.c and used only for the kernel IBT feature. To reduce ifdeffery, move it to it's own file. In future patches, functionality will be added to make this handler also handle user shadow stack faults. So name the file cet.c.

No functional change.

...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#30 lore raw

PATCH v7 8/41 x86/shstk: Add user control-protection fault handler

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.

There already exists a control-protection fault handler for handling kernel IBT faults. Refactor this fault handler into separate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode. To avoid ifdeffery, put them both in a new file cet.c, which is compiled in the case of either of the two CET features supported in the
kernel: kernel IBT or user mode shadow stack. Move some static inline functions from traps.c into a header so they can be used in cet.c.

Opportunistically fix a comment in the kernel IBT part of the fault handler that is on the end of the line instead of preceding it.

Keep the same behavior for the kernel side of the fault handler, except for converting a BUG to a WARN in the case of a #CP happening when the feature is missing. This unifies the behavior with the new shadow stack code, and also prevents the kernel from crashing under this situation which is potentially recoverable.

The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler.

---
v7:
- Adjust alignment of WARN statement

v6:
- Split into separate patches (Kees)
- Change to "x86/shstk" in commit log (Boris)

v5:
- Move to separate file to avoid ifdeffery (Boris)
- Improvements to commit log (Boris)
- Rename control_protection_err (Boris)
- Move comment from end of line in IBT fault handler (Boris)

v3:
- Shorten user/kernel #CP handler function names (peterz)
- Restore CP_ENDBR check to kernel handler (peterz)
- Utilize CONFIG_X86_CET (Kees)
- Unify "unexpected" warnings (Andrew Cooper)
- Use 2d array for error code chars (Andrew Cooper)
- Add comment about why to read SSP MSR before enabling interrupts

v2:
- Integrate with kernel IBT fault handler
- Update printed messages. (Dave)
- Remove array_index_nospec() usage. (Dave)
- Remove IBT messages. (Dave)
- Add enclave error code bit processing it case it can get triggered somehow.
- Add extra "unknown" in control_protection_err.

---
 arch/arm/kernel/signal.c                 |  2 +-
 arch/arm64/kernel/signal.c               |  2 +-
 arch/arm64/kernel/signal32.c             |  2 +-
 arch/sparc/kernel/signal32.c             |  2 +-
 arch/sparc/kernel/signal_64.c            |  2 +-
 arch/x86/include/asm/disabled-features.h |  8 +-
 arch/x86/include/asm/idtentry.h          |  2 +-
 arch/x86/include/asm/traps.h             | 12 +++
 arch/x86/kernel/cet.c                    | 94 +++++++++++++++++++++---
 arch/x86/kernel/idt.c                    |  2 +-
 arch/x86/kernel/signal_32.c              |  2 +-
 arch/x86/kernel/signal_64.c              |  2 +-
 arch/x86/kernel/traps.c                  | 12 ---
 arch/x86/xen/enlighten_pv.c              |  2 +-
 arch/x86/xen/xen-asm.S                   |  2 +-
 include/uapi/asm-generic/siginfo.h       |  3 +-
 16 files changed, 117 insertions(+), 34 deletions(-)
arch/arm/kernel/signal.c +1 -1
index e07f359254c3..9a3c9de5ac5e 100644
@@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/arm64/kernel/signal.c +1 -1
index 06a02707f488..19b6b292892c 100644
@@ -1341,7 +1341,7 @@ void __init minsigstksz_setup(void)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/arm64/kernel/signal32.c +1 -1
index 4700f8522d27..bbd542704730 100644
@@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/sparc/kernel/signal32.c +1 -1
index dad38960d1a8..82da8a2d769d 100644
@@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/sparc/kernel/signal_64.c +1 -1
index 570e43e6fda5..b4e410976e0d 100644
@@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/x86/include/asm/disabled-features.h +7 -1
index 505f78ddca82..652e366b68a0 100644
@@ -105,6 +105,12 @@
 #define DISABLE_USER_SHSTK	(1 << (X86_FEATURE_USER_SHSTK & 31))
 #endif
 
+#ifdef CONFIG_X86_KERNEL_IBT
+#define DISABLE_IBT	0
+#else
+#define DISABLE_IBT	(1 << (X86_FEATURE_IBT & 31))
+#endif
+
 /*
  * Make sure to add features to the correct mask
  */
@@ -128,7 +134,7 @@
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
 			 DISABLE_ENQCMD)
 #define DISABLED_MASK17	0
-#define DISABLED_MASK18	0
+#define DISABLED_MASK18	(DISABLE_IBT)
 #define DISABLED_MASK19	0
 #define DISABLED_MASK20	0
 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 21)
arch/x86/include/asm/idtentry.h +1 -1
index 72184b0b2219..69e26f48d027 100644
@@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF,	xenpv_exc_double_fault);
 #endif
 
 /* #CP */
-#ifdef CONFIG_X86_KERNEL_IBT
+#ifdef CONFIG_X86_CET
 DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_CP,	exc_control_protection);
 #endif
 
arch/x86/include/asm/traps.h +12 -0
index 47ecfff2c83d..75e0dabf0c45 100644
@@ -47,4 +47,16 @@ void __noreturn handle_stack_overflow(struct pt_regs *regs,
 				      struct stack_info *info);
 #endif
 
+static inline void cond_local_irq_enable(struct pt_regs *regs)
+{
+	if (regs->flags & X86_EFLAGS_IF)
+		local_irq_enable();
+}
+
+static inline void cond_local_irq_disable(struct pt_regs *regs)
+{
+	if (regs->flags & X86_EFLAGS_IF)
+		local_irq_disable();
+}
+
 #endif /* _ASM_X86_TRAPS_H */
arch/x86/kernel/cet.c +85 -9

113 lines not on this page · raw message

arch/x86/kernel/idt.c +1 -1
index a58c6bc1cd68..5074b8420359 100644
@@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = {
 	ISTG(X86_TRAP_MC,		asm_exc_machine_check, IST_INDEX_MCE),
 #endif
 
-#ifdef CONFIG_X86_KERNEL_IBT
+#ifdef CONFIG_X86_CET
 	INTG(X86_TRAP_CP,		asm_exc_control_protection),
 #endif
 
arch/x86/kernel/signal_32.c +1 -1
index 9027fc088f97..c12624bc82a3 100644
@@ -402,7 +402,7 @@ int ia32_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)
 */
 static_assert(NSIGILL  == 11);
 static_assert(NSIGFPE  == 15);
-static_assert(NSIGSEGV == 9);
+static_assert(NSIGSEGV == 10);
 static_assert(NSIGBUS  == 5);
 static_assert(NSIGTRAP == 6);
 static_assert(NSIGCHLD == 6);
arch/x86/kernel/signal_64.c +1 -1
index 13a1e6083837..0e808c72bf7e 100644
@@ -403,7 +403,7 @@ void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact)
 */
 static_assert(NSIGILL  == 11);
 static_assert(NSIGFPE  == 15);
-static_assert(NSIGSEGV == 9);
+static_assert(NSIGSEGV == 10);
 static_assert(NSIGBUS  == 5);
 static_assert(NSIGTRAP == 6);
 static_assert(NSIGCHLD == 6);
arch/x86/kernel/traps.c +0 -12
index cc223e60aba2..18fb9d620824 100644
@@ -77,18 +77,6 @@
 
 DECLARE_BITMAP(system_vectors, NR_VECTORS);
 
-static inline void cond_local_irq_enable(struct pt_regs *regs)
-{
-	if (regs->flags & X86_EFLAGS_IF)
-		local_irq_enable();
-}
-
-static inline void cond_local_irq_disable(struct pt_regs *regs)
-{
-	if (regs->flags & X86_EFLAGS_IF)
-		local_irq_disable();
-}
-
 __always_inline int is_valid_bugaddr(unsigned long addr)
 {
 	if (addr < TASK_SIZE_MAX)
arch/x86/xen/enlighten_pv.c +1 -1
index bb59cc6ddb2d..9c29cd5393cc 100644
@@ -640,7 +640,7 @@ static struct trap_array_entry trap_array[] = {
 	TRAP_ENTRY(exc_coprocessor_error,		false ),
 	TRAP_ENTRY(exc_alignment_check,			false ),
 	TRAP_ENTRY(exc_simd_coprocessor_error,		false ),
-#ifdef CONFIG_X86_KERNEL_IBT
+#ifdef CONFIG_X86_CET
 	TRAP_ENTRY(exc_control_protection,		false ),
 #endif
 };
arch/x86/xen/xen-asm.S +1 -1
index 4a184f6e4e4d..7cdcb4ce6976 100644
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault
 xen_pv_trap asm_exc_spurious_interrupt_bug
 xen_pv_trap asm_exc_coprocessor_error
 xen_pv_trap asm_exc_alignment_check
-#ifdef CONFIG_X86_KERNEL_IBT
+#ifdef CONFIG_X86_CET
 xen_pv_trap asm_exc_control_protection
 #endif
 #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +2 -1
index ffbe4cec9f32..0f52d0ac47c5 100644
@@ -242,7 +242,8 @@ typedef struct siginfo {
 #define SEGV_ADIPERR	7	/* Precise MCD exception */
 #define SEGV_MTEAERR	8	/* Asynchronous ARM MTE error */
 #define SEGV_MTESERR	9	/* Synchronous ARM MTE exception */
-#define NSIGSEGV	9
+#define SEGV_CPERR	10	/* Control protection fault */
+#define NSIGSEGV	10
 
 /*
  * SIGBUS si_codes
Signature · 1 line
2.17.1
maintainer
#31 lore raw

Re: [PATCH v7 08/41] x86/shstk: Add user control-protection fault handler

Quoted 5 lines · Rick Edgecombe
arch/x86/kernel/cet.c +0 -0
index 7ad22b705b64..cc10d8be9d74 100644

arch/x86/kernel/cet.c line 8

Quoted 9 lines
arch/x86/kernel/cet.c +0 -4
@@ -4,10 +4,6 @@
 #include <asm/bugs.h>
 #include <asm/traps.h>
 
-static __ro_after_init bool ibt_fatal = true;
-
-extern void ibt_selftest_ip(void); /* code label defined in asm below */
-
 enum cp_error_code {
 	CP_EC        = (1 << 15) - 1,

From a previous review:

"That looks like a mask, so

CP_EC_MASK

I guess."

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#32 lore raw

Re: [PATCH v7 08/41] x86/shstk: Add user control-protection fault handler

Quoted 19 lines · Borislav Petkov
Quoted 13 lines · Rick Edgecombe
arch/x86/kernel/cet.c +0 -3
index 7ad22b705b64..cc10d8be9d74 100644
@@ -4,10 +4,6 @@
  #include <asm/bugs.h>
  #include <asm/traps.h>
 
-static __ro_after_init bool ibt_fatal = true;
-
-extern void ibt_selftest_ip(void); /* code label defined in asm

below */
-

Quoted 2 lines
  enum cp_error_code {
       CP_EC        = (1 << 15) - 1,

From a previous review:

Quoted 5 lines

"That looks like a mask, so

CP_EC_MASK

I guess."

It is from the existing code:

https://lore.kernel.org/lkml/[email protected]/#t

The rename certainly doesn't belong in the code move patch, but I took the previous discussion to mean it didn't belong in this patch either. Do you want me to add it to this one or a separate one?

maintainer
#33 lore raw

Re: [PATCH v7 08/41] x86/shstk: Add user control-protection fault handler

Quoted 1 line · Rick P

Do you want me to add it to this one or a separate one?

Don't bother. I'll fix it up myself in the future, if I notice it again. No need to disturb the set just for that.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#34 lore raw

PATCH v7 9/41 x86/mm: Remove _PAGE_DIRTY from kernel RO pages

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages.

In normal cases, it can be helpful to create Write=1 PTEs as also Dirty=1 if HW dirty tracking is not needed, because if the Dirty bit is not already set the CPU has to set Dirty=1 when the memory gets written to. This creates additional work for the CPU. So traditional wisdom was to simply set the Dirty bit whenever you didn't care about it. However, it was never really very helpful for read-only kernel memory.

When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, so avoiding kernel Write=0,Dirty=1 memory is not strictly needed for any functional reason. But having Write=0,Dirty=1 kernel memory doesn't have any functional benefit either, so to reduce ambiguity between shadow stack and regular Write=0 pages, remove Dirty=1 from any kernel Write=0 PTEs.

---
v6:
- Also remove dirty from newly added set_memory_rox()

v5:
- Spelling and grammar in commit log (Boris)

v3:
- Update commit log (Andrew Cooper, Peterz)

v2:
- Normalize PTE bit descriptions between patches

---
 arch/x86/include/asm/pgtable_types.h | 6 +++---
 arch/x86/mm/pat/set_memory.c         | 4 ++--
 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
index 447d4bee25c4..0646ad00178b 100644
@@ -192,10 +192,10 @@ enum page_cache_mode {
 #define _KERNPG_TABLE		 (__PP|__RW|   0|___A|   0|___D|   0|   0| _ENC)
 #define _PAGE_TABLE_NOENC	 (__PP|__RW|_USR|___A|   0|___D|   0|   0)
 #define _PAGE_TABLE		 (__PP|__RW|_USR|___A|   0|___D|   0|   0| _ENC)
-#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|___D|   0|___G)
-#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|___D|   0|___G)
+#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|   0|   0|___G)
+#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|   0|   0|___G)
 #define __PAGE_KERNEL_NOCACHE	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __NC)
-#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|___D|   0|___G)
+#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|   0|   0|___G)
 #define __PAGE_KERNEL_LARGE	 (__PP|__RW|   0|___A|__NX|___D|_PSE|___G)
 #define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW|   0|___A|   0|___D|_PSE|___G)
 #define __PAGE_KERNEL_WP	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __WP)
arch/x86/mm/pat/set_memory.c +2 -2
index 356758b7d4b4..1b5c0dc9f32b 100644
@@ -2073,12 +2073,12 @@ int set_memory_nx(unsigned long addr, int numpages)
 
 int set_memory_ro(unsigned long addr, int numpages)
 {
-	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0);
+	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW | _PAGE_DIRTY), 0);
 }
 
 int set_memory_rox(unsigned long addr, int numpages)
 {
-	pgprot_t clr = __pgprot(_PAGE_RW);
+	pgprot_t clr = __pgprot(_PAGE_RW | _PAGE_DIRTY);
 
 	if (__supported_pte_mask & _PAGE_NX)
 		clr.pgprot |= _PAGE_NX;
Signature · 1 line
2.17.1
#35 lore raw

PATCH v7 10/41 x86/mm: Move pmd_write(), pud_write() up in the file

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

To prepare the introduction of _PAGE_SAVED_DIRTY, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes.

---
 arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
arch/x86/include/asm/pgtable.h +12 -12
index 7425f32e5293..56eea96502c6 100644
@@ -160,6 +160,18 @@ static inline int pte_write(pte_t pte)
 	return pte_flags(pte) & _PAGE_RW;
 }
 
+#define pmd_write pmd_write
+static inline int pmd_write(pmd_t pmd)
+{
+	return pmd_flags(pmd) & _PAGE_RW;
+}
+
+#define pud_write pud_write
+static inline int pud_write(pud_t pud)
+{
+	return pud_flags(pud) & _PAGE_RW;
+}
+
 static inline int pte_huge(pte_t pte)
 {
 	return pte_flags(pte) & _PAGE_PSE;
@@ -1120,12 +1132,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma,
 				  unsigned long address, pmd_t *pmdp);
 
 
-#define pmd_write pmd_write
-static inline int pmd_write(pmd_t pmd)
-{
-	return pmd_flags(pmd) & _PAGE_RW;
-}
-
 #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR
 static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr,
 				       pmd_t *pmdp)
@@ -1155,12 +1161,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp);
 }
 
-#define pud_write pud_write
-static inline int pud_write(pud_t pud)
-{
-	return pud_flags(pud) & _PAGE_RW;
-}
-
 #ifndef pmdp_establish
 #define pmdp_establish pmdp_establish
 static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
Signature · 1 line
2.17.1
#36 lore raw

PATCH v7 11/41 mm: Introduce pte_mkwrite_kernel()

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One of these changes is to allow for pte_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pte_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pte_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pte_mkwrite_kernel() and switch the kernel users over to it. Have pte_mkwrite() and pte_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pte_mkwrite() take a VMA.

Only do this for architectures that need it because they call pte_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst.

---
Hi Non-x86 Arch’s,

x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.

Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.

Testing was just 0-day build testing.

Hopefully that is enough context. Thanks!

[0] https://lore.kernel.org/lkml/[email protected]/#t

v6:
- New patch

---
 arch/arm64/include/asm/pgtable.h | 7 ++++++-
 arch/arm64/mm/trans_pgd.c        | 4 ++--
 arch/s390/include/asm/pgtable.h  | 7 ++++++-
 arch/s390/mm/pageattr.c          | 2 +-
 arch/x86/include/asm/pgtable.h   | 7 ++++++-
 arch/x86/xen/mmu_pv.c            | 2 +-
 6 files changed, 22 insertions(+), 7 deletions(-)
arch/arm64/include/asm/pgtable.h +6 -1
index b6ba466e2e8a..cccf8885792e 100644
@@ -180,13 +180,18 @@ static inline pmd_t set_pmd_bit(pmd_t pmd, pgprot_t prot)
 	return pmd;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite_kernel(pte_t pte)
 {
 	pte = set_pte_bit(pte, __pgprot(PTE_WRITE));
 	pte = clear_pte_bit(pte, __pgprot(PTE_RDONLY));
 	return pte;
 }
 
+static inline pte_t pte_mkwrite(pte_t pte)
+{
+	return pte_mkwrite_kernel(pte);
+}
+
 static inline pte_t pte_mkclean(pte_t pte)
 {
 	pte = clear_pte_bit(pte, __pgprot(PTE_DIRTY));
arch/arm64/mm/trans_pgd.c +2 -2
index 4ea2eefbc053..5c07e68d80ea 100644
@@ -40,7 +40,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
 		 * read only (code, rodata). Clear the RDONLY bit from
 		 * the temporary mappings we use during restore.
 		 */
-		set_pte(dst_ptep, pte_mkwrite(pte));
+		set_pte(dst_ptep, pte_mkwrite_kernel(pte));
 	} else if (debug_pagealloc_enabled() && !pte_none(pte)) {
 		/*
 		 * debug_pagealloc will removed the PTE_VALID bit if
@@ -53,7 +53,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr)
 		 */
 		BUG_ON(!pfn_valid(pte_pfn(pte)));
 
-		set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte)));
+		set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_kernel(pte)));
 	}
 }
 
arch/s390/include/asm/pgtable.h +6 -1
index 2c70b4d1263d..d4943f2d3f00 100644
@@ -1005,7 +1005,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
 	return set_pte_bit(pte, __pgprot(_PAGE_PROTECT));
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite_kernel(pte_t pte)
 {
 	pte = set_pte_bit(pte, __pgprot(_PAGE_WRITE));
 	if (pte_val(pte) & _PAGE_DIRTY)
@@ -1013,6 +1013,11 @@ static inline pte_t pte_mkwrite(pte_t pte)
 	return pte;
 }
 
+static inline pte_t pte_mkwrite(pte_t pte)
+{
+	return pte_mkwrite_kernel(pte);
+}
+
 static inline pte_t pte_mkclean(pte_t pte)
 {
 	pte = clear_pte_bit(pte, __pgprot(_PAGE_DIRTY));
arch/s390/mm/pageattr.c +1 -1
index 85195c18b2e8..4ee5fe5caa23 100644
@@ -96,7 +96,7 @@ static int walk_pte_level(pmd_t *pmdp, unsigned long addr, unsigned long end,
 		if (flags & SET_MEMORY_RO)
 			new = pte_wrprotect(new);
 		else if (flags & SET_MEMORY_RW)
-			new = pte_mkwrite(pte_mkdirty(new));
+			new = pte_mkwrite_kernel(pte_mkdirty(new));
 		if (flags & SET_MEMORY_NX)
 			new = set_pte_bit(new, __pgprot(_PAGE_NOEXEC));
 		else if (flags & SET_MEMORY_X)
arch/x86/include/asm/pgtable.h +6 -1
index 56eea96502c6..3607f2572f9e 100644
@@ -364,11 +364,16 @@ static inline pte_t pte_mkyoung(pte_t pte)
 	return pte_set_flags(pte, _PAGE_ACCESSED);
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite_kernel(pte_t pte)
 {
 	return pte_set_flags(pte, _PAGE_RW);
 }
 
+static inline pte_t pte_mkwrite(pte_t pte)
+{
+	return pte_mkwrite_kernel(pte);
+}
+
 static inline pte_t pte_mkhuge(pte_t pte)
 {
 	return pte_set_flags(pte, _PAGE_PSE);
arch/x86/xen/mmu_pv.c +1 -1
index ee29fb558f2e..a23f04243c19 100644
@@ -150,7 +150,7 @@ void make_lowmem_page_readwrite(void *vaddr)
 	if (pte == NULL)
 		return;		/* vaddr missing */
 
-	ptev = pte_mkwrite(*pte);
+	ptev = pte_mkwrite_kernel(*pte);
 
 	if (HYPERVISOR_update_va_mapping(address, ptev, 0))
 		BUG();
Signature · 1 line
2.17.1
#37 lore raw

PATCH v7 12/41 s390/mm: Introduce pmd_mkwrite_kernel()

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One of these changes is to allow for pmd_mkwrite() to create different types of writable memory (the existing conventionally writable type and also the new shadow stack type). Future patches will convert pmd_mkwrite() to take a VMA in order to facilitate this, however there are places in the kernel where pmd_mkwrite() is called outside of the context of a VMA. These are for kernel memory. So create a new variant called pmd_mkwrite_kernel() and switch the kernel users over to it. Have pmd_mkwrite() and pmd_mkwrite_kernel() be the same for now. Future patches will introduce changes to make pmd_mkwrite() take a VMA.

Only do this for architectures that need it because they call pmd_mkwrite() in arch code without an associated VMA. Since it will only currently be used in arch code, so do not include it in arch_pgtable_helpers.rst.

---
Hi Non-x86 Arch’s,

x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.

Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.

Testing was just 0-day build testing.

Hopefully that is enough context. Thanks!

[0] https://lore.kernel.org/lkml/[email protected]/#t

v6:
- New patch

---
 arch/s390/include/asm/pgtable.h | 7 ++++++-
 arch/s390/mm/pageattr.c         | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)
arch/s390/include/asm/pgtable.h +6 -1
index d4943f2d3f00..deeb918cae1d 100644
@@ -1491,7 +1491,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)
 	return set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_PROTECT));
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd)
 {
 	pmd = set_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_WRITE));
 	if (pmd_val(pmd) & _SEGMENT_ENTRY_DIRTY)
@@ -1499,6 +1499,11 @@ static inline pmd_t pmd_mkwrite(pmd_t pmd)
 	return pmd;
 }
 
+static inline pmd_t pmd_mkwrite(pmd_t pmd)
+{
+	return pmd_mkwrite_kernel(pmd);
+}
+
 static inline pmd_t pmd_mkclean(pmd_t pmd)
 {
 	pmd = clear_pmd_bit(pmd, __pgprot(_SEGMENT_ENTRY_DIRTY));
arch/s390/mm/pageattr.c +1 -1
index 4ee5fe5caa23..7b6967dfacd0 100644
@@ -146,7 +146,7 @@ static void modify_pmd_page(pmd_t *pmdp, unsigned long addr,
 	if (flags & SET_MEMORY_RO)
 		new = pmd_wrprotect(new);
 	else if (flags & SET_MEMORY_RW)
-		new = pmd_mkwrite(pmd_mkdirty(new));
+		new = pmd_mkwrite_kernel(pmd_mkdirty(new));
 	if (flags & SET_MEMORY_NX)
 		new = set_pmd_bit(new, __pgprot(_SEGMENT_ENTRY_NOEXEC));
 	else if (flags & SET_MEMORY_X)
Signature · 1 line
2.17.1
#38 lore raw

PATCH v7 13/41 mm: Make pte_mkwrite() take a VMA

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call pte_mkwrite().

In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So make pte_mkwrite() take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.

Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().

No functional change.

---
Hi Non-x86 Arch’s,

x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.

Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.

Testing was just 0-day build testing.

Hopefully that is enough context. Thanks!

[0] https://lore.kernel.org/lkml/[email protected]/#t

v6:
- New patch

---
 Documentation/mm/arch_pgtable_helpers.rst    |  9 ++++++---
 arch/alpha/include/asm/pgtable.h             |  6 +++++-
 arch/arc/include/asm/hugepage.h              |  2 +-
 arch/arc/include/asm/pgtable-bits-arcv2.h    |  7 ++++++-
 arch/arm/include/asm/pgtable-3level.h        |  7 ++++++-
 arch/arm/include/asm/pgtable.h               |  2 +-
 arch/arm64/include/asm/pgtable.h             |  4 ++--
 arch/csky/include/asm/pgtable.h              |  2 +-
 arch/hexagon/include/asm/pgtable.h           |  2 +-
 arch/ia64/include/asm/pgtable.h              |  2 +-
 arch/loongarch/include/asm/pgtable.h         |  4 ++--
 arch/m68k/include/asm/mcf_pgtable.h          |  2 +-
 arch/m68k/include/asm/motorola_pgtable.h     |  6 +++++-
 arch/m68k/include/asm/sun3_pgtable.h         |  6 +++++-
 arch/microblaze/include/asm/pgtable.h        |  2 +-
 arch/mips/include/asm/pgtable.h              |  6 +++---
 arch/nios2/include/asm/pgtable.h             |  2 +-
 arch/openrisc/include/asm/pgtable.h          |  2 +-
 arch/parisc/include/asm/pgtable.h            |  6 +++++-
 arch/powerpc/include/asm/book3s/32/pgtable.h |  2 +-
 arch/powerpc/include/asm/book3s/64/pgtable.h |  4 ++--
 arch/powerpc/include/asm/nohash/32/pgtable.h |  2 +-
 arch/powerpc/include/asm/nohash/32/pte-8xx.h |  2 +-
 arch/powerpc/include/asm/nohash/64/pgtable.h |  2 +-
 arch/riscv/include/asm/pgtable.h             |  6 +++---
 arch/s390/include/asm/hugetlb.h              |  4 ++--
 arch/s390/include/asm/pgtable.h              |  4 ++--
 arch/sh/include/asm/pgtable_32.h             | 10 ++++++++--
 arch/sparc/include/asm/pgtable_32.h          |  2 +-
 arch/sparc/include/asm/pgtable_64.h          |  6 +++---
 arch/um/include/asm/pgtable.h                |  2 +-
 arch/x86/include/asm/pgtable.h               |  6 ++++--
 arch/xtensa/include/asm/pgtable.h            |  2 +-
 include/asm-generic/hugetlb.h                |  4 ++--
 include/linux/mm.h                           |  2 +-
 mm/debug_vm_pgtable.c                        | 16 ++++++++--------
 mm/huge_memory.c                             |  6 +++---
 mm/hugetlb.c                                 |  4 ++--
 mm/memory.c                                  |  4 ++--
 mm/migrate_device.c                          |  2 +-
 mm/mprotect.c                                |  2 +-
 mm/userfaultfd.c                             |  2 +-
 42 files changed, 106 insertions(+), 69 deletions(-)
Documentation/mm/arch_pgtable_helpers.rst +6 -3
index 30d9a09f01f4..78ac3ff2fe1d 100644
@@ -46,7 +46,8 @@ PTE Page Table Helpers
 +---------------------------+--------------------------------------------------+
 | pte_mkclean               | Creates a clean PTE                              |
 +---------------------------+--------------------------------------------------+
-| pte_mkwrite               | Creates a writable PTE                           |
+| pte_mkwrite               | Creates a writable PTE of the type specified by  |
+|                           | the VMA.                                         |
 +---------------------------+--------------------------------------------------+
 | pte_wrprotect             | Creates a write protected PTE                    |
 +---------------------------+--------------------------------------------------+
@@ -118,7 +119,8 @@ PMD Page Table Helpers
 +---------------------------+--------------------------------------------------+
 | pmd_mkclean               | Creates a clean PMD                              |
 +---------------------------+--------------------------------------------------+
-| pmd_mkwrite               | Creates a writable PMD                           |
+| pmd_mkwrite               | Creates a writable PMD of the type specified by  |
+|                           | the VMA.                                         |
 +---------------------------+--------------------------------------------------+
 | pmd_wrprotect             | Creates a write protected PMD                    |
 +---------------------------+--------------------------------------------------+
@@ -222,7 +224,8 @@ HugeTLB Page Table Helpers
 +---------------------------+--------------------------------------------------+
 | huge_pte_mkdirty          | Creates a dirty HugeTLB                          |
 +---------------------------+--------------------------------------------------+
-| huge_pte_mkwrite          | Creates a writable HugeTLB                       |
+| huge_pte_mkwrite          | Creates a writable HugeTLB of the type specified |
+|                           | by the VMA.                                      |
 +---------------------------+--------------------------------------------------+
 | huge_pte_wrprotect        | Creates a write protected HugeTLB                |
 +---------------------------+--------------------------------------------------+
arch/alpha/include/asm/pgtable.h +5 -1
index ba43cb841d19..fb5d207c2a89 100644
@@ -256,9 +256,13 @@ extern inline int pte_young(pte_t pte)		{ return pte_val(pte) & _PAGE_ACCESSED;
 extern inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) |= _PAGE_FOW; return pte; }
 extern inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~(__DIRTY_BITS); return pte; }
 extern inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~(__ACCESS_BITS); return pte; }
-extern inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) &= ~_PAGE_FOW; return pte; }
 extern inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= __DIRTY_BITS; return pte; }
 extern inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= __ACCESS_BITS; return pte; }
+extern inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) &= ~_PAGE_FOW;
+	return pte;
+}
 
 /*
  * The smp_rmb() in the following functions are required to order the load of
arch/arc/include/asm/hugepage.h +1 -1
index 5001b796fb8d..223a96967188 100644
@@ -21,7 +21,7 @@ static inline pmd_t pte_pmd(pte_t pte)
 }
 
 #define pmd_wrprotect(pmd)	pte_pmd(pte_wrprotect(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
 #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
 #define pmd_mkold(pmd)		pte_pmd(pte_mkold(pmd_pte(pmd)))
 #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/arc/include/asm/pgtable-bits-arcv2.h +6 -1
index 6e9f8ca6d6a1..a5b8bc955015 100644
@@ -87,7 +87,6 @@
 
 PTE_BIT_FUNC(mknotpresent,     &= ~(_PAGE_PRESENT));
 PTE_BIT_FUNC(wrprotect,	&= ~(_PAGE_WRITE));
-PTE_BIT_FUNC(mkwrite,	|= (_PAGE_WRITE));
 PTE_BIT_FUNC(mkclean,	&= ~(_PAGE_DIRTY));
 PTE_BIT_FUNC(mkdirty,	|= (_PAGE_DIRTY));
 PTE_BIT_FUNC(mkold,	&= ~(_PAGE_ACCESSED));
@@ -95,6 +94,12 @@ PTE_BIT_FUNC(mkyoung,	|= (_PAGE_ACCESSED));
 PTE_BIT_FUNC(mkspecial,	|= (_PAGE_SPECIAL));
 PTE_BIT_FUNC(mkhuge,	|= (_PAGE_HW_SZ));
 
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= (_PAGE_WRITE);
+	return pte;
+}
+
 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 {
 	return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot));
arch/arm/include/asm/pgtable-3level.h +6 -1
index 106049791500..df071a807610 100644
@@ -202,11 +202,16 @@ static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; }
 
 PMD_BIT_FUNC(wrprotect,	|= L_PMD_SECT_RDONLY);
 PMD_BIT_FUNC(mkold,	&= ~PMD_SECT_AF);
-PMD_BIT_FUNC(mkwrite,   &= ~L_PMD_SECT_RDONLY);
 PMD_BIT_FUNC(mkdirty,   |= L_PMD_SECT_DIRTY);
 PMD_BIT_FUNC(mkclean,   &= ~L_PMD_SECT_DIRTY);
 PMD_BIT_FUNC(mkyoung,   |= PMD_SECT_AF);
 
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
+{
+	pmd_val(pmd) |= L_PMD_SECT_RDONLY;
+	return pmd;
+}
+
 #define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
 
 #define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
arch/arm/include/asm/pgtable.h +1 -1
index a58ccbb406ad..39ad1ae1308d 100644
@@ -227,7 +227,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
 	return set_pte_bit(pte, __pgprot(L_PTE_RDONLY));
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return clear_pte_bit(pte, __pgprot(L_PTE_RDONLY));
 }
arch/arm64/include/asm/pgtable.h +2 -2
index cccf8885792e..913bf370f74a 100644
@@ -187,7 +187,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return pte_mkwrite_kernel(pte);
 }
@@ -492,7 +492,7 @@ static inline int pmd_trans_huge(pmd_t pmd)
 #define pmd_cont(pmd)		pte_cont(pmd_pte(pmd))
 #define pmd_wrprotect(pmd)	pte_pmd(pte_wrprotect(pmd_pte(pmd)))
 #define pmd_mkold(pmd)		pte_pmd(pte_mkold(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
 #define pmd_mkclean(pmd)	pte_pmd(pte_mkclean(pmd_pte(pmd)))
 #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
 #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/csky/include/asm/pgtable.h +1 -1
index d4042495febc..c2f92c991e37 100644
@@ -176,7 +176,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	if (pte_val(pte) & _PAGE_MODIFIED)
arch/hexagon/include/asm/pgtable.h +1 -1
index 59393613d086..14ab9c789c0e 100644
@@ -300,7 +300,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
 }
 
 /* pte_mkwrite - mark page as writable */
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	return pte;
arch/ia64/include/asm/pgtable.h +1 -1
index 21c97e31a28a..f879dd626da6 100644
@@ -268,7 +268,7 @@ ia64_phys_addr_valid (unsigned long addr)
  * access rights:
  */
 #define pte_wrprotect(pte)	(__pte(pte_val(pte) & ~_PAGE_AR_RW))
-#define pte_mkwrite(pte)	(__pte(pte_val(pte) | _PAGE_AR_RW))
+#define pte_mkwrite(pte, vma)	(__pte(pte_val(pte) | _PAGE_AR_RW))
 #define pte_mkold(pte)		(__pte(pte_val(pte) & ~_PAGE_A))
 #define pte_mkyoung(pte)	(__pte(pte_val(pte) | _PAGE_A))
 #define pte_mkclean(pte)	(__pte(pte_val(pte) & ~_PAGE_D))
arch/loongarch/include/asm/pgtable.h +2 -2
index d28fb9dbec59..ebf645f40298 100644
@@ -390,7 +390,7 @@ static inline pte_t pte_mkdirty(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	if (pte_val(pte) & _PAGE_MODIFIED)
@@ -490,7 +490,7 @@ static inline int pmd_write(pmd_t pmd)
 	return !!(pmd_val(pmd) & _PAGE_WRITE);
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	pmd_val(pmd) |= _PAGE_WRITE;
 	if (pmd_val(pmd) & _PAGE_MODIFIED)
arch/m68k/include/asm/mcf_pgtable.h +1 -1
index 13741c1245e1..37d77e055016 100644
@@ -211,7 +211,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= CF_PAGE_WRITABLE;
 	return pte;
arch/m68k/include/asm/motorola_pgtable.h +5 -1
index ec0dc19ab834..c4e8eb76286d 100644
@@ -155,7 +155,6 @@ static inline int pte_young(pte_t pte)		{ return pte_val(pte) & _PAGE_ACCESSED;
 static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) |= _PAGE_RONLY; return pte; }
 static inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~_PAGE_DIRTY; return pte; }
 static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~_PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) &= ~_PAGE_RONLY; return pte; }
 static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
 static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
 static inline pte_t pte_mknocache(pte_t pte)
@@ -168,6 +167,11 @@ static inline pte_t pte_mkcache(pte_t pte)
 	pte_val(pte) = (pte_val(pte) & _CACHEMASK040) | m68k_supervisor_cachemode;
 	return pte;
 }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) &= ~_PAGE_RONLY;
+	return pte;
+}
 
 #define swapper_pg_dir kernel_pg_dir
 extern pgd_t kernel_pg_dir[128];
arch/m68k/include/asm/sun3_pgtable.h +5 -1
index e582b0484a55..2a06bea51a1e 100644
@@ -143,10 +143,14 @@ static inline int pte_young(pte_t pte)		{ return pte_val(pte) & SUN3_PAGE_ACCESS
 static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_WRITEABLE; return pte; }
 static inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_MODIFIED; return pte; }
 static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_WRITEABLE; return pte; }
 static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_MODIFIED; return pte; }
 static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_ACCESSED; return pte; }
 static inline pte_t pte_mknocache(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_NOCACHE; return pte; }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= SUN3_PAGE_WRITEABLE;
+	return pte;
+}
 // use this version when caches work...
 //static inline pte_t pte_mkcache(pte_t pte)	{ pte_val(pte) &= SUN3_PAGE_NOCACHE; return pte; }
 // until then, use:
arch/microblaze/include/asm/pgtable.h +1 -1
index d1b8272abcd9..5b83e82f8d7e 100644
@@ -266,7 +266,7 @@ static inline pte_t pte_mkread(pte_t pte) \
 	{ pte_val(pte) |= _PAGE_USER; return pte; }
 static inline pte_t pte_mkexec(pte_t pte) \
 	{ pte_val(pte) |= _PAGE_USER | _PAGE_EXEC; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte) \
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) \
 	{ pte_val(pte) |= _PAGE_RW; return pte; }
 static inline pte_t pte_mkdirty(pte_t pte) \
 	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
arch/mips/include/asm/pgtable.h +3 -3
index 791389bf3c12..06efd567144a 100644
@@ -309,7 +309,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte.pte_low |= _PAGE_WRITE;
 	if (pte.pte_low & _PAGE_MODIFIED) {
@@ -364,7 +364,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	if (pte_val(pte) & _PAGE_MODIFIED)
@@ -626,7 +626,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)
 	return pmd;
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	pmd_val(pmd) |= _PAGE_WRITE;
 	if (pmd_val(pmd) & _PAGE_MODIFIED)
arch/nios2/include/asm/pgtable.h +1 -1
index 0f5c2564e9f5..edd458518e0e 100644
@@ -129,7 +129,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	return pte;
arch/openrisc/include/asm/pgtable.h +1 -1
index 3eb9b9555d0d..fd40aec189d1 100644
@@ -250,7 +250,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	pte_val(pte) |= _PAGE_WRITE;
 	return pte;
arch/parisc/include/asm/pgtable.h +5 -1
index e2950f5db7c9..89f62137e67f 100644
@@ -331,8 +331,12 @@ static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~_PAGE_ACCESSED; retu
 static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) &= ~_PAGE_WRITE; return pte; }
 static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
 static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) |= _PAGE_WRITE; return pte; }
 static inline pte_t pte_mkspecial(pte_t pte)	{ pte_val(pte) |= _PAGE_SPECIAL; return pte; }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= _PAGE_WRITE;
+	return pte;
+}
 
 /*
  * Huge pte definitions.
arch/powerpc/include/asm/book3s/32/pgtable.h +1 -1
index 7bf1fe7297c6..10d9a1d2aca9 100644
@@ -498,7 +498,7 @@ static inline pte_t pte_mkpte(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) | _PAGE_RW);
 }
arch/powerpc/include/asm/book3s/64/pgtable.h +2 -2
index 4acc9690f599..be0636522d36 100644
@@ -600,7 +600,7 @@ static inline pte_t pte_mkexec(pte_t pte)
 	return __pte_raw(pte_raw(pte) | cpu_to_be64(_PAGE_EXEC));
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	/*
 	 * write implies read, hence set both
@@ -1071,7 +1071,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)
 #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
 #define pmd_mkclean(pmd)	pte_pmd(pte_mkclean(pmd_pte(pmd)))
 #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
 
 #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
 #define pmd_soft_dirty(pmd)    pte_soft_dirty(pmd_pte(pmd))
arch/powerpc/include/asm/nohash/32/pgtable.h +1 -1
index fec56d965f00..7bfbcb9ba55b 100644
@@ -171,7 +171,7 @@ void unmap_kernel_page(unsigned long va);
 	do { pte_update(mm, addr, ptep, ~0, 0, 0); } while (0)
 
 #ifndef pte_mkwrite
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) | _PAGE_RW);
 }
arch/powerpc/include/asm/nohash/32/pte-8xx.h +1 -1
index 1a89ebdc3acc..f32450eb270a 100644
@@ -101,7 +101,7 @@ static inline int pte_write(pte_t pte)
 
 #define pte_write pte_write
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) & ~_PAGE_RO);
 }
arch/powerpc/include/asm/nohash/64/pgtable.h +1 -1
index 287e25864ffa..589009555877 100644
@@ -85,7 +85,7 @@
 #ifndef __ASSEMBLY__
 /* pte_clear moved to later in this file */
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) | _PAGE_RW);
 }
arch/riscv/include/asm/pgtable.h +3 -3
index d8d8de0ded99..fed1b81fbe07 100644
@@ -338,7 +338,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
 
 /* static inline pte_t pte_mkread(pte_t pte) */
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) | _PAGE_WRITE);
 }
@@ -624,9 +624,9 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 	return pte_pmd(pte_mkyoung(pmd_pte(pmd)));
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
-	return pte_pmd(pte_mkwrite(pmd_pte(pmd)));
+	return pte_pmd(pte_mkwrite(pmd_pte(pmd), vma));
 }
 
 static inline pmd_t pmd_wrprotect(pmd_t pmd)
arch/s390/include/asm/hugetlb.h +2 -2
index ccdbccfde148..558f7eef9c4d 100644
@@ -102,9 +102,9 @@ static inline int huge_pte_dirty(pte_t pte)
 	return pte_dirty(pte);
 }
 
-static inline pte_t huge_pte_mkwrite(pte_t pte)
+static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	return pte_mkwrite(pte);
+	return pte_mkwrite(pte, vma);
 }
 
 static inline pte_t huge_pte_mkdirty(pte_t pte)
arch/s390/include/asm/pgtable.h +2 -2
index deeb918cae1d..8f2c743da0eb 100644
@@ -1013,7 +1013,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
 	return pte;
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return pte_mkwrite_kernel(pte);
 }
@@ -1499,7 +1499,7 @@ static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd)
 	return pmd;
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	return pmd_mkwrite_kernel(pmd);
 }
arch/sh/include/asm/pgtable_32.h +8 -2
index 21952b094650..9f2dcb9eafc8 100644
@@ -351,6 +351,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte)
 
 #define PTE_BIT_FUNC(h,fn,op) \
 static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; }
+#define PTE_BIT_FUNC_VMA(h,fn,op) \
+static inline pte_t pte_##fn(pte_t pte, struct vm_area_struct *vma) \
+{ \
+	pte.pte_##h op; \
+	return pte; \
+}
 
 #ifdef CONFIG_X2TLB
 /*
@@ -359,11 +365,11 @@ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; }
  * kernel permissions), we attempt to couple them a bit more sanely here.
  */
 PTE_BIT_FUNC(high, wrprotect, &= ~(_PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE));
-PTE_BIT_FUNC(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE);
+PTE_BIT_FUNC_VMA(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE);
 PTE_BIT_FUNC(high, mkhuge, |= _PAGE_SZHUGE);
 #else
 PTE_BIT_FUNC(low, wrprotect, &= ~_PAGE_RW);
-PTE_BIT_FUNC(low, mkwrite, |= _PAGE_RW);
+PTE_BIT_FUNC_VMA(low, mkwrite, |= _PAGE_RW);
 PTE_BIT_FUNC(low, mkhuge, |= _PAGE_SZHUGE);
 #endif
 
arch/sparc/include/asm/pgtable_32.h +1 -1
index d4330e3c57a6..3e8836179456 100644
@@ -241,7 +241,7 @@ static inline pte_t pte_mkold(pte_t pte)
 	return __pte(pte_val(pte) & ~SRMMU_REF);
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return __pte(pte_val(pte) | SRMMU_WRITE);
 }
arch/sparc/include/asm/pgtable_64.h +3 -3
index 2dc8d4641734..c5cd5c03f557 100644
@@ -466,7 +466,7 @@ static inline pte_t pte_mkclean(pte_t pte)
 	return __pte(val);
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	unsigned long val = pte_val(pte), mask;
 
@@ -756,11 +756,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 	return __pmd(pte_val(pte));
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	pte_t pte = __pte(pmd_val(pmd));
 
-	pte = pte_mkwrite(pte);
+	pte = pte_mkwrite(pte, vma);
 
 	return __pmd(pte_val(pte));
 }
arch/um/include/asm/pgtable.h +1 -1
index a70d1618eb35..963479c133b7 100644
@@ -207,7 +207,7 @@ static inline pte_t pte_mkyoung(pte_t pte)
 	return(pte);
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	if (unlikely(pte_get_bits(pte,  _PAGE_RW)))
 		return pte;
arch/x86/include/asm/pgtable.h +4 -2
index 3607f2572f9e..66c514808276 100644
@@ -369,7 +369,9 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
 	return pte_set_flags(pte, _PAGE_RW);
 }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+struct vm_area_struct;
+
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	return pte_mkwrite_kernel(pte);
 }
@@ -470,7 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 	return pmd_set_flags(pmd, _PAGE_ACCESSED);
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	return pmd_set_flags(pmd, _PAGE_RW);
 }
arch/xtensa/include/asm/pgtable.h +1 -1
index fc7a14884c6c..d72632d9c53c 100644
@@ -262,7 +262,7 @@ static inline pte_t pte_mkdirty(pte_t pte)
 	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
 static inline pte_t pte_mkyoung(pte_t pte)
 	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 	{ pte_val(pte) |= _PAGE_WRITABLE; return pte; }
 
 #define pgprot_noncached(prot) \
include/asm-generic/hugetlb.h +2 -2
index d7f6335d3999..e86c830728de 100644
@@ -20,9 +20,9 @@ static inline unsigned long huge_pte_dirty(pte_t pte)
 	return pte_dirty(pte);
 }
 
-static inline pte_t huge_pte_mkwrite(pte_t pte)
+static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	return pte_mkwrite(pte);
+	return pte_mkwrite(pte, vma);
 }
 
 #ifndef __HAVE_ARCH_HUGE_PTE_WRPROTECT
include/linux/mm.h +1 -1
index 1f79667824eb..af652444fbba 100644
@@ -1163,7 +1163,7 @@ void free_compound_page(struct page *page);
 static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
 	if (likely(vma->vm_flags & VM_WRITE))
-		pte = pte_mkwrite(pte);
+		pte = pte_mkwrite(pte, vma);
 	return pte;
 }
 
mm/debug_vm_pgtable.c +8 -8
index af59cc7bd307..7bc5592900bc 100644
@@ -109,10 +109,10 @@ static void __init pte_basic_tests(struct pgtable_debug_args *args, int idx)
 	WARN_ON(!pte_same(pte, pte));
 	WARN_ON(!pte_young(pte_mkyoung(pte_mkold(pte))));
 	WARN_ON(!pte_dirty(pte_mkdirty(pte_mkclean(pte))));
-	WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte))));
+	WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte), args->vma)));
 	WARN_ON(pte_young(pte_mkold(pte_mkyoung(pte))));
 	WARN_ON(pte_dirty(pte_mkclean(pte_mkdirty(pte))));
-	WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte))));
+	WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte, args->vma))));
 	WARN_ON(pte_dirty(pte_wrprotect(pte_mkclean(pte))));
 	WARN_ON(!pte_dirty(pte_wrprotect(pte_mkdirty(pte))));
 }
@@ -153,7 +153,7 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args)
 	pte = pte_mkclean(pte);
 	set_pte_at(args->mm, args->vaddr, args->ptep, pte);
 	flush_dcache_page(page);
-	pte = pte_mkwrite(pte);
+	pte = pte_mkwrite(pte, args->vma);
 	pte = pte_mkdirty(pte);
 	ptep_set_access_flags(args->vma, args->vaddr, args->ptep, pte, 1);
 	pte = ptep_get(args->ptep);
@@ -199,10 +199,10 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)
 	WARN_ON(!pmd_same(pmd, pmd));
 	WARN_ON(!pmd_young(pmd_mkyoung(pmd_mkold(pmd))));
 	WARN_ON(!pmd_dirty(pmd_mkdirty(pmd_mkclean(pmd))));
-	WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd))));
+	WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd), args->vma)));
 	WARN_ON(pmd_young(pmd_mkold(pmd_mkyoung(pmd))));
 	WARN_ON(pmd_dirty(pmd_mkclean(pmd_mkdirty(pmd))));
-	WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd))));
+	WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd, args->vma))));
 	WARN_ON(pmd_dirty(pmd_wrprotect(pmd_mkclean(pmd))));
 	WARN_ON(!pmd_dirty(pmd_wrprotect(pmd_mkdirty(pmd))));
 	/*
@@ -253,7 +253,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args)
 	pmd = pmd_mkclean(pmd);
 	set_pmd_at(args->mm, vaddr, args->pmdp, pmd);
 	flush_dcache_page(page);
-	pmd = pmd_mkwrite(pmd);
+	pmd = pmd_mkwrite(pmd, args->vma);
 	pmd = pmd_mkdirty(pmd);
 	pmdp_set_access_flags(args->vma, vaddr, args->pmdp, pmd, 1);
 	pmd = READ_ONCE(*args->pmdp);
@@ -928,8 +928,8 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args)
 	pte = mk_huge_pte(page, args->page_prot);
 
 	WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte)));
-	WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte))));
-	WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte))));
+	WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte), args->vma)));
+	WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte, args->vma))));
 
 #ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB
 	pte = pfn_pte(args->fixed_pmd_pfn, args->page_prot);
mm/huge_memory.c +3 -3
index 4fc43859e59a..aaf815838144 100644
@@ -555,7 +555,7 @@ __setup("transparent_hugepage=", setup_transparent_hugepage);
 pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
 	if (likely(vma->vm_flags & VM_WRITE))
-		pmd = pmd_mkwrite(pmd);
+		pmd = pmd_mkwrite(pmd, vma);
 	return pmd;
 }
 
@@ -1580,7 +1580,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	pmd = pmd_modify(oldpmd, vma->vm_page_prot);
 	pmd = pmd_mkyoung(pmd);
 	if (writable)
-		pmd = pmd_mkwrite(pmd);
+		pmd = pmd_mkwrite(pmd, vma);
 	set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
@@ -1926,7 +1926,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	/* See change_pte_range(). */
 	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
 	    can_change_pmd_writable(vma, addr, entry))
-		entry = pmd_mkwrite(entry);
+		entry = pmd_mkwrite(entry, vma);
 
 	ret = HPAGE_PMD_NR;
 	set_pmd_at(mm, addr, pmd, entry);
mm/hugetlb.c +2 -2
index 07abcb6eb203..6af471bdcff8 100644
@@ -4900,7 +4900,7 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,
 
 	if (writable) {
 		entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page,
-					 vma->vm_page_prot)));
+					 vma->vm_page_prot)), vma);
 	} else {
 		entry = huge_pte_wrprotect(mk_huge_pte(page,
 					   vma->vm_page_prot));
@@ -4916,7 +4916,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma,
 {
 	pte_t entry;
 
-	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)));
+	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)), vma);
 	if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1))
 		update_mmu_cache(vma, address, ptep);
 }
mm/memory.c +2 -2
index f456f3b5049c..d0972d2d6f36 100644
@@ -4067,7 +4067,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 	entry = mk_pte(&folio->page, vma->vm_page_prot);
 	entry = pte_sw_mkyoung(entry);
 	if (vma->vm_flags & VM_WRITE)
-		entry = pte_mkwrite(pte_mkdirty(entry));
+		entry = pte_mkwrite(pte_mkdirty(entry), vma);
 
 	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
 			&vmf->ptl);
@@ -4755,7 +4755,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	pte = pte_modify(old_pte, vma->vm_page_prot);
 	pte = pte_mkyoung(pte);
 	if (writable)
-		pte = pte_mkwrite(pte);
+		pte = pte_mkwrite(pte, vma);
 	ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, pte);
 	update_mmu_cache(vma, vmf->address, vmf->pte);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
mm/migrate_device.c +1 -1
index d30c9de60b0d..df3f5e9d5f76 100644
@@ -646,7 +646,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
 		}
 		entry = mk_pte(page, vma->vm_page_prot);
 		if (vma->vm_flags & VM_WRITE)
-			entry = pte_mkwrite(pte_mkdirty(entry));
+			entry = pte_mkwrite(pte_mkdirty(entry), vma);
 	}
 
 	ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
mm/mprotect.c +1 -1
index 1d4843c97c2a..381163a41e88 100644
@@ -198,7 +198,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
 			    !pte_write(ptent) &&
 			    can_change_pte_writable(vma, addr, ptent))
-				ptent = pte_mkwrite(ptent);
+				ptent = pte_mkwrite(ptent, vma);
 
 			ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent);
 			if (pte_needs_flush(oldpte, ptent))
mm/userfaultfd.c +1 -1
index 53c3d916ff66..3db6f87c0aca 100644
@@ -75,7 +75,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
 	if (page_in_cache && !vm_shared)
 		writable = false;
 	if (writable)
-		_dst_pte = pte_mkwrite(_dst_pte);
+		_dst_pte = pte_mkwrite(_dst_pte, dst_vma);
 	if (wp_copy)
 		_dst_pte = pte_mkuffd_wp(_dst_pte);
 
Signature · 1 line
2.17.1
#39 lore raw

Re: [PATCH v7 13/41] mm: Make pte_mkwrite() take a VMA

Le 27/02/2023 à 23:29, Rick Edgecombe a écrit :

Quoted 11 lines

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call pte_mkwrite().

Quoted 6 lines

In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So make pte_mkwrite() take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.

Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().

I'm not sure it is a good idea to add a second argument to pte_mkwrite(). All pte_mkxxxx() only take a pte and nothing else.

I think you should do the same as commit d9ed9faac283 ("mm: add new arch_make_huge_pte() method for tile support")

Christophe

Quoted 921 lines

No functional change.

---
Hi Non-x86 Arch’s,

x86 has a feature that allows for the creation of a special type of writable memory (shadow stack) that is only writable in limited specific ways. Previously, changes were proposed to core MM code to teach it to decide when to create normally writable memory or the special shadow stack writable memory, but David Hildenbrand suggested[0] to change pXX_mkwrite() to take a VMA, so awareness of shadow stack memory can be moved into x86 code.

Since pXX_mkwrite() is defined in every arch, it requires some tree-wide changes. So that is why you are seeing some patches out of a big x86 series pop up in your arch mailing list. There is no functional change. After this refactor, the shadow stack series goes on to use the arch helpers to push shadow stack memory details inside arch/x86.

Testing was just 0-day build testing.

Hopefully that is enough context. Thanks!

[0] https://lore.kernel.org/lkml/[email protected]/#t

v6:
- New patch

---
  Documentation/mm/arch_pgtable_helpers.rst    |  9 ++++++---
  arch/alpha/include/asm/pgtable.h             |  6 +++++-
  arch/arc/include/asm/hugepage.h              |  2 +-
  arch/arc/include/asm/pgtable-bits-arcv2.h    |  7 ++++++-
  arch/arm/include/asm/pgtable-3level.h        |  7 ++++++-
  arch/arm/include/asm/pgtable.h               |  2 +-
  arch/arm64/include/asm/pgtable.h             |  4 ++--
  arch/csky/include/asm/pgtable.h              |  2 +-
  arch/hexagon/include/asm/pgtable.h           |  2 +-
  arch/ia64/include/asm/pgtable.h              |  2 +-
  arch/loongarch/include/asm/pgtable.h         |  4 ++--
  arch/m68k/include/asm/mcf_pgtable.h          |  2 +-
  arch/m68k/include/asm/motorola_pgtable.h     |  6 +++++-
  arch/m68k/include/asm/sun3_pgtable.h         |  6 +++++-
  arch/microblaze/include/asm/pgtable.h        |  2 +-
  arch/mips/include/asm/pgtable.h              |  6 +++---
  arch/nios2/include/asm/pgtable.h             |  2 +-
  arch/openrisc/include/asm/pgtable.h          |  2 +-
  arch/parisc/include/asm/pgtable.h            |  6 +++++-
  arch/powerpc/include/asm/book3s/32/pgtable.h |  2 +-
  arch/powerpc/include/asm/book3s/64/pgtable.h |  4 ++--
  arch/powerpc/include/asm/nohash/32/pgtable.h |  2 +-
  arch/powerpc/include/asm/nohash/32/pte-8xx.h |  2 +-
  arch/powerpc/include/asm/nohash/64/pgtable.h |  2 +-
  arch/riscv/include/asm/pgtable.h             |  6 +++---
  arch/s390/include/asm/hugetlb.h              |  4 ++--
  arch/s390/include/asm/pgtable.h              |  4 ++--
  arch/sh/include/asm/pgtable_32.h             | 10 ++++++++--
  arch/sparc/include/asm/pgtable_32.h          |  2 +-
  arch/sparc/include/asm/pgtable_64.h          |  6 +++---
  arch/um/include/asm/pgtable.h                |  2 +-
  arch/x86/include/asm/pgtable.h               |  6 ++++--
  arch/xtensa/include/asm/pgtable.h            |  2 +-
  include/asm-generic/hugetlb.h                |  4 ++--
  include/linux/mm.h                           |  2 +-
  mm/debug_vm_pgtable.c                        | 16 ++++++++--------
  mm/huge_memory.c                             |  6 +++---
  mm/hugetlb.c                                 |  4 ++--
  mm/memory.c                                  |  4 ++--
  mm/migrate_device.c                          |  2 +-
  mm/mprotect.c                                |  2 +-
  mm/userfaultfd.c                             |  2 +-
  42 files changed, 106 insertions(+), 69 deletions(-)
Documentation/mm/arch_pgtable_helpers.rst +6 -3
index 30d9a09f01f4..78ac3ff2fe1d 100644
@@ -46,7 +46,8 @@ PTE Page Table Helpers
  +---------------------------+--------------------------------------------------+
  | pte_mkclean               | Creates a clean PTE                              |
  +---------------------------+--------------------------------------------------+
-| pte_mkwrite               | Creates a writable PTE                           |
+| pte_mkwrite               | Creates a writable PTE of the type specified by  |
+|                           | the VMA.                                         |
  +---------------------------+--------------------------------------------------+
  | pte_wrprotect             | Creates a write protected PTE                    |
  +---------------------------+--------------------------------------------------+
@@ -118,7 +119,8 @@ PMD Page Table Helpers
  +---------------------------+--------------------------------------------------+
  | pmd_mkclean               | Creates a clean PMD                              |
  +---------------------------+--------------------------------------------------+
-| pmd_mkwrite               | Creates a writable PMD                           |
+| pmd_mkwrite               | Creates a writable PMD of the type specified by  |
+|                           | the VMA.                                         |
  +---------------------------+--------------------------------------------------+
  | pmd_wrprotect             | Creates a write protected PMD                    |
  +---------------------------+--------------------------------------------------+
@@ -222,7 +224,8 @@ HugeTLB Page Table Helpers
  +---------------------------+--------------------------------------------------+
  | huge_pte_mkdirty          | Creates a dirty HugeTLB                          |
  +---------------------------+--------------------------------------------------+
-| huge_pte_mkwrite          | Creates a writable HugeTLB                       |
+| huge_pte_mkwrite          | Creates a writable HugeTLB of the type specified |
+|                           | by the VMA.                                      |
  +---------------------------+--------------------------------------------------+
  | huge_pte_wrprotect        | Creates a write protected HugeTLB                |
  +---------------------------+--------------------------------------------------+
arch/alpha/include/asm/pgtable.h +5 -1
index ba43cb841d19..fb5d207c2a89 100644
@@ -256,9 +256,13 @@ extern inline int pte_young(pte_t pte)		{ return pte_val(pte) & _PAGE_ACCESSED;
  extern inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) |= _PAGE_FOW; return pte; }
  extern inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~(__DIRTY_BITS); return pte; }
  extern inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~(__ACCESS_BITS); return pte; }
-extern inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) &= ~_PAGE_FOW; return pte; }
  extern inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= __DIRTY_BITS; return pte; }
  extern inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= __ACCESS_BITS; return pte; }
+extern inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) &= ~_PAGE_FOW;
+	return pte;
+}
 
  /*
   * The smp_rmb() in the following functions are required to order the load of
arch/arc/include/asm/hugepage.h +1 -1
index 5001b796fb8d..223a96967188 100644
@@ -21,7 +21,7 @@ static inline pmd_t pte_pmd(pte_t pte)
  }
 
  #define pmd_wrprotect(pmd)	pte_pmd(pte_wrprotect(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
  #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
  #define pmd_mkold(pmd)		pte_pmd(pte_mkold(pmd_pte(pmd)))
  #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/arc/include/asm/pgtable-bits-arcv2.h +6 -1
index 6e9f8ca6d6a1..a5b8bc955015 100644
@@ -87,7 +87,6 @@
 
  PTE_BIT_FUNC(mknotpresent,     &= ~(_PAGE_PRESENT));
  PTE_BIT_FUNC(wrprotect,	&= ~(_PAGE_WRITE));
-PTE_BIT_FUNC(mkwrite,	|= (_PAGE_WRITE));
  PTE_BIT_FUNC(mkclean,	&= ~(_PAGE_DIRTY));
  PTE_BIT_FUNC(mkdirty,	|= (_PAGE_DIRTY));
  PTE_BIT_FUNC(mkold,	&= ~(_PAGE_ACCESSED));
@@ -95,6 +94,12 @@ PTE_BIT_FUNC(mkyoung,	|= (_PAGE_ACCESSED));
  PTE_BIT_FUNC(mkspecial,	|= (_PAGE_SPECIAL));
  PTE_BIT_FUNC(mkhuge,	|= (_PAGE_HW_SZ));
 
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= (_PAGE_WRITE);
+	return pte;
+}
+
  static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
  {
  	return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot));
arch/arm/include/asm/pgtable-3level.h +6 -1
index 106049791500..df071a807610 100644
@@ -202,11 +202,16 @@ static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; }
 
  PMD_BIT_FUNC(wrprotect,	|= L_PMD_SECT_RDONLY);
  PMD_BIT_FUNC(mkold,	&= ~PMD_SECT_AF);
-PMD_BIT_FUNC(mkwrite,   &= ~L_PMD_SECT_RDONLY);
  PMD_BIT_FUNC(mkdirty,   |= L_PMD_SECT_DIRTY);
  PMD_BIT_FUNC(mkclean,   &= ~L_PMD_SECT_DIRTY);
  PMD_BIT_FUNC(mkyoung,   |= PMD_SECT_AF);
 
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
+{
+	pmd_val(pmd) |= L_PMD_SECT_RDONLY;
+	return pmd;
+}
+
  #define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
 
  #define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
arch/arm/include/asm/pgtable.h +1 -1
index a58ccbb406ad..39ad1ae1308d 100644
@@ -227,7 +227,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
  	return set_pte_bit(pte, __pgprot(L_PTE_RDONLY));
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return clear_pte_bit(pte, __pgprot(L_PTE_RDONLY));
  }
arch/arm64/include/asm/pgtable.h +2 -2
index cccf8885792e..913bf370f74a 100644
@@ -187,7 +187,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return pte_mkwrite_kernel(pte);
  }
@@ -492,7 +492,7 @@ static inline int pmd_trans_huge(pmd_t pmd)
  #define pmd_cont(pmd)		pte_cont(pmd_pte(pmd))
  #define pmd_wrprotect(pmd)	pte_pmd(pte_wrprotect(pmd_pte(pmd)))
  #define pmd_mkold(pmd)		pte_pmd(pte_mkold(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
  #define pmd_mkclean(pmd)	pte_pmd(pte_mkclean(pmd_pte(pmd)))
  #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
  #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
arch/csky/include/asm/pgtable.h +1 -1
index d4042495febc..c2f92c991e37 100644
@@ -176,7 +176,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	if (pte_val(pte) & _PAGE_MODIFIED)
arch/hexagon/include/asm/pgtable.h +1 -1
index 59393613d086..14ab9c789c0e 100644
@@ -300,7 +300,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
  }
 
  /* pte_mkwrite - mark page as writable */
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	return pte;
arch/ia64/include/asm/pgtable.h +1 -1
index 21c97e31a28a..f879dd626da6 100644
@@ -268,7 +268,7 @@ ia64_phys_addr_valid (unsigned long addr)
   * access rights:
   */
  #define pte_wrprotect(pte)	(__pte(pte_val(pte) & ~_PAGE_AR_RW))
-#define pte_mkwrite(pte)	(__pte(pte_val(pte) | _PAGE_AR_RW))
+#define pte_mkwrite(pte, vma)	(__pte(pte_val(pte) | _PAGE_AR_RW))
  #define pte_mkold(pte)		(__pte(pte_val(pte) & ~_PAGE_A))
  #define pte_mkyoung(pte)	(__pte(pte_val(pte) | _PAGE_A))
  #define pte_mkclean(pte)	(__pte(pte_val(pte) & ~_PAGE_D))
arch/loongarch/include/asm/pgtable.h +2 -2
index d28fb9dbec59..ebf645f40298 100644
@@ -390,7 +390,7 @@ static inline pte_t pte_mkdirty(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	if (pte_val(pte) & _PAGE_MODIFIED)
@@ -490,7 +490,7 @@ static inline int pmd_write(pmd_t pmd)
  	return !!(pmd_val(pmd) & _PAGE_WRITE);
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	pmd_val(pmd) |= _PAGE_WRITE;
  	if (pmd_val(pmd) & _PAGE_MODIFIED)
arch/m68k/include/asm/mcf_pgtable.h +1 -1
index 13741c1245e1..37d77e055016 100644
@@ -211,7 +211,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= CF_PAGE_WRITABLE;
  	return pte;
arch/m68k/include/asm/motorola_pgtable.h +5 -1
index ec0dc19ab834..c4e8eb76286d 100644
@@ -155,7 +155,6 @@ static inline int pte_young(pte_t pte)		{ return pte_val(pte) & _PAGE_ACCESSED;
  static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) |= _PAGE_RONLY; return pte; }
  static inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~_PAGE_DIRTY; return pte; }
  static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~_PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) &= ~_PAGE_RONLY; return pte; }
  static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
  static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
  static inline pte_t pte_mknocache(pte_t pte)
@@ -168,6 +167,11 @@ static inline pte_t pte_mkcache(pte_t pte)
  	pte_val(pte) = (pte_val(pte) & _CACHEMASK040) | m68k_supervisor_cachemode;
  	return pte;
  }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) &= ~_PAGE_RONLY;
+	return pte;
+}
 
  #define swapper_pg_dir kernel_pg_dir
  extern pgd_t kernel_pg_dir[128];
arch/m68k/include/asm/sun3_pgtable.h +5 -1
index e582b0484a55..2a06bea51a1e 100644
@@ -143,10 +143,14 @@ static inline int pte_young(pte_t pte)		{ return pte_val(pte) & SUN3_PAGE_ACCESS
  static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_WRITEABLE; return pte; }
  static inline pte_t pte_mkclean(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_MODIFIED; return pte; }
  static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~SUN3_PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_WRITEABLE; return pte; }
  static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_MODIFIED; return pte; }
  static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_ACCESSED; return pte; }
  static inline pte_t pte_mknocache(pte_t pte)	{ pte_val(pte) |= SUN3_PAGE_NOCACHE; return pte; }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= SUN3_PAGE_WRITEABLE;
+	return pte;
+}
  // use this version when caches work...
  //static inline pte_t pte_mkcache(pte_t pte)	{ pte_val(pte) &= SUN3_PAGE_NOCACHE; return pte; }
  // until then, use:
arch/microblaze/include/asm/pgtable.h +1 -1
index d1b8272abcd9..5b83e82f8d7e 100644
@@ -266,7 +266,7 @@ static inline pte_t pte_mkread(pte_t pte) \
  	{ pte_val(pte) |= _PAGE_USER; return pte; }
  static inline pte_t pte_mkexec(pte_t pte) \
  	{ pte_val(pte) |= _PAGE_USER | _PAGE_EXEC; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte) \
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma) \
  	{ pte_val(pte) |= _PAGE_RW; return pte; }
  static inline pte_t pte_mkdirty(pte_t pte) \
  	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
arch/mips/include/asm/pgtable.h +3 -3
index 791389bf3c12..06efd567144a 100644
@@ -309,7 +309,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte.pte_low |= _PAGE_WRITE;
  	if (pte.pte_low & _PAGE_MODIFIED) {
@@ -364,7 +364,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	if (pte_val(pte) & _PAGE_MODIFIED)
@@ -626,7 +626,7 @@ static inline pmd_t pmd_wrprotect(pmd_t pmd)
  	return pmd;
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	pmd_val(pmd) |= _PAGE_WRITE;
  	if (pmd_val(pmd) & _PAGE_MODIFIED)
arch/nios2/include/asm/pgtable.h +1 -1
index 0f5c2564e9f5..edd458518e0e 100644
@@ -129,7 +129,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	return pte;
arch/openrisc/include/asm/pgtable.h +1 -1
index 3eb9b9555d0d..fd40aec189d1 100644
@@ -250,7 +250,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	pte_val(pte) |= _PAGE_WRITE;
  	return pte;
arch/parisc/include/asm/pgtable.h +5 -1
index e2950f5db7c9..89f62137e67f 100644
@@ -331,8 +331,12 @@ static inline pte_t pte_mkold(pte_t pte)	{ pte_val(pte) &= ~_PAGE_ACCESSED; retu
  static inline pte_t pte_wrprotect(pte_t pte)	{ pte_val(pte) &= ~_PAGE_WRITE; return pte; }
  static inline pte_t pte_mkdirty(pte_t pte)	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
  static inline pte_t pte_mkyoung(pte_t pte)	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)	{ pte_val(pte) |= _PAGE_WRITE; return pte; }
  static inline pte_t pte_mkspecial(pte_t pte)	{ pte_val(pte) |= _PAGE_SPECIAL; return pte; }
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	pte_val(pte) |= _PAGE_WRITE;
+	return pte;
+}
 
  /*
   * Huge pte definitions.
arch/powerpc/include/asm/book3s/32/pgtable.h +1 -1
index 7bf1fe7297c6..10d9a1d2aca9 100644
@@ -498,7 +498,7 @@ static inline pte_t pte_mkpte(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) | _PAGE_RW);
  }
arch/powerpc/include/asm/book3s/64/pgtable.h +2 -2
index 4acc9690f599..be0636522d36 100644
@@ -600,7 +600,7 @@ static inline pte_t pte_mkexec(pte_t pte)
  	return __pte_raw(pte_raw(pte) | cpu_to_be64(_PAGE_EXEC));
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	/*
  	 * write implies read, hence set both
@@ -1071,7 +1071,7 @@ static inline pte_t *pmdp_ptep(pmd_t *pmd)
  #define pmd_mkdirty(pmd)	pte_pmd(pte_mkdirty(pmd_pte(pmd)))
  #define pmd_mkclean(pmd)	pte_pmd(pte_mkclean(pmd_pte(pmd)))
  #define pmd_mkyoung(pmd)	pte_pmd(pte_mkyoung(pmd_pte(pmd)))
-#define pmd_mkwrite(pmd)	pte_pmd(pte_mkwrite(pmd_pte(pmd)))
+#define pmd_mkwrite(pmd, vma)	pte_pmd(pte_mkwrite(pmd_pte(pmd), (vma)))
 
  #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
  #define pmd_soft_dirty(pmd)    pte_soft_dirty(pmd_pte(pmd))
arch/powerpc/include/asm/nohash/32/pgtable.h +1 -1
index fec56d965f00..7bfbcb9ba55b 100644
@@ -171,7 +171,7 @@ void unmap_kernel_page(unsigned long va);
  	do { pte_update(mm, addr, ptep, ~0, 0, 0); } while (0)
 
  #ifndef pte_mkwrite
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) | _PAGE_RW);
  }
arch/powerpc/include/asm/nohash/32/pte-8xx.h +1 -1
index 1a89ebdc3acc..f32450eb270a 100644
@@ -101,7 +101,7 @@ static inline int pte_write(pte_t pte)
 
  #define pte_write pte_write
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) & ~_PAGE_RO);
  }
arch/powerpc/include/asm/nohash/64/pgtable.h +1 -1
index 287e25864ffa..589009555877 100644
@@ -85,7 +85,7 @@
  #ifndef __ASSEMBLY__
  /* pte_clear moved to later in this file */
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) | _PAGE_RW);
  }
arch/riscv/include/asm/pgtable.h +3 -3
index d8d8de0ded99..fed1b81fbe07 100644
@@ -338,7 +338,7 @@ static inline pte_t pte_wrprotect(pte_t pte)
 
  /* static inline pte_t pte_mkread(pte_t pte) */
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) | _PAGE_WRITE);
  }
@@ -624,9 +624,9 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
  	return pte_pmd(pte_mkyoung(pmd_pte(pmd)));
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
-	return pte_pmd(pte_mkwrite(pmd_pte(pmd)));
+	return pte_pmd(pte_mkwrite(pmd_pte(pmd), vma));
  }
 
  static inline pmd_t pmd_wrprotect(pmd_t pmd)
arch/s390/include/asm/hugetlb.h +2 -2
index ccdbccfde148..558f7eef9c4d 100644
@@ -102,9 +102,9 @@ static inline int huge_pte_dirty(pte_t pte)
  	return pte_dirty(pte);
  }
 
-static inline pte_t huge_pte_mkwrite(pte_t pte)
+static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
-	return pte_mkwrite(pte);
+	return pte_mkwrite(pte, vma);
  }
 
  static inline pte_t huge_pte_mkdirty(pte_t pte)
arch/s390/include/asm/pgtable.h +2 -2
index deeb918cae1d..8f2c743da0eb 100644
@@ -1013,7 +1013,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
  	return pte;
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return pte_mkwrite_kernel(pte);
  }
@@ -1499,7 +1499,7 @@ static inline pmd_t pmd_mkwrite_kernel(pmd_t pmd)
  	return pmd;
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	return pmd_mkwrite_kernel(pmd);
  }
arch/sh/include/asm/pgtable_32.h +8 -2
index 21952b094650..9f2dcb9eafc8 100644
@@ -351,6 +351,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte)
 
  #define PTE_BIT_FUNC(h,fn,op) \
  static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; }
+#define PTE_BIT_FUNC_VMA(h,fn,op) \
+static inline pte_t pte_##fn(pte_t pte, struct vm_area_struct *vma) \
+{ \
+	pte.pte_##h op; \
+	return pte; \
+}
 
  #ifdef CONFIG_X2TLB
  /*
@@ -359,11 +365,11 @@ static inline pte_t pte_##fn(pte_t pte) { pte.pte_##h op; return pte; }
   * kernel permissions), we attempt to couple them a bit more sanely here.
   */
  PTE_BIT_FUNC(high, wrprotect, &= ~(_PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE));
-PTE_BIT_FUNC(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE);
+PTE_BIT_FUNC_VMA(high, mkwrite, |= _PAGE_EXT_USER_WRITE | _PAGE_EXT_KERN_WRITE);
  PTE_BIT_FUNC(high, mkhuge, |= _PAGE_SZHUGE);
  #else
  PTE_BIT_FUNC(low, wrprotect, &= ~_PAGE_RW);
-PTE_BIT_FUNC(low, mkwrite, |= _PAGE_RW);
+PTE_BIT_FUNC_VMA(low, mkwrite, |= _PAGE_RW);
  PTE_BIT_FUNC(low, mkhuge, |= _PAGE_SZHUGE);
  #endif
 
arch/sparc/include/asm/pgtable_32.h +1 -1
index d4330e3c57a6..3e8836179456 100644
@@ -241,7 +241,7 @@ static inline pte_t pte_mkold(pte_t pte)
  	return __pte(pte_val(pte) & ~SRMMU_REF);
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return __pte(pte_val(pte) | SRMMU_WRITE);
  }
arch/sparc/include/asm/pgtable_64.h +3 -3
index 2dc8d4641734..c5cd5c03f557 100644
@@ -466,7 +466,7 @@ static inline pte_t pte_mkclean(pte_t pte)
  	return __pte(val);
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	unsigned long val = pte_val(pte), mask;
 
@@ -756,11 +756,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
  	return __pmd(pte_val(pte));
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	pte_t pte = __pte(pmd_val(pmd));
 
-	pte = pte_mkwrite(pte);
+	pte = pte_mkwrite(pte, vma);
 
  	return __pmd(pte_val(pte));
  }
arch/um/include/asm/pgtable.h +1 -1
index a70d1618eb35..963479c133b7 100644
@@ -207,7 +207,7 @@ static inline pte_t pte_mkyoung(pte_t pte)
  	return(pte);
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	if (unlikely(pte_get_bits(pte,  _PAGE_RW)))
  		return pte;
arch/x86/include/asm/pgtable.h +4 -2
index 3607f2572f9e..66c514808276 100644
@@ -369,7 +369,9 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
  	return pte_set_flags(pte, _PAGE_RW);
  }
 
-static inline pte_t pte_mkwrite(pte_t pte)
+struct vm_area_struct;
+
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	return pte_mkwrite_kernel(pte);
  }
@@ -470,7 +472,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
  	return pmd_set_flags(pmd, _PAGE_ACCESSED);
  }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd)
+static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	return pmd_set_flags(pmd, _PAGE_RW);
  }
arch/xtensa/include/asm/pgtable.h +1 -1
index fc7a14884c6c..d72632d9c53c 100644
@@ -262,7 +262,7 @@ static inline pte_t pte_mkdirty(pte_t pte)
  	{ pte_val(pte) |= _PAGE_DIRTY; return pte; }
  static inline pte_t pte_mkyoung(pte_t pte)
  	{ pte_val(pte) |= _PAGE_ACCESSED; return pte; }
-static inline pte_t pte_mkwrite(pte_t pte)
+static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  	{ pte_val(pte) |= _PAGE_WRITABLE; return pte; }
 
  #define pgprot_noncached(prot) \
include/asm-generic/hugetlb.h +2 -2
index d7f6335d3999..e86c830728de 100644
@@ -20,9 +20,9 @@ static inline unsigned long huge_pte_dirty(pte_t pte)
  	return pte_dirty(pte);
  }
 
-static inline pte_t huge_pte_mkwrite(pte_t pte)
+static inline pte_t huge_pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
-	return pte_mkwrite(pte);
+	return pte_mkwrite(pte, vma);
  }
 
  #ifndef __HAVE_ARCH_HUGE_PTE_WRPROTECT
include/linux/mm.h +1 -1
index 1f79667824eb..af652444fbba 100644
@@ -1163,7 +1163,7 @@ void free_compound_page(struct page *page);
  static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
  {
  	if (likely(vma->vm_flags & VM_WRITE))
-		pte = pte_mkwrite(pte);
+		pte = pte_mkwrite(pte, vma);
  	return pte;
  }
 
mm/debug_vm_pgtable.c +8 -8
index af59cc7bd307..7bc5592900bc 100644
@@ -109,10 +109,10 @@ static void __init pte_basic_tests(struct pgtable_debug_args *args, int idx)
  	WARN_ON(!pte_same(pte, pte));
  	WARN_ON(!pte_young(pte_mkyoung(pte_mkold(pte))));
  	WARN_ON(!pte_dirty(pte_mkdirty(pte_mkclean(pte))));
-	WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte))));
+	WARN_ON(!pte_write(pte_mkwrite(pte_wrprotect(pte), args->vma)));
  	WARN_ON(pte_young(pte_mkold(pte_mkyoung(pte))));
  	WARN_ON(pte_dirty(pte_mkclean(pte_mkdirty(pte))));
-	WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte))));
+	WARN_ON(pte_write(pte_wrprotect(pte_mkwrite(pte, args->vma))));
  	WARN_ON(pte_dirty(pte_wrprotect(pte_mkclean(pte))));
  	WARN_ON(!pte_dirty(pte_wrprotect(pte_mkdirty(pte))));
  }
@@ -153,7 +153,7 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args)
  	pte = pte_mkclean(pte);
  	set_pte_at(args->mm, args->vaddr, args->ptep, pte);
  	flush_dcache_page(page);
-	pte = pte_mkwrite(pte);
+	pte = pte_mkwrite(pte, args->vma);
  	pte = pte_mkdirty(pte);
  	ptep_set_access_flags(args->vma, args->vaddr, args->ptep, pte, 1);
  	pte = ptep_get(args->ptep);
@@ -199,10 +199,10 @@ static void __init pmd_basic_tests(struct pgtable_debug_args *args, int idx)
  	WARN_ON(!pmd_same(pmd, pmd));
  	WARN_ON(!pmd_young(pmd_mkyoung(pmd_mkold(pmd))));
  	WARN_ON(!pmd_dirty(pmd_mkdirty(pmd_mkclean(pmd))));
-	WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd))));
+	WARN_ON(!pmd_write(pmd_mkwrite(pmd_wrprotect(pmd), args->vma)));
  	WARN_ON(pmd_young(pmd_mkold(pmd_mkyoung(pmd))));
  	WARN_ON(pmd_dirty(pmd_mkclean(pmd_mkdirty(pmd))));
-	WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd))));
+	WARN_ON(pmd_write(pmd_wrprotect(pmd_mkwrite(pmd, args->vma))));
  	WARN_ON(pmd_dirty(pmd_wrprotect(pmd_mkclean(pmd))));
  	WARN_ON(!pmd_dirty(pmd_wrprotect(pmd_mkdirty(pmd))));
  	/*
@@ -253,7 +253,7 @@ static void __init pmd_advanced_tests(struct pgtable_debug_args *args)
  	pmd = pmd_mkclean(pmd);
  	set_pmd_at(args->mm, vaddr, args->pmdp, pmd);
  	flush_dcache_page(page);
-	pmd = pmd_mkwrite(pmd);
+	pmd = pmd_mkwrite(pmd, args->vma);
  	pmd = pmd_mkdirty(pmd);
  	pmdp_set_access_flags(args->vma, vaddr, args->pmdp, pmd, 1);
  	pmd = READ_ONCE(*args->pmdp);
@@ -928,8 +928,8 @@ static void __init hugetlb_basic_tests(struct pgtable_debug_args *args)
  	pte = mk_huge_pte(page, args->page_prot);
 
  	WARN_ON(!huge_pte_dirty(huge_pte_mkdirty(pte)));
-	WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte))));
-	WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte))));
+	WARN_ON(!huge_pte_write(huge_pte_mkwrite(huge_pte_wrprotect(pte), args->vma)));
+	WARN_ON(huge_pte_write(huge_pte_wrprotect(huge_pte_mkwrite(pte, args->vma))));
 
  #ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB
  	pte = pfn_pte(args->fixed_pmd_pfn, args->page_prot);
mm/huge_memory.c +3 -3
index 4fc43859e59a..aaf815838144 100644
@@ -555,7 +555,7 @@ __setup("transparent_hugepage=", setup_transparent_hugepage);
  pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
  {
  	if (likely(vma->vm_flags & VM_WRITE))
-		pmd = pmd_mkwrite(pmd);
+		pmd = pmd_mkwrite(pmd, vma);
  	return pmd;
  }
 
@@ -1580,7 +1580,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	pmd = pmd_modify(oldpmd, vma->vm_page_prot);
  	pmd = pmd_mkyoung(pmd);
  	if (writable)
-		pmd = pmd_mkwrite(pmd);
+		pmd = pmd_mkwrite(pmd, vma);
  	set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
  	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
  	spin_unlock(vmf->ptl);
@@ -1926,7 +1926,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
  	/* See change_pte_range(). */
  	if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) && !pmd_write(entry) &&
  	    can_change_pmd_writable(vma, addr, entry))
-		entry = pmd_mkwrite(entry);
+		entry = pmd_mkwrite(entry, vma);
 
  	ret = HPAGE_PMD_NR;
  	set_pmd_at(mm, addr, pmd, entry);
mm/hugetlb.c +2 -2
index 07abcb6eb203..6af471bdcff8 100644
@@ -4900,7 +4900,7 @@ static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,
 
  	if (writable) {
  		entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page,
-					 vma->vm_page_prot)));
+					 vma->vm_page_prot)), vma);
  	} else {
  		entry = huge_pte_wrprotect(mk_huge_pte(page,
  					   vma->vm_page_prot));
@@ -4916,7 +4916,7 @@ static void set_huge_ptep_writable(struct vm_area_struct *vma,
  {
  	pte_t entry;
 
-	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)));
+	entry = huge_pte_mkwrite(huge_pte_mkdirty(huge_ptep_get(ptep)), vma);
  	if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1))
  		update_mmu_cache(vma, address, ptep);
  }
mm/memory.c +2 -2
index f456f3b5049c..d0972d2d6f36 100644
@@ -4067,7 +4067,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
  	entry = mk_pte(&folio->page, vma->vm_page_prot);
  	entry = pte_sw_mkyoung(entry);
  	if (vma->vm_flags & VM_WRITE)
-		entry = pte_mkwrite(pte_mkdirty(entry));
+		entry = pte_mkwrite(pte_mkdirty(entry), vma);
 
  	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
  			&vmf->ptl);
@@ -4755,7 +4755,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	pte = pte_modify(old_pte, vma->vm_page_prot);
  	pte = pte_mkyoung(pte);
  	if (writable)
-		pte = pte_mkwrite(pte);
+		pte = pte_mkwrite(pte, vma);
  	ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, pte);
  	update_mmu_cache(vma, vmf->address, vmf->pte);
  	pte_unmap_unlock(vmf->pte, vmf->ptl);
mm/migrate_device.c +1 -1
index d30c9de60b0d..df3f5e9d5f76 100644
@@ -646,7 +646,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
  		}
  		entry = mk_pte(page, vma->vm_page_prot);
  		if (vma->vm_flags & VM_WRITE)
-			entry = pte_mkwrite(pte_mkdirty(entry));
+			entry = pte_mkwrite(pte_mkdirty(entry), vma);
  	}
 
  	ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
mm/mprotect.c +1 -1
index 1d4843c97c2a..381163a41e88 100644
@@ -198,7 +198,7 @@ static long change_pte_range(struct mmu_gather *tlb,
  			if ((cp_flags & MM_CP_TRY_CHANGE_WRITABLE) &&
  			    !pte_write(ptent) &&
  			    can_change_pte_writable(vma, addr, ptent))
-				ptent = pte_mkwrite(ptent);
+				ptent = pte_mkwrite(ptent, vma);
 
  			ptep_modify_prot_commit(vma, addr, pte, oldpte, ptent);
  			if (pte_needs_flush(oldpte, ptent))
mm/userfaultfd.c +1 -1
index 53c3d916ff66..3db6f87c0aca 100644
@@ -75,7 +75,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
  	if (page_in_cache && !vm_shared)
  		writable = false;
  	if (writable)
-		_dst_pte = pte_mkwrite(_dst_pte);
+		_dst_pte = pte_mkwrite(_dst_pte, dst_vma);
  	if (wp_copy)
  		_dst_pte = pte_mkuffd_wp(_dst_pte);
 
#40 lore raw

Re: [PATCH v7 13/41] mm: Make pte_mkwrite() take a VMA

Quoted 19 lines · Christophe Leroy

Le 27/02/2023 à 23:29, Rick Edgecombe a écrit :

Quoted 16 lines

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One of these unusual properties is that shadow stack memory is writable, but only in limited ways. These limits are applied via a specific PTE bit combination. Nevertheless, the memory is writable, and core mm code will need to apply the writable permissions in the typical paths that call pte_mkwrite().

In addition to VM_WRITE, the shadow stack VMA's will have a flag denoting that they are special shadow stack flavor of writable memory. So make pte_mkwrite() take a VMA, so that the x86 implementation of it can know to create regular writable memory or shadow stack memory.

Quoted 4 lines
Quoted 1 line

Apply the same changes for pmd_mkwrite() and huge_pte_mkwrite().

I'm not sure it is a good idea to add a second argument to pte_mkwrite(). All pte_mkxxxx() only take a pte and nothing else.

We touched on this in previous revisions and so far there was no strong push back. This turned out to be cleaner and easier than the alternatives we evaluated.

pte_modify(), for example, takes another argument. Sure, we could try thinking about passing something else than a VMA to identify the writability type, but I am not convinced that will look particularly better.

Quoted 4 lines

I think you should do the same as commit d9ed9faac283 ("mm: add new arch_make_huge_pte() method for tile support")

We already have 3 architectures intending to support shadow stacks in one way or the other. Replacing all pte_mkwrite() with arch_pte_mkwrite() doesn't sound particularly appealing to me.

Signature · 9 lines
Thanks,

David / dhildenb


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#42 lore raw

PATCH v7 14/41 x86/mm: Introduce _PAGE_SAVED_DIRTY

In reply to Rick Edgecombe

Some OSes have a greater dependence on software available bits in PTEs than Linux. That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1. So in order to support shadow stack memory, Linux should avoid creating memory with this PTE bit combination unless it intends for it to be shadow stack.

The reason it's lightly used is that Dirty=1 is normally set by HW _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Dirty=1 *and* generate the fault, resulting in a Write=0,Dirty=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.

So that leaves Write=0,Dirty=1 PTEs created in software. To avoid inadvertently created shadow stack memory, in places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_SAVED_DIRTY in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,SavedDirty=1 except for shadow stack, which is Write=0,Dirty=1.

There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_SAVED_DIRTY. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.

Implement only the infrastructure for _PAGE_SAVED_DIRTY. Changes to actually begin creating _PAGE_SAVED_DIRTY PTEs will follow once other pieces are in place.

---
v7:
- Use lightly edited comment verbiage from (David Hildenbrand)
- Update commit log to reduce verbosity (David Hildenbrand)

v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Add _PAGE_SAVED_DIRTY to _PAGE_CHG_MASK

v5:
- Fix log, comments and whitespace (Boris)
- Remove capitalization on shadow stack (Boris)

v4:
- Teach pte_flags_need_flush() about _PAGE_COW bit
- Break apart patch for better bisectability

v3:
- Add comment around _PAGE_TABLE in response to comment from (Andrew Cooper)
- Check for PSE in pmd_shstk (Andrew Cooper)
- Get to the point quicker in commit log (Andrew Cooper)
- Clarify and reorder commit log for why the PTE bit examples have multiple entries. Apply same changes for comment. (peterz)
- Fix comment that implied dirty bit for COW was a specific x86 thing (peterz)
- Fix swapping of Write/Dirty (PeterZ)

---
 arch/x86/include/asm/pgtable.h       | 79 ++++++++++++++++++++++++++++
 arch/x86/include/asm/pgtable_types.h | 50 +++++++++++++++---
 arch/x86/include/asm/tlbflush.h      |  3 +-
 3 files changed, 123 insertions(+), 9 deletions(-)
arch/x86/include/asm/pgtable.h +79 -0
index 66c514808276..7360783f2140 100644
@@ -301,6 +301,45 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)
 	return native_make_pte(v & ~clear);
 }
 
+/*
+ * Write protection operations can result in Dirty=1,Write=0 PTEs. But in the
+ * case of X86_FEATURE_USER_SHSTK, the software SavedDirty bit is used, since
+ * the Dirty=1,Write=0 will result in the memory being treated as shadow stack
+ * by the HW. So when creating dirty, write-protected memory, a software bit is
+ * used _PAGE_BIT_SAVED_DIRTY. The following functions pte_mksaveddirty() and
+ * pte_clear_saveddirty() take a conventional dirty, write-protected PTE
+ * (Write=0,Dirty=1) and transition it to the shadow stack compatible
+ * version. (Write=0,SavedDirty=1).
+ */
+static inline pte_t pte_mksaveddirty(pte_t pte)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pte;
+
+	pte = pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_set_flags(pte, _PAGE_SAVED_DIRTY);
+}
+
+static inline pte_t pte_clear_saveddirty(pte_t pte)
+{
+	/*
+	 * _PAGE_SAVED_DIRTY is unnecessary on !X86_FEATURE_USER_SHSTK kernels,
+	 * since the HW dirty bit can be used without creating shadow stack
+	 * memory. See the _PAGE_SAVED_DIRTY definition for more details.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pte;
+
+	/*
+	 * PTE is getting copied-on-write, so it will be dirtied
+	 * if writable, or made shadow stack if shadow stack and
+	 * being copied on access. Set the dirty bit for both
+	 * cases.
+	 */
+	pte = pte_set_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_SAVED_DIRTY);
+}
+
 static inline pte_t pte_wrprotect(pte_t pte)
 {
 	return pte_clear_flags(pte, _PAGE_RW);
@@ -420,6 +459,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)
 	return native_make_pmd(v & ~clear);
 }
 
+/* See comments above pte_mksaveddirty() */
+static inline pmd_t pmd_mksaveddirty(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pmd;
+
+	pmd = pmd_clear_flags(pmd, _PAGE_DIRTY);
+	return pmd_set_flags(pmd, _PAGE_SAVED_DIRTY);
+}
+
+/* See comments above pte_mksaveddirty() */
+static inline pmd_t pmd_clear_saveddirty(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pmd;
+
+	pmd = pmd_set_flags(pmd, _PAGE_DIRTY);
+	return pmd_clear_flags(pmd, _PAGE_SAVED_DIRTY);
+}
+
 static inline pmd_t pmd_wrprotect(pmd_t pmd)
 {
 	return pmd_clear_flags(pmd, _PAGE_RW);
@@ -491,6 +550,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)
 	return native_make_pud(v & ~clear);
 }
 
+/* See comments above pte_mksaveddirty() */
+static inline pud_t pud_mksaveddirty(pud_t pud)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pud;
+
+	pud = pud_clear_flags(pud, _PAGE_DIRTY);
+	return pud_set_flags(pud, _PAGE_SAVED_DIRTY);
+}
+
+/* See comments above pte_mksaveddirty() */
+static inline pud_t pud_clear_saveddirty(pud_t pud)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return pud;
+
+	pud = pud_set_flags(pud, _PAGE_DIRTY);
+	return pud_clear_flags(pud, _PAGE_SAVED_DIRTY);
+}
+
 static inline pud_t pud_mkold(pud_t pud)
 {
 	return pud_clear_flags(pud, _PAGE_ACCESSED);
arch/x86/include/asm/pgtable_types.h +42 -8

82 lines not on this page · raw message

arch/x86/include/asm/tlbflush.h +2 -1
index cda3118f3b27..6c5ef14060a8 100644
@@ -273,7 +273,8 @@ static inline bool pte_flags_need_flush(unsigned long oldflags,
 	const pteval_t flush_on_clear = _PAGE_DIRTY | _PAGE_PRESENT |
 					_PAGE_ACCESSED;
 	const pteval_t software_flags = _PAGE_SOFTW1 | _PAGE_SOFTW2 |
-					_PAGE_SOFTW3 | _PAGE_SOFTW4;
+					_PAGE_SOFTW3 | _PAGE_SOFTW4 |
+					_PAGE_SAVED_DIRTY;
 	const pteval_t flush_on_change = _PAGE_RW | _PAGE_USER | _PAGE_PWT |
 			  _PAGE_PCD | _PAGE_PSE | _PAGE_GLOBAL | _PAGE_PAT |
 			  _PAGE_PAT_LARGE | _PAGE_PKEY_BIT0 | _PAGE_PKEY_BIT1 |
Signature · 1 line
2.17.1
maintainer
#43 lore raw

Re: [PATCH v7 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY

Quoted 33 lines · Rick Edgecombe
arch/x86/include/asm/pgtable_types.h +11 -1
index 0646ad00178b..56b374d1bffb 100644
@@ -21,7 +21,8 @@
 #define _PAGE_BIT_SOFTW2	10	/* " */
 #define _PAGE_BIT_SOFTW3	11	/* " */
 #define _PAGE_BIT_PAT_LARGE	12	/* On 2MB or 1GB pages */
-#define _PAGE_BIT_SOFTW4	58	/* available for programmer */
+#define _PAGE_BIT_SOFTW4	57	/* available for programmer */
+#define _PAGE_BIT_SOFTW5	58	/* available for programmer */
 #define _PAGE_BIT_PKEY_BIT0	59	/* Protection Keys, bit 1/4 */
 #define _PAGE_BIT_PKEY_BIT1	60	/* Protection Keys, bit 2/4 */
 #define _PAGE_BIT_PKEY_BIT2	61	/* Protection Keys, bit 3/4 */
@@ -34,6 +35,15 @@
 #define _PAGE_BIT_SOFT_DIRTY	_PAGE_BIT_SOFTW3 /* software dirty tracking */
 #define _PAGE_BIT_DEVMAP	_PAGE_BIT_SOFTW4
 
+/*
+ * Indicates a Saved Dirty bit page.
+ */
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+#define _PAGE_BIT_SAVED_DIRTY		_PAGE_BIT_SOFTW5 /* Saved Dirty bit */
+#else
+#define _PAGE_BIT_SAVED_DIRTY		0
+#endif
+
 /* If _PAGE_BIT_PRESENT is clear, we use these: */
 /* - if the user mapped it with PROT_NONE; pte_present gives true */
 #define _PAGE_BIT_PROTNONE	_PAGE_BIT_GLOBAL
@@ -117,6 +127,25 @@
 #define _PAGE_SOFTW4	(_AT(pteval_t, 0))
 #endif

arch/x86/include/asm/pgtable_types.h line 138

Quoted 10 lines
arch/x86/include/asm/pgtable_types.h +9 -0
 
+/*
+ * The hardware requires shadow stack to be Write=0,Dirty=1. However,
+ * there are valid cases where the kernel might create read-only PTEs that
+ * are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty  tracking). In
+ * this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW-dirty bit,
+ * to avoid creating a wrong "shadow stack" PTEs. Such PTEs have
+ * (Write=0,SavedDirty=1,Dirty=0) set.
+ *
+ * Note that on processors without shadow stack support, the 

.git/rebase-apply/patch:154: trailing whitespace.
* Note that on processors without shadow stack support, the
warning: 1 line adds whitespace errors.

Hm, apparently git checks for that too - not only trailing empty lines.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#44 lore raw

Re: [PATCH v7 14/41] x86/mm: Introduce _PAGE_SAVED_DIRTY

Quoted 63 lines · Borislav Petkov
Quoted 49 lines · Rick Edgecombe
arch/x86/include/asm/pgtable_types.h +2 -1
index 0646ad00178b..56b374d1bffb 100644
@@ -21,7 +21,8 @@
  #define _PAGE_BIT_SOFTW2     10      /* " */
  #define _PAGE_BIT_SOFTW3     11      /* " */
  #define _PAGE_BIT_PAT_LARGE  12      /* On 2MB or 1GB pages */
-#define _PAGE_BIT_SOFTW4     58      /* available for programmer
 */
+#define _PAGE_BIT_SOFTW4     57      /* available for programmer
 */
+#define _PAGE_BIT_SOFTW5     58      /* available for programmer
 */

#define _PAGE_BIT_PKEY_BIT0 59 /* Protection Keys, bit 1/4
*/
#define _PAGE_BIT_PKEY_BIT1 60 /* Protection Keys, bit 2/4
*/
#define _PAGE_BIT_PKEY_BIT2 61 /* Protection Keys, bit 3/4
*/

arch/x86/include/asm/pgtable_types.h +0 -0
@@ -34,6 +35,15 @@
  #define _PAGE_BIT_SOFT_DIRTY _PAGE_BIT_SOFTW3 /* software dirty

tracking */
#define _PAGE_BIT_DEVMAP _PAGE_BIT_SOFTW4

arch/x86/include/asm/pgtable_types.h +5 -0
+/*
+ * Indicates a Saved Dirty bit page.
+ */
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+#define _PAGE_BIT_SAVED_DIRTY                _PAGE_BIT_SOFTW5 /*

Saved Dirty bit */

arch/x86/include/asm/pgtable_types.h +4 -0
+#else
+#define _PAGE_BIT_SAVED_DIRTY                0
+#endif
+
  /* If _PAGE_BIT_PRESENT is clear, we use these: */
  /* - if the user mapped it with PROT_NONE; pte_present gives true

*/
#define _PAGE_BIT_PROTNONE _PAGE_BIT_GLOBAL

arch/x86/include/asm/pgtable_types.h +3 -0
@@ -117,6 +127,25 @@
  #define _PAGE_SOFTW4 (_AT(pteval_t, 0))
  #endif
 
+/*
+ * The hardware requires shadow stack to be Write=0,Dirty=1.
 However,
+ * there are valid cases where the kernel might create read-only

PTEs that

Quoted 8 lines

+ * are dirty (e.g., fork(), mprotect(), uffd-wp(), soft-dirty tracking). In
+ * this case, the _PAGE_SAVED_DIRTY bit is used instead of the HW- dirty bit,
+ * to avoid creating a wrong "shadow stack" PTEs. Such PTEs have
+ * (Write=0,SavedDirty=1,Dirty=0) set.
+ *
+ * Note that on processors without shadow stack support, the

.git/rebase-apply/patch:154: trailing whitespace.
* Note that on processors without shadow stack support, the
warning: 1 line adds whitespace errors.

Quoted 2 lines

Hm, apparently git checks for that too - not only trailing empty lines.

Weird. And oops on the space. Just wondering how checkpatch missed this. It didn't, just was in a pile of false positives on that patch and I didn't notice it in there.

#45 lore raw

PATCH v7 15/41 x86/mm: Update ptep/pmdp_set_wrprotect() for _PAGE_SAVED_DIRTY

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When shadow stack is in use, Write=0,Dirty=1 PTE are preserved for shadow stack. Copy-on-write PTEs then have Write=0,SavedDirty=1.

When a PTE goes from Write=1,Dirty=1 to Write=0,SavedDirty=1, it could become a transient shadow stack PTE in two cases:

1. Some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting shadow stack, and a TLB flush is not necessary.

2. When _PAGE_DIRTY is replaced with _PAGE_SAVED_DIRTY non-atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.

In the case of pmdp_set_wrprotect(), for nopmd configs the ->pmd operated on does not exist and the logic would need to be different. Although the extra functionality will normally be optimized out when user shadow stacks are not configured, also exclude it in the preprocessor stage so that it will still compile. User shadow stack is not supported there by Linux anyway. Leave the cpu_feature_enabled() check so that the functionality also gets disabled based on runtime detection of the feature.

Similarly, compile it out in ptep_set_wrprotect() due to a clang warning on i386. Like above, the code path should get optimized out on i386 since shadow stack is not supported on 32 bit kernels, but this makes the compiler happy.

Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.

---
v6:
- Fix comment and log to update for _PAGE_COW being replaced with _PAGE_SAVED_DIRTY.

v5:
- Commit log verbiage and formatting (Boris)
- Remove capitalization on shadow stack (Boris)
- Fix i386 warning on recent clang

v3:
- Remove unnecessary #ifdef (Dave Hansen)

v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)

---
 arch/x86/include/asm/pgtable.h | 35 ++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
arch/x86/include/asm/pgtable.h +35 -0
index 7360783f2140..349fcab0405a 100644
@@ -1192,6 +1192,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 static inline void ptep_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pte_t *ptep)
 {
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) {
+		pte_t old_pte, new_pte;
+
+		old_pte = READ_ONCE(*ptep);
+		do {
+			new_pte = pte_wrprotect(old_pte);
+		} while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte));
+
+		return;
+	}
+#endif
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte);
 }
 
@@ -1244,6 +1261,24 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
 static inline void pmdp_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pmd_t *pmdp)
 {
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK)) {
+		pmd_t old_pmd, new_pmd;
+
+		old_pmd = READ_ONCE(*pmdp);
+		do {
+			new_pmd = pmd_wrprotect(old_pmd);
+		} while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd));
+
+		return;
+	}
+#endif
+
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp);
 }
 
Signature · 1 line
2.17.1
#46 lore raw

PATCH v7 16/41 x86/mm: Start actually marking _PAGE_SAVED_DIRTY

In reply to Rick Edgecombe

The recently introduced _PAGE_SAVED_DIRTY should be used instead of the HW Dirty bit whenever a PTE is Write=0, in order to not inadvertently create shadow stack PTEs. Update pte_mk*() helpers to do this, and apply the same changes to pmd and pud.

For pte_modify() this is a bit trickier. It takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one.

Modify it to also move _PAGE_DIRTY to _PAGE_SAVED_DIRTY. To avoid creating Write=0,Dirty=1 PTEs, pte_modify() needs to avoid:
1. Marking Write=0 PTEs Dirty=1
2. Marking Dirty=1 PTEs Write=0

The first case cannot happen as the existing behavior of pte_modify() is to filter out any Dirty bit passed in newprot. Handle the second case by shifting _PAGE_DIRTY=1 to _PAGE_SAVED_DIRTY=1 if the PTE was write protected by the pte_modify() call. Apply the same changes to pmd_modify().

---
v6:
- Rename _PAGE_COW to _PAGE_SAVED_DIRTY (David Hildenbrand)
- Open code _PAGE_SAVED_DIRTY part in pte_modify() (Boris)
- Change the logic so the open coded part is not too ugly
- Merge pte_modify() patch with this one because of the above

v4:
- Break part patch for better bisectability

---
 arch/x86/include/asm/pgtable.h | 168 ++++++++++++++++++++++++++++-----
 1 file changed, 145 insertions(+), 23 deletions(-)
arch/x86/include/asm/pgtable.h +145 -23
index 349fcab0405a..05dfdbdf96b4 100644
@@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags;
  * The following only work if pte_present() is true.
  * Undefined behaviour if not..
  */
-static inline int pte_dirty(pte_t pte)
+static inline bool pte_dirty(pte_t pte)
 {
-	return pte_flags(pte) & _PAGE_DIRTY;
+	return pte_flags(pte) & _PAGE_DIRTY_BITS;
+}
+
+static inline bool pte_shstk(pte_t pte)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return false;
+
+	return (pte_flags(pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY;
 }
 
 static inline int pte_young(pte_t pte)
@@ -134,9 +142,18 @@ static inline int pte_young(pte_t pte)
 	return pte_flags(pte) & _PAGE_ACCESSED;
 }
 
-static inline int pmd_dirty(pmd_t pmd)
+static inline bool pmd_dirty(pmd_t pmd)
 {
-	return pmd_flags(pmd) & _PAGE_DIRTY;
+	return pmd_flags(pmd) & _PAGE_DIRTY_BITS;
+}
+
+static inline bool pmd_shstk(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		return false;
+
+	return (pmd_flags(pmd) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) ==
+	       (_PAGE_DIRTY | _PAGE_PSE);
 }
 
 #define pmd_young pmd_young
@@ -145,9 +162,9 @@ static inline int pmd_young(pmd_t pmd)
 	return pmd_flags(pmd) & _PAGE_ACCESSED;
 }
 
-static inline int pud_dirty(pud_t pud)
+static inline bool pud_dirty(pud_t pud)
 {
-	return pud_flags(pud) & _PAGE_DIRTY;
+	return pud_flags(pud) & _PAGE_DIRTY_BITS;
 }
 
 static inline int pud_young(pud_t pud)
@@ -157,13 +174,21 @@ static inline int pud_young(pud_t pud)
 
 static inline int pte_write(pte_t pte)
 {
-	return pte_flags(pte) & _PAGE_RW;
+	/*
+	 * Shadow stack pages are logically writable, but do not have
+	 * _PAGE_RW.  Check for them separately from _PAGE_RW itself.
+	 */
+	return (pte_flags(pte) & _PAGE_RW) || pte_shstk(pte);
 }
 
 #define pmd_write pmd_write
 static inline int pmd_write(pmd_t pmd)
 {
-	return pmd_flags(pmd) & _PAGE_RW;
+	/*
+	 * Shadow stack pages are logically writable, but do not have
+	 * _PAGE_RW.  Check for them separately from _PAGE_RW itself.
+	 */
+	return (pmd_flags(pmd) & _PAGE_RW) || pmd_shstk(pmd);
 }
 
 #define pud_write pud_write
@@ -342,7 +367,16 @@ static inline pte_t pte_clear_saveddirty(pte_t pte)
 
 static inline pte_t pte_wrprotect(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_RW);
+	pte = pte_clear_flags(pte, _PAGE_RW);
+
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PTE (Write=0,Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pte_dirty(pte))
+		pte = pte_mksaveddirty(pte);
+	return pte;
 }
 
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
@@ -380,7 +414,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)
 
 static inline pte_t pte_mkclean(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_DIRTY_BITS);
 }
 
 static inline pte_t pte_mkold(pte_t pte)
@@ -395,7 +429,19 @@ static inline pte_t pte_mkexec(pte_t pte)
 
 static inline pte_t pte_mkdirty(pte_t pte)
 {
-	return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pteval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating Dirty=1,Write=0 PTEs */
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pte_write(pte))
+		dirty = _PAGE_SAVED_DIRTY;
+
+	return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY);
+}
+
+static inline pte_t pte_mkwrite_shstk(pte_t pte)
+{
+	/* pte_clear_saveddirty() also sets Dirty=1 */
+	return pte_clear_saveddirty(pte);
 }
 
 static inline pte_t pte_mkyoung(pte_t pte)
@@ -412,7 +458,12 @@ struct vm_area_struct;
 
 static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	return pte_mkwrite_kernel(pte);
+	pte = pte_mkwrite_kernel(pte);
+
+	if (pte_dirty(pte))
+		pte = pte_clear_saveddirty(pte);
+
+	return pte;
 }
 
 static inline pte_t pte_mkhuge(pte_t pte)
@@ -481,7 +532,15 @@ static inline pmd_t pmd_clear_saveddirty(pmd_t pmd)
 
 static inline pmd_t pmd_wrprotect(pmd_t pmd)
 {
-	return pmd_clear_flags(pmd, _PAGE_RW);
+	pmd = pmd_clear_flags(pmd, _PAGE_RW);
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PMD (RW=0, Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pmd_dirty(pmd))
+		pmd = pmd_mksaveddirty(pmd);
+	return pmd;
 }
 
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
@@ -508,12 +567,23 @@ static inline pmd_t pmd_mkold(pmd_t pmd)
 
 static inline pmd_t pmd_mkclean(pmd_t pmd)
 {
-	return pmd_clear_flags(pmd, _PAGE_DIRTY);
+	return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS);
 }
 
 static inline pmd_t pmd_mkdirty(pmd_t pmd)
 {
-	return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pmdval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating (HW)Dirty=1, Write=0 PMDs */
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pmd_write(pmd))
+		dirty = _PAGE_SAVED_DIRTY;
+
+	return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY);
+}
+
+static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd)
+{
+	return pmd_clear_saveddirty(pmd);
 }
 
 static inline pmd_t pmd_mkdevmap(pmd_t pmd)
@@ -533,7 +603,12 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 
 static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
-	return pmd_set_flags(pmd, _PAGE_RW);
+	pmd = pmd_set_flags(pmd, _PAGE_RW);
+
+	if (pmd_dirty(pmd))
+		pmd = pmd_clear_saveddirty(pmd);
+
+	return pmd;
 }
 
 static inline pud_t pud_set_flags(pud_t pud, pudval_t set)
@@ -577,17 +652,32 @@ static inline pud_t pud_mkold(pud_t pud)
 
 static inline pud_t pud_mkclean(pud_t pud)
 {
-	return pud_clear_flags(pud, _PAGE_DIRTY);
+	return pud_clear_flags(pud, _PAGE_DIRTY_BITS);
 }
 
 static inline pud_t pud_wrprotect(pud_t pud)
 {
-	return pud_clear_flags(pud, _PAGE_RW);
+	pud = pud_clear_flags(pud, _PAGE_RW);
+
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PUD (RW=0, Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pud_dirty(pud))
+		pud = pud_mksaveddirty(pud);
+	return pud;
 }
 
 static inline pud_t pud_mkdirty(pud_t pud)
 {
-	return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pudval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating (HW)Dirty=1, Write=0 PUDs */
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && !pud_write(pud))
+		dirty = _PAGE_SAVED_DIRTY;
+
+	return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY);
 }
 
 static inline pud_t pud_mkdevmap(pud_t pud)
@@ -607,7 +697,11 @@ static inline pud_t pud_mkyoung(pud_t pud)
 
 static inline pud_t pud_mkwrite(pud_t pud)
 {
-	return pud_set_flags(pud, _PAGE_RW);
+	pud = pud_set_flags(pud, _PAGE_RW);
+
+	if (pud_dirty(pud))
+		pud = pud_clear_saveddirty(pud);
+	return pud;
 }
 
 #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
@@ -724,6 +818,8 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask);
 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 {
 	pteval_t val = pte_val(pte), oldval = val;
+	bool wr_protected;
+	pte_t pte_result;
 
 	/*
 	 * Chop off the NX bit (if present), and add the NX portion of
@@ -732,17 +828,43 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 	val &= _PAGE_CHG_MASK;
 	val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);
-	return __pte(val);
+
+	pte_result = __pte(val);
+
+	/*
+	 * Do the saveddirty fixup if the PTE was just write protected and
+	 * it's dirty.
+	 */
+	wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW);
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && wr_protected &&
+	    (val & _PAGE_DIRTY))
+		pte_result = pte_mksaveddirty(pte_result);
+
+	return pte_result;
 }
 
 static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 {
 	pmdval_t val = pmd_val(pmd), oldval = val;
+	bool wr_protected;
+	pmd_t pmd_result;
 
-	val &= _HPAGE_CHG_MASK;
+	val &= (_HPAGE_CHG_MASK & ~_PAGE_DIRTY);
 	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
-	return __pmd(val);
+
+	pmd_result = __pmd(val);
+
+	/*
+	 * Do the saveddirty fixup if the PMD was just write protected and
+	 * it's dirty.
+	 */
+	wr_protected = (oldval & _PAGE_RW) && !(val & _PAGE_RW);
+	if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) && wr_protected &&
+	    (val & _PAGE_DIRTY))
+		pmd_result = pmd_mksaveddirty(pmd_result);
+
+	return pmd_result;
 }
 
 /*
Signature · 1 line
2.17.1
#47 lore raw

PATCH v7 17/41 mm: Move VM_UFFD_MINOR_BIT from 37 to 38

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Future patches will introduce a new VM flag VM_SHADOW_STACK that will be VM_HIGH_ARCH_BIT_5. VM_HIGH_ARCH_BIT_1 through VM_HIGH_ARCH_BIT_4 are bits 32-36, and bit 37 is the unrelated VM_UFFD_MINOR_BIT. For the sake of order, make all VM_HIGH_ARCH_BITs stay together by moving VM_UFFD_MINOR_BIT from 37 to 38. This will allow VM_SHADOW_STACK to be introduced as 37.

---
 include/linux/mm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
index af652444fbba..a1b31caae013 100644
@@ -377,7 +377,7 @@ extern unsigned int kobjsize(const void *objp);
 #endif
 
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
-# define VM_UFFD_MINOR_BIT	37
+# define VM_UFFD_MINOR_BIT	38
 # define VM_UFFD_MINOR		BIT(VM_UFFD_MINOR_BIT)	/* UFFD minor faults */
 #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
 # define VM_UFFD_MINOR		VM_NONE
Signature · 1 line
2.17.1
#48 lore raw

PATCH v7 18/41 mm: Introduce VM_SHADOW_STACK for shadow stack memory

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

New hardware extensions implement support for shadow stack memory, such as x86 Control-flow Enforcement Technology (CET). Add a new VM flag to identify these areas, for example, to be used to properly indicate shadow stack PTEs to the hardware.

Shadow stack VMA creation will be tightly controlled and limited to anonymous memory to make the implementation simpler and since that is all that is required. The solution will rely on pte_mkwrite() to create the shadow stack PTEs, so it will not be required for vm_get_page_prot() to learn how to create shadow stack memory. For this reason document that VM_SHADOW_STACK should not be mixed with VM_SHARED.

---
v7:
- Use lightly edited commit log verbiage from (David Hildenbrand)
- Add explanation for VM_SHARED limitation (David Hildenbrand)

v6:
- Add comment about VM_SHADOW_STACK not being allowed with VM_SHARED (David Hildenbrand)

v3:
- Drop arch specific change in arch_vma_name(). The memory can show as anonymous (Kirill)
- Change CONFIG_ARCH_HAS_SHADOW_STACK to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags() (Boris)

---
 Documentation/filesystems/proc.rst | 1 +
 fs/proc/task_mmu.c                 | 3 +++
 include/linux/mm.h                 | 8 ++++++++
 3 files changed, 12 insertions(+)
Documentation/filesystems/proc.rst +1 -0
index 9d5fd9424e8b..8b314df7ccdf 100644
@@ -564,6 +564,7 @@ encoded manner. The codes are the following:
     mt    arm64 MTE allocation tags are enabled
     um    userfaultfd missing tracking
     uw    userfaultfd wr-protect tracking
+    ss    shadow stack page
     ==    =======================================
 
 Note that there is no guarantee that every flag and associated mnemonic will
fs/proc/task_mmu.c +3 -0
index 6a96e1713fd5..324b092c2ac9 100644
@@ -711,6 +711,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
 		[ilog2(VM_UFFD_MINOR)]	= "ui",
 #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+		[ilog2(VM_SHADOW_STACK)] = "ss",
+#endif
 	};
 	size_t i;
 
include/linux/mm.h +8 -0
index a1b31caae013..097544afb1aa 100644
@@ -326,11 +326,13 @@ extern unsigned int kobjsize(const void *objp);
 #define VM_HIGH_ARCH_BIT_2	34	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_3	35	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_4	36	/* bit only usable on 64-bit architectures */
+#define VM_HIGH_ARCH_BIT_5	37	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_0	BIT(VM_HIGH_ARCH_BIT_0)
 #define VM_HIGH_ARCH_1	BIT(VM_HIGH_ARCH_BIT_1)
 #define VM_HIGH_ARCH_2	BIT(VM_HIGH_ARCH_BIT_2)
 #define VM_HIGH_ARCH_3	BIT(VM_HIGH_ARCH_BIT_3)
 #define VM_HIGH_ARCH_4	BIT(VM_HIGH_ARCH_BIT_4)
+#define VM_HIGH_ARCH_5	BIT(VM_HIGH_ARCH_BIT_5)
 #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -346,6 +348,12 @@ extern unsigned int kobjsize(const void *objp);
 #endif
 #endif /* CONFIG_ARCH_HAS_PKEYS */
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+# define VM_SHADOW_STACK	VM_HIGH_ARCH_5 /* Should not be set with VM_SHARED */
+#else
+# define VM_SHADOW_STACK	VM_NONE
+#endif
+
 #if defined(CONFIG_X86)
 # define VM_PAT		VM_ARCH_1	/* PAT reserves whole VMA at once (x86) */
 #elif defined(CONFIG_PPC)
Signature · 1 line
2.17.1
#49 lore raw

PATCH v7 19/41 x86/mm: Check shadow stack page fault errors

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).

Shadow stack accesses to shadow-stack mappings can result in faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.

The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.

This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.

For the purpose of making this clearer, consider the following example. If a process has a shadow stack, and forks, the shadow stack PTEs will become read-only due to COW. If the CPU in one process performs a shadow stack read access to the shadow stack, for example executing a RET and causing the CPU to read the shadow stack copy of the return address, then in order for the fault to be resolved the PTE will need to be set with shadow stack permissions. But then the memory would be changeable from userspace (from CALL, RET, WRSS, etc). So this scenario needs to trigger COW, otherwise the shared page would be changeable from both processes.

Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.

---
v7:
- Update comment in fault handler (David Hildenbrand)

v6:
- Update comment due to rename of Cow bit to SavedDirty

v5:
- Add description of COW example (Boris)
- Replace "permissioned" (Boris)
- Remove capitalization of shadow stack (Boris)

v4:
- Further improve comment talking about FAULT_FLAG_WRITE (Peterz)

v3:
- Improve comment talking about using FAULT_FLAG_WRITE (Peterz)

---
 arch/x86/include/asm/trap_pf.h |  2 ++
 arch/x86/mm/fault.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
arch/x86/include/asm/trap_pf.h +2 -0
index 10b1de500ab1..afa524325e55 100644
@@ -11,6 +11,7 @@
  *   bit 3 ==				1: use of reserved bit detected
  *   bit 4 ==				1: fault was an instruction fetch
  *   bit 5 ==				1: protection keys block access
+ *   bit 6 ==				1: shadow stack access fault
  *   bit 15 ==				1: SGX MMU page-fault
  */
 enum x86_pf_error_code {
@@ -20,6 +21,7 @@ enum x86_pf_error_code {
 	X86_PF_RSVD	=		1 << 3,
 	X86_PF_INSTR	=		1 << 4,
 	X86_PF_PK	=		1 << 5,
+	X86_PF_SHSTK	=		1 << 6,
 	X86_PF_SGX	=		1 << 15,
 };
 
arch/x86/mm/fault.c +31 -0
index a498ae1fbe66..776b92339cfe 100644
@@ -1117,8 +1117,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)
 				       (error_code & X86_PF_INSTR), foreign))
 		return 1;
 
+	/*
+	 * Shadow stack accesses (PF_SHSTK=1) are only permitted to
+	 * shadow stack VMAs. All other accesses result in an error.
+	 */
+	if (error_code & X86_PF_SHSTK) {
+		if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK)))
+			return 1;
+		if (unlikely(!(vma->vm_flags & VM_WRITE)))
+			return 1;
+		return 0;
+	}
+
 	if (error_code & X86_PF_WRITE) {
 		/* write, present and write, not present: */
+		if (unlikely(vma->vm_flags & VM_SHADOW_STACK))
+			return 1;
 		if (unlikely(!(vma->vm_flags & VM_WRITE)))
 			return 1;
 		return 0;
@@ -1310,6 +1324,23 @@ void do_user_addr_fault(struct pt_regs *regs,
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
 
+	/*
+	 * For conventionally writable pages, a read can be serviced with a
+	 * read only PTE. But for shadow stack, there isn't a concept of
+	 * read-only shadow stack memory. If it a PTE has the shadow stack
+	 * permission, it can be modified via CALL and RET instructions. So
+	 * core MM needs to fault in a writable PTE and do things it already
+	 * does for write faults.
+	 *
+	 * Shadow stack accesses (read or write) need to be serviced with
+	 * shadow stack permission memory, which always include write
+	 * permissions. So in the case of a shadow stack read access, treat it
+	 * as a WRITE fault. This will make sure that MM will prepare
+	 * everything (e.g., break COW) such that maybe_mkwrite() can create a
+	 * proper shadow stack PTE.
+	 */
+	if (error_code & X86_PF_SHSTK)
+		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_WRITE)
 		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_INSTR)
Signature · 1 line
2.17.1
maintainer
#50 lore raw

Re: [PATCH v7 19/41] x86/mm: Check shadow stack page fault errors

arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1330

Quoted 8 lines · Rick Edgecombe
arch/x86/mm/fault.c +4 -0
@@ -1310,6 +1324,23 @@ void do_user_addr_fault(struct pt_regs *regs,
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
 
+	/*
+	 * For conventionally writable pages, a read can be serviced with a
+	 * read only PTE. But for shadow stack, there isn't a concept of
+	 * read-only shadow stack memory. If it a PTE has the shadow stack

s/it //

Quoted 19 lines
arch/x86/mm/fault.c +13 -0
+	 * permission, it can be modified via CALL and RET instructions. So
+	 * core MM needs to fault in a writable PTE and do things it already
+	 * does for write faults.
+	 *
+	 * Shadow stack accesses (read or write) need to be serviced with
+	 * shadow stack permission memory, which always include write
+	 * permissions. So in the case of a shadow stack read access, treat it
+	 * as a WRITE fault. This will make sure that MM will prepare
+	 * everything (e.g., break COW) such that maybe_mkwrite() can create a
+	 * proper shadow stack PTE.
+	 */
+	if (error_code & X86_PF_SHSTK)
+		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_WRITE)
 		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_INSTR)
Signature · 1 line
2.17.1
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#51 lore raw

Re: [PATCH v7 19/41] x86/mm: Check shadow stack page fault errors

Quoted 11 lines · Borislav Petkov

arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1330

Quoted 8 lines · Rick Edgecombe
arch/x86/mm/fault.c +4 -0
@@ -1310,6 +1324,23 @@ void do_user_addr_fault(struct pt_regs *regs,
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
 
+	/*
+	 * For conventionally writable pages, a read can be serviced with a
+	 * read only PTE. But for shadow stack, there isn't a concept of
+	 * read-only shadow stack memory. If it a PTE has the shadow stack

s/it //

Quoted 10 lines

arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1340

Quoted 10 lines
arch/x86/mm/fault.c +10 -0
+	 * permission, it can be modified via CALL and RET instructions. So
+	 * core MM needs to fault in a writable PTE and do things it already
+	 * does for write faults.
+	 *
+	 * Shadow stack accesses (read or write) need to be serviced with
+	 * shadow stack permission memory, which always include write
+	 * permissions. So in the case of a shadow stack read access, treat it
+	 * as a WRITE fault. This will make sure that MM will prepare
+	 * everything (e.g., break COW) such that maybe_mkwrite() can create a
+	 * proper shadow stack PTE.

I ended up just chopping that top paragraph out and rewording it a bit. I think this still expresses the intent in a lot less space:

/*
* Read-only permissions can not be expressed in shadow stack PTEs.
* Treat all shadow stack accesses as WRITE faults. This ensures
* that the MM will prepare everything (e.g., break COW) such that
* maybe_mkwrite() can create a proper shadow stack PTE. */

#52 lore raw

PATCH v7 20/41 x86/mm: Teach pte_mkwrite() about stack memory

In reply to Rick Edgecombe

If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So when it is made writable with pte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now that pte_mkwrite() takes a VMA, and places where shadow stack memory might be created pass one, pte_mkwrite() can know when it should do this.

So make pte_mkwrite() create shadow stack memory when the VMA has the VM_SHADOW_STACK flag. Do the same thing for pmd_mkwrite().

This requires referencing VM_SHADOW_STACK in these functions, which are currently defined in pgtable.h, however mm.h (where VM_SHADOW_STACK is located) can't be pulled in without causing problems for files that reference pgtable.h. So also move pte/pmd_mkwrite() into pgtable.c, where they can safely reference VM_SHADOW_STACK.

---
v6:
- New patch

---
 arch/x86/include/asm/pgtable.h | 20 ++------------------
 arch/x86/mm/pgtable.c          | 26 ++++++++++++++++++++++++++
 2 files changed, 28 insertions(+), 18 deletions(-)
arch/x86/include/asm/pgtable.h +2 -18
index 05dfdbdf96b4..d81e7ec27507 100644
@@ -456,15 +456,7 @@ static inline pte_t pte_mkwrite_kernel(pte_t pte)
 
 struct vm_area_struct;
 
-static inline pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
-{
-	pte = pte_mkwrite_kernel(pte);
-
-	if (pte_dirty(pte))
-		pte = pte_clear_saveddirty(pte);
-
-	return pte;
-}
+pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma);
 
 static inline pte_t pte_mkhuge(pte_t pte)
 {
@@ -601,15 +593,7 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 	return pmd_set_flags(pmd, _PAGE_ACCESSED);
 }
 
-static inline pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
-{
-	pmd = pmd_set_flags(pmd, _PAGE_RW);
-
-	if (pmd_dirty(pmd))
-		pmd = pmd_clear_saveddirty(pmd);
-
-	return pmd;
-}
+pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
 
 static inline pud_t pud_set_flags(pud_t pud, pudval_t set)
 {
arch/x86/mm/pgtable.c +26 -0
index e4f499eb0f29..98856bcc8102 100644
@@ -880,3 +880,29 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
 
 #endif /* CONFIG_X86_64 */
 #endif	/* CONFIG_HAVE_ARCH_HUGE_VMAP */
+
+pte_t pte_mkwrite(pte_t pte, struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return pte_mkwrite_shstk(pte);
+
+	pte = pte_mkwrite_kernel(pte);
+
+	if (pte_dirty(pte))
+		pte = pte_clear_saveddirty(pte);
+
+	return pte;
+}
+
+pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return pmd_mkwrite_shstk(pmd);
+
+	pmd = pmd_set_flags(pmd, _PAGE_RW);
+
+	if (pmd_dirty(pmd))
+		pmd = pmd_clear_saveddirty(pmd);
+
+	return pmd;
+}
Signature · 1 line
2.17.1
maintainer
#53 lore raw

Re: [PATCH v7 20/41] x86/mm: Teach pte_mkwrite() about stack memory

Quoted 5 lines · Rick Edgecombe

If a VMA has the VM_SHADOW_STACK flag, it is shadow stack memory. So when it is made writable with pte_mkwrite(), it should create shadow stack memory, not conventionally writable memory. Now that pte_mkwrite() takes a VMA, and places where shadow stack memory might be created pass one, pte_mkwrite() can know when it should do this.

^^^^

This sentence needs rewriting.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#54 lore raw

PATCH v7 21/41 mm: Add guard pages around a shadow stack.

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks.

Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.

The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:

	addq    $0x80, %rsp

However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:

READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8); // read+discard last popped stack element

The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.

This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work.

---
v5:
- Fix typo in commit log

v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)

v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)

Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.

---
 include/linux/mm.h | 31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)
include/linux/mm.h +26 -5
index 097544afb1aa..6a093daced88 100644
@@ -3107,15 +3107,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
 	return mtree_load(&mm->mm_mt, addr);
 }
 
+static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+
+	/*
+	 * Shadow stack pointer is moved by CALL, RET, and INCSSPQ.
+	 * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB
+	 * and touches the first and the last element in the range, which
+	 * triggers a page fault if the range is not in a shadow stack.
+	 * Because of this, creating 4-KB guard pages around a shadow
+	 * stack prevents these instructions from going beyond.
+	 *
+	 * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+	 * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+	 */
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return PAGE_SIZE;
+
+	return 0;
+}
+
 static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
 {
+	unsigned long gap = stack_guard_start_gap(vma);
 	unsigned long vm_start = vma->vm_start;
 
-	if (vma->vm_flags & VM_GROWSDOWN) {
-		vm_start -= stack_guard_gap;
-		if (vm_start > vma->vm_start)
-			vm_start = 0;
-	}
+	vm_start -= gap;
+	if (vm_start > vma->vm_start)
+		vm_start = 0;
 	return vm_start;
 }
 
Signature · 1 line
2.17.1
maintainer
#55 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

Just typos:

Quoted 10 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to

						^
						instruction

s/ssp/SSP/g

Quoted 4 lines

different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such

VMAs

Quoted 5 lines

that there will always be a gap between adjacent shadow stacks.

Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET

"can be incremented or decremented"

Quoted 13 lines

can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.

The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:

	addq    $0x80, %rsp

However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:

Quoted 6 lines

READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8); // read+discard last popped stack element

The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to

				  ^
				  ,
Quoted 5 lines

prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.

This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is

s/stack's/stacks/

Quoted 43 lines

unneeded and adds a strange difference to how most regular stacks work.

---
v5:
- Fix typo in commit log

v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)

v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)

Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.

---
 include/linux/mm.h | 31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)
include/linux/mm.h +2 -0
index 097544afb1aa..6a093daced88 100644
@@ -3107,15 +3107,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
 	return mtree_load(&mm->mm_mt, addr);
 }
 
+static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{

include/linux/mm.h struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) · line 3121

Quoted 10 lines
include/linux/mm.h +10 -0
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+
+	/*
+	 * Shadow stack pointer is moved by CALL, RET, and INCSSPQ.
+	 * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB
+	 * and touches the first and the last element in the range, which
+	 * triggers a page fault if the range is not in a shadow stack.
+	 * Because of this, creating 4-KB guard pages around a shadow
+	 * stack prevents these instructions from going beyond.

I'd prefer the equivalant explanation above from the commit message - it is more precise.

Quoted 9 lines
include/linux/mm.h +9 -0
+	 *
+	 * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+	 * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+	 */
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return PAGE_SIZE;
+
+	return 0;
+}
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#56 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

Quoted 1 line · Borislav Petkov

Just typos:

All seem reasonable to me. Thanks.

For using the log verbiage for the comment, it is quite big. Does something like this seem reasonable?

/*
* The shadow stack pointer(SSP) is moved by CALL, RET, and INCSSPQ.
* The INCSSP instruction can increment the shadow stack pointer. It
* is the shadow stack analog of an instruction like:
*
* addq $0x80, %rsp
*
* However, there is one important difference between an ADD on %rsp
* and INCSSP. In addition to modifying SSP, INCSSP also reads from the
* memory of the first and last elements that were "popped". It can be
* thought of as acting like this:
*
* READ_ONCE(ssp); // read+discard top element on stack
* ssp += nr_to_pop * 8; // move the shadow stack
* READ_ONCE(ssp-8); // read+discard last popped stack element
*
* The maximum distance INCSSP can move the SSP is 2040 bytes, before
* it would read the memory. Therefore a single page gap will be enough
* to prevent any operation from shifting the SSP to an adjacent stack,
* since it would have to land in the gap at least once, causing a
* fault.
*
* Prevent using INCSSP to move the SSP between shadow stacks by
* having a PAGE_SIZE gaurd gap.
*/

maintainer
#57 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

Quoted 7 lines · Rick P
Quoted 1 line · Borislav Petkov

Just typos:

All seem reasonable to me. Thanks.

For using the log verbiage for the comment, it is quite big. Does something like this seem reasonable?

Yeah, it does. I wouldn't want to lose that explanation in a commit message.

However, this special aspect pertains to the shstk implementation in x86 but the code is generic mm and such arch-specific comments are kinda unfitting there.

I wonder if it would be better if you could stick that explanation somewhere in arch/x86/ and only refer to it in a short comment above VM_SHADOW_STACK check in stack_guard_start_gap()...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#58 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

Quoted 12 lines · Borislav Petkov
Quoted 7 lines · Rick P
Quoted 1 line · Borislav Petkov

Just typos:

All seem reasonable to me. Thanks.

For using the log verbiage for the comment, it is quite big. Does something like this seem reasonable?

Yeah, it does. I wouldn't want to lose that explanation in a commit message.

Quoted 7 lines

However, this special aspect pertains to the shstk implementation in x86 but the code is generic mm and such arch-specific comments are kinda unfitting there.

I wonder if it would be better if you could stick that explanation somewhere in arch/x86/ and only refer to it in a short comment above VM_SHADOW_STACK check in stack_guard_start_gap()...

+1

Signature · 3 lines
Thanks,

David / dhildenb
#59 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

Quoted 19 lines · David Hildenbrand
Quoted 18 lines · Borislav Petkov
Quoted 7 lines · Rick P
Quoted 1 line · Borislav Petkov

Just typos:

All seem reasonable to me. Thanks.

For using the log verbiage for the comment, it is quite big. Does something like this seem reasonable?

Yeah, it does. I wouldn't want to lose that explanation in a commit message.

However, this special aspect pertains to the shstk implementation in x86
but the code is generic mm and such arch-specific comments are kinda
unfitting there.

Quoted 6 lines
Quoted 4 lines

I wonder if it would be better if you could stick that explanation somewhere in arch/x86/ and only refer to it in a short comment above
VM_SHADOW_STACK check in stack_guard_start_gap()...

+1

I can't find a good place for it in the arch code. Basically there is no arch/x86 functionality that has to do with guard pages. The closest is pte_mkwrite() because it at least references VM_SHADOW_STACK but it doesn't really fit.

We could to add an arch version of stack_guard_start_gap() but we had that and removed it for other style reasons. Code duplication IIRC.

So I thought to just move it elsewhere in mm.h where VM_SHADOW_STACK is defined.

#60 lore raw

Re: [PATCH v7 21/41] mm: Add guard pages around a shadow stack.

In reply to Rick Edgecombe
Quoted 93 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the ssp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks.

Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.

The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:

        addq    $0x80, %rsp

However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:

READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8); // read+discard last popped stack element

The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.

This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work.

---
v5:
- Fix typo in commit log

v4:
- Drop references to 32 bit instructions
- Switch to generic code to drop __weak (Peterz)

v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)

Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.

---
 include/linux/mm.h | 31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)
include/linux/mm.h +8 -0
index 097544afb1aa..6a093daced88 100644
@@ -3107,15 +3107,36 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
        return mtree_load(&mm->mm_mt, addr);
 }
 
+static inline unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{
+       if (vma->vm_flags & VM_GROWSDOWN)
+               return stack_guard_gap;
+
+       /*
+        * Shadow stack pointer is moved by CALL, RET, and INCSSPQ.
+        * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB

include/linux/mm.h struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr) · line 3127

Quoted 10 lines
include/linux/mm.h +10 -0
+        * and touches the first and the last element in the range, which
+        * triggers a page fault if the range is not in a shadow stack.
+        * Because of this, creating 4-KB guard pages around a shadow
+        * stack prevents these instructions from going beyond.
+        *
+        * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+        * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+        */
+       if (vma->vm_flags & VM_SHADOW_STACK)
+               return PAGE_SIZE;

This is an arch agnostic header file. Can we remove VM_SHADOW_STACK from here? and instead
have arch_is_shadow_stack which consumes vma flags and returns true or false. This allows different architectures to choose their own encoding of vma flags to represent a shadow stack.

Quoted 23 lines
include/linux/mm.h +8 -6
+
+       return 0;
+}
+
 static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
 {
+       unsigned long gap = stack_guard_start_gap(vma);
        unsigned long vm_start = vma->vm_start;
 
-       if (vma->vm_flags & VM_GROWSDOWN) {
-               vm_start -= stack_guard_gap;
-               if (vm_start > vma->vm_start)
-                       vm_start = 0;
-       }
+       vm_start -= gap;
+       if (vm_start > vma->vm_start)
+               vm_start = 0;
        return vm_start;
 }
 
--

2.17.1

#61 lore raw

PATCH v7 22/41 mm/mmap: Add shadow stack pages to memory accounting

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping().

---
v7:
- Change is_stack_mapping() to know about VM_SHADOW_STACK so the additions in vm_stat_account() can be dropped. (David Hildenbrand)

v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill)

v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.

Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.

Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().

---
 mm/internal.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/internal.h +4 -4
index 7920a8b7982e..1d13d5580f64 100644
@@ -491,14 +491,14 @@ static inline bool is_exec_mapping(vm_flags_t flags)
 }
 
 /*
- * Stack area - automatically grows in one direction
+ * Stack area
  *
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations.
  */
 static inline bool is_stack_mapping(vm_flags_t flags)
 {
-	return (flags & VM_STACK) == VM_STACK;
+	return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);
 }
 
 /*
Signature · 1 line
2.17.1
maintainer
#62 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 9 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping().

That last sentence is superfluous.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#63 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 10 lines · Borislav Petkov
Quoted 9 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.

Quoted 4 lines
Quoted 2 lines

Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping().

That last sentence is superfluous.

Before this version it was open coded, but David Hildenbrand suggested this is_stack_mapping() solution. Should it be explained more, or just dropped?

maintainer
#64 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 3 lines · Rick P

Before this version it was open coded, but David Hildenbrand suggested this is_stack_mapping() solution. Should it be explained more, or just dropped?

Well, "adding a VM_SHADOW_STACK check in is_stack_mapping()" is what's in the diff already and it is kinda obvious. So why write what the patch does when one can simply look at the diff?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
Acked-by on v7 22/41
#65 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

In reply to Rick Edgecombe
Quoted 47 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping().

---
v7:
- Change is_stack_mapping() to know about VM_SHADOW_STACK so the additions in vm_stat_account() can be dropped. (David Hildenbrand)

v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill)

v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.

Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.

Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().

---
  mm/internal.h | 8 ++++----
  1 file changed, 4 insertions(+), 4 deletions(-)
mm/internal.h +0 -0
index 7920a8b7982e..1d13d5580f64 100644

mm/internal.h static inline bool is_exec_mapping(vm_flags_t flags) · line 494

Quoted 5 lines
mm/internal.h +1 -1
@@ -491,14 +491,14 @@ static inline bool is_exec_mapping(vm_flags_t flags)
  }
 
  /*
- * Stack area - automatically grows in one direction
+ * Stack area

Maybe "Stack area (including shadow stacks)"

Signature · 3 lines
Thanks,

David / dhildenb
#66 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

In reply to Rick Edgecombe
Quoted 53 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Account shadow stack pages to stack memory. Do this by adding a VM_SHADOW_STACK check in is_stack_mapping().

---
v7:
- Change is_stack_mapping() to know about VM_SHADOW_STACK so the additions in vm_stat_account() can be dropped. (David Hildenbrand)

v3:
- Remove unneeded VM_SHADOW_STACK check in accountable_mapping() (Kirill)

v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.

Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.

Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().

---
 mm/internal.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/internal.h +1 -1
index 7920a8b7982e..1d13d5580f64 100644
@@ -491,14 +491,14 @@ static inline bool is_exec_mapping(vm_flags_t flags)
 }
 
 /*
- * Stack area - automatically grows in one direction
+ * Stack area

mm/internal.h static inline bool is_exec_mapping(vm_flags_t flags) · line 501

Quoted 10 lines
mm/internal.h +3 -3
  *
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations.
  */
 static inline bool is_stack_mapping(vm_flags_t flags)
 {
-       return (flags & VM_STACK) == VM_STACK;
+       return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);

Same comment here. VM_SHADOW_STACK is an x86 specific way of encoding a shadow stack.
Instead let's have a proxy here which allows architectures to have their own encodings to represent a shadow stack.

Quoted 6 lines
mm/internal.h +0 -1
 }
 
 /*
--

2.17.1

#67 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 13 lines · Deepak Gupta

mm/internal.h static inline bool is_exec_mapping(vm_flags_t flags) · line 501

Quoted 13 lines

/*
- * Stack area - automatically grows in one direction
+ * Stack area
*
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations. */
static inline bool is_stack_mapping(vm_flags_t flags) {

mm/internal.h +1 -1
-       return (flags & VM_STACK) == VM_STACK;
+       return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);
Quoted 4 lines

Same comment here. VM_SHADOW_STACK is an x86 specific way of encoding a shadow stack.
Instead let's have a proxy here which allows architectures to have their own encodings to represent a shadow stack.

This doesn't _preclude_ another architecture from coming along and doing that, right? I'd just prefer that shadow stack architecture #2 comes along and refactors this in precisely the way _they_ need it.

#68 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 15 lines · Dave Hansen
Quoted 13 lines · Deepak Gupta
Quoted 13 lines

/*
- * Stack area - automatically grows in one direction
+ * Stack area
*
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations. */
static inline bool is_stack_mapping(vm_flags_t flags) {

mm/internal.h +1 -1
-       return (flags & VM_STACK) == VM_STACK;
+       return ((flags & VM_STACK) == VM_STACK) || (flags & VM_SHADOW_STACK);
Quoted 8 lines
Quoted 4 lines

Same comment here. VM_SHADOW_STACK is an x86 specific way of encoding a shadow stack.
Instead let's have a proxy here which allows architectures to have their own encodings to represent a shadow stack.

This doesn't _preclude_ another architecture from coming along and doing that, right? I'd just prefer that shadow stack architecture #2 comes along and refactors this in precisely the way _they_ need it.

There are two issues here
- Encoding of shadow stack: Another arch can choose different encoding. And yes, another architecture can come in and re-factor it. But so
much thought and work has been given to x86 implementation to keep
shadow stack to not impact arch agnostic parts of the kernel. So
why creep it in here.

- VM_SHADOW_STACK is coming out of the VM_HIGH_ARCH_XX bit position which makes it arch specific.

If re-factor takes care then I would say the 2nd issue still exists, it's better to keep it away from arch agnostic code.

#69 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 40 lines · Deepak Gupta
Quoted 24 lines · Dave Hansen
Quoted 16 lines · Deepak Gupta
Quoted 14 lines

/*
- * Stack area - automatically grows in one direction
+ * Stack area
*
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations. */
static inline bool is_stack_mapping(vm_flags_t flags) {

(quoted diff) +1 -1
-       return (flags & VM_STACK) == VM_STACK;
+       return ((flags & VM_STACK) == VM_STACK) || (flags &
Quoted 1 line

VM_SHADOW_STACK);

Quoted 5 lines

Same comment here. VM_SHADOW_STACK is an x86 specific way of encoding a shadow stack.
Instead let's have a proxy here which allows architectures to have
their own encodings to represent a shadow stack.

Quoted 5 lines

This doesn't _preclude_ another architecture from coming along and doing
that, right? I'd just prefer that shadow stack architecture #2 comes
along and refactors this in precisely the way _they_ need it.

There are two issues here
- Encoding of shadow stack: Another arch can choose different
encoding.
And yes, another architecture can come in and re-factor it. But so
much thought and work has been given to x86 implementation to keep
shadow stack to not impact arch agnostic parts of the kernel. So
why creep it in here.

Quoted 4 lines

- VM_SHADOW_STACK is coming out of the VM_HIGH_ARCH_XX bit position which makes it arch specific.

VM_SHADOW_STACK is defined like this (trimmed for clarity):
#ifdef CONFIG_X86_USER_SHADOW_STACK

# define VM_SHADOW_STACK	VM_HIGH_ARCH_5
#else
# define VM_SHADOW_STACK	VM_NONE
#endif

Also, we actually had an is_shadow_stack_mapping(vma) in the past, but it was dropped from other feedback. I think it might be too soon to say whether other implementations won't end up with a similar vma flag, so this would be premature refactoring. If not though, a helper like that seems like a reasonable solution.

#70 lore raw

Re: [PATCH v7 22/41] mm/mmap: Add shadow stack pages to memory accounting

Quoted 47 lines · Rick P
Quoted 40 lines · Deepak Gupta
Quoted 24 lines · Dave Hansen
Quoted 16 lines · Deepak Gupta
Quoted 14 lines

/*
- * Stack area - automatically grows in one direction
+ * Stack area
*
- * VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
- * do_mmap() forbids all other combinations.
+ * VM_GROWSUP, VM_GROWSDOWN VMAs are always private
+ * anonymous. do_mmap() forbids all other combinations. */
static inline bool is_stack_mapping(vm_flags_t flags) {

(quoted diff) +1 -1
-       return (flags & VM_STACK) == VM_STACK;
+       return ((flags & VM_STACK) == VM_STACK) || (flags &
Quoted 1 line

VM_SHADOW_STACK);

Quoted 5 lines

Same comment here. VM_SHADOW_STACK is an x86 specific way of encoding a shadow stack.
Instead let's have a proxy here which allows architectures to have
their own encodings to represent a shadow stack.

Quoted 5 lines

This doesn't _preclude_ another architecture from coming along and doing
that, right? I'd just prefer that shadow stack architecture #2 comes
along and refactors this in precisely the way _they_ need it.

There are two issues here
- Encoding of shadow stack: Another arch can choose different
encoding.
And yes, another architecture can come in and re-factor it. But so
much thought and work has been given to x86 implementation to keep
shadow stack to not impact arch agnostic parts of the kernel. So
why creep it in here.

Quoted 4 lines

- VM_SHADOW_STACK is coming out of the VM_HIGH_ARCH_XX bit position which makes it arch specific.

Quoted 6 lines

VM_SHADOW_STACK is defined like this (trimmed for clarity):
#ifdef CONFIG_X86_USER_SHADOW_STACK

# define VM_SHADOW_STACK        VM_HIGH_ARCH_5
#else
# define VM_SHADOW_STACK        VM_NONE
#endif

Ok.

Quoted 3 lines

Also, we actually had an is_shadow_stack_mapping(vma) in the past, but it was dropped from other feedback.

looks like I've been late to the party.
IMHO, that was the right approach.

Quoted 2 lines
#71 lore raw

PATCH v7 23/41 mm: Re-introduce vm_flags to do_mmap()

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by:

commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()").

There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap().

---
 fs/aio.c           |  2 +-
 include/linux/mm.h |  3 ++-
 ipc/shm.c          |  2 +-
 mm/mmap.c          | 10 +++++-----
 mm/nommu.c         |  4 ++--
 mm/util.c          |  2 +-
 6 files changed, 12 insertions(+), 11 deletions(-)
fs/aio.c +1 -1
index b0b17bd098bb..4a7576989719 100644
@@ -558,7 +558,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)
 
 	ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size,
 				 PROT_READ | PROT_WRITE,
-				 MAP_SHARED, 0, &unused, NULL);
+				 MAP_SHARED, 0, 0, &unused, NULL);
 	mmap_write_unlock(mm);
 	if (IS_ERR((void *)ctx->mmap_base)) {
 		ctx->mmap_size = 0;
include/linux/mm.h +2 -1
index 6a093daced88..87e46a9e0e93 100644
@@ -3014,7 +3014,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr,
 	struct list_head *uf);
 extern unsigned long do_mmap(struct file *file, unsigned long addr,
 	unsigned long len, unsigned long prot, unsigned long flags,
-	unsigned long pgoff, unsigned long *populate, struct list_head *uf);
+	vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate,
+	struct list_head *uf);
 extern int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
 			 unsigned long start, size_t len, struct list_head *uf,
 			 bool downgrade);
ipc/shm.c +1 -1
index 60e45e7045d4..576a543b7cff 100644
@@ -1662,7 +1662,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,
 			goto invalid;
 	}
 
-	addr = do_mmap(file, addr, size, prot, flags, 0, &populate, NULL);
+	addr = do_mmap(file, addr, size, prot, flags, 0, 0, &populate, NULL);
 	*raddr = addr;
 	err = 0;
 	if (IS_ERR_VALUE(addr))
mm/mmap.c +5 -5
index 20f21f0949dd..eedae44dfc78 100644
@@ -1191,11 +1191,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,
  */
 unsigned long do_mmap(struct file *file, unsigned long addr,
 			unsigned long len, unsigned long prot,
-			unsigned long flags, unsigned long pgoff,
-			unsigned long *populate, struct list_head *uf)
+			unsigned long flags, vm_flags_t vm_flags,
+			unsigned long pgoff, unsigned long *populate,
+			struct list_head *uf)
 {
 	struct mm_struct *mm = current->mm;
-	vm_flags_t vm_flags;
 	int pkey = 0;
 
 	validate_mm(mm);
@@ -1256,7 +1256,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
 	 * to. we assume access permissions have been handled by the open
 	 * of the memory object, so we don't do any here.
 	 */
-	vm_flags = calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) |
+	vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) |
 			mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC;
 
 	if (flags & MAP_LOCKED)
@@ -2829,7 +2829,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
 
 	file = get_file(vma->vm_file);
 	ret = do_mmap(vma->vm_file, start, size,
-			prot, flags, pgoff, &populate, NULL);
+			prot, flags, 0, pgoff, &populate, NULL);
 	fput(file);
 out:
 	mmap_write_unlock(mm);
mm/nommu.c +2 -2
index 57ba243c6a37..f6ddd084671f 100644
@@ -1002,6 +1002,7 @@ unsigned long do_mmap(struct file *file,
 			unsigned long len,
 			unsigned long prot,
 			unsigned long flags,
+			vm_flags_t vm_flags,
 			unsigned long pgoff,
 			unsigned long *populate,
 			struct list_head *uf)
@@ -1009,7 +1010,6 @@ unsigned long do_mmap(struct file *file,
 	struct vm_area_struct *vma;
 	struct vm_region *region;
 	struct rb_node *rb;
-	vm_flags_t vm_flags;
 	unsigned long capabilities, result;
 	int ret;
 	VMA_ITERATOR(vmi, current->mm, 0);
@@ -1029,7 +1029,7 @@ unsigned long do_mmap(struct file *file,
 
 	/* we've determined that we can make the mapping, now translate what we
 	 * now know into VMA flags */
-	vm_flags = determine_vm_flags(file, prot, flags, capabilities);
+	vm_flags |= determine_vm_flags(file, prot, flags, capabilities);
 
 
 	/* we're going to need to record the mapping */
mm/util.c +1 -1
index b8ed9dbc7fd5..a93e832f4065 100644
@@ -539,7 +539,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
 	if (!ret) {
 		if (mmap_write_lock_killable(mm))
 			return -EINTR;
-		ret = do_mmap(file, addr, len, prot, flag, pgoff, &populate,
+		ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate,
 			      &uf);
 		mmap_write_unlock(mm);
 		userfaultfd_unmap_complete(mm, &uf);
Signature · 1 line
2.17.1
#72 lore raw

PATCH v7 24/41 mm: Don't allow write GUPs to shadow stack memory

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.

Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.

---
v3:
- Add comment in __pte_access_permitted() (Dave)
- Remove unneeded shadow stack specific check in
__pte_access_permitted() (Jann)

---
 arch/x86/include/asm/pgtable.h | 5 +++++
 mm/gup.c                       | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +5 -0
index d81e7ec27507..2e3d8cca1195 100644
@@ -1638,6 +1638,11 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write)
 {
 	unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
+	/*
+	 * Write=0,Dirty=1 PTEs are shadow stack, which the kernel
+	 * shouldn't generally allow access to, but since they
+	 * are already Write=0, the below logic covers both cases.
+	 */
 	if (write)
 		need_pte_bits |= _PAGE_RW;
 
mm/gup.c +1 -1
index eab18ba045db..e7c7bcc0e268 100644
@@ -978,7 +978,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 		return -EFAULT;
 
 	if (write) {
-		if (!(vm_flags & VM_WRITE)) {
+		if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {
 			if (!(gup_flags & FOLL_FORCE))
 				return -EFAULT;
 			/* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
Signature · 1 line
2.17.1
maintainer
#73 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

Quoted 7 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result

									  ^
									  ,
Quoted 2 lines

there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a

"stacks"

or "to write to a shadow stack via..."

Quoted 1 line

little less exposed, block writable GUPs for shadow stack VMAs.

GUPs?

I supposed this means "prevent get_user_pages() from pinning pages to which the corresponding VMA is a shadow stack one."?

Or something like that which is less mm-internal speak...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#74 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

Quoted 7 lines · Borislav Petkov
Quoted 5 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

Quoted 7 lines
Quoted 2 lines

Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result

                                                                          ^
                                                                          ,
Quoted 2 lines

there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a

Is there an alternate mechanism, or do we still want to allow FOLL_FORCE so that debuggers can write it?

--Andy

#75 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

Quoted 21 lines · Andy Lutomirski
Quoted 20 lines · Borislav Petkov
Quoted 8 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.

Quoted 4 lines

Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result

        ^
                                                                   
        ,
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

there remain many ways for userspace to trigger the kernel to write to
shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a

Is there an alternate mechanism, or do we still want to allow FOLL_FORCE so that debuggers can write it?

Yes, GDB shadow stack support uses it via both ptrace poke and /proc/pid/mem apparently. So some ability to write through is needed for debuggers. But not CRIU actually. It uses WRSS.

There was also some discussion[0] previously about how apps might prefer to block /proc/self/mem for general security reasons. Blocking shadow stack writes while you allow text writes is probably not that impactful security-wise. So I thought it would be better to leave the logic simpler. Then when /proc/self/mem could be locked down per the discussion, shadow stack can be locked down the same way.

[0]
https://lore.kernel.org/lkml/[email protected]/

maintainer
#76 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

Quoted 34 lines · Rick P
Quoted 21 lines · Andy Lutomirski
Quoted 20 lines · Borislav Petkov
Quoted 8 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function
properly.

Quoted 4 lines

Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result

        ^
                                                                   
        ,
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

there remain many ways for userspace to trigger the kernel to write to
shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a

Is there an alternate mechanism, or do we still want to allow FOLL_FORCE so that debuggers can write it?

Yes, GDB shadow stack support uses it via both ptrace poke and /proc/pid/mem apparently. So some ability to write through is needed for debuggers. But not CRIU actually. It uses WRSS.

Quoted 6 lines

There was also some discussion[0] previously about how apps might prefer to block /proc/self/mem for general security reasons. Blocking shadow stack writes while you allow text writes is probably not that impactful security-wise. So I thought it would be better to leave the logic simpler. Then when /proc/self/mem could be locked down per the discussion, shadow stack can be locked down the same way.

Ah, I am guilty of reading your changelog but not the code.

You said:

Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.

I read that as *denying* FOLL_FORCE. Maybe clarify the changelog?

Quoted 3 lines

[0]
https://lore.kernel.org/lkml/[email protected]/

#77 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

Quoted 68 lines · Andy Lutomirski
Quoted 42 lines · Rick P
Quoted 27 lines · Andy Lutomirski
Quoted 25 lines · Borislav Petkov
Quoted 9 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has
some
unusual properties, which requires some core mm changes to function
properly.

Quoted 6 lines

Shadow stack memory is writable only in very specific, controlled
ways.
However, since it is writable, the kernel treats it as such. As a
result

         ^
                                                               
     
         ,
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

there remain many ways for userspace to trigger the kernel to write to
shadow stack's via get_user_pages(, FOLL_WRITE) operations. To
make this a

Is there an alternate mechanism, or do we still want to allow FOLL_FORCE so that debuggers can write it?

Yes, GDB shadow stack support uses it via both ptrace poke and /proc/pid/mem apparently. So some ability to write through is needed
for debuggers. But not CRIU actually. It uses WRSS.

Quoted 10 lines

There was also some discussion[0] previously about how apps might prefer to block /proc/self/mem for general security reasons. Blocking
shadow stack writes while you allow text writes is probably not that
impactful security-wise. So I thought it would be better to leave the
logic simpler. Then when /proc/self/mem could be locked down per the
discussion, shadow stack can be locked down the same way.

Ah, I am guilty of reading your changelog but not the code.

You said:

Shadow stack memory is writable only in very specific, controlled ways.
However, since it is writable, the kernel treats it as such. As a result
there remain many ways for userspace to trigger the kernel to write to
shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a
little less exposed, block writable GUPs for shadow stack VMAs.

Quoted 1 line

I read that as *denying* FOLL_FORCE. Maybe clarify the changelog?

I think maybe some helpful text missed the quote in Boris comment about other issues: "Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections."

But, yea, the tenses are hard to parse. Maybe something like this:
The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

In userspace, shadow stack memory is writable only in very specific, controlled ways. However, since userspace can, even in the limited ways, modify shadow stack contents, the kernel treats it as writable memory. As a result, without additional work there would remain many ways for userspace to trigger the kernel to write arbitrary data to shadow stacks via get_user_pages(, FOLL_WRITE) based operations. To help userspace protect their shadow stacks, make this a little less exposed by blocking writable get_user_pages() operations for shadow stack VMAs.

Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections. This is required for debugging use cases.

#78 lore raw

Re: [PATCH v7 24/41] mm: Don't allow write GUPs to shadow stack memory

In reply to Rick Edgecombe

mm/gup.c static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) · line 981

Quoted 10 lines · Rick Edgecombe
mm/gup.c +1 -1
index eab18ba045db..e7c7bcc0e268 100644
@@ -978,7 +978,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
                return -EFAULT;
 
        if (write) {
-               if (!(vm_flags & VM_WRITE)) {
+               if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {

I think I missed this in the review.
VM_SHADOW_STACK is an x86 specific vmaflag to represent a shadow stack VMA. Since this is arch agnostic code. Can we instead have is_arch_shadow_stack_vma which consumes vma flags and returns true. This allows different architectures to choose whatever encoding of the vma flag to represent a shadow stack.

Quoted 6 lines
mm/gup.c +0 -1
                        if (!(gup_flags & FOLL_FORCE))
                                return -EFAULT;
                        /* hugetlb does not support FOLL_FORCE|FOLL_WRITE. */
--

2.17.1

#79 lore raw

PATCH v7 25/41 x86/mm: Introduce MAP_ABOVE4G

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly.

One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals.

On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated outside of the 32 bit address space. This way userspace will trigger a

general protection fault which will in turn trigger a segfault if it

tries to transition to 32 bit mode with shadow stack enabled.

This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by.

So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though.

If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall.

Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's own randomization in the bottom up case.

For MAP_32BIT, only the bottom up search path is used. For MAP_ABOVE4G both are potentially valid, so both are used. In the bottomup search path, the default behavior is already consistent with MAP_ABOVE4G since mmap base should be above 4GB.

Without MAP_ABOVE4G, the shadow stack will already normally be above 4GB. So without introducing MAP_ABOVE4G, trying to transition to 32 bit mode with shadow stack enabled would usually segfault anyway. This is already pretty decent guard rails. But the addition of MAP_ABOVE4G is some small complexity spent to make it make it more complete.

---
v5:
- New patch

---
 arch/x86/include/uapi/asm/mman.h | 1 +
 arch/x86/kernel/sys_x86_64.c     | 6 +++++-
 include/linux/mman.h             | 4 ++++
 3 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/mman.h +1 -0
index 775dbd3aff73..5a0256e73f1e 100644
@@ -3,6 +3,7 @@
 #define _ASM_X86_MMAN_H
 
 #define MAP_32BIT	0x40		/* only give out 32bit addresses */
+#define MAP_ABOVE4G	0x80		/* only map above 4GB */
 
 #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
 #define arch_calc_vm_prot_bits(prot, key) (		\
arch/x86/kernel/sys_x86_64.c +5 -1
index 8cc653ffdccd..06378b5682c1 100644
@@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,
 
 	info.flags = VM_UNMAPPED_AREA_TOPDOWN;
 	info.length = len;
-	info.low_limit = PAGE_SIZE;
+	if (!in_32bit_syscall() && (flags & MAP_ABOVE4G))
+		info.low_limit = 0x100000000;
+	else
+		info.low_limit = PAGE_SIZE;
+
 	info.high_limit = get_mmap_base(0);
 
 	/*
include/linux/mman.h +4 -0
index cee1e4b566d8..40d94411d492 100644
@@ -15,6 +15,9 @@
 #ifndef MAP_32BIT
 #define MAP_32BIT 0
 #endif
+#ifndef MAP_ABOVE4G
+#define MAP_ABOVE4G 0
+#endif
 #ifndef MAP_HUGE_2MB
 #define MAP_HUGE_2MB 0
 #endif
@@ -50,6 +53,7 @@
 		| MAP_STACK \
 		| MAP_HUGETLB \
 		| MAP_32BIT \
+		| MAP_ABOVE4G \
 		| MAP_HUGE_2MB \
 		| MAP_HUGE_1GB)
 
Signature · 1 line
2.17.1
maintainer
#80 lore raw

Re: [PATCH v7 25/41] x86/mm: Introduce MAP_ABOVE4G

Quoted 12 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which require some core mm changes to function properly.

One of the properties is that the shadow stack pointer (SSP), which is a CPU register that points to the shadow stack like the stack pointer points to the stack, can't be pointing outside of the 32 bit address space when the CPU is executing in 32 bit mode. It is desirable to prevent executing in 32 bit mode when shadow stack is enabled because the kernel can't easily support 32 bit signals.

Quoted 4 lines

On x86 it is possible to transition to 32 bit mode without any special interaction with the kernel, by doing a "far call" to a 32 bit segment. So the shadow stack implementation can use this address space behavior as a feature, by enforcing that shadow stack memory is always crated

								^^^^^^^

"created"

and I'd say "mapped" or "allocated" here. "Created" sounds weird.

Quoted 10 lines

outside of the 32 bit address space. This way userspace will trigger a

general protection fault which will in turn trigger a segfault if it

tries to transition to 32 bit mode with shadow stack enabled.

This provides a clean error generating border for the user if they try attempt to do 32 bit mode shadow stack, rather than leave the kernel in a half working state for userspace to be surprised by.

So to allow future shadow stack enabling patches to map shadow stacks out of the 32 bit address space, introduce MAP_ABOVE4G. The behavior

I guess this needs to be documented in the mmap() manpage too.

Quoted 9 lines

is pretty much like MAP_32BIT, except that it has the opposite address range. The are a few differences though.

If both MAP_32BIT and MAP_ABOVE4G are provided, the kernel will use the MAP_ABOVE4G behavior. Like MAP_32BIT, MAP_ABOVE4G is ignored in a 32 bit syscall.

Since the default search behavior is top down, the normal kaslr base can be used for MAP_ABOVE4G. This is unlike MAP_32BIT which has to add it's

								     ^^^^

"its"

Quoted 1 line

own randomization in the bottom up case.

...

arch/x86/kernel/sys_x86_64.c arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, · line 197

Quoted 11 lines
arch/x86/kernel/sys_x86_64.c +2 -1
index 8cc653ffdccd..06378b5682c1 100644
@@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0,
 
 	info.flags = VM_UNMAPPED_AREA_TOPDOWN;
 	info.length = len;
-	info.low_limit = PAGE_SIZE;
+	if (!in_32bit_syscall() && (flags & MAP_ABOVE4G))
+		info.low_limit = 0x100000000;

We have a human readable define for that: SZ_4G

Quoted 6 lines
arch/x86/kernel/sys_x86_64.c +3 -0
+	else
+		info.low_limit = PAGE_SIZE;
+
 	info.high_limit = get_mmap_base(0);
 
 	/*
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#81 lore raw

Re: [PATCH v7 25/41] x86/mm: Introduce MAP_ABOVE4G

Quoted 8 lines · Borislav Petkov
Quoted 8 lines
arch/x86/kernel/sys_x86_64.c +0 -0
index 8cc653ffdccd..06378b5682c1 100644
@@ -193,7 +193,11 @@ arch_get_unmapped_area_topdown(struct file
 *filp, const unsigned long addr0,
 
Quoted 7 lines

arch/x86/kernel/sys_x86_64.c arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, · line 197

Quoted 5 lines
arch/x86/kernel/sys_x86_64.c +2 -1
       info.flags = VM_UNMAPPED_AREA_TOPDOWN;
       info.length = len;
-     info.low_limit = PAGE_SIZE;
+     if (!in_32bit_syscall() && (flags & MAP_ABOVE4G))
+             info.low_limit = 0x100000000;

We have a human readable define for that: SZ_4G

Uhh, yes that's much better. And the typos. Thanks.

#82 lore raw

PATCH v7 26/41 mm: Warn on shadow stack memory in wrong vma

In reply to Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured.

In order to check if a pte is shadow stack in core mm code, add two arch breakouts arch_check_zapped_pte/pmd(). This will allow shadow stack specific code to be kept in arch/x86.

---
v6:
- Add arch breakout to remove shstk from core MM code.

v5:
- Fix typo in commit log

v3:
- New patch

---
 arch/x86/include/asm/pgtable.h |  6 ++++++
 arch/x86/mm/pgtable.c          | 12 ++++++++++++
 include/linux/pgtable.h        | 14 ++++++++++++++
 mm/huge_memory.c               |  1 +
 mm/memory.c                    |  1 +
 5 files changed, 34 insertions(+)
arch/x86/include/asm/pgtable.h +6 -0
index 2e3d8cca1195..e5b3dce0d9fe 100644
@@ -1684,6 +1684,12 @@ static inline bool arch_has_hw_pte_young(void)
 	return true;
 }
 
+#define arch_check_zapped_pte arch_check_zapped_pte
+void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte);
+
+#define arch_check_zapped_pmd arch_check_zapped_pmd
+void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd);
+
 #ifdef CONFIG_XEN_PV
 #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young
 static inline bool arch_has_hw_nonleaf_pmd_young(void)
arch/x86/mm/pgtable.c +12 -0
index 98856bcc8102..afab0bc7862b 100644
@@ -906,3 +906,15 @@ pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 
 	return pmd;
 }
+
+void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte)
+{
+	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
+			pte_shstk(pte));
+}
+
+void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd)
+{
+	VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) &&
+			pmd_shstk(pmd));
+}
include/linux/pgtable.h +14 -0
index c63cd44777ec..4a8970b9fb11 100644
@@ -291,6 +291,20 @@ static inline bool arch_has_hw_pte_young(void)
 }
 #endif
 
+#ifndef arch_check_zapped_pte
+static inline void arch_check_zapped_pte(struct vm_area_struct *vma,
+					 pte_t pte)
+{
+}
+#endif
+
+#ifndef arch_check_zapped_pmd
+static inline void arch_check_zapped_pmd(struct vm_area_struct *vma,
+					 pmd_t pmd)
+{
+}
+#endif
+
 #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR
 static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
 				       unsigned long address,
mm/huge_memory.c +1 -0
index aaf815838144..24797be05fcb 100644
@@ -1689,6 +1689,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	 */
 	orig_pmd = pmdp_huge_get_and_clear_full(vma, addr, pmd,
 						tlb->fullmm);
+	arch_check_zapped_pmd(vma, orig_pmd);
 	tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
 	if (vma_is_special_huge(vma)) {
 		if (arch_needs_pgtable_deposit())
mm/memory.c +1 -0
index d0972d2d6f36..c953c2c4588c 100644
@@ -1389,6 +1389,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 				continue;
 			ptent = ptep_get_and_clear_full(mm, addr, pte,
 							tlb->fullmm);
+			arch_check_zapped_pte(vma, ptent);
 			tlb_remove_tlb_entry(tlb, pte, addr);
 			zap_install_uffd_wp_if_needed(vma, addr, pte, details,
 						      ptent);
Signature · 1 line
2.17.1
maintainer
#83 lore raw

Re: [PATCH v7 26/41] mm: Warn on shadow stack memory in wrong vma

Quoted 16 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new type of memory called shadow stack. This shadow stack memory has some unusual properties, which requires some core mm changes to function properly.

One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by the kernel on x86. Previous patches have changed the kernel to now avoid creating these PTEs unless they are for shadow stack memory. In case any missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic, warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non-shadow stack VMAs when they are being zapped. This won't catch transient cases but should have decent coverage. It will be compiled out when shadow stack is not configured.

Quoted 1 line

In order to check if a pte is shadow stack in core mm code, add two arch

s/pte/PTE/

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#84 lore raw

Re: [PATCH v7 26/41] mm: Warn on shadow stack memory in wrong vma

Quoted 26 lines · Borislav Petkov
Quoted 25 lines · Rick Edgecombe

The x86 Control-flow Enforcement Technology (CET) feature includes a new
type of memory called shadow stack. This shadow stack memory has some
unusual properties, which requires some core mm changes to function properly.

One sharp edge is that PTEs that are both Write=0 and Dirty=1 are treated as shadow by the CPU, but this combination used to be created by
the kernel on x86. Previous patches have changed the kernel to now avoid
creating these PTEs unless they are for shadow stack memory. In case any
missed corners of the kernel are still creating PTEs like this for non-shadow stack memory, and to catch any re-introductions of the logic,
warn if any shadow stack PTEs (Write=0, Dirty=1) are found in non- shadow
stack VMAs when they are being zapped. This won't catch transient cases
but should have decent coverage. It will be compiled out when shadow
stack is not configured.

Quoted 4 lines
Quoted 2 lines

In order to check if a pte is shadow stack in core mm code, add two arch

s/pte/PTE/

Yes, it matches the rest.

#85 lore raw

PATCH v7 27/41 x86/mm: Warn if create Write=0,Dirty=1 with raw prot

In reply to Rick Edgecombe

When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected.

However, when a PTE is created by a raw prot using mk_pte(), mk_pte() can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift.

The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning to make sure no new mk_pte() start doing this.

---
v6:
- New patch (Note, this has already been a useful warning, it caught the newly added set_memory_rox() doing this)

---
 arch/x86/include/asm/pgtable.h | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +9 -1
index e5b3dce0d9fe..7142f99d3fbb 100644
@@ -1032,7 +1032,15 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
  * (Currently stuck as a macro because of indirect forward reference
  * to linux/mm.h:page_to_nid())
  */
-#define mk_pte(page, pgprot)   pfn_pte(page_to_pfn(page), (pgprot))
+#define mk_pte(page, pgprot)						 \
+({									 \
+	pgprot_t __pgprot = pgprot;					 \
+									 \
+	WARN_ON_ONCE(cpu_feature_enabled(X86_FEATURE_USER_SHSTK) &&	 \
+		    (pgprot_val(__pgprot) & (_PAGE_DIRTY | _PAGE_RW)) == \
+		    _PAGE_DIRTY);					 \
+	pfn_pte(page_to_pfn(page), __pgprot);				 \
+})
 
 static inline int pmd_bad(pmd_t pmd)
 {
Signature · 1 line
2.17.1
maintainer Reviewed-by on v7 27/41
#86 lore raw

Re: [PATCH v7 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot

Quoted 18 lines · Rick Edgecombe

When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected.

However, when a PTE is created by a raw prot using mk_pte(), mk_pte() can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift.

The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning to make sure no new mk_pte() start doing this.

Quoted 5 lines
Signature · 1 line
Kees Cook
maintainer
#87 lore raw

Re: [PATCH v7 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot

In reply to Rick Edgecombe
Quoted 1 line · Rick Edgecombe

When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as

			   ^
			   in
Quoted 10 lines

shadow stack memory. So for shadow stack memory this bit combination is valid, but when Dirty=1,Write=1 (conventionally writable) memory is being write protected, the kernel has been taught to transition the Dirty=1 bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is not intended to be a writable shadow stack entry, it is supposed to be write protected.

Quoted 8 lines

However, when a PTE is created by a raw prot using mk_pte(), mk_pte() can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or needing the SavedDirty shift.

The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning to make sure no new mk_pte() start doing this.

Might wanna add the note from below here:

"... start doing this, like, for example, set_memory_rox() did."

Quoted 10 lines

---
v6:
- New patch (Note, this has already been a useful warning, it caught the newly added set_memory_rox() doing this)

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#88 lore raw

Re: [PATCH v7 27/41] x86/mm: Warn if create Write=0,Dirty=1 with raw prot

Quoted 6 lines · Borislav Petkov
Quoted 2 lines · Rick Edgecombe

When user shadow stack is use, Write=0,Dirty=1 is treated by the CPU as

                           ^
                           in

Oops, yes.

Quoted 23 lines
Quoted 23 lines

shadow stack memory. So for shadow stack memory this bit combination is
valid, but when Dirty=1,Write=1 (conventionally writable) memory is being
write protected, the kernel has been taught to transition the Dirty=1
bit to SavedDirty=1, to avoid inadvertently creating shadow stack memory. It does this inside pte_wrprotect() because it knows the PTE is
not intended to be a writable shadow stack entry, it is supposed to be
write protected.

However, when a PTE is created by a raw prot using mk_pte(), mk_pte()
can't know whether to adjust Dirty=1 to SavedDirty=1. It can't distinguish between the caller intending to create a shadow stack PTE or
needing the SavedDirty shift.

Quoted 8 lines
Quoted 4 lines

The kernel has been updated to not do this, and so Write=0,Dirty=1 memory should only be created by the pte_mkfoo() helpers. Add a warning
to make sure no new mk_pte() start doing this.

Might wanna add the note from below here:

"... start doing this, like, for example, set_memory_rox() did."

Fine by me.

Thanks.

#89 lore raw

PATCH v7 28/41 x86: Introduce userspace API for shadow stack

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

- ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().

This is preparation patch. It does not implement any features.

---
v4:
- Remove references to CET and replace with shadow stack (Peterz)

v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)

---
 arch/x86/include/asm/processor.h  |  6 +++++
 arch/x86/include/asm/shstk.h      | 21 +++++++++++++++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/Makefile          |  2 ++
 arch/x86/kernel/process_64.c      |  7 ++++-
 arch/x86/kernel/shstk.c           | 44 +++++++++++++++++++++++++++++++
 6 files changed, 85 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/shstk.h
 create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/processor.h +6 -0
index 8d73004e4cac..bd16e012b3e9 100644
@@ -28,6 +28,7 @@ struct vm86;
 #include <asm/unwind_hints.h>
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
+#include <asm/shstk.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -475,6 +476,11 @@ struct thread_struct {
 	 */
 	u32			pkru;
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+	unsigned long		features;
+	unsigned long		features_locked;
+#endif
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/asm/shstk.h new file +21 -0
new file mode 100644
index 000000000000..ec753809f074
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_SHSTK_H
+#define _ASM_X86_SHSTK_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+struct task_struct;
+
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+long shstk_prctl(struct task_struct *task, int option, unsigned long features);
+void reset_thread_features(void);
+#else
+static inline long shstk_prctl(struct task_struct *task, int option,
+			       unsigned long arg2) { return -EINVAL; }
+static inline void reset_thread_features(void) {}
+#endif /* CONFIG_X86_USER_SHADOW_STACK */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_SHSTK_H */
arch/x86/include/uapi/asm/prctl.h +6 -0
index 500b96e71f18..b2b3b7200b2d 100644
@@ -20,4 +20,10 @@
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
 
+/* Don't use 0x3001-0x3004 because of old glibcs */
+
+#define ARCH_SHSTK_ENABLE		0x5001
+#define ARCH_SHSTK_DISABLE		0x5002
+#define ARCH_SHSTK_LOCK			0x5003
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +2 -0
index 92446f1dedd7..b366641703e3 100644
@@ -146,6 +146,8 @@ obj-$(CONFIG_CALL_THUNKS)		+= callthunks.o
 
 obj-$(CONFIG_X86_CET)			+= cet.o
 
+obj-$(CONFIG_X86_USER_SHADOW_STACK)	+= shstk.o
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/process_64.c +6 -1
index 4e34b3b68ebd..71094c8a305f 100644
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,
 		load_gs_index(__USER_DS);
 	}
 
+	reset_thread_features();
+
 	loadsegment(fs, 0);
 	loadsegment(es, _ds);
 	loadsegment(ds, _ds);
@@ -830,7 +832,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 	case ARCH_MAP_VDSO_64:
 		return prctl_map_vdso(&vdso_image_64, arg2);
 #endif
-
+	case ARCH_SHSTK_ENABLE:
+	case ARCH_SHSTK_DISABLE:
+	case ARCH_SHSTK_LOCK:
+		return shstk_prctl(task, option, arg2);
 	default:
 		ret = -EINVAL;
 		break;
arch/x86/kernel/shstk.c new file +44 -0

44 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#90 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 6 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

"... or a negative value on error."

Quoted 2 lines

- ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

ditto.

What is the use case of the feature locking?

I'm under the simple assumption that once shstk is enabled for an app, it remains so. I guess my question is rather, what's the use case for enabling shadow stack and then disabling it later for an app...?

Quoted 4 lines

The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().

This is preparation patch. It does not implement any features.

That belongs under the "---" line I guess.

Quoted 37 lines

---
v4:
- Remove references to CET and replace with shadow stack (Peterz)

v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that would be compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)

---
 arch/x86/include/asm/processor.h  |  6 +++++
 arch/x86/include/asm/shstk.h      | 21 +++++++++++++++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/Makefile          |  2 ++
 arch/x86/kernel/process_64.c      |  7 ++++-
 arch/x86/kernel/shstk.c           | 44 +++++++++++++++++++++++++++++++
 6 files changed, 85 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/shstk.h
 create mode 100644 arch/x86/kernel/shstk.c

...

Quoted 12 lines
arch/x86/kernel/shstk.c new file +12 -0
+long shstk_prctl(struct task_struct *task, int option, unsigned long features)
+{
+	if (option == ARCH_SHSTK_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;
+
+	/* Do not allow to change locked features */

arch/x86/kernel/shstk.c line 40

Quoted 10 lines
arch/x86/kernel/shstk.c new file +10 -0
+	if (features & task->thread.features_locked)
+		return -EPERM;
+
+	/* Only support enabling/disabling one feature at a time. */
+	if (hweight_long(features) > 1)
+		return -EINVAL;
+
+	if (option == ARCH_SHSTK_DISABLE) {
+		return -EINVAL;
+	}

{} braces left over from some previous version. Can go now.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#91 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 9 lines · Borislav Petkov
Quoted 6 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_SHSTK_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

"... or a negative value on error."

Sure.

Quoted 4 lines
Quoted 3 lines

- ARCH_SHSTK_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

Quoted 8 lines

ditto.

What is the use case of the feature locking?

I'm under the simple assumption that once shstk is enabled for an app,
it remains so. I guess my question is rather, what's the use case for enabling shadow stack and then disabling it later for an app...?

This would be for things like the "permissive mode", where glibc determines that it has to do something like dlopen() an unsupporting DSO much later.

But being able to late lock the features is required for the working behavior of glibc as well. Glibc enables shadow stack very early, then disables it later if it finds that any of the normal dynamic libraries don't support it. It only locks shadow stack after this point even in non-permissive mode.

The selftest also does a lot of enabling and disabling.

Quoted 8 lines
Quoted 6 lines

The features are handled per-thread and inherited over fork(2)/clone(2),
but reset on exec().

This is preparation patch. It does not implement any features.

That belongs under the "---" line I guess.

Oh, yes.

Quoted 61 lines
Quoted 41 lines

---
v4:
- Remove references to CET and replace with shadow stack (Peterz)

v3:
- Move shstk.c Makefile changes earlier (Kees)
- Add #ifdef around features_locked and features (Kees)
- Encapsulate features reset earlier in reset_thread_features() so features and features_locked are not referenced in code that
would be
compiled !CONFIG_X86_USER_SHADOW_STACK. (Kees)
- Fix typo in commit log (Kees)
- Switch arch_prctl() numbers to avoid conflict with LAM

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander
Potapenko)
- Make CET only (tglx)

---
 arch/x86/include/asm/processor.h  |  6 +++++
 arch/x86/include/asm/shstk.h      | 21 +++++++++++++++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/Makefile          |  2 ++
 arch/x86/kernel/process_64.c      |  7 ++++-
 arch/x86/kernel/shstk.c           | 44
+++++++++++++++++++++++++++++++
 6 files changed, 85 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/shstk.h
 create mode 100644 arch/x86/kernel/shstk.c

...

Quoted 17 lines

+long shstk_prctl(struct task_struct *task, int option, unsigned long features)

arch/x86/kernel/shstk.c new file +15 -0
+{
+	if (option == ARCH_SHSTK_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;
+
+	/* Do not allow to change locked features */
+	if (features & task->thread.features_locked)
+		return -EPERM;
+
+	/* Only support enabling/disabling one feature at a time. */
Quoted 9 lines

arch/x86/kernel/shstk.c line 40

Quoted 6 lines
arch/x86/kernel/shstk.c new file +6 -0
+	if (hweight_long(features) > 1)
+		return -EINVAL;
+
+	if (option == ARCH_SHSTK_DISABLE) {
+		return -EINVAL;
+	}

{} braces left over from some previous version. Can go now.

This was intentional, but I wasn't sure on it. It makes the diff cleaner in later patches, is the reason.

maintainer
#92 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 9 lines · Rick P

This would be for things like the "permissive mode", where glibc determines that it has to do something like dlopen() an unsupporting DSO much later.

But being able to late lock the features is required for the working behavior of glibc as well. Glibc enables shadow stack very early, then disables it later if it finds that any of the normal dynamic libraries don't support it. It only locks shadow stack after this point even in non-permissive mode.

So this all sounds weird. Especially from a user point of view.

Now let's imagine there's a Linux user called Boris and he goes and buys a CPU which supports shadow stack, gets a distro which has shadow stack enabled. All good.

Now, at some point he loads a program which pulls in an old library which hasn't been enabled for shadow stack yet.

In the name of not breaking stuff, his glibc is configured in permissive mode by default so that program loads and shadow stack for it is disabled.

And Boris doesn't even know and continues on his merry way thinking that he has all that cool ROP protection.

So where is the knob that says, "disable permissive mode"?

Or at least where does the user get a warning saying, "hey, this app doesn't do shadow stack and we disabled it for ya so that it can still work"?

Or am I way off?

I hope you're catching my drift. Because if there's no enforcement of shstk and we do this permissive mode by default, this whole overhead is just a unnecessary nuisance...

But maybe that'll come later and I should keep going through the set...

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#93 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 11 lines · Borislav Petkov

So this all sounds weird. Especially from a user point of view.

Now let's imagine there's a Linux user called Boris and he goes and buys
a CPU which supports shadow stack, gets a distro which has shadow stack
enabled. All good.

Now, at some point he loads a program which pulls in an old library which hasn't been enabled for shadow stack yet.

Quoted 8 lines

In the name of not breaking stuff, his glibc is configured in permissive
mode by default so that program loads and shadow stack for it is disabled.

And Boris doesn't even know and continues on his merry way thinking that
he has all that cool ROP protection.

There is a proc that shows if shadow stack is enabled in a thread. It does indeed come later in the series.

Quoted 2 lines

So where is the knob that says, "disable permissive mode"?

glibc has an environment variable that can change the loader's behavior. There is also a compile time config for the default mode. But this "permissive mode" is a glibc thing. The kernel doesn't implement it per-se, just provides building blocks.

Quoted 7 lines

Or at least where does the user get a warning saying, "hey, this app doesn't do shadow stack and we disabled it for ya so that it can still
work"?

Or am I way off?

I don't think so. The whole "when to enable shadow stack" question is thornier than it might seem though, and what we have here is based on some trade offs in the details.

Quoted 5 lines

I hope you're catching my drift. Because if there's no enforcement of shstk and we do this permissive mode by default, this whole overhead is
just a unnecessary nuisance...

In the existing glibc patches, and this is highly related to glibc behavior because the decisions around enabling and locking have been pushed there, there are two reasons why shadow stack would get disabled on an supporting executable after it gets enabled.
1. An executable is loaded and one of the shared objects (the ones that come out of ldd) does not support shadow stack
2. An executable is loaded in permissive mode, and much later during execution dlopen()s a DSO that does not support shadow stack.

One of the challenges with enabling shadow stack is you only start recording the shadow stack history when you enable it. If you enable it at some point, and then return from that function you underflow the shadow stack and get a violation. So if the shadow stack will be locked, it has to be enabled at the earliest point it might return to at some point (for example after returning from main()).

So in 1, the existing logic of glibc is to enable shadow stack at the very beginning of the loader. Then go through the whole loading/linking process. If problems are found, disable shadow stack. If no problems are found, then lock it.

I've wondered if this super early glibc enabling behavior is really needed and if they could enable it after processing the linked libraries in the elf. Then save the work of enabling and disabling shadow stack for situations that don't support it. To me this is the big wart in the whole thing, but I don't think the kernel can help resolve it. If glibc can enable later, then we can combine the locking and enabling into a single operation. But it only saves a syscall and it might prevent some other libc that needs to do things like glibc does currently, from being able to make it work at all.

In 2, the enabling happens like normal and the locking is skipped, so that shadow stack can be enabled during a dlopen(). But glibc permissive mode promises more than it delivers. Since it can only disable shadow stack per-thread, it leaves the other threads enabled. Making a working permissive mode is sort of an unsolved problem. There are some proposals to make it work in just userspace, and some that would need additional kernel support. If you are interested I can go into why per-process disabling is not straightforward.

So the locking is needed for the basic support in 1 and the weak permissive mode in 2 uses it. I am considering this series to support 1, but people may end up using 2 to get some permissive-ness. In general the idea of this API is to push the enabling decisions into userspace because that is where the information for making the decision is known. We previously tried to add some batch operations to improve the performance, but tglx had suggested to start with something simple. So we end up with this simple composable API.

maintainer
#94 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 2 lines · Rick P

There is a proc that shows if shadow stack is enabled in a thread. It does indeed come later in the series.

Not good enough:

1. buried somewhere in proc where no one knows about it

2. it is per thread so user needs to grep *all*

Quoted 3 lines

... We previously tried to add some batch operations to improve the performance, but tglx had suggested to start with something simple. So we end up with this simple composable API.

I agree with starting simple and thanks for explaining this in detail.

TBH, though, it already sounds like a mess to me. I guess a mess we'll have to deal with because there will always be this case of some shared object/lib not being enabled for shstk because of raisins.

And TBH #2, I would've done it even simpler: if some shared object can't do shadow stack, we disable it for the whole process. I mean, what's the point? Only some of the stack is shadowed so an attacker could find a way to keep the process perhaps run this shstk-unsupporting shared object more/longer and ROP its way around the system.

But I tend to oversimplify things sometimes so...

What I'd like to have, though, is a kernel cmdline param which disables permissive mode and userspace can't do anything about it. So that once you boot your kernel, you can know that everything that runs on the machine has shstk and is properly protected.

Also, it'll allow for faster fixing of all those shared objects to use shstk by way of political pressure.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#95 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

+Joao regarding mixed mode designs

Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Rick P

There is a proc that shows if shadow stack is enabled in a thread. It
does indeed come later in the series.

Not good enough:

1. buried somewhere in proc where no one knows about it

2. it is per thread so user needs to grep *all*

See "x86: Expose thread features in /proc/$PID/status" for the patch. We could emit something in dmesg I guess? The logic would be:
- Record the presence of elf SHSTK bit on exec
- On shadow stack disable, if it had the elf bit, pr_info("bad!")

Quoted 7 lines
Quoted 6 lines

... We previously tried to add some batch operations to improve
the
performance, but tglx had suggested to start with something
simple.
So we end up with this simple composable API.

Quoted 7 lines

I agree with starting simple and thanks for explaining this in detail.

TBH, though, it already sounds like a mess to me. I guess a mess we'll
have to deal with because there will always be this case of some shared object/lib not being enabled for shstk because of raisins.

The compatibility problems are totally the mess in this whole thing. When you try to look at a "permissive" mode that actually works it gets even more complex. Joao and I have been banging our heads on that problem for months.

But there are some expected users of this that say: we compile and check our known set of binaries, we won't get any surprises. So it's more of a distro problem.

Quoted 6 lines

And TBH #2, I would've done it even simpler: if some shared object can't
do shadow stack, we disable it for the whole process. I mean, what's the
point?

You mean a late loaded dlopen()ed DSO? The enabling logic can't know this will happen ahead of time.

If you mean if the shared objects in the elf all support shadow stack, then this is what happens. The complication is that the loader wants to enable shadow stack before it has checked the elf libs so it doesn't underflow the shadow stack when it returns from the function that does this checking.

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled, don't underflow because it was enabled in step 1 and we have return addresses from 2 on the shadow stack

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

But it depends on the loader code design which I don't know well enough.

Quoted 3 lines

Only some of the stack is shadowed so an attacker could find a way to keep the process perhaps run this shstk-unsupporting shared object more/longer and ROP its way around the system.

I hope non-permissive mode is the standard usage eventually.

Quoted 9 lines

But I tend to oversimplify things sometimes so...

What I'd like to have, though, is a kernel cmdline param which disables
permissive mode and userspace can't do anything about it. So that once
you boot your kernel, you can know that everything that runs on the machine has shstk and is properly protected.

Szabolcs Nagy was commenting something similar in another thread, for supporting kernel enforced security policies. I think the way to do it would have the kernel detect the the elf bit itself (like it used to) and enable shadow stack on exec. If you can't rely on userspace to call in to enable it, it's not clear at what point the kernel should check that it did.

But then if you trigger off of the elf bit in the kernel, you get all the regression issues of the old glibcs at that point. But it is already an "I don't care if I crash" mode, so...

I think if you trust your libc, glibc could implement this in userspace too. It would be useful even as as testing override.

Quoted 4 lines

Also, it'll allow for faster fixing of all those shared objects to use
shstk by way of political pressure.

#96 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 68 lines · Rick P

+Joao regarding mixed mode designs

Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Rick P

There is a proc that shows if shadow stack is enabled in a thread. It
does indeed come later in the series.

Not good enough:

1. buried somewhere in proc where no one knows about it

2. it is per thread so user needs to grep *all*

See "x86: Expose thread features in /proc/$PID/status" for the patch. We could emit something in dmesg I guess? The logic would be:
- Record the presence of elf SHSTK bit on exec
- On shadow stack disable, if it had the elf bit, pr_info("bad!")

Quoted 7 lines
Quoted 6 lines

... We previously tried to add some batch operations to improve
the
performance, but tglx had suggested to start with something
simple.
So we end up with this simple composable API.

Quoted 7 lines

I agree with starting simple and thanks for explaining this in detail.

TBH, though, it already sounds like a mess to me. I guess a mess we'll
have to deal with because there will always be this case of some shared object/lib not being enabled for shstk because of raisins.

The compatibility problems are totally the mess in this whole thing. When you try to look at a "permissive" mode that actually works it gets even more complex. Joao and I have been banging our heads on that problem for months.

But there are some expected users of this that say: we compile and check our known set of binaries, we won't get any surprises. So it's more of a distro problem.

Quoted 6 lines

And TBH #2, I would've done it even simpler: if some shared object can't
do shadow stack, we disable it for the whole process. I mean, what's the
point?

You mean a late loaded dlopen()ed DSO? The enabling logic can't know this will happen ahead of time.

If you mean if the shared objects in the elf all support shadow stack, then this is what happens. The complication is that the loader wants to enable shadow stack before it has checked the elf libs so it doesn't underflow the shadow stack when it returns from the function that does this checking.

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled, don't underflow because it was enabled in step 1 and we have return addresses from 2 on the shadow stack

Quoted 4 lines

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

I will try it out.

Quoted 37 lines

But it depends on the loader code design which I don't know well enough.

Quoted 3 lines

Only some of the stack is shadowed so an attacker could find a way to keep the process perhaps run this shstk-unsupporting shared object more/longer and ROP its way around the system.

I hope non-permissive mode is the standard usage eventually.

Quoted 9 lines

But I tend to oversimplify things sometimes so...

What I'd like to have, though, is a kernel cmdline param which disables
permissive mode and userspace can't do anything about it. So that once
you boot your kernel, you can know that everything that runs on the machine has shstk and is properly protected.

Szabolcs Nagy was commenting something similar in another thread, for supporting kernel enforced security policies. I think the way to do it would have the kernel detect the the elf bit itself (like it used to) and enable shadow stack on exec. If you can't rely on userspace to call in to enable it, it's not clear at what point the kernel should check that it did.

But then if you trigger off of the elf bit in the kernel, you get all the regression issues of the old glibcs at that point. But it is already an "I don't care if I crash" mode, so...

I think if you trust your libc, glibc could implement this in userspace too. It would be useful even as as testing override.

Quoted 4 lines

Also, it'll allow for faster fixing of all those shared objects to use
shstk by way of political pressure.

Signature · 1 line
H.J.
#97 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 71 lines · H.J. Lu
Quoted 68 lines · Rick P

+Joao regarding mixed mode designs

Quoted 10 lines · Borislav Petkov
Quoted 3 lines · Rick P

There is a proc that shows if shadow stack is enabled in a thread. It
does indeed come later in the series.

Not good enough:

1. buried somewhere in proc where no one knows about it

2. it is per thread so user needs to grep *all*

See "x86: Expose thread features in /proc/$PID/status" for the patch. We could emit something in dmesg I guess? The logic would be:
- Record the presence of elf SHSTK bit on exec
- On shadow stack disable, if it had the elf bit, pr_info("bad!")

Quoted 7 lines
Quoted 6 lines

... We previously tried to add some batch operations to improve
the
performance, but tglx had suggested to start with something
simple.
So we end up with this simple composable API.

Quoted 7 lines

I agree with starting simple and thanks for explaining this in detail.

TBH, though, it already sounds like a mess to me. I guess a mess we'll
have to deal with because there will always be this case of some shared object/lib not being enabled for shstk because of raisins.

The compatibility problems are totally the mess in this whole thing. When you try to look at a "permissive" mode that actually works it gets even more complex. Joao and I have been banging our heads on that problem for months.

But there are some expected users of this that say: we compile and check our known set of binaries, we won't get any surprises. So it's more of a distro problem.

Quoted 6 lines

And TBH #2, I would've done it even simpler: if some shared object can't
do shadow stack, we disable it for the whole process. I mean, what's the
point?

You mean a late loaded dlopen()ed DSO? The enabling logic can't know this will happen ahead of time.

If you mean if the shared objects in the elf all support shadow stack, then this is what happens. The complication is that the loader wants to enable shadow stack before it has checked the elf libs so it doesn't underflow the shadow stack when it returns from the function that does this checking.

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled, don't underflow because it was enabled in step 1 and we have return addresses from 2 on the shadow stack

Quoted 7 lines
Quoted 4 lines

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

I will try it out.

Currently glibc enables shadow stack as early as possible. There are only a few places where a function call in glibc never returns.
We can enable shadow stack just before calling main. There are
quite some code paths without shadow stack protection. Is this
an issue?

H.J.

#98 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 15 lines · H.J. Lu
Quoted 15 lines
Quoted 14 lines

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled, don't
underflow because it was enabled in step 1 and we have return addresses
from 2 on the shadow stack

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

Quoted 8 lines
Quoted 2 lines

I will try it out.

Currently glibc enables shadow stack as early as possible. There are only a few places where a function call in glibc never returns.
We can enable shadow stack just before calling main. There are
quite some code paths without shadow stack protection. Is this
an issue?

Thanks for checking. Hmm, does the loader get attacked?

#99 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 20 lines · Rick P
Quoted 18 lines · H.J. Lu
Quoted 17 lines
Quoted 10 lines

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled, don't
underflow because it was enabled in step 1 and we have return addresses
from 2 on the shadow stack

Quoted 4 lines

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

I will try it out.

Quoted 7 lines
Quoted 5 lines

Currently glibc enables shadow stack as early as possible. There are only a few places where a function call in glibc never returns.
We can enable shadow stack just before calling main. There are
quite some code paths without shadow stack protection. Is this
an issue?

Thanks for checking. Hmm, does the loader get attacked?

Not I know of. But there are user codes from .init_array
and .preinit_array which are executed before main. In theory,
an attack can happen before main.

Signature · 1 line
H.J.
#100 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

Quoted 32 lines · H.J. Lu
Quoted 30 lines · Rick P
Quoted 27 lines · H.J. Lu
Quoted 12 lines
Quoted 12 lines

So it does:
1. Enable shadow stack
2. Call elf libs checking functions
3. If all good, lock shadow stack. Else, disable shadow stack.
4. Return from elf checking functions and if shstk is enabled,
don't
underflow because it was enabled in step 1 and we have return addresses
from 2 on the shadow stack

Quoted 8 lines
Quoted 5 lines

I'm wondering if this can't be improved in glibc to look like:
1. Check elf libs, and record it somewhere
2. Wait until just the right spot
3. If all good, enable and lock shadow stack.

I will try it out.

Currently glibc enables shadow stack as early as possible. There are only a few places where a function call in glibc never returns.
We can enable shadow stack just before calling main. There are
quite some code paths without shadow stack protection. Is this
an issue?

Quoted 5 lines
Quoted 1 line

Thanks for checking. Hmm, does the loader get attacked?

Not I know of. But there are user codes from .init_array
and .preinit_array which are executed before main. In theory,
an attack can happen before main.

Hmm, it would be nice to not add any startup overhead to non-shadow stack binaries. I guess it's a tradeoff. Might be worth asking around.

But you can't just enable shadow stack before any user code? It has to go something like?
1. Execute init codes
2. Check elf libs
3. Enable SHSTK

Or what if you just did the enable-disable dance if the execing binary itself has shadow stack. If it doesn't have shadow stack, the elf libs won't change the decision.

maintainer
#101 lore raw

Re: [PATCH v7 28/41] x86: Introduce userspace API for shadow stack

In reply to Edgecombe, Rick P
Quoted 2 lines · Rick P

See "x86: Expose thread features in /proc/$PID/status" for the patch. We could emit something in dmesg I guess? The logic would be:

dmesg is just flaky: ring buffer can get overwritten, users don't see it, ...

Quoted 4 lines

The compatibility problems are totally the mess in this whole thing. When you try to look at a "permissive" mode that actually works it gets even more complex. Joao and I have been banging our heads on that problem for months.

Oh yeah, I'm soo NOT jealous. :-\

Quoted 3 lines

But there are some expected users of this that say: we compile and check our known set of binaries, we won't get any surprises. So it's more of a distro problem.

I'm guessing what will happen here is that distros will gradually enable shstk and once it is ubiquitous, there will be no reason to disable it at all.

Quoted 2 lines

You mean a late loaded dlopen()ed DSO? The enabling logic can't know this will happen ahead of time.

No, I meant the case where you start with shstk enabled and later disable it when some lib does not support it.

From now on that whole process is marked as "cannot use shstk anymore" and any other shared object that tries to use shstk simply doesn't get it.

But meh, this makes the situation even more convoluted as the stuff that has loaded before the first shstk-not-supporting lib, already uses shstk.

So you have half and half.

What a mess.

Quoted 1 line

I hope non-permissive mode is the standard usage eventually.

Yah.

Quoted 2 lines

I think if you trust your libc, glibc could implement this in userspace too. It would be useful even as as testing override.

No, you cannot trust any userspace. And there are other libc's beside glibc.

This should be a kernel parameter. I'm not saying we should do it now but we should do it at some point.

So that user Boris again, he installs his new shiny distro, he checks that all the use cases and software he uses there is already shstk-enabled and then he goes and builds the kernel with

CONFIG_X86_USER_SHADOW_STACK_STRICT=y

or supplies a cmdline param and from now on, nothing can run without shstk. No checking, no trusting, no nothing.

We fail any thread creation which doesn't init shstk.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#102 lore raw

PATCH v7 29/41 x86/shstk: Add user-mode shadow stack support

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).

Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.

32 bit shadow stack is not expected to have many users and it will complicate the signal implementation. So do not support IA32 emulation or x32.

---
v7:
- Add explanation for not supporting 32 bit in commit log (Boris)

v5:
- Switch to EOPNOTSUPP
- Use MAP_ABOVE4G
- Move set_clr_bits_msrl() to patch where it is first used

v4:
- Just set MSR_IA32_U_CET when disabling shadow stack, since we don't have IBT yet. (Peterz)

v3:
- Use define for set_clr_bits_msrl() (Kees)
- Make some functions static (Kees)
- Change feature_foo() to features_foo() (Kees)
- Centralize shadow stack size rlimit checks (Kees)
- Disable x32 support

v2:
- Get rid of unnecessary shstk->base checks
- Don't support IA32 emulation

---
 arch/x86/include/asm/processor.h  |   2 +
 arch/x86/include/asm/shstk.h      |   7 ++
 arch/x86/include/uapi/asm/prctl.h |   3 +
 arch/x86/kernel/shstk.c           | 145 ++++++++++++++++++++++++++++++
 4 files changed, 157 insertions(+)
arch/x86/include/asm/processor.h +2 -0
index bd16e012b3e9..ff98cd6d5af2 100644
@@ -479,6 +479,8 @@ struct thread_struct {
 #ifdef CONFIG_X86_USER_SHADOW_STACK
 	unsigned long		features;
 	unsigned long		features_locked;
+
+	struct thread_shstk	shstk;
 #endif
 
 	/* Floating point and extended processor state */
arch/x86/include/asm/shstk.h +7 -0
index ec753809f074..2b1f7c9b9995 100644
@@ -8,12 +8,19 @@
 struct task_struct;
 
 #ifdef CONFIG_X86_USER_SHADOW_STACK
+struct thread_shstk {
+	u64	base;
+	u64	size;
+};
+
 long shstk_prctl(struct task_struct *task, int option, unsigned long features);
 void reset_thread_features(void);
+void shstk_free(struct task_struct *p);
 #else
 static inline long shstk_prctl(struct task_struct *task, int option,
 			       unsigned long arg2) { return -EINVAL; }
 static inline void reset_thread_features(void) {}
+static inline void shstk_free(struct task_struct *p) {}
 #endif /* CONFIG_X86_USER_SHADOW_STACK */
 
 #endif /* __ASSEMBLY__ */
arch/x86/include/uapi/asm/prctl.h +3 -0
index b2b3b7200b2d..7dfd9dc00509 100644
@@ -26,4 +26,7 @@
 #define ARCH_SHSTK_DISABLE		0x5002
 #define ARCH_SHSTK_LOCK			0x5003
 
+/* ARCH_SHSTK_ features bits */
+#define ARCH_SHSTK_SHSTK		(1ULL <<  0)
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/shstk.c +145 -0

159 lines not on this page · raw message

Signature · 1 line
2.17.1
#103 lore raw

PATCH v7 30/41 x86/shstk: Handle thread shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways.

With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.

For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.

For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.

Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.

For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().

Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.

Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent.

During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner.

---
v3:
- Fix update_fpu_shstk() stub (Mike Rapoport)
- Fix chunks around alloc_shstk() in wrong patch (Kees)
- Fix stack_size/flags swap (Kees)
- Use centralized stack size logic (Kees)

v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx)

v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.

Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)

---
 arch/x86/include/asm/fpu/sched.h   |  3 ++-
 arch/x86/include/asm/mmu_context.h |  2 ++
 arch/x86/include/asm/shstk.h       |  7 +++++
 arch/x86/kernel/fpu/core.c         | 41 +++++++++++++++++++++++++++-
 arch/x86/kernel/process.c          | 18 ++++++++++++-
 arch/x86/kernel/shstk.c            | 43 ++++++++++++++++++++++++++++--
 6 files changed, 109 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/sched.h +2 -1
index c2d6cd78ed0c..3c2903bbb456 100644
@@ -11,7 +11,8 @@
 
 extern void save_fpregs_to_fpstate(struct fpu *fpu);
 extern void fpu__drop(struct fpu *fpu);
-extern int  fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal);
+extern int  fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,
+		      unsigned long shstk_addr);
 extern void fpu_flush_thread(void);
 
 /*
arch/x86/include/asm/mmu_context.h +2 -0
index e01aa74a6de7..9714f08d941b 100644
@@ -147,6 +147,8 @@ do {						\
 #else
 #define deactivate_mm(tsk, mm)			\
 do {						\
+	if (!tsk->vfork_done)			\
+		shstk_free(tsk);		\
 	load_gs_index(0);			\
 	loadsegment(fs, 0);			\
 } while (0)
arch/x86/include/asm/shstk.h +7 -0
index 2b1f7c9b9995..1399f4df098b 100644
@@ -15,11 +15,18 @@ struct thread_shstk {
 
 long shstk_prctl(struct task_struct *task, int option, unsigned long features);
 void reset_thread_features(void);
+int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
+			     unsigned long stack_size,
+			     unsigned long *shstk_addr);
 void shstk_free(struct task_struct *p);
 #else
 static inline long shstk_prctl(struct task_struct *task, int option,
 			       unsigned long arg2) { return -EINVAL; }
 static inline void reset_thread_features(void) {}
+static inline int shstk_alloc_thread_stack(struct task_struct *p,
+					   unsigned long clone_flags,
+					   unsigned long stack_size,
+					   unsigned long *shstk_addr) { return 0; }
 static inline void shstk_free(struct task_struct *p) {}
 #endif /* CONFIG_X86_USER_SHADOW_STACK */
 
arch/x86/kernel/fpu/core.c +40 -1

54 lines not on this page · raw message

arch/x86/kernel/process.c +17 -1
index b650cde3f64d..bf703f53fa49 100644
@@ -48,6 +48,7 @@
 #include <asm/frame.h>
 #include <asm/unwind.h>
 #include <asm/tdx.h>
+#include <asm/shstk.h>
 
 #include "process.h"
 
@@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk)
 
 	free_vm86(t);
 
+	shstk_free(tsk);
 	fpu__drop(fpu);
 }
 
@@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	struct inactive_task_frame *frame;
 	struct fork_frame *fork_frame;
 	struct pt_regs *childregs;
+	unsigned long shstk_addr = 0;
 	int ret = 0;
 
 	childregs = task_pt_regs(p);
@@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	frame->flags = X86_EFLAGS_FIXED;
 #endif
 
-	fpu_clone(p, clone_flags, args->fn);
+	/* Allocate a new shadow stack for pthread if needed */
+	ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size,
+				       &shstk_addr);
+	if (ret)
+		return ret;
+
+	fpu_clone(p, clone_flags, args->fn, shstk_addr);
 
 	/* Kernel thread ? */
 	if (unlikely(p->flags & PF_KTHREAD)) {
@@ -220,6 +229,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	if (!ret && unlikely(test_tsk_thread_flag(current, TIF_IO_BITMAP)))
 		io_bitmap_share(p);
 
+	/*
+	 * If copy_thread() if failing, don't leak the shadow stack possibly
+	 * allocated in shstk_alloc_thread_stack() above.
+	 */
+	if (ret)
+		shstk_free(p);
+
 	return ret;
 }
 
arch/x86/kernel/shstk.c +41 -2

61 lines not on this page · raw message

Signature · 1 line
2.17.1
#104 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 8 lines · Rick Edgecombe

For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().

Quoted 4 lines

Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.

this commit message implies there is protection against the vfork child clobbering the parent's shadow stack, but actually the child can INCSSP (or longjmp) and then clobber it.

so the glibc code just tries to catch bugs and accidents not a strong security mechanism. i'd skip this paragraph.

#105 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 23 lines · Szabolcs Nagy
Quoted 14 lines · Rick Edgecombe

For shadow stack enabled vfork(), the parent and child can share the same
shadow stack, like they can share a normal stack. Since the parent is
suspended until the child terminates, the child will not interfere with
the parent while executing as long as it doesn't return from the vfork()
and overwrite up the shadow stack. The child can safely overwrite down
the shadow stack, as the parent can just overwrite this later. So CET does
not add any additional limitations for vfork().

Quoted 7 lines

Userspace implementing posix vfork() can actually prevent the child from
returning from the vfork() calling function, using CET. Glibc does this
by adjusting the shadow stack pointer in the child, so that the child
receives a #CP if it tries to return from vfork() calling function.

Quoted 4 lines

this commit message implies there is protection against the vfork child clobbering the parent's shadow stack, but actually the child can INCSSP (or longjmp) and then clobber it.

It's true the vfork child could use INCSSP and clobber to create problems, so it is not a strong guarantee of shadow stack integrity. But that's not claimed either. It does "prevent the child from returning from the vfork() calling function" as much as shadow stack protections apply, which I think would be reasonably understood. The vfork child could also use wrss to write the return address to the shadow stack and actually return, or disable shadow stack and return, as other ways to create problems.

Quoted 3 lines

so the glibc code just tries to catch bugs and accidents not a strong security mechanism. i'd skip this paragraph.

Yep. I think it's very much a "nice to have" thing and not intended for security. The paragraph is an aside anyway, because it is specifics about another project. I don't have any objection to dropping it if the opportunity comes up. IIRC it was added because someone thought vfork couldn't work with shadow stack, so people might like to have the details of how can be done.

I wouldn't even be too bothered if the discussed glibc behavior was dropped either. vfork() can go wrong many ways regardless of shadow stack. Is it worth the extra special behavior? Maybe just barely...

maintainer
#106 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

In reply to Rick Edgecombe
Quoted 5 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled

"non-CET"

Quoted 29 lines

in two ways.

With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.

For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.

For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.

Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.

For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().

Quoted 1 line

Userspace implementing posix vfork() can actually prevent the child from

"POSIX"

...

Quoted 25 lines
arch/x86/kernel/fpu/core.c +17 -0
index f851558b673f..bc3de4aeb661 100644
@@ -552,8 +552,41 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu)
 	}
 }
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;
+
+	xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave,
+				XFEATURE_CET_USER);
+
+	/*
+	 * If there is a non-zero ssp, then 'dst' must be configured with a shadow
+	 * stack and the fpu state should be up to date since it was just copied
+	 * from the parent in fpu_clone(). So there must be a valid non-init CET
+	 * state location in the buffer.

arch/x86/kernel/fpu/core.c static inline void fpu_inherit_perms(struct fpu *dst_fpu) · line 581

Quoted 10 lines
arch/x86/kernel/fpu/core.c +10 -0
+	 */
+	if (WARN_ON_ONCE(!xstate))
+		return 1;
+
+	xstate->user_ssp = (u64)ssp;
+
+	return 0;
+}
+#else
+static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr)
								      ^^^^^^^^^^^

ssp, like above.

Better yet:

static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp) {

#ifdef CONFIG_X86_USER_SHADOW_STACK
	...
#endif
	return 0;
}

and less ifdeffery.

Quoted 54 lines
arch/x86/kernel/fpu/core.c +13 -1
+{
+	return 0;
+}
+#endif
+
 /* Clone current's FPU state on fork */
-int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal)
+int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,
+	      unsigned long ssp)
 {
 	struct fpu *src_fpu = &current->thread.fpu;
 	struct fpu *dst_fpu = &dst->thread.fpu;
@@ -613,6 +646,12 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal)
 	if (use_xsave())
 		dst_fpu->fpstate->regs.xsave.header.xfeatures &= ~XFEATURE_MASK_PASID;
 
+	/*
+	 * Update shadow stack pointer, in case it changed during clone.
+	 */
+	if (update_fpu_shstk(dst, ssp))
+		return 1;
+
 	trace_x86_fpu_copy_src(src_fpu);
 	trace_x86_fpu_copy_dst(dst_fpu);
 
arch/x86/kernel/process.c +3 -0
index b650cde3f64d..bf703f53fa49 100644
@@ -48,6 +48,7 @@
 #include <asm/frame.h>
 #include <asm/unwind.h>
 #include <asm/tdx.h>
+#include <asm/shstk.h>
 
 #include "process.h"
 
@@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk)
 
 	free_vm86(t);
 
+	shstk_free(tsk);
 	fpu__drop(fpu);
 }
 
@@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	struct inactive_task_frame *frame;
 	struct fork_frame *fork_frame;
 	struct pt_regs *childregs;
+	unsigned long shstk_addr = 0;
 	int ret = 0;
 
 	childregs = task_pt_regs(p);

arch/x86/kernel/process.c int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) · line 182

Quoted 7 lines
arch/x86/kernel/process.c +3 -1
@@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	frame->flags = X86_EFLAGS_FIXED;
 #endif
 
-	fpu_clone(p, clone_flags, args->fn);
+	/* Allocate a new shadow stack for pthread if needed */
+	ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size,
+				       &shstk_addr);

That function will return 0 even if shstk_addr hasn't been written in it and you will continue merrily and call

	fpu_clone(..., shstk_addr=0);

why don't you return the shadow stack address or negative on error instead of adding an I/O parameter which is pretty much always nasty to deal with.

Quoted 7 lines
arch/x86/kernel/process.c +4 -0
+	if (ret)
+		return ret;
+
+	fpu_clone(p, clone_flags, args->fn, shstk_addr);
 
 	/* Kernel thread ? */
 	if (unlikely(p->flags & PF_KTHREAD)) {

...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#107 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 11 lines · Borislav Petkov
Quoted 8 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a process is duplicated, but the child shares the address space with
the parent, there is potential for the threads sharing a single stack to
cause conflicts for each other. In the normal non-cet case this is handled

"non-CET"

Sure.

Quoted 46 lines
Quoted 46 lines

in two ways.

With regular CLONE_VM a new stack is provided by userspace such that the
parent and child have different stacks.

For vfork, the parent is suspended until the child exits. So as long as
the child doesn't return from the vfork()/CLONE_VFORK calling function and
sticks to a limited set of operations, the parent and child can share the
same stack.

For shadow stack, these scenarios present similar sharing problems. For the
CLONE_VM case, the child and the parent must have separate shadow stacks.
Instead of changing clone to take a shadow stack, have the kernel just
allocate one and switch to it.

Use stack_size passed from clone3() syscall for thread shadow stack size. A
compat-mode thread shadow stack size is further reduced to 1/4. This
allows more threads to run in a 32-bit address space. The clone() does not
pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.

For shadow stack enabled vfork(), the parent and child can share the same
shadow stack, like they can share a normal stack. Since the parent is
suspended until the child terminates, the child will not interfere with
the parent while executing as long as it doesn't return from the vfork()
and overwrite up the shadow stack. The child can safely overwrite down
the shadow stack, as the parent can just overwrite this later. So CET does
not add any additional limitations for vfork().

Quoted 4 lines
Quoted 2 lines

Userspace implementing posix vfork() can actually prevent the child from

"POSIX"

Ok.

Quoted 55 lines

...

Quoted 41 lines
arch/x86/kernel/fpu/core.c +14 -0
index f851558b673f..bc3de4aeb661 100644
@@ -552,8 +552,41 @@ static inline void fpu_inherit_perms(struct
 fpu *dst_fpu)
 	}
 }
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long
 ssp)
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;
+
+	xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave,
+				XFEATURE_CET_USER);
+
+	/*
+	 * If there is a non-zero ssp, then 'dst' must be configured

with a shadow

arch/x86/kernel/fpu/core.c +1 -0
+	 * stack and the fpu state should be up to date since it was

just copied

arch/x86/kernel/fpu/core.c +1 -0
+	 * from the parent in fpu_clone(). So there must be a valid

non-init CET

arch/x86/kernel/fpu/core.c +11 -0
+	 * state location in the buffer.
+	 */
+	if (WARN_ON_ONCE(!xstate))
+		return 1;
+
+	xstate->user_ssp = (u64)ssp;
+
+	return 0;
+}
+#else
+static int update_fpu_shstk(struct task_struct *dst, unsigned long
Quoted 1 line

shstk_addr)

								      ^

^^^^^^^^^^
ssp, like above.

Better yet:

static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)
{

Quoted 7 lines
#ifdef CONFIG_X86_USER_SHADOW_STACK
	...
#endif
	return 0;
}

and less ifdeffery.

Sure. Sometimes people tell me to only ifdef out whole functions to make it easier to read. I suppose in this case it's not hard to see.

Quoted 77 lines
Quoted 63 lines
arch/x86/kernel/fpu/core.c +5 -1
+{
+	return 0;
+}
+#endif
+
 /* Clone current's FPU state on fork */
-int fpu_clone(struct task_struct *dst, unsigned long clone_flags,

bool minimal)
+int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,

arch/x86/kernel/fpu/core.c +3 -0
+	      unsigned long ssp)
 {
 	struct fpu *src_fpu = &current->thread.fpu;
 	struct fpu *dst_fpu = &dst->thread.fpu;
@@ -613,6 +646,12 @@ int fpu_clone(struct task_struct *dst,
 unsigned long clone_flags, bool minimal)
 	if (use_xsave())
 		dst_fpu->fpstate->regs.xsave.header.xfeatures &=
 ~XFEATURE_MASK_PASID;
 
+	/*
+	 * Update shadow stack pointer, in case it changed during

clone.

arch/x86/kernel/fpu/core.c +4 -0
+	 */
+	if (update_fpu_shstk(dst, ssp))
+		return 1;
+
 	trace_x86_fpu_copy_src(src_fpu);
 	trace_x86_fpu_copy_dst(dst_fpu);
arch/x86/kernel/process.c +3 -0
index b650cde3f64d..bf703f53fa49 100644
@@ -48,6 +48,7 @@
 #include <asm/frame.h>
 #include <asm/unwind.h>
 #include <asm/tdx.h>
+#include <asm/shstk.h>
 
 #include "process.h"
 
@@ -119,6 +120,7 @@ void exit_thread(struct task_struct *tsk)
 
 	free_vm86(t);
 
+	shstk_free(tsk);
 	fpu__drop(fpu);
 }
 
@@ -140,6 +142,7 @@ int copy_thread(struct task_struct *p, const
 struct kernel_clone_args *args)
 	struct inactive_task_frame *frame;
 	struct fork_frame *fork_frame;
 	struct pt_regs *childregs;
+	unsigned long shstk_addr = 0;
 	int ret = 0;
 
arch/x86/kernel/process.c +0 -0
 	childregs = task_pt_regs(p);

arch/x86/kernel/process.c int copy_thread(struct task_struct *p, const · line 184

Quoted 9 lines
arch/x86/kernel/process.c +3 -1
@@ -174,7 +177,13 @@ int copy_thread(struct task_struct *p, const
 struct kernel_clone_args *args)
 	frame->flags = X86_EFLAGS_FIXED;
 #endif
 
-	fpu_clone(p, clone_flags, args->fn);
+	/* Allocate a new shadow stack for pthread if needed */
+	ret = shstk_alloc_thread_stack(p, clone_flags, args-
 >stack_size,
+				       &shstk_addr);

That function will return 0 even if shstk_addr hasn't been written in it
and you will continue merrily and call

Quoted 6 lines
	fpu_clone(..., shstk_addr=0);

why don't you return the shadow stack address or negative on error instead of adding an I/O parameter which is pretty much always nasty to
deal with.

On a shadow stack allocation error, we fail the copy_thread(). When shadow stack is enabled, the app might be able to handle a clone failure, but would not be able to handle starting a new thread without getting a new shadow stack.

So in your suggestion I guess we would have two types of failure one that signifies shadow stack is enabled and the allocation failed, and another that signifies that shadow stack is not enabled, so zero needs to be passed into fpu_clone()?

We need the output param in shstk_alloc_thread_stack() because we need to update the SSP to the new shadow stack. If we want to make the non- shadow stack case handled differently, I think the extra conditionals are worse, like:
/* Allocate a new shadow stack for pthread if needed */
ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size,

				&shstk_addr);
if (ret == -EOPNOTSUPP)
	fpu_clone(p, clone_flags, args->fn, 0);
else if (ret < 0)
	return ret;
else
	fpu_clone(p, clone_flags, args->fn, shstk_addr);

Do you think?

It used to be that shstk_alloc_thread_stack() reached into FPU internals to do the SSP update itself. Then the ability to do this was removed. So I came up with an interface for allowing features to modify XSAVE buffers from outside the FPU code. On further discussion, letting code outside the FPU have flexible access to the XSAVE buffer could constrain the FPU code from adding optimizations. So Thomas suggested to pass the SSP along into FPU code so that the FPU modification could be all monolithic and flexible.

If the default SSP value logic is too hidden, what about some clearer code and comments, like this?

arch/x86/kernel/process.c +0 -0
index bf703f53fa49..bd123527fcca 100644
@@ -142,7 +142,7 @@ int copy_thread(struct task_struct *p, const struct

kernel_clone_args *args)

        struct inactive_task_frame *frame;
        struct fork_frame *fork_frame;
        struct pt_regs *childregs;
-       unsigned long shstk_addr = 0;
+       unsigned long new_ssp;
        int ret = 0;
 
        childregs = task_pt_regs(p);
@@ -177,13 +177,18 @@ int copy_thread(struct task_struct *p, const
struct kernel_clone_args *args)
        frame->flags = X86_EFLAGS_FIXED;

#endif

- /* Allocate a new shadow stack for pthread if needed */
+ /*
+ * Allocate a new shadow stack for thread if needed. If shadow
stack,
+ * is disabled, new_ssp will remain 0, and fpu_clone() will
know not to
+ * update it.
+ */
+ new_ssp = 0;
ret = shstk_alloc_thread_stack(p, clone_flags, args-

Quoted 1 line

stack_size,

- &shstk_addr);

+                                      &new_ssp);
        if (ret)
                return ret;

- fpu_clone(p, clone_flags, args->fn, shstk_addr);
+ fpu_clone(p, clone_flags, args->fn, new_ssp);

        /* Kernel thread ? */
        if (unlikely(p->flags & PF_KTHREAD)) {
maintainer
#108 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

On Wed, Mar 08, 2023 at 08:03:17PM +0000, Edgecombe, Rick P wrote:

Btw,

pls try to trim your replies as I need ot scroll through pages of quoted text to find the response.

Quoted 2 lines

Sure. Sometimes people tell me to only ifdef out whole functions to make it easier to read. I suppose in this case it's not hard to see.

Yeah, the less ifdeffery we have, the better.

Quoted 2 lines

If the default SSP value logic is too hidden, what about some clearer code and comments, like this?

The problem with this function is that it needs to return three things:

* success:
** 0
or
** shadow stack address
* failure: due to allocation.

How about this below instead? (totally untested ofc):

arch/x86/kernel/process.c +4 -5
index bf703f53fa49..6e323d4e32fc 100644
@@ -142,7 +142,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	struct inactive_task_frame *frame;
 	struct fork_frame *fork_frame;
 	struct pt_regs *childregs;
-	unsigned long shstk_addr = 0;
+	unsigned long shstk_addr;
 	int ret = 0;
 
 	childregs = task_pt_regs(p);
@@ -178,10 +178,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 #endif
 
 	/* Allocate a new shadow stack for pthread if needed */
-	ret = shstk_alloc_thread_stack(p, clone_flags, args->stack_size,
-				       &shstk_addr);
-	if (ret)
-		return ret;
+	shstk_addr = shstk_alloc_thread_stack(p, clone_flags, args->stack_size);
+	if (IS_ERR_VALUE(shstk_addr))
+		return PTR_ERR((void *)shstk_addr);
 
 	fpu_clone(p, clone_flags, args->fn, shstk_addr);
 
arch/x86/kernel/shstk.c +4 -6
index 13c02747386f..b1668b499e9a 100644
@@ -157,8 +157,8 @@ void reset_thread_features(void)
 	current->thread.features_locked = 0;
 }
 
-int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
-			     unsigned long stack_size, unsigned long *shstk_addr)
+unsigned long shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
+				       unsigned long stack_size)
 {
 	struct thread_shstk *shstk = &tsk->thread.shstk;
 	unsigned long addr, size;
@@ -180,14 +180,12 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
 	size = adjust_shstk_size(stack_size);
 	addr = alloc_shstk(size);
 	if (IS_ERR_VALUE(addr))
-		return PTR_ERR((void *)addr);
+		return addr;
 
 	shstk->base = addr;
 	shstk->size = size;
 
-	*shstk_addr = addr + size;
-
-	return 0;
+	return addr + size;
 }
 
 static unsigned long get_user_shstk_addr(void)
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#109 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 7 lines · Borislav Petkov

On Wed, Mar 08, 2023 at 08:03:17PM +0000, Edgecombe, Rick P wrote:

Btw,

pls try to trim your replies as I need ot scroll through pages of quoted
text to find the response.

Sure sorry.

[...]

Quoted 8 lines
Quoted 4 lines

If the default SSP value logic is too hidden, what about some clearer
code and comments, like this?

The problem with this function is that it needs to return three things:

Quoted 7 lines

* success:
** 0
or
** shadow stack address
* failure: due to allocation.

How about this below instead? (totally untested ofc):

Ah, I see what you were saying now. It looks like it will work to me if you think it is better stylistically.

maintainer
#110 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 2 lines · Rick P

Ah, I see what you were saying now. It looks like it will work to me if you think it is better stylistically.

Yeah, having a function return an error *and* an I/O parameter at the same time is more complicated and error prone instead of when you have a single retval and only input parameters.

I'd say.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#111 lore raw

Re: [PATCH v7 30/41] x86/shstk: Handle thread shadow stack

Quoted 11 lines · Borislav Petkov
Quoted 3 lines · Rick P

Ah, I see what you were saying now. It looks like it will work to me if
you think it is better stylistically.

Yeah, having a function return an error *and* an I/O parameter at the same time is more complicated and error prone instead of when you have
a single retval and only input parameters.

I'd say.

Yea, I agree it's better this way, and at first I just missed your point. By "if you think it's better", I just meant that if someone told me to do it the other way I wouldn't die on the hill.

#112 lore raw

PATCH v7 31/41 x86/shstk: Introduce routines modifying shstk

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stacks are normally written to via CALL/RET or specific CET instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.

To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.

---
v5:
- Fix typo in commit log

v3:
- Drop shstk_check_rstor_token()
- Fail put_shstk_data() if bit 63 is set in the data (Kees)
- Add comment in create_rstor_token() (Kees)
- Pull in create_rstor_token() changes from future patch (Kees)

v2:
- Add data helpers for writing to shadow stack.

v1:
- Use xsave helpers.

---
 arch/x86/include/asm/special_insns.h | 13 +++++
 arch/x86/kernel/shstk.c              | 73 ++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
index de48d1389936..d6cd9344f6c7 100644
@@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);
+	return 0;
+fail:
+	return -EFAULT;
+}
+#endif /* CONFIG_X86_USER_SHADOW_STACK */
+
 #define nop() asm volatile ("nop")
 
 static inline void serialize(void)
arch/x86/kernel/shstk.c +73 -0

91 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#113 lore raw

Re: [PATCH v7 31/41] x86/shstk: Introduce routines modifying shstk

Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stacks are normally written to via CALL/RET or specific CET

				       ^
				       indirectly.
Quoted 2 lines

instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only

"However, sometimes the kernel will need to..."

Quoted 15 lines

WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.

Quoted 4 lines

To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a

s/is //

Quoted 1 line

valid restore token.

...

Quoted 8 lines
arch/x86/include/asm/special_insns.h +0 -0
index de48d1389936..d6cd9344f6c7 100644
@@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 214

Quoted 10 lines
arch/x86/include/asm/special_insns.h +10 -0
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);
+	return 0;
+fail:
+	return -EFAULT;

Nice!

Quoted 6 lines
arch/x86/include/asm/special_insns.h +3 -0
+}
+#endif /* CONFIG_X86_USER_SHADOW_STACK */
+
 #define nop() asm volatile ("nop")
 
 static inline void serialize(void)

...

arch/x86/include/asm/special_insns.h

Quoted 3 lines
arch/x86/include/asm/special_insns.h +3 -0
+static int put_shstk_data(u64 __user *addr, u64 data)
+{
+	if (WARN_ON_ONCE(data & BIT(63)))

Dunno, maybe something like:

/*
* A comment explaining what that is...
*/
#define SHSTK_SIGRETURN_TOKEN BIT_ULL(63)

or so?

And use that instead of that magical bit 63.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#114 lore raw

Re: [PATCH v7 31/41] x86/shstk: Introduce routines modifying shstk

Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stacks are normally written to via CALL/RET or specific CET

                                       ^
                                       indirectly.

Dunno here, RSTORSSP/SAVEPREVSSP are kind of direct.

Quoted 6 lines
Quoted 4 lines

instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only

"However, sometimes the kernel will need to..."

Ok.

Quoted 33 lines
Quoted 24 lines

WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and
the address in a token must point directly above the token, which is within
the same shadow stack. This is distinctively different from other pointers
on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is
a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel
will need something to denote the point in the stack where sigreturn may be
called. This will prevent attackers calling sigreturn at arbitrary places
in the stack, in order to help prevent SROP attacks.

Quoted 8 lines

To do this, something that can only be written by the kernel needs to be
placed on the shadow stack. This can be accomplished by setting bit 63 in
the frame written to the shadow stack. Userspace return addresses can't
have this bit set as it is in the kernel range. It is also can't be a

Quoted 1 line

s/is //

Yep, thanks.

Quoted 42 lines
Quoted 2 lines

valid restore token.

...

Quoted 9 lines
arch/x86/include/asm/special_insns.h +0 -0
index de48d1389936..d6cd9344f6c7 100644
@@ -202,6 +202,19 @@ static inline void clwb(volatile void *__p)
               : [pax] "a" (p));
  }
 

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 214

Quoted 10 lines
arch/x86/include/asm/special_insns.h +10 -0
+#ifdef CONFIG_X86_USER_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+     asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+                       _ASM_EXTABLE(1b, %l[fail])
+                       :: [addr] "r" (addr), [val] "r" (val)
+                       :: fail);
+     return 0;
+fail:
+     return -EFAULT;

Nice!

Quoted 6 lines
arch/x86/include/asm/special_insns.h +3 -0
+}
+#endif /* CONFIG_X86_USER_SHADOW_STACK */
+
  #define nop() asm volatile ("nop")
 
  static inline void serialize(void)

...

arch/x86/include/asm/special_insns.h

Quoted 3 lines
arch/x86/include/asm/special_insns.h +3 -0
+static int put_shstk_data(u64 __user *addr, u64 data)
+{
+     if (WARN_ON_ONCE(data & BIT(63)))

Dunno, maybe something like:

Quoted 8 lines

/*
* A comment explaining what that is...
*/
#define SHSTK_SIGRETURN_TOKEN BIT_ULL(63)

or so?

And use that instead of that magical bit 63.

Seems very reasonable. Since we are calling this the "data format", I might go with SHSTK_DATA_BIT.

maintainer
#115 lore raw

Re: [PATCH v7 31/41] x86/shstk: Introduce routines modifying shstk

Quoted 14 lines · Rick P
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stacks are normally written to via CALL/RET or specific CET

                                       ^
                                       indirectly.

Dunno here, RSTORSSP/SAVEPREVSSP are kind of direct.

Quoted 3 lines
Quoted 3 lines

instructions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the

						  ^^^^^^^^^

Yes, I was trying to make the contrast more obvious because you say "directly" here.

But not too important.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#116 lore raw

PATCH v7 32/41 x86/shstk: Handle signals for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.

One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.

The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.

For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.

So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.

In sigreturn, verify SSP is stored in the data format and pop the shadow stack.

---
v3:
- Drop shstk_setup_rstor_token() (Kees)
- Drop x32 signal support, since x32 support is dropped

v2:
- Switch to new shstk signal format

v1:
- Use xsave helpers.
- Expand commit log.

Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.

---
 arch/x86/include/asm/shstk.h |  5 ++
 arch/x86/kernel/shstk.c      | 98 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/signal.c     |  1 +
 arch/x86/kernel/signal_64.c  |  6 +++
 4 files changed, 110 insertions(+)
arch/x86/include/asm/shstk.h +5 -0
index 1399f4df098b..acee68d30a07 100644
@@ -6,6 +6,7 @@
 #include <linux/types.h>
 
 struct task_struct;
+struct ksignal;
 
 #ifdef CONFIG_X86_USER_SHADOW_STACK
 struct thread_shstk {
@@ -19,6 +20,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
 			     unsigned long stack_size,
 			     unsigned long *shstk_addr);
 void shstk_free(struct task_struct *p);
+int setup_signal_shadow_stack(struct ksignal *ksig);
+int restore_signal_shadow_stack(void);
 #else
 static inline long shstk_prctl(struct task_struct *task, int option,
 			       unsigned long arg2) { return -EINVAL; }
@@ -28,6 +31,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p,
 					   unsigned long stack_size,
 					   unsigned long *shstk_addr) { return 0; }
 static inline void shstk_free(struct task_struct *p) {}
+static inline int setup_signal_shadow_stack(struct ksignal *ksig) { return 0; }
+static inline int restore_signal_shadow_stack(void) { return 0; }
 #endif /* CONFIG_X86_USER_SHADOW_STACK */
 
 #endif /* __ASSEMBLY__ */
arch/x86/kernel/shstk.c +98 -0

104 lines not on this page · raw message

arch/x86/kernel/signal.c +1 -0
index 004cb30b7419..356253e85ce9 100644
@@ -40,6 +40,7 @@
 #include <asm/syscall.h>
 #include <asm/sigframe.h>
 #include <asm/signal.h>
+#include <asm/shstk.h>
 
 static inline int is_ia32_compat_frame(struct ksignal *ksig)
 {
arch/x86/kernel/signal_64.c +6 -0
index 0e808c72bf7e..cacf2ede6217 100644
@@ -175,6 +175,9 @@ int x64_setup_rt_frame(struct ksignal *ksig, struct pt_regs *regs)
 	frame = get_sigframe(ksig, regs, sizeof(struct rt_sigframe), &fp);
 	uc_flags = frame_uc_flags(regs);
 
+	if (setup_signal_shadow_stack(ksig))
+		return -EFAULT;
+
 	if (!user_access_begin(frame, sizeof(*frame)))
 		return -EFAULT;
 
@@ -260,6 +263,9 @@ SYSCALL_DEFINE0(rt_sigreturn)
 	if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
 		goto badframe;
 
+	if (restore_signal_shadow_stack())
+		goto badframe;
+
 	if (restore_altstack(&frame->uc.uc_stack))
 		goto badframe;
 
Signature · 1 line
2.17.1
maintainer
#117 lore raw

Re: [PATCH v7 32/41] x86/shstk: Handle signals for shadow stack

Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack

s/normally //

Quoted 1 line

before handling it. For shadow stacks, since the shadow stack only track's

"tracks"

Quoted 3 lines

return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.

"visible to userspace"

s/which //

Quoted 6 lines

One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.

Quoted 6 lines

The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.

For this token we can use the shadow stack data format defined earlier.

		^^^

Please use passive voice in your commit message: no "we" or "I", etc.

Quoted 11 lines

Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.

So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.

In sigreturn, verify SSP is stored in the data format and pop the shadow stack.

...

Quoted 8 lines
arch/x86/kernel/shstk.c +0 -0
index 13c02747386f..40f0a55762a9 100644
@@ -232,6 +232,104 @@ static int get_shstk_data(unsigned long *data, unsigned long __user *addr)
 	return 0;
 }
 

arch/x86/kernel/shstk.c static int get_shstk_data(unsigned long *data, unsigned long __user *addr) · line 244

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+static int shstk_push_sigframe(unsigned long *ssp)
+{
+	unsigned long target_ssp = *ssp;
+
+	/* Token must be aligned */
+	if (!IS_ALIGNED(*ssp, 8))
+		return -EINVAL;
+
+	if (!IS_ALIGNED(target_ssp, 8))
+		return -EINVAL;

Those two statements are identical AFAICT.

Quoted 6 lines
arch/x86/kernel/shstk.c +6 -0
+	*ssp -= SS_FRAME_SIZE;
+	if (put_shstk_data((void *__user)*ssp, target_ssp))
+		return -EFAULT;
+
+	return 0;
+}
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#118 lore raw

Re: [PATCH v7 32/41] x86/shstk: Handle signals for shadow stack

Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack

s/normally //

It is trying to say "When a signal is handled without shadow stack, the context is pushed to the stack"

Quoted 5 lines
Quoted 3 lines

before handling it. For shadow stacks, since the shadow stack only track's

"tracks"

Right.

Quoted 7 lines
Quoted 5 lines

return addresses, there isn't any state that needs to be pushed. However,
there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.

"visible to userspace"

Sure.

Quoted 2 lines

s/which //

Ok.

Quoted 18 lines
Quoted 18 lines

One is to make sure the restorer address is written to shadow stack, since
the signal handler (if not changing ucontext) returns to the restorer, and
the restorer calls sigreturn. So add the restorer on the shadow stack
before handling the signal, so there is not a conflict when the signal
handler returns to the restorer.

The other thing to do is to place some type of checkable token on the
thread's shadow stack before handling the signal and check it during
sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.

Quoted 6 lines
Quoted 2 lines

For this token we can use the shadow stack data format defined earlier.

		^^^

Please use passive voice in your commit message: no "we" or "I", etc.

Argh, right. And it looks like I wrote this one.

Quoted 28 lines
Quoted 16 lines

Have the data pushed be the previous SSP. In the future the sigreturn
might want to return back to a different stack. Storing the SSP (instead
of a restore offset or something) allows for future functionality that
may want to restore to a different stack.

So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.

In sigreturn, verify SSP is stored in the data format and pop the shadow
stack.

...

Quoted 9 lines
arch/x86/kernel/shstk.c +0 -0
index 13c02747386f..40f0a55762a9 100644
@@ -232,6 +232,104 @@ static int get_shstk_data(unsigned long
 *data, unsigned long __user *addr)
 	return 0;
 }
 
Quoted 12 lines

arch/x86/kernel/shstk.c static int get_shstk_data(unsigned long *data, unsigned long __user *addr) · line 244

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+static int shstk_push_sigframe(unsigned long *ssp)
+{
+	unsigned long target_ssp = *ssp;
+
+	/* Token must be aligned */
+	if (!IS_ALIGNED(*ssp, 8))
+		return -EINVAL;
+
+	if (!IS_ALIGNED(target_ssp, 8))
+		return -EINVAL;

Those two statements are identical AFAICT.

Uhh, yes they are. Not sure what happened here.

Quoted 9 lines
Quoted 7 lines
arch/x86/kernel/shstk.c +6 -0
+	*ssp -= SS_FRAME_SIZE;
+	if (put_shstk_data((void *__user)*ssp, target_ssp))
+		return -EFAULT;
+
+	return 0;
+}
maintainer
#119 lore raw

Re: [PATCH v7 32/41] x86/shstk: Handle signals for shadow stack

Quoted 11 lines · Rick P
Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack

s/normally //

It is trying to say "When a signal is handled without shadow stack, the context is pushed to the stack"

Yeah, I see that. But "normally" is implicit in the "normal" case, without shadow stack.

So you don't really need to say "normally". :-)

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#120 lore raw

PATCH v7 33/41 x86/shstk: Introduce map_shadow_stack syscall

In reply to Rick Edgecombe

When operating with shadow stacks enabled, the kernel will automatically allocate shadow stacks for new threads, however in some cases userspace will need additional shadow stacks. The main example of this is the ucontext family of functions, which require userspace allocating and pivoting to userspace managed stacks.

Unlike most other user memory permissions, shadow stacks need to be provisioned with special data in order to be useful. They need to be setup with a restore token so that userspace can pivot to them via the RSTORSSP instruction. But, the security design of shadow stack's is that they should not be written to except in limited circumstances. This presents a problem for userspace, as to how userspace can provision this special data, without allowing for the shadow stack to be generally writable.

Previously, a new PROT_SHADOW_STACK was attempted, which could be mprotect()ed from RW permissions after the data was provisioned. This was found to not be secure enough, as other thread's could write to the shadow stack during the writable window.

The kernel can use a special instruction, WRUSS, to write directly to userspace shadow stacks. So the solution can be that memory can be mapped as shadow stack permissions from the beginning (never generally writable in userspace), and the kernel itself can write the restore token.

First, a new madvise() flag was explored, which could operate on the PROT_SHADOW_STACK memory. This had a couple downsides:
1. Extra checks were needed in mprotect() to prevent writable memory from ever becoming PROT_SHADOW_STACK.
2. Extra checks/vma state were needed in the new madvise() to prevent restore tokens being written into the middle of pre-used shadow stacks. It is ideal to prevent restore tokens being added at arbitrary locations, so the check was to make sure the shadow stack had never been written to.
3. It stood out from the rest of the madvise flags, as more of direct action than a hint at future desired behavior.

So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.

The following example demonstrates how to create a new shadow stack with map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

---
v5:
- Fix addr/mapped_addr (Kees)
- Switch to EOPNOTSUPP (Kees suggested ENOTSUPP, but checkpatch suggests this)
- Return error for addresses below 4G

v3:
- Change syscall common -> 64 (Kees)
- Use bit shift notation instead of 0x1 for uapi header (Kees)
- Call do_mmap() with MAP_FIXED_NOREPLACE (Kees)
- Block unsupported flags (Kees)
- Require size >= 8 to set token (Kees)

v2:
- Change syscall to take address like mmap() for CRIU's usage

v1:
- New patch (replaces PROT_SHADOW_STACK).

---
 arch/x86/entry/syscalls/syscall_64.tbl |  1 +
 arch/x86/include/uapi/asm/mman.h       |  3 ++
 arch/x86/kernel/shstk.c                | 59 ++++++++++++++++++++++----
 include/linux/syscalls.h               |  1 +
 include/uapi/asm-generic/unistd.h      |  2 +-
 kernel/sys_ni.c                        |  1 +
 6 files changed, 58 insertions(+), 9 deletions(-)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
index c84d12608cd2..f65c671ce3b1 100644
@@ -372,6 +372,7 @@
 448	common	process_mrelease	sys_process_mrelease
 449	common	futex_waitv		sys_futex_waitv
 450	common	set_mempolicy_home_node	sys_set_mempolicy_home_node
+451	64	map_shadow_stack	sys_map_shadow_stack
 
 #
 # Due to a historical design error, certain syscalls are numbered differently
arch/x86/include/uapi/asm/mman.h +3 -0
index 5a0256e73f1e..8148bdddbd2c 100644
@@ -13,6 +13,9 @@
 		((key) & 0x8 ? VM_PKEY_BIT3 : 0))
 #endif
 
+/* Flags for map_shadow_stack(2) */
+#define SHADOW_STACK_SET_TOKEN	(1ULL << 0)	/* Set up a restore token in the shadow stack */
+
 #include <asm-generic/mman.h>
 
 #endif /* _ASM_X86_MMAN_H */
arch/x86/kernel/shstk.c +51 -8

96 lines not on this page · raw message

include/linux/syscalls.h +1 -0
index 33a0ee3bcb2e..392dc11e3556 100644
@@ -1058,6 +1058,7 @@ asmlinkage long sys_memfd_secret(unsigned int flags);
 asmlinkage long sys_set_mempolicy_home_node(unsigned long start, unsigned long len,
 					    unsigned long home_node,
 					    unsigned long flags);
+asmlinkage long sys_map_shadow_stack(unsigned long addr, unsigned long size, unsigned int flags);
 
 /*
  * Architecture-specific system calls
include/uapi/asm-generic/unistd.h +1 -1
index 45fa180cc56a..b12940ec5926 100644
@@ -887,7 +887,7 @@ __SYSCALL(__NR_futex_waitv, sys_futex_waitv)
 __SYSCALL(__NR_set_mempolicy_home_node, sys_set_mempolicy_home_node)
 
 #undef __NR_syscalls
-#define __NR_syscalls 451
+#define __NR_syscalls 452
 
 /*
  * 32 bit systems traditionally used different
kernel/sys_ni.c +1 -0
index 860b2dcf3ac4..cb9aebd34646 100644
@@ -381,6 +381,7 @@ COND_SYSCALL(vm86old);
 COND_SYSCALL(modify_ldt);
 COND_SYSCALL(vm86);
 COND_SYSCALL(kexec_file_load);
+COND_SYSCALL(map_shadow_stack);
 
 /* s390 */
 COND_SYSCALL(s390_pci_mmio_read);
Signature · 1 line
2.17.1
#121 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 8 lines

So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.

...

Quoted 3 lines

The following example demonstrates how to create a new shadow stack with map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

most of the mmap features need not be available (EINVAL) when MAP_SHADOW_STACK is specified.

the main drawback is running out of mmap flags so extension is limited. (but the new syscall has limitations too).

#122 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 35 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 14 lines

So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.

...

Quoted 5 lines

The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Quoted 5 lines

most of the mmap features need not be available (EINVAL) when MAP_SHADOW_STACK is specified.

the main drawback is running out of mmap flags so extension is limited. (but the new syscall has limitations too).

Deepak Gupta (working on riscv shadow stack) asked something similar. Can you see if this thread answers your questions?

https://lore.kernel.org/lkml/[email protected]/

#123 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

In reply to Szabolcs Nagy
Quoted 16 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 8 lines

So rather than repurpose two existing syscalls (mmap, madvise) that don't quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext is the primary motivator, userspace may have other unforeseen reasons to setup it's own shadow stacks using the WRSS instruction. Towards this provide a flag so that stacks can be optionally setup securely for the common case of ucontext without enabling WRSS. Or potentially have the kernel set up the shadow stack in some new way.

...

Quoted 3 lines

The following example demonstrates how to create a new shadow stack with map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

Quoted 8 lines

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Discussion on this topic in v6
https://lore.kernel.org/all/[email protected]/

Again I know earlier CET patches had protection flag and somehow due to pushback on mailing list, it was adopted to go for special syscall because no one else had shadow stack.

Seeing a response from Szabolcs, I am assuming arm4 would also want to follow using mmap to manufacture shadow stack. For reference RFC patches for risc-v shadow stack, use a new protection flag = PROT_SHADOWSTACK.
https://lore.kernel.org/lkml/[email protected]/

I know earlier discussion had been that we let this go and do a re-factor later as other arch support trickle in. But as I thought more on this and I think it may just be messy from user mode point of view as well to have cognition of two different ways of creating shadow stack. One would be special syscall (in current libc) and another mmap (whenever future re-factor happens)

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

Quoted 6 lines

most of the mmap features need not be available (EINVAL) when MAP_SHADOW_STACK is specified.

the main drawback is running out of mmap flags so extension is limited. (but the new syscall has limitations too).

#124 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 29 lines · Deepak Gupta
Quoted 28 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 14 lines

So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.

...

Quoted 5 lines

The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

Quoted 9 lines
Quoted 6 lines

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Discussion on this topic in v6

https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Again I know earlier CET patches had protection flag and somehow due to pushback
on mailing list,
it was adopted to go for special syscall because no one else
had shadow stack.

Quoted 6 lines

Seeing a response from Szabolcs, I am assuming arm4 would also want to follow
using mmap to manufacture shadow stack. For reference RFC patches for risc-v shadow stack,
use a new protection flag = PROT_SHADOWSTACK.

https://lore.kernel.org/lkml/[email protected]/

Quoted 11 lines

I know earlier discussion had been that we let this go and do a re- factor later as other
arch support trickle in. But as I thought more on this and I think it may just be
messy from user mode point of view as well to have cognition of two different ways of
creating shadow stack. One would be special syscall (in current libc) and another mmap
(whenever future re-factor happens)

Quoted 2 lines

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

There is sort of two things intermixed here when we talk about a PROT_SHADOW_STACK.

One is: what is the interface for specifying how the shadow stack should be provisioned with data? Right now there are two ways supported, all zero or with an X86 shadow stack restore token at the end. Then there was already some conversation about a third type. In which case the question would be is using mmap MAP_ flags the right place for this? How many types of initialization will be needed in the end and what is the overlap between the architectures?

The other thing is: should shadow stack memory creation be tightly controlled? For example in x86 we limit this to anonymous memory, etc. Some reasons for this are x86 specific, but some are not. So if we disallow most of the options why allow the interface to take them? And then you are in the position of carefully maintaining a list of not- allowed options instead letting a list of allowed options sit there.

The only benefit I've heard is that it saves creating a new syscall, but it also saves several MAP_ flags. That, and that the RFC for riscv did a PROT_SHADOW_STACK to start. So, yes, two people asked the same question, but I'm still not seeing any benefits. Can you give the pros and cons please?

BTW, in glibc map_shadow_stack is called from arch code. So I think userspace wise, for this to affect other architectures there would need to be some code that could do things generically, with somehow the shadow stack pivot abstracted but the shadow stack allocation not.

#125 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 71 lines · Rick P
Quoted 29 lines · Deepak Gupta
Quoted 28 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 14 lines

So rather than repurpose two existing syscalls (mmap, madvise) that don't
quite fit, just implement a new map_shadow_stack syscall to allow userspace to map and setup new shadow stacks in one step. While ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards this
provide a flag so that stacks can be optionally setup securely for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.

...

Quoted 5 lines

The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

Quoted 9 lines
Quoted 6 lines

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Discussion on this topic in v6

https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Again I know earlier CET patches had protection flag and somehow due to pushback
on mailing list,
it was adopted to go for special syscall because no one else
had shadow stack.

Quoted 6 lines

Seeing a response from Szabolcs, I am assuming arm4 would also want to follow
using mmap to manufacture shadow stack. For reference RFC patches for risc-v shadow stack,
use a new protection flag = PROT_SHADOWSTACK.

https://lore.kernel.org/lkml/[email protected]/

Quoted 11 lines

I know earlier discussion had been that we let this go and do a re- factor later as other
arch support trickle in. But as I thought more on this and I think it may just be
messy from user mode point of view as well to have cognition of two different ways of
creating shadow stack. One would be special syscall (in current libc) and another mmap
(whenever future re-factor happens)

Quoted 2 lines

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

There is sort of two things intermixed here when we talk about a PROT_SHADOW_STACK.

Quoted 7 lines

One is: what is the interface for specifying how the shadow stack should be provisioned with data? Right now there are two ways supported, all zero or with an X86 shadow stack restore token at the end. Then there was already some conversation about a third type. In which case the question would be is using mmap MAP_ flags the right place for this? How many types of initialization will be needed in the end and what is the overlap between the architectures?

First of all, arches can choose to have token at the bottom or not.

Token serve following purposes
- It allows one to put desired value in shadow stack pointer in safe/secure manner.
Note: x86 doesn't provide any opcode encoding to value in SSP register. So having a token is kind of a necessity because x86 doesn't easily allow writing shadow stack.

- A token at the bottom acts marker / barrier and can be useful in debugging

- If (and a big *if*) we ever reach a point in future where return address is only pushed on shadow stack (x86 should have motivation to do this because less uops on call/ret), a token at the bottom (bottom means lower address) is ensuring sure shot way of getting a fault when exhausted.

Current RISCV zisslpcfi proposal doesn't define CPU based tokens because it's RISC. It allows mechanisms using which software can define formatting of token for itself. Not sure of what ARM is doing.

Now coming to the point of all zero v/s shadow stack token. Why not always have token at the bottom?

In case of x86, Why need for two ways and why not always have a token at the bottom. The way x86 is going, user mode is responsible for establishing shadow stack and thus whenever shadow stack is created then if x86 kernel implementation always place a token at the base/bottom.

Now user mode can do following:--
- If it has access to WRSS, it can sure go ahead and create a token of its choosing and overwrite kernel created token. and then do RSTORSSP on it's own created token.

- If it doesn't have access to WRSS (and dont need to create its own token), it can do RSTORSSP on this. As soon as it does, no other thread in process can restore to it. On fork, you get the same un-restorable token.

So why not always have a token at the bottom.
This is my plan for riscv implementation as well (to have a token at the bottom)

Quoted 7 lines

The other thing is: should shadow stack memory creation be tightly controlled? For example in x86 we limit this to anonymous memory, etc. Some reasons for this are x86 specific, but some are not. So if we disallow most of the options why allow the interface to take them? And then you are in the position of carefully maintaining a list of not- allowed options instead letting a list of allowed options sit there.

I am new to linux kernel and thus may be not able to follow the argument of limiting to anonymous memory.

Why is limiting it to anonymous memory a problem. IIRC, ARM's PROT_MTE is applicable only to anonymous memory. I can probably find few more examples.

Eventually syscall will also go ahead and use memory management code to perform mapping. So I didn't understand the reasoning here. The way syscall can limit it to anonymous memory, why mmap can't do the same if it sees PROT_SHADOWSTACK.

Quoted 6 lines

The only benefit I've heard is that it saves creating a new syscall, but it also saves several MAP_ flags. That, and that the RFC for riscv did a PROT_SHADOW_STACK to start. So, yes, two people asked the same question, but I'm still not seeing any benefits. Can you give the pros and cons please?

Again the way syscall will limit it to anonymous memory, Why mmap can't do same? There is precedence for it (like PROT_MTE is applicable only to anonymous memory)

So if it can be done, then why introduce a new syscall?

Quoted 5 lines

BTW, in glibc map_shadow_stack is called from arch code. So I think userspace wise, for this to affect other architectures there would need to be some code that could do things generically, with somehow the shadow stack pivot abstracted but the shadow stack allocation not.

Agreed, yes it can be done in a way where it won't put tax on other architectures.

But what about fragmentation within x86. Will x86 always choose to use system call method map shadow stack. If future re-factor results in x86 also use mmap method. Isn't it a mess for x86 glibc to figure out what to do; whether to use system call or mmap?

#126 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 43 lines · Deepak Gupta
Quoted 42 lines · Rick P
Quoted 41 lines · Deepak Gupta
Quoted 39 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 19 lines

So rather than repurpose two existing syscalls (mmap, madvise)
that don't
quite fit, just implement a new map_shadow_stack syscall to allow
userspace to map and setup new shadow stacks in one step. While
ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards
this
provide a flag so that stacks can be optionally setup securely
for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.

...

Quoted 5 lines

The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Discussion on this topic in v6

https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Again I know earlier CET patches had protection flag and somehow due
to pushback
on mailing list,
it was adopted to go for special syscall because no one else
had shadow stack.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

Seeing a response from Szabolcs, I am assuming arm4 would also want
to follow
using mmap to manufacture shadow stack. For reference RFC patches for
risc-v shadow stack,
use a new protection flag = PROT_SHADOWSTACK.

https://lore.kernel.org/lkml/[email protected]/

Quoted 53 lines
Quoted 21 lines
Quoted 15 lines

I know earlier discussion had been that we let this go and do a re-
factor later as other
arch support trickle in. But as I thought more on this and I think it
may just be
messy from user mode point of view as well to have cognition of two
different ways of
creating shadow stack. One would be special syscall (in current libc)
and another mmap
(whenever future re-factor happens)

Quoted 2 lines

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

There is sort of two things intermixed here when we talk about a PROT_SHADOW_STACK.

Quoted 10 lines

One is: what is the interface for specifying how the shadow stack should be provisioned with data? Right now there are two ways supported, all zero or with an X86 shadow stack restore token at the
end. Then there was already some conversation about a third type. In
which case the question would be is using mmap MAP_ flags the right place for this? How many types of initialization will be needed in the
end and what is the overlap between the architectures?

First of all, arches can choose to have token at the bottom or not.

Token serve following purposes
- It allows one to put desired value in shadow stack pointer in
safe/secure manner.
Note: x86 doesn't provide any opcode encoding to value in SSP
register. So having
a token is kind of a necessity because x86 doesn't easily allow
writing shadow stack.

- A token at the bottom acts marker / barrier and can be useful in
debugging

- If (and a big *if*) we ever reach a point in future where return
address is only pushed
on shadow stack (x86 should have motivation to do this because
less uops on call/ret),
a token at the bottom (bottom means lower address) is ensuring
sure shot way of getting
a fault when exhausted.

Quoted 5 lines

Current RISCV zisslpcfi proposal doesn't define CPU based tokens because it's RISC.
It allows mechanisms using which software can define formatting of token for itself.
Not sure of what ARM is doing.

Ok, so riscv doesn't need to have the kernel write the token, but x86 does.

Quoted 3 lines

Now coming to the point of all zero v/s shadow stack token. Why not always have token at the bottom?

With WRSS you can setup the shadow stack however you want. So the user would then have to take care to erase the token if they didn't want it. Not the end of the world, but kind of clunky if there is no reason for it.

Quoted 8 lines

In case of x86, Why need for two ways and why not always have a token at the bottom.
The way x86 is going, user mode is responsible for establishing shadow stack and thus
whenever shadow stack is created then if x86 kernel implementation always place a token
at the base/bottom.

There was also some discussion recently of adding a token AND an end of stack marker, as a potential solution for backtracing in ucontext stacks. In this case it could cause an ABI break to just start adding the end of stack marker where the token was, and so would require a new map_shadow_stack flag.

Quoted 29 lines

Now user mode can do following:--
- If it has access to WRSS, it can sure go ahead and create a token
of its choosing and
overwrite kernel created token. and then do RSTORSSP on it's own
created token.

- If it doesn't have access to WRSS (and dont need to create its
own token), it can do
RSTORSSP on this. As soon as it does, no other thread in process
can restore to it.
On fork, you get the same un-restorable token.

So why not always have a token at the bottom.
This is my plan for riscv implementation as well (to have a token at the bottom)

Quoted 11 lines

The other thing is: should shadow stack memory creation be tightly controlled? For example in x86 we limit this to anonymous memory, etc.
Some reasons for this are x86 specific, but some are not. So if we disallow most of the options why allow the interface to take them? And
then you are in the position of carefully maintaining a list of not-
allowed options instead letting a list of allowed options sit there.

Quoted 7 lines

I am new to linux kernel and thus may be not able to follow the argument of
limiting to anonymous memory.

Why is limiting it to anonymous memory a problem. IIRC, ARM's PROT_MTE is applicable
only to anonymous memory. I can probably find few more examples.

Oh I see, they have a special arch VMA flag VM_MTE_ALLOWED that only gets set if all the rules are followed. Then PROT_MTE can only be set on that to set VM_MTE. That is kind of nice because certain other special situations can choose to support it.

It does take another arch vma flag though. For x86 I guess I would need to figure out how to squeeze VM_SHADOW_STACK into other flags to have a free flag to use the same method. It also only supports mprotect() and shadow stack would only want to support mmap(). And you still have the initialization stuff to plumb through. Yea, I think the PROT_MTE is a good thing to consider, but it's not super obvious to me how similar the logic would be for shadow stack.

The question I'm asking though is, not "can mmap code and rules be changed to enforce the required limitations?". I think it is yes. But the question is "why is that plumbing better than a new syscall?". I guess to get a better idea, the mmap solution would need to get POCed. I had half done this at one point, but abandoned the approach.

For your question about why limit it, the special x86 case is the Dirty=1,Write=0 PTE bit combination for shadow stacks. So for shadow stack you could have some confusion about whether a PTE is actually dirty for writeback, etc. I wouldn't say it's known to be impossible to do MAP_SHARED, but it has not been fully analyzed enough to know what the changes would be. There were some solvable concrete issues that tipped the scale as well. It was also not expected to be a common usage, if at all.

The non-x86, general reasons for it, are for a smaller benefit. It blocks a lot of ways shadow stack memory could be written to. Like say you have a memory mapped writable file, and you also map it shadow stack. So it has better security properties depending on what your threat model is.

Quoted 37 lines

Eventually syscall will also go ahead and use memory management code to
perform mapping. So I didn't understand the reasoning here. The way syscall
can limit it to anonymous memory, why mmap can't do the same if it sees
PROT_SHADOWSTACK.

Quoted 10 lines

The only benefit I've heard is that it saves creating a new syscall,
but it also saves several MAP_ flags. That, and that the RFC for riscv
did a PROT_SHADOW_STACK to start. So, yes, two people asked the same
question, but I'm still not seeing any benefits. Can you give the pros
and cons please?

Again the way syscall will limit it to anonymous memory, Why mmap can't do same?
There is precedence for it (like PROT_MTE is applicable only to anonymous memory)

So if it can be done, then why introduce a new syscall?

Quoted 6 lines

BTW, in glibc map_shadow_stack is called from arch code. So I think userspace wise, for this to affect other architectures there would need
to be some code that could do things generically, with somehow the shadow stack pivot abstracted but the shadow stack allocation not.

Agreed, yes it can be done in a way where it won't put tax on other architectures.

Quoted 8 lines

But what about fragmentation within x86. Will x86 always choose to use system call
method map shadow stack. If future re-factor results in x86 also use mmap method.
Isn't it a mess for x86 glibc to figure out what to do; whether to use system call
or mmap?

Ok, so this is the downside I guess. What happens if we want to support the other types of memory in the future and end up using mmap for this? Then we have 15-20 lines of extra syscall wrapping code to maintain to support legacy.

For the mmap solution, we have the downside of using extra MAP_ flags, and *some* amount of currently unknown vm_flag and address range logic, plus mmap arch breakouts to add to core MM. Like I said earlier, you would need to POC it out to see how bad that looks and get some core MM feedback on the new type of MAP flag usage. But, syscalls being pretty straightforward, it would probably be *some* amount of added complexity _now_ to support something that might happen in the future. I'm not seeing either one as a landslide win.

It's kind of an eternal software design philosophical question, isn't it? How much work should you do to prepare for things that might be needed in the future? From what I've seen the balance in the kernel seems to be to try not to paint yourself in to an ABI corner, but otherwise let the kernel evolve naturally in response to real usages. If anyone wants to correct this, please do. But otherwise I think the new syscall is aligned with that.

TBH, you are making me wonder if I'm missing something. It seems you strongly don't prefer this approach, but I'm not hearing any huge potential negative impacts. And you also say it won't tax the riscv implementation. Is this just something just smells bad here? Or it would shrink the riscv series?

#127 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 131 lines · Rick P
Quoted 43 lines · Deepak Gupta
Quoted 42 lines · Rick P
Quoted 41 lines · Deepak Gupta
Quoted 39 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 19 lines

So rather than repurpose two existing syscalls (mmap, madvise)
that don't
quite fit, just implement a new map_shadow_stack syscall to allow
userspace to map and setup new shadow stacks in one step. While
ucontext
is the primary motivator, userspace may have other unforeseen reasons to
setup it's own shadow stacks using the WRSS instruction. Towards
this
provide a flag so that stacks can be optionally setup securely
for the
common case of ucontext without enabling WRSS. Or potentially have the
kernel set up the shadow stack in some new way.

...

Quoted 5 lines

The following example demonstrates how to create a new shadow stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for it but could be squeezed in).

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Discussion on this topic in v6

https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Again I know earlier CET patches had protection flag and somehow due
to pushback
on mailing list,
it was adopted to go for special syscall because no one else
had shadow stack.

Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

Seeing a response from Szabolcs, I am assuming arm4 would also want
to follow
using mmap to manufacture shadow stack. For reference RFC patches for
risc-v shadow stack,
use a new protection flag = PROT_SHADOWSTACK.

https://lore.kernel.org/lkml/[email protected]/

Quoted 53 lines
Quoted 21 lines
Quoted 15 lines

I know earlier discussion had been that we let this go and do a re-
factor later as other
arch support trickle in. But as I thought more on this and I think it
may just be
messy from user mode point of view as well to have cognition of two
different ways of
creating shadow stack. One would be special syscall (in current libc)
and another mmap
(whenever future re-factor happens)

Quoted 2 lines

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

There is sort of two things intermixed here when we talk about a PROT_SHADOW_STACK.

Quoted 10 lines

One is: what is the interface for specifying how the shadow stack should be provisioned with data? Right now there are two ways supported, all zero or with an X86 shadow stack restore token at the
end. Then there was already some conversation about a third type. In
which case the question would be is using mmap MAP_ flags the right place for this? How many types of initialization will be needed in the
end and what is the overlap between the architectures?

First of all, arches can choose to have token at the bottom or not.

Token serve following purposes
- It allows one to put desired value in shadow stack pointer in
safe/secure manner.
Note: x86 doesn't provide any opcode encoding to value in SSP
register. So having
a token is kind of a necessity because x86 doesn't easily allow
writing shadow stack.

- A token at the bottom acts marker / barrier and can be useful in
debugging

- If (and a big *if*) we ever reach a point in future where return
address is only pushed
on shadow stack (x86 should have motivation to do this because
less uops on call/ret),
a token at the bottom (bottom means lower address) is ensuring
sure shot way of getting
a fault when exhausted.

Quoted 5 lines

Current RISCV zisslpcfi proposal doesn't define CPU based tokens because it's RISC.
It allows mechanisms using which software can define formatting of token for itself.
Not sure of what ARM is doing.

Ok, so riscv doesn't need to have the kernel write the token, but x86 does.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

Now coming to the point of all zero v/s shadow stack token. Why not always have token at the bottom?

With WRSS you can setup the shadow stack however you want. So the user would then have to take care to erase the token if they didn't want it. Not the end of the world, but kind of clunky if there is no reason for it.

Yes but kernel always assumes the user is going to use the token. It' upto the user to decide whether they want to use the restore token or not. If they've WRSS capability security posture is anyways diluted. An attacker who would be clever enough to re-use RSTORSSP present in address space to restore using kernel prepared token, should anyways can be clever enough to use WRSS as well.

It kind of makes shadow stack creation simpler for kernel to always place the token. This point is irrespective of whether to use system call or mmap.

Quoted 10 lines
Quoted 9 lines

In case of x86, Why need for two ways and why not always have a token at the bottom.
The way x86 is going, user mode is responsible for establishing shadow stack and thus
whenever shadow stack is created then if x86 kernel implementation always place a token
at the base/bottom.

Quoted 5 lines

There was also some discussion recently of adding a token AND an end of stack marker, as a potential solution for backtracing in ucontext stacks. In this case it could cause an ABI break to just start adding the end of stack marker where the token was, and so would require a new map_shadow_stack flag.

Was this discussed why restore token itself can't be used as marker for end of stack (if we assume there is always going to be one at the bottom). It's a unique value. An address pointing to itself.

Quoted 34 lines
Quoted 34 lines

Now user mode can do following:--
- If it has access to WRSS, it can sure go ahead and create a token
of its choosing and
overwrite kernel created token. and then do RSTORSSP on it's own
created token.

- If it doesn't have access to WRSS (and dont need to create its
own token), it can do
RSTORSSP on this. As soon as it does, no other thread in process
can restore to it.
On fork, you get the same un-restorable token.

So why not always have a token at the bottom.
This is my plan for riscv implementation as well (to have a token at the bottom)

Quoted 11 lines

The other thing is: should shadow stack memory creation be tightly controlled? For example in x86 we limit this to anonymous memory, etc.
Some reasons for this are x86 specific, but some are not. So if we disallow most of the options why allow the interface to take them? And
then you are in the position of carefully maintaining a list of not-
allowed options instead letting a list of allowed options sit there.

I am new to linux kernel and thus may be not able to follow the argument of
limiting to anonymous memory.

Quoted 8 lines
Quoted 3 lines

Why is limiting it to anonymous memory a problem. IIRC, ARM's PROT_MTE is applicable
only to anonymous memory. I can probably find few more examples.

Oh I see, they have a special arch VMA flag VM_MTE_ALLOWED that only gets set if all the rules are followed. Then PROT_MTE can only be set on that to set VM_MTE. That is kind of nice because certain other special situations can choose to support it.

That's because MTE is different. It allows to assign tags to existing virtual memory. So one need to know whether a memory can have tags assigned.

Quoted 8 lines

It does take another arch vma flag though. For x86 I guess I would need to figure out how to squeeze VM_SHADOW_STACK into other flags to have a free flag to use the same method. It also only supports mprotect() and shadow stack would only want to support mmap(). And you still have the initialization stuff to plumb through. Yea, I think the PROT_MTE is a good thing to consider, but it's not super obvious to me how similar the logic would be for shadow stack.

I dont think you need another VMA flag. Memory tagging allows adding tags to existing virtual memory. That's why having mprotect makes sense for MTE. In shadow stack case, there is no requirement of changing a shadow stack to regular memory or vice-versa.

All that's needed to change is mmap. mprotect should fail. Syscall approach gives that benefit by default because there is no protection flag for shadow stack.

I was giving example that any feature which gives new meaning to virtual memory has been able to work with existing memory mapping APIs without the need of new system call (including whether you're dealing with anonymous memory).

Quoted 7 lines

The question I'm asking though is, not "can mmap code and rules be changed to enforce the required limitations?". I think it is yes. But the question is "why is that plumbing better than a new syscall?". I guess to get a better idea, the mmap solution would need to get POCed. I had half done this at one point, but abandoned the approach.

Quoted 8 lines

For your question about why limit it, the special x86 case is the Dirty=1,Write=0 PTE bit combination for shadow stacks. So for shadow stack you could have some confusion about whether a PTE is actually dirty for writeback, etc. I wouldn't say it's known to be impossible to do MAP_SHARED, but it has not been fully analyzed enough to know what the changes would be. There were some solvable concrete issues that tipped the scale as well. It was also not expected to be a common usage, if at all.

I am not sure how confusion of D=1,W=0 is not completely taken away by syscall approach. It'll always be there. One can only do things to minimize the chances.

In case of syscall approach, syscall makes sure that

flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G

This can be easily checked in arch specific landing function for mmap.

Additionally, If you always have the token at base, you don't need that ABI between user and kernel.

Quoted 6 lines

The non-x86, general reasons for it, are for a smaller benefit. It blocks a lot of ways shadow stack memory could be written to. Like say you have a memory mapped writable file, and you also map it shadow stack. So it has better security properties depending on what your threat model is.

I wouldn't say any architecture should allow such primitives. It kind of defeats the purpose for shadow stack. Yes if some sort of secure memory is needed, there may be new ISA extensions for that.

Quoted 69 lines
Quoted 38 lines

Eventually syscall will also go ahead and use memory management code to
perform mapping. So I didn't understand the reasoning here. The way syscall
can limit it to anonymous memory, why mmap can't do the same if it sees
PROT_SHADOWSTACK.

Quoted 10 lines

The only benefit I've heard is that it saves creating a new syscall,
but it also saves several MAP_ flags. That, and that the RFC for riscv
did a PROT_SHADOW_STACK to start. So, yes, two people asked the same
question, but I'm still not seeing any benefits. Can you give the pros
and cons please?

Again the way syscall will limit it to anonymous memory, Why mmap can't do same?
There is precedence for it (like PROT_MTE is applicable only to anonymous memory)

So if it can be done, then why introduce a new syscall?

Quoted 6 lines

BTW, in glibc map_shadow_stack is called from arch code. So I think userspace wise, for this to affect other architectures there would need
to be some code that could do things generically, with somehow the shadow stack pivot abstracted but the shadow stack allocation not.

Agreed, yes it can be done in a way where it won't put tax on other architectures.

Quoted 8 lines

But what about fragmentation within x86. Will x86 always choose to use system call
method map shadow stack. If future re-factor results in x86 also use mmap method.
Isn't it a mess for x86 glibc to figure out what to do; whether to use system call
or mmap?

Ok, so this is the downside I guess. What happens if we want to support the other types of memory in the future and end up using mmap for this? Then we have 15-20 lines of extra syscall wrapping code to maintain to support legacy.

For the mmap solution, we have the downside of using extra MAP_ flags, and *some* amount of currently unknown vm_flag and address range logic, plus mmap arch breakouts to add to core MM. Like I said earlier, you would need to POC it out to see how bad that looks and get some core MM feedback on the new type of MAP flag usage. But, syscalls being pretty straightforward, it would probably be *some* amount of added complexity _now_ to support something that might happen in the future. I'm not seeing either one as a landslide win.

It's kind of an eternal software design philosophical question, isn't it? How much work should you do to prepare for things that might be needed in the future? From what I've seen the balance in the kernel seems to be to try not to paint yourself in to an ABI corner, but otherwise let the kernel evolve naturally in response to real usages. If anyone wants to correct this, please do. But otherwise I think the new syscall is aligned with that.

Quoted 5 lines

TBH, you are making me wonder if I'm missing something. It seems you strongly don't prefer this approach, but I'm not hearing any huge potential negative impacts. And you also say it won't tax the riscv implementation. Is this just something just smells bad here? Or it would shrink the riscv series?

No you're not missing anything. It's just wierdness of adding a system call
which enforces certain MAP_XX flags and pretty much mapping API.
And difference between architectures on how they will create shadow stack. + if x86 chooses to use mmap in future, then there is ugliness in user mode to decide which method to choose.

And yes you got it right, to some extent there is my own selfishness playing out as well here to reduce riscv patches.

#128 lore raw

Re: [PATCH v7 33/41] x86/shstk: Introduce map_shadow_stack syscall

Quoted 54 lines · Deepak Gupta
Quoted 53 lines · Rick P
Quoted 52 lines · Deepak Gupta
Quoted 50 lines · Rick P
Quoted 49 lines · Deepak Gupta
Quoted 46 lines · Szabolcs Nagy
Quoted 1 line · Rick Edgecombe

Previously, a new PROT_SHADOW_STACK was attempted,

...

Quoted 22 lines

So rather than repurpose two existing syscalls (mmap, madvise)
that don't
quite fit, just implement a new map_shadow_stack syscall to
allow
userspace to map and setup new shadow stacks in one step. While
ucontext
is the primary motivator, userspace may have other unforeseen
reasons to
setup it's own shadow stacks using the WRSS instruction. Towards
this
provide a flag so that stacks can be optionally setup securely
for the
common case of ucontext without enabling WRSS. Or potentially
have the
kernel set up the shadow stack in some new way.

...

Quoted 6 lines

The following example demonstrates how to create a new shadow
stack with
map_shadow_stack:
void *shstk = map_shadow_stack(addr, stack_size, SHADOW_STACK_SET_TOKEN);

i think

mmap(addr, size, PROT_READ, MAP_ANON|MAP_SHADOW_STACK, -1, 0);

could do the same with less disruption to users (new syscalls
are harder to deal with than new flags). it would do the guard page and initial token setup too (there is no flag for
it but could be squeezed in).

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Discussion on this topic in v6

https://lore.kernel.org/all/[email protected]/

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Again I know earlier CET patches had protection flag and somehow
due
to pushback
on mailing list,
it was adopted to go for special syscall because no one else
had shadow stack.

Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Seeing a response from Szabolcs, I am assuming arm4 would also
want
to follow

Quoted 10 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

using mmap to manufacture shadow stack. For reference RFC patches
for
risc-v shadow stack,
use a new protection flag = PROT_SHADOWSTACK.

https://lore.kernel.org/lkml/[email protected]/

Quoted 101 lines
Quoted 83 lines
Quoted 68 lines
Quoted 25 lines
Quoted 18 lines

I know earlier discussion had been that we let this go and do a
re-
factor later as other
arch support trickle in. But as I thought more on this and I think it
may just be
messy from user mode point of view as well to have cognition of
two
different ways of
creating shadow stack. One would be special syscall (in current
libc)
and another mmap
(whenever future re-factor happens)

Quoted 2 lines

If it's not too late, it would be more wise to take mmap approach rather than special syscall approach.

There is sort of two things intermixed here when we talk about a
PROT_SHADOW_STACK.

Quoted 15 lines

One is: what is the interface for specifying how the shadow stack
should be provisioned with data? Right now there are two ways supported, all zero or with an X86 shadow stack restore token at
the
end. Then there was already some conversation about a third type.
In
which case the question would be is using mmap MAP_ flags the right
place for this? How many types of initialization will be needed in
the
end and what is the overlap between the architectures?

First of all, arches can choose to have token at the bottom or not.

Token serve following purposes
- It allows one to put desired value in shadow stack pointer in
safe/secure manner.
Note: x86 doesn't provide any opcode encoding to value in SSP
register. So having
a token is kind of a necessity because x86 doesn't easily
allow
writing shadow stack.

- A token at the bottom acts marker / barrier and can be useful
in
debugging

- If (and a big *if*) we ever reach a point in future where
return
address is only pushed
on shadow stack (x86 should have motivation to do this
because
less uops on call/ret),
a token at the bottom (bottom means lower address) is
ensuring
sure shot way of getting
a fault when exhausted.

Quoted 6 lines

Current RISCV zisslpcfi proposal doesn't define CPU based tokens because it's RISC.
It allows mechanisms using which software can define formatting of
token for itself.
Not sure of what ARM is doing.

Ok, so riscv doesn't need to have the kernel write the token, but x86
does.

Quoted 3 lines

Now coming to the point of all zero v/s shadow stack token. Why not always have token at the bottom?

Quoted 7 lines

With WRSS you can setup the shadow stack however you want. So the user
would then have to take care to erase the token if they didn't want it.
Not the end of the world, but kind of clunky if there is no reason for
it.

Yes but kernel always assumes the user is going to use the token. It' upto the user
to decide whether they want to use the restore token or not. If they've WRSS capability
security posture is anyways diluted. An attacker who would be clever enough to
re-use RSTORSSP present in address space to restore using kernel prepared token, should
anyways can be clever enough to use WRSS as well.

Quoted 3 lines

It kind of makes shadow stack creation simpler for kernel to always place the token.
This point is irrespective of whether to use system call or mmap.

Think about like CRIU restoring the shadow stack, or other special cases like that. Userspace can always overwrite the token, but this involves some amount of extra work (extra writes, earlier faulting in the page, etc). It is clunky and very negligibly worse.

Quoted 22 lines
Quoted 13 lines
Quoted 12 lines

In case of x86, Why need for two ways and why not always have a token
at the bottom.
The way x86 is going, user mode is responsible for establishing shadow stack and thus
whenever shadow stack is created then if x86 kernel implementation
always place a token
at the base/bottom.

Quoted 8 lines

There was also some discussion recently of adding a token AND an end of
stack marker, as a potential solution for backtracing in ucontext stacks. In this case it could cause an ABI break to just start adding
the end of stack marker where the token was, and so would require a new
map_shadow_stack flag.

Quoted 5 lines

Was this discussed why restore token itself can't be used as marker for
end of stack (if we assume there is always going to be one at the bottom).
It's a unique value. An address pointing to itself.

I thought the same thing at first, but it gets clobbered during the pivot and push.

Quoted 74 lines
Quoted 48 lines
Quoted 40 lines

Now user mode can do following:--
- If it has access to WRSS, it can sure go ahead and create a
token
of its choosing and
overwrite kernel created token. and then do RSTORSSP on it's
own
created token.

- If it doesn't have access to WRSS (and dont need to create
its
own token), it can do
RSTORSSP on this. As soon as it does, no other thread in
process
can restore to it.
On fork, you get the same un-restorable token.

So why not always have a token at the bottom.
This is my plan for riscv implementation as well (to have a token at
the bottom)

Quoted 15 lines

The other thing is: should shadow stack memory creation be tightly
controlled? For example in x86 we limit this to anonymous memory,
etc.
Some reasons for this are x86 specific, but some are not. So if we
disallow most of the options why allow the interface to take them?
And
then you are in the position of carefully maintaining a list of not-
allowed options instead letting a list of allowed options sit there.

Quoted 7 lines

I am new to linux kernel and thus may be not able to follow the argument of
limiting to anonymous memory.

Why is limiting it to anonymous memory a problem. IIRC, ARM's PROT_MTE is applicable
only to anonymous memory. I can probably find few more examples.

Quoted 6 lines

Oh I see, they have a special arch VMA flag VM_MTE_ALLOWED that only
gets set if all the rules are followed. Then PROT_MTE can only be set
on that to set VM_MTE. That is kind of nice because certain other special situations can choose to support it.

That's because MTE is different. It allows to assign tags to existing virtual memory. So one need to know whether a memory can have tags assigned.

Quoted 14 lines

It does take another arch vma flag though. For x86 I guess I would need
to figure out how to squeeze VM_SHADOW_STACK into other flags to have a
free flag to use the same method. It also only supports mprotect() and
shadow stack would only want to support mmap(). And you still have the
initialization stuff to plumb through. Yea, I think the PROT_MTE is a
good thing to consider, but it's not super obvious to me how similar
the logic would be for shadow stack.

Quoted 3 lines

I dont think you need another VMA flag. Memory tagging allows adding tags
to existing virtual memory.

...need another VMA flag to use the existing mmap arch breakouts in the same way as VM_MTE. Of course changing mmap makes other solutions possible.

Quoted 4 lines

That's why having mprotect makes sense for MTE.
In shadow stack case, there is no requirement of changing a shadow stack
to regular memory or vice-versa.

uffd needs mprotect internals. You might take a look at it in regards to your VM_WRITE/mprotect blocking approach for riscv. I was imagining, even if mmap was the syscall, mprotect() would not be blocked in the x86 case at least. The mprotect() blocking is a separate thing than the syscall, right?

Quoted 41 lines

All that's needed to change is mmap. mprotect should fail. Syscall
approach gives that benefit by default because there is no protection flag
for shadow stack.

I was giving example that any feature which gives new meaning to virtual memory
has been able to work with existing memory mapping APIs without the need of new
system call (including whether you're dealing with anonymous memory).

Quoted 10 lines

The question I'm asking though is, not "can mmap code and rules be changed to enforce the required limitations?". I think it is yes. But
the question is "why is that plumbing better than a new syscall?". I
guess to get a better idea, the mmap solution would need to get POCed.
I had half done this at one point, but abandoned the approach.

Quoted 11 lines

For your question about why limit it, the special x86 case is the Dirty=1,Write=0 PTE bit combination for shadow stacks. So for shadow
stack you could have some confusion about whether a PTE is actually dirty for writeback, etc. I wouldn't say it's known to be impossible to
do MAP_SHARED, but it has not been fully analyzed enough to know what
the changes would be. There were some solvable concrete issues that tipped the scale as well. It was also not expected to be a common usage, if at all.

I am not sure how confusion of D=1,W=0 is not completely taken away by
syscall approach. It'll always be there. One can only do things to minimize
the chances.

Quoted 6 lines

In case of syscall approach, syscall makes sure that

flags = MAP_ANONYMOUS | MAP_PRIVATE | MAP_ABOVE4G

This can be easily checked in arch specific landing function for mmap.

Right, this is why I listed two types of things in the mix here. The memory features supported, and what the syscall is. You asked why limit the memory features, so that is the explanation.

Quoted 15 lines

Additionally, If you always have the token at base, you don't need that ABI
between user and kernel.

Quoted 7 lines

The non-x86, general reasons for it, are for a smaller benefit. It blocks a lot of ways shadow stack memory could be written to. Like say
you have a memory mapped writable file, and you also map it shadow stack. So it has better security properties depending on what your threat model is.

Quoted 5 lines

I wouldn't say any architecture should allow such primitives. It kind of defeats
the purpose for shadow stack. Yes if some sort of secure memory is needed, there may
be new ISA extensions for that.

Yea, seems reasonable to prevent this regardless of the extra x86 reasons, if that is what you are saying. It depends on people's threat models (as always in security).

Quoted 102 lines
Quoted 95 lines
Quoted 49 lines

Eventually syscall will also go ahead and use memory management code
to
perform mapping. So I didn't understand the reasoning here. The way
syscall
can limit it to anonymous memory, why mmap can't do the same if it
sees
PROT_SHADOWSTACK.

Quoted 12 lines

The only benefit I've heard is that it saves creating a new syscall,
but it also saves several MAP_ flags. That, and that the RFC for
riscv
did a PROT_SHADOW_STACK to start. So, yes, two people asked the same
question, but I'm still not seeing any benefits. Can you give the
pros
and cons please?

Again the way syscall will limit it to anonymous memory, Why mmap can't do same?
There is precedence for it (like PROT_MTE is applicable only to anonymous memory)

So if it can be done, then why introduce a new syscall?

Quoted 10 lines

BTW, in glibc map_shadow_stack is called from arch code. So I think
userspace wise, for this to affect other architectures there would
need
to be some code that could do things generically, with somehow the
shadow stack pivot abstracted but the shadow stack allocation not.

Agreed, yes it can be done in a way where it won't put tax on other
architectures.

Quoted 11 lines

But what about fragmentation within x86. Will x86 always choose to
use system call
method map shadow stack. If future re-factor results in x86 also use
mmap method.
Isn't it a mess for x86 glibc to figure out what to do; whether to
use system call
or mmap?

Ok, so this is the downside I guess. What happens if we want to support
the other types of memory in the future and end up using mmap for this?
Then we have 15-20 lines of extra syscall wrapping code to maintain to
support legacy.

For the mmap solution, we have the downside of using extra MAP_ flags,
and *some* amount of currently unknown vm_flag and address range logic,
plus mmap arch breakouts to add to core MM. Like I said earlier, you
would need to POC it out to see how bad that looks and get some core MM
feedback on the new type of MAP flag usage. But, syscalls being pretty
straightforward, it would probably be *some* amount of added complexity
_now_ to support something that might happen in the future. I'm not seeing either one as a landslide win.

It's kind of an eternal software design philosophical question, isn't
it? How much work should you do to prepare for things that might be needed in the future? From what I've seen the balance in the kernel seems to be to try not to paint yourself in to an ABI corner, but otherwise let the kernel evolve naturally in response to real usages.
If anyone wants to correct this, please do. But otherwise I think the
new syscall is aligned with that.

Quoted 6 lines

TBH, you are making me wonder if I'm missing something. It seems you
strongly don't prefer this approach, but I'm not hearing any huge potential negative impacts. And you also say it won't tax the riscv implementation. Is this just something just smells bad here? Or it would shrink the riscv series?

Quoted 8 lines

No you're not missing anything. It's just wierdness of adding a system call
which enforces certain MAP_XX flags and pretty much mapping API. And difference between architectures on how they will create shadow stack. +
if x86 chooses to use mmap in future, then there is ugliness in user mode to
decide which method to choose.

Ok, I think I will leave it given it's entirely in arch/x86. It just got some special error codes in the other thread today too.

Quoted 5 lines

And yes you got it right, to some extent there is my own selfishness playing out
as well here to reduce riscv patches.

Feel free to join the map_shadow_stack party. :)