Skip to contentSkip to conversation

Shadowstacks for userspace

PATCH Superseded Rick Edgecombe 23 participants 241 messages Sep 29, 2022 – Nov 3, 2022 Page 1 of 2

Subsystems (23)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover Shadowstacks for userspace

Hi,

This is an overdue followup to the “Shadow stacks for userspace” CET series. Thanks for all the comments on the first version [0]. They drove a decent amount of changes for v2. Since it has been awhile, I’ll try to summarize the areas that got major changes since last time. Smaller changes are listed in each patch.

The coverletter is organized into the following sections:
1. Shadow Stack Memory Solution
2. FPU API
3. Alt Shadow Stacks
4. Compatibility of Existing Binaries/Enabling Interface
5. CRIU Support
6. Bigger Selftest

Last time, two bigger pieces of new functionality were requested (Alt shadow stack and CRIU support). Alt shadow stack support was requested, not because there was an immediate need, but more because of the risk of signal ABI decisions made now, creating implementation problems if alt shadow stacks were added later.

A POC for alt shadow stacks may be enough to gauge this risk. CRIU support may also not be something critical for day one, if glibc disables all existing binaries as described in section 4. So I marked the patches at the end that support those two things as RFC/OPTIONAL. The earlier patches will support a smaller, basic initial implementation. So I’m wondering if we could consider just enabling the basics upstream first, assuming the RFC pieces here look passable.

1. Shadow Stack Memory Solution
===============================
Dave had a lot of questions and feedback about how shadow stack memory is handled, including why shadow stack VMAs were not VM_WRITE. These questions prompted a revisit of the design, and in the end shadow stack’s were switched to be VM_WRITE. I’ve tried to summarize how shadow stack memory is supposed to work, with some examples of how MM features interact with shadow stack memory.

Shadow Stack Memory Summary
---------------------------
Integrating shadow stack memory into the kernel has two main challenges. One, Write=0,Dirty=1 PTEs are already created by the kernel, and now they can’t be or they will inadvertently create shadow stack memory.

And, two, shadow stack memory fits strangely into the existing concepts of Copy-On-Write and “writable” memory. It is *sort of* writable, in that it can be changed by userspace, but sort of not in that it has Write=0 and can’t be written by normal mov-type accesses. So we still have the “writable” memory we always had, but now we also have another type of memory that is changeable from userspace. Another weird aspect is that memory has to be shadow stack, in order to serve a “shadow stack read”, so a shadow stack read also needs to cause something like a Copy-On-Write, as the result will be changeable from userspace.

Dealing with the new meaning of Dirty and Write bits
----------------------------------------------------
The first issue is solved with creating PAGE_COW using a software PTE bit. This is hidden inside the pgtable.h helpers, such that it *mostly* (more on this later) happens without changing core code. Basically in pte_wrprotect() will clear Dirty and set Cow=1, if the pte was dirty. In pte_mkdirty(), it set’s COW if the PTE was Write=0. Then pte_dirty() returns true for Dirty=1 or Cow=1. Since this requires a little extra work, this behavior is compiled out when shadow stack support is not enabled for the kernel.

Dealing with a new type of writable memory
------------------------------------------
The other side of the problem - dealing with the concept-splitting new type of userspace changeable memory - leaves a bit more loose ends. Probably the most important thing is that we don’t want the kernel thinking that shadow stack memory is protected from changes from userspace. But we also don’t want the kernel to treat it like normal writable memory in some ways either, for example to get confused and inadvertently make it writable in the normal (PTE Write=1) sense.

The solution here is to treat shadow stack memory as a special class of writable memory by updating places where memory is made writable to be aware of it, and treat all shadow stack accesses as if they are writes.

Shadow stack accesses are always treated as write faults because even shadow stack reads need to be made (shadow stack) writable in order to service them. Logic creating PTE’s then decides whether to create shadow stack or normal writable memory by the VMA type. Most of this is encapsulated in maybe_mkwrite() but some differentiation needs to be open coded where pte_mkwrite() is called directly.

Shadow stack VMA’s are a special type of writable and so they are created as VM_WRITE | VM_SHADOW_STACK. The benefit of making them also VM_WRITE is that there is some existing logic around using VM_WRITE to make decisions in the kernel that apply to shadow stack memory as well.
- Scheduling code decides whether to migrate a VMA depending on whether
it’s VM_WRITE. The same reasoning should apply for shadow stack memory.
- While there is no current interface for mmap()ing files as shadow stack, various drivers enforce non-writable mappings by checking !VM_WRITE and clearing VM_MAYWRITE. Because there is no longer a way to mmap() something arbitrarily as shadow stack, this can’t be hit. But this un-hittable wrong logic makes the design confusing and brittle.

The downside of having shadow stack memory have VM_WRITE is that any logic that assumes VM_WRITE means normally writable, for example open coded like:
if (flags & VM_WRITE)
pte_mkwrite()
...will no longer be correct. It will need to be changed to have additional logic that knows about shadow stack. It turns out there are not too many of these cases and so this series just adds the logic.

This solution for this second issue also tweaks the behavior of pte_write() and pte_dirty(). pte_write() check’s whether a pte is writable or not, previously this was only the case when Write=1, but now pte_write() also returns true for shadow stack memory.

There are some additional areas that are probably worth commenting on:

COW
---
When a shadow stack page is shared as part of COW, it becomes read-only, just like normally writable memory would be. As part of the Dirty bit solution described above, pte_wrprotect() will move Dirty=1 to COW=1. This will leave the PTE in a read-only state automatically. Then when it takes a shadow stack access, it will perform COW, copying the page and making it writable. Logic added as part of the shadow stack memory solution will detect that the VMA is shadow stack and make the PTE a shadow stack PTE.

mprotect()/VM_WRITE
-------------------
Shadow stack memory doesn’t have a PROT flag. It is created either internally in the kernel or via a special syscall. When it is created this way, the VMA gets VM_WRITE|VM_SHADOW_STACK. However, some functionality of the kernel will remove VM_WRITE, for example mprotect(). When this happens the memory is expected to be read only. So without any intervention, there may be a VMA that is VM_SHADOW_STACK and not VM_WRITE. We could try to prevent this from happening, (for example block mprotect() from operating on shadow stack memory), however some things like userfaulfd call mprotect internally and depend on it to work.

So mprotect()ing shadow stack memory can make it read-only (non-shadow stack). It can then become shadow stack again by mprotect()ing it with PROT_WRITE. It always keeps the VM_SHADOW_STACK, so that it can never become normally writable memory.

GUP
---
Shadow stack memory is generally treated as writable by the kernel, but it behaves differently then other writable memory with respect to GUP. FOLL_WRITE will not GUP shadow stack memory unless FOLL_FORCE is also set. Shadow stack memory is writable from the perspective of being changeable by userspace, but it is also protected memory from userspace’s perspective. So preventing it from being writable via FOLL_WRITE help’s make it harder for userspace to arbitrarily write to it. However, like read-only memory, FOLL_FORCE can still write through it. This means shadow stacks can be written to via things like “/proc/self/mem”. Apps that want extra security will have to prevent access to kernel features that can write with FOLL_FORCE.

2. FPU API
==========
The last version of this had an interface for modifying the FPU state in either the buffer or the registers to try to minimize saves and restores. Shortly after that, Thomas experimented with a different fpu optimization that was incompatible with how the interface kept state in the caller. So it doesn't seem like a robust interface and for this version the optimization piece of the API is dropped in this series, and the force restore technique is used again.

3. Alt Shadow Stacks
====================
Andy Lutomirski asked about alt shadow stack support. The following describes the design of shadow stack support for signals and alt shadow stacks.

Signal handling and shadow stacks
---------------------------------
Signals push information about the execution context to the stack that will handle the signal. The data pushed is use to restore registers and other state after the signal. In the case of handling the signal on a normal stack, the stack just needs to be unwound over the stack frame, but in the case of alt stacks, the saved stack pointer is important for the sigreturn to find it’s way back to the thread stack. With shadow stack there is a new type of stack pointer, the shadow stack pointer (SSP), that needs to be restored. Just like the regular stack pointer, it needs to be saved somewhere in order to implement shadow alt stacks. Beyond supporting basic functionality, it would be nice if shadow stack’s could make sigreturn oriented programming (SROP) attacks harder.

Alt stacks
----------
The automatically-created thread shadow stacks are sized such that shadow stack overflows should not normally be expected. However, especially since userspace can create and pivot to arbitrarily sized shadow stacks and we now optionally have WRSS, overflows are not impossible. To cover the case of shadow stack overflow, user’s may want to handle a signal on an alternate shadow stack.

Normal signal alt stacks had problems with using swapcontext() in the signal handler. Apps couldn’t do it safely, because a subsequent signal would overwrite the previous signal’s stack. The kernel would see the current stack pointer was not on the shadow stack (since it swapcontext()ed off of it), so would restart the signal from the end of the alt stack, clobbering the previous signal. The solution was to create a new flag that would change the signal behavior to disable alt stack switching while on the alt stack. Then new signals would be pushed onto the alt stack. On sigreturn, when the sigframe for the first signal that switched to the alt stack is encountered, the alt signal stack would be re-enabled. Then subsequent signals would start at the end of the alt stack again.

For regular alt stacks, this swapcontext() capable behavior is enabled by having the kernel clear its copy of the alt signal stack address and length after this data is saved to the sigframe. So when the first sigframe on the alt stack is sigreturn-ed, the alt stack is automatically restored.

In order to support swapcontext() on alt shadow stacks, we can have something similar where we push the SSP, alt shadow stack base and length to some kind of shadow stack sigframe. This leaves the question of where to push this data.

SROP
----
Similar to normal returns, sigreturn’s can be security sensitive. One exploit technique (SROP) is to call sigreturn directly with the stack pointer at a forged sigframe. So this involves being somewhere else on the stack, than a real kernel placed sigframe. These attacks can be made harder by placing something on the protected shadow stack to signify that a specific location on the shadow stack corresponds to where sigreturn is supposed to be called. The kernel can check for this token during sigreturn, and then sigreturn can’t be called at arbitrary places on the stack.

Shadow stack signal format
--------------------------
So to handle alt shadow stacks we need to push some data onto a stack. To prevent SROP we need to push something to the shadow stack that the kernel can know it must have placed there itself. To support both we can push a special shadow stack sigframe to the shadow stack that contains the necessary alt stack restore data, in a format that couldn't possibly occur naturally. To be extra careful, this data should be written such that it can't be used as a regular shadow stack return address or a shadow stack tokens. To make sure it can’t be used, data is pushed with the high bit (bit 63) set. This bit is a linear address bit in both the token format and a normal return address, so it should not conflict with anything. It puts any return address in the kernel half of the address space, so would never be created naturally by a userspace program. It will not be a valid restore token either, as the kernel address will never be pointing to the previous frame in the shadow stack.

When a signal hits, the format pushed to the stack that is handling the signal is four 8 byte values (since we are 64 bit only):
|1...old SSP|1...alt stack size|1...alt stack base|0|

The zero (without high bit set) at the end is pushed to act as a guard frame. An attacker cannot restore from a point where the frame processed would span two shadow stack sigframes because the kernel would detect the missing high bit.

setjmp()/longjmp()
------------------
In past designs for userspace shadow stacks, shadow alt stacks were not supported. Since there was only one shadow stack, longjmp() could jump out of a signal by using incssp to unwind the SSP to the place where the setjmp() was called. In order to support longjmp() off of an alt shadow stack, a restore token could be pushed to the original stack before switching to the alt stack. Userspace could search the alt stack for the alt stack sigframe to find the restore token, then restore back to it and continue unwinding. However, the main point of alt shadow stacks is to handle shadow stack overflows. So requiring there be space to push a token would prevent the feature from being used for it’s main purpose. So in this design nothing is pushed to the old stack.

Since shadow alt stacks are a new feature, longjmp()ing from an alt shadow stack will simply not be supported. If a libc want’s to support this it will need to enable WRSS and write it’s own restore token. This could likely even let it jump straight back to the setjmp() point and skip the whole incssp piece. It could even work for longjmp() after a swapcontext(). So this kernel design makes longjmp() support a security/compatibility tradeoff that the kernel is not entirely in charge of making.

sigaltshstk() syscall
---------------------
The sigaltstack() syscall works pretty well and is familiar interface, so sigaltshstk() is just a copy. It uses the same stack_t struct for transferring the shadow stack point, size and flags. For the flags however, it will not honor the meaning of the existing flags. Future flags may not have sensible meanings for shadow stack, so sigaltshstk() will start from scratch for flag meanings. As long as we are making new flag meanings, we can make SS_AUTODISARM the default behavior for sigaltshstk(), and not require a flag. Today the only flag supported is SS_DISABLE, and a !SS_AUTODISARM mode is not supported.

sigaltshstk() is separate from sigaltstack(). You can have one without the other, neither or both together. Because the shadow stack specific state is pushed to the shadow stack, the two features don’t need to know about each other.

Preventing use as an arbitrary “set SSP”
----------------------------------------
So now when a signal hits it will jump to the location specified in sigaltshstk(). Currently (without WRSS), userspace doesn’t have the ability to arbitrarily set the SSP. But telling the kernel to set the SSP to an arbitrary point on signal is kind of like that. So there would be a weakening of the shadow stack protections unless additional checks are made. With the SS_AUTODISARM-style behavior, the SSP will only jump to the shadow stack if the SSP is not already on the shadow stack, otherwise it will just push the SSP. So we really only need to worry about the transition to the start of the alt shadow stack. So the kernel checks for a token whenever transitioning to the alt stack from a place other than the alt stack. This token can be placed when doing the allocation using the existing map_shadow_stack syscall.

RFC
---
Lastly, Andy Lutomirski raised the issue of alt shadow stacks (I think) out of concern that we might settle on an ABI that wouldn’t support them if there was later demand. The ABI of the sigreturn token was actually changed to support alt shadow stacks here. So if this whole series feels like a lot of code, I wanted to toss out the option of settling on how we could do alt shadow stacks someday, but then leave the implementation until later.

4. Compatibility of Existing Binaries/Enabling Interface
========================================================
The last version of this dealt with the problem of old glib’s breaking against future upstream shadow stack enabled kernels. Unfortunately, more userspace issues have been found. In anticipation of kernel support, some distro’s have been apparently force compiling applications with shadow stack support. Of course compiling with shadow stack really mostly means marking the elf header bit as “this binary supports shadow stack”. And having this bit doesn’t necessarily mean that the binary actually supports shadow stack. In the case of JITing or other custom stack switching programs, it often doesn’t. I have come across at least one popular distro package that completely fails to even start up, so there are likely more issues hidden in less common code paths. None of these apps will break until glibc is updated to use the new kernel API for enabling shadow stack. They will simply not run with shadow stack.

Waiting until glibc updates to break packages might not technically be a kernel regression, but it’s not good either. With the current kernel API, the decision of which binaries to enable shadow stack is left to userspace. So to prevent breakages my plan is to engage the glibc community to detect and not enable CET for these old binaries as part of the upstream of glibc CET support that will work with the new kernel interface. Then only enable CET on future more carefully compiled binaries. This will also lessen the impact of old CRIU’s (pre-Mike’s changes) failing to save shadow stack enabled programs, as most existing binaries wouldn't all turn on with CET at once.

5. CRIU Support
===============
Big thanks to Mike Rapoport for a POC [1] that fixes CRIU to work with processes that enable shadow stacks. The general design is to allow CET features to be unlocked via ptrace only, then WRSS can be used to manipulate the shadow stack to allow CRIU’s sigreturn-oriented operation to continue to work. He needed a few tweaks to the kernel in order for CRIU to do this, including the general CET ptrace support that was missing in recent postings of CET. So this is added back in, as well as his new UNLOCK ptrace-only arch_prctl(). With the new plan of not trying to enable shadow stack for most apps all at once, I wonder if this functionality might also be a good candidate for a fast follow up. Note, this CRIU POC will need to be updated to target the final signal shadow stack format.

6. Bigger Selftest
==================
A new selftest that exercises the shadow stack kernel features without any special glibc requirements. It manually enables shadow stack with the arch_prctl() and exercises shadow stack arch_prctl(), shadow stack MM, userfaultfd, signal, and the 2 new syscalls.

[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/

Kirill A. Shutemov (2):
x86: Introduce userspace API for CET enabling
x86: Expose thread features status in /proc/$PID/arch_status

Mike Rapoport (1):
x86/cet/shstk: Add ARCH_CET_UNLOCK

Rick Edgecombe (11):
x86/fpu: Add helper for modifying xstate
mm: Don't allow write GUPs to shadow stack memory x86/cet/shstk: Introduce map_shadow_stack syscall x86/cet/shstk: Support wrss for userspace x86/cet/shstk: Wire in CET interface selftests/x86: Add shadow stack test x86/cpufeatures: Limit shadow stack to Intel CPUs
x86: Separate out x86_regset for 32 and 64 bit
x86: Improve formatting of user_regset arrays x86/fpu: Add helper for initing features
x86: Add alt shadow stack support

Yu-cheng Yu (25):
Documentation/x86: Add CET description
x86/cet/shstk: Add Kconfig option for Shadow Stack x86/cpufeatures: Add CPU feature flags for shadow stacks x86/cpufeatures: Enable CET CR4 bit for shadow stack x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states x86/cet: Add user control-protection fault handler x86/mm: Remove _PAGE_DIRTY from kernel RO pages x86/mm: Move pmd_write(), pud_write() up in the file x86/mm: Introduce _PAGE_COW
x86/mm: Update pte_modify for _PAGE_COW
x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for
transition from _PAGE_DIRTY to _PAGE_COW
mm: Move VM_UFFD_MINOR_BIT from 37 to 38
mm: Introduce VM_SHADOW_STACK for shadow stack memory x86/mm: Check Shadow Stack page fault errors
x86/mm: Update maybe_mkwrite() for shadow stack
mm: Fixup places that call pte_mkwrite() directly
mm: Add guard pages around a shadow stack.
mm/mmap: Add shadow stack pages to memory accounting mm/mprotect: Exclude shadow stack from preserve_write
mm: Re-introduce vm_flags to do_mmap()
x86/cet/shstk: Add user-mode shadow stack support x86/cet/shstk: Handle thread shadow stack x86/cet/shstk: Introduce routines modifying shstk x86/cet/shstk: Handle signals for shadow stack x86/cet: Add PTRACE interface for CET

 Documentation/filesystems/proc.rst            |   1 +
 Documentation/x86/cet.rst                     | 143 ++++
 Documentation/x86/index.rst                   |   1 +
 arch/arm/kernel/signal.c                      |   2 +-
 arch/arm64/kernel/signal.c                    |   2 +-
 arch/arm64/kernel/signal32.c                  |   2 +-
 arch/sparc/kernel/signal32.c                  |   2 +-
 arch/sparc/kernel/signal_64.c                 |   2 +-
 arch/x86/Kconfig                              |  18 +
 arch/x86/Kconfig.assembler                    |   5 +
 arch/x86/entry/syscalls/syscall_64.tbl        |   2 +
 arch/x86/ia32/ia32_signal.c                   |   1 +
 arch/x86/include/asm/cet.h                    |  49 ++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/disabled-features.h      |   8 +-
 arch/x86/include/asm/fpu/api.h                |   6 +
 arch/x86/include/asm/fpu/regset.h             |   7 +-
 arch/x86/include/asm/fpu/sched.h              |   3 +-
 arch/x86/include/asm/fpu/types.h              |  14 +-
 arch/x86/include/asm/fpu/xstate.h             |   6 +-
 arch/x86/include/asm/idtentry.h               |   2 +-
 arch/x86/include/asm/mmu_context.h            |   2 +
 arch/x86/include/asm/msr-index.h              |   5 +
 arch/x86/include/asm/msr.h                    |  11 +
 arch/x86/include/asm/pgtable.h                | 314 ++++++++-
 arch/x86/include/asm/pgtable_types.h          |  48 +-
 arch/x86/include/asm/processor.h              |  11 +
 arch/x86/include/asm/special_insns.h          |  13 +
 arch/x86/include/asm/trap_pf.h                |   2 +
 arch/x86/include/uapi/asm/mman.h              |   2 +
 arch/x86/include/uapi/asm/prctl.h             |  10 +
 arch/x86/kernel/Makefile                      |   4 +
 arch/x86/kernel/cpu/common.c                  |  30 +-
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/fpu/core.c                    |  59 +-
 arch/x86/kernel/fpu/regset.c                  |  95 +++
 arch/x86/kernel/fpu/xstate.c                  | 198 +++---
 arch/x86/kernel/fpu/xstate.h                  |   6 +
 arch/x86/kernel/idt.c                         |   2 +-
 arch/x86/kernel/proc.c                        |  63 ++
 arch/x86/kernel/process.c                     |  24 +-
 arch/x86/kernel/process_64.c                  |   8 +-
 arch/x86/kernel/ptrace.c                      | 188 +++--
 arch/x86/kernel/shstk.c                       | 628 +++++++++++++++++
 arch/x86/kernel/signal.c                      |  10 +
 arch/x86/kernel/signal_compat.c               |   2 +-
 arch/x86/kernel/traps.c                       |  98 ++-
 arch/x86/mm/fault.c                           |  21 +
 arch/x86/mm/mmap.c                            |  25 +
 arch/x86/mm/pat/set_memory.c                  |   2 +-
 arch/x86/xen/enlighten_pv.c                   |   2 +-
 arch/x86/xen/xen-asm.S                        |   2 +-
 fs/aio.c                                      |   2 +-
 fs/proc/task_mmu.c                            |   3 +
 include/linux/mm.h                            |  38 +-
 include/linux/pgtable.h                       |  14 +
 include/linux/syscalls.h                      |   2 +
 include/uapi/asm-generic/siginfo.h            |   3 +-
 include/uapi/asm-generic/unistd.h             |   2 +-
 include/uapi/linux/elf.h                      |   1 +
 ipc/shm.c                                     |   2 +-
 kernel/sys_ni.c                               |   2 +
 mm/gup.c                                      |   2 +-
 mm/huge_memory.c                              |  16 +-
 mm/memory.c                                   |   3 +-
 mm/migrate_device.c                           |   3 +-
 mm/mmap.c                                     |  22 +-
 mm/mprotect.c                                 |   7 +
 mm/nommu.c                                    |   4 +-
 mm/userfaultfd.c                              |  10 +-
 mm/util.c                                     |   2 +-
 tools/testing/selftests/x86/Makefile          |   4 +-
 .../testing/selftests/x86/test_shadow_stack.c | 646 ++++++++++++++++++
 73 files changed, 2670 insertions(+), 281 deletions(-)
 create mode 100644 Documentation/x86/cet.rst
 create mode 100644 arch/x86/include/asm/cet.h
 create mode 100644 arch/x86/kernel/proc.c
 create mode 100644 arch/x86/kernel/shstk.c
 create mode 100644 tools/testing/selftests/x86/test_shadow_stack.c

base-commit: f76349cf41451c5c42a99f18a9163377e4b364ff

Signature · 1 line
2.17.1
#1 lore raw

PATCH v2 1/39 Documentation/x86: Add CET description

From: Yu-cheng Yu <[email protected]>

Introduce a new document on Control-flow Enforcement Technology (CET).

---

v2:
- Updated to new arch_prctl() API
- Add bit about new proc status

v1:
- Update and clarify the docs.
- Moved kernel parameters documentation to other patch.

 Documentation/x86/cet.rst   | 140 ++++++++++++++++++++++++++++++++++++
 Documentation/x86/index.rst |   1 +
 2 files changed, 141 insertions(+)
 create mode 100644 Documentation/x86/cet.rst
Documentation/x86/cet.rst new file +140 -0
new file mode 100644
index 000000000000..4a0dfb6830f9
@@ -0,0 +1,140 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=========================================
+Control-flow Enforcement Technology (CET)
+=========================================
+
+Overview
+========
+
+Control-flow Enforcement Technology (CET) is term referring to several
+related x86 processor features that provides protection against control
+flow hijacking attacks. The HW feature itself can be set up to protect
+both applications and the kernel. Only user-mode protection is implemented
+in the 64-bit kernel.
+
+CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
+a secondary stack allocated from memory and cannot be directly modified by
+applications. When executing a CALL instruction, the processor pushes the
+return address to both the normal stack and the shadow stack. Upon
+function return, the processor pops the shadow stack copy and compares it
+to the normal stack copy. If the two differ, the processor raises a
+control-protection fault. Indirect branch tracking verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking
+and only Shadow Stack is currently supported in the kernel.
+
+The Kconfig options is X86_SHADOW_STACK, and it can be disabled with
+the kernel parameter clearcpuid, like this: "clearcpuid=shstk".
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1
+or later are required. To build a CET-enabled application, GLIBC v2.28 or
+later is also required.
+
+At run time, /proc/cpuinfo shows CET features if the processor supports
+CET.
+
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF header and can be
+verified from readelf/llvm-readelf output:
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications directly. Applications must
+enable them using the interface descriped in section 4. Typically this
+would be done in dynamic loader or static runtime objects, as is the case
+in glibc.
+
+Backward Compatibility
+======================
+
+GLIBC provides a few CET tunables via the GLIBC_TUNABLES environment
+variable:
+
+GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-WRSS
+    Turn off SHSTK/WRSS.
+
+GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>
+    This controls how dlopen() handles SHSTK legacy libraries::
+
+        on         - continue with SHSTK enabled;
+        permissive - continue with SHSTK off.
+
+Details can be found in the GLIBC manual pages.
+
+CET arch_prctl()'s
+==================
+
+Elf features should be enabled by the loader using the below arch_prctl's.
+
+arch_prctl(ARCH_CET_ENABLE, unsigned int feature)
+    Enable a single feature specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_DISABLE, unsigned int feature)
+    Disable features specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_LOCK, unsigned int features)
+    Lock in features at their current enabled or disabled status.
+
+The return values are as following:
+    On success, return 0. On error, errno can be::
+
+        -EPERM if any of the passed feature are locked.
+        -EOPNOTSUPP if the feature is not supported by the hardware or
+         disabled by kernel parameter.
+        -EINVAL arguments (non existing feature, etc)
+
+Currently shadow stack and WRSS are supported via this interface. WRSS
+can only be enabled with shadow stack, and is automatically disabled
+if shadow stack is disabled.
+
+Proc status
+===========
+To check if an application is actually running with shadow stack, the
+user can read the /proc/$PID/arch_status. It will report "wrss" or
+"shstk" depending on what is enabled.
+
+The implementation of the Shadow Stack
+======================================
+
+Shadow Stack size
+-----------------
+
+A task's shadow stack is allocated from memory to a fixed size of
+MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to
+the maximum size of the normal stack, but capped to 4 GB. However,
+a compat-mode application's address space is smaller, each of its thread's
+shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).
+
+Signal
+------
+
+By default, the main program and its signal handlers use the same shadow
+stack. Because the shadow stack stores only return addresses, a large
+shadow stack covers the condition that both the program stack and the
+signal alternate stack run out.
+
+The kernel creates a restore token for the shadow stack and pushes the
+restorer address to the shadow stack. Then verifies that token when
+restoring from the signal handler.
+
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a
+shadow access triggers a page fault with the shadow stack access bit set
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page fault
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow stack
+for the new thread.
Documentation/x86/index.rst +1 -0
index c73d133fd37c..9ac03055c4b5 100644
@@ -22,6 +22,7 @@ x86-specific Documentation
    mtrr
    pat
    intel-hfi
+   cet
    iommu
    intel_txt
    amd-memory-encryption
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 130 lines · Rick Edgecombe

+.. SPDX-License-Identifier: GPL-2.0

(quoted diff) +129 -0
+
+=========================================
+Control-flow Enforcement Technology (CET)
+=========================================
+
+Overview
+========
+
+Control-flow Enforcement Technology (CET) is term referring to several
+related x86 processor features that provides protection against control
+flow hijacking attacks. The HW feature itself can be set up to protect
+both applications and the kernel. Only user-mode protection is implemented
+in the 64-bit kernel.
+
+CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
+a secondary stack allocated from memory and cannot be directly modified by
+applications. When executing a CALL instruction, the processor pushes the
+return address to both the normal stack and the shadow stack. Upon
+function return, the processor pops the shadow stack copy and compares it
+to the normal stack copy. If the two differ, the processor raises a
+control-protection fault. Indirect branch tracking verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking
+and only Shadow Stack is currently supported in the kernel.
+
+The Kconfig options is X86_SHADOW_STACK, and it can be disabled with
+the kernel parameter clearcpuid, like this: "clearcpuid=shstk".
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1
+or later are required. To build a CET-enabled application, GLIBC v2.28 or
+later is also required.
+
+At run time, /proc/cpuinfo shows CET features if the processor supports
+CET.
+
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF header and can be
+verified from readelf/llvm-readelf output:
+
+    readelf -n <application> | grep -a SHSTK
+        properties: x86 feature: SHSTK
+
+The kernel does not process these applications directly. Applications must
+enable them using the interface descriped in section 4. Typically this
+would be done in dynamic loader or static runtime objects, as is the case
+in glibc.
+
+Backward Compatibility
+======================
+
+GLIBC provides a few CET tunables via the GLIBC_TUNABLES environment
+variable:
+
+GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-WRSS
+    Turn off SHSTK/WRSS.
+
+GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>
+    This controls how dlopen() handles SHSTK legacy libraries::
+
+        on         - continue with SHSTK enabled;
+        permissive - continue with SHSTK off.
+
+Details can be found in the GLIBC manual pages.
+
+CET arch_prctl()'s
+==================
+
+Elf features should be enabled by the loader using the below arch_prctl's.
+
+arch_prctl(ARCH_CET_ENABLE, unsigned int feature)
+    Enable a single feature specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_DISABLE, unsigned int feature)
+    Disable features specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_LOCK, unsigned int features)
+    Lock in features at their current enabled or disabled status.
+
+The return values are as following:
+    On success, return 0. On error, errno can be::
+
+        -EPERM if any of the passed feature are locked.
+        -EOPNOTSUPP if the feature is not supported by the hardware or
+         disabled by kernel parameter.
+        -EINVAL arguments (non existing feature, etc)
+
+Currently shadow stack and WRSS are supported via this interface. WRSS
+can only be enabled with shadow stack, and is automatically disabled
+if shadow stack is disabled.
+
+Proc status
+===========
+To check if an application is actually running with shadow stack, the
+user can read the /proc/$PID/arch_status. It will report "wrss" or
+"shstk" depending on what is enabled.
+
+The implementation of the Shadow Stack
+======================================
+
+Shadow Stack size
+-----------------
+
+A task's shadow stack is allocated from memory to a fixed size of
+MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to
+the maximum size of the normal stack, but capped to 4 GB. However,
+a compat-mode application's address space is smaller, each of its thread's
+shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).
+
+Signal
+------
+
+By default, the main program and its signal handlers use the same shadow
+stack. Because the shadow stack stores only return addresses, a large
+shadow stack covers the condition that both the program stack and the
+signal alternate stack run out.
+
+The kernel creates a restore token for the shadow stack and pushes the
+restorer address to the shadow stack. Then verifies that token when
+restoring from the signal handler.
+
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a

Review

Quoted 10 lines
(quoted diff) +10 -0
+shadow access triggers a page fault with the shadow stack access bit set
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page fault
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow stack
+for the new thread.

The documentation above can be improved (both grammar and formatting):

---- >8 ----

Documentation/x86/cet.rst +59 -50
index 6b270a24ebc3a2..f691f7995cf088 100644
@@ -15,92 +15,101 @@ in the 64-bit kernel.
 
 CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
 a secondary stack allocated from memory and cannot be directly modified by
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the processor pushes the
 return address to both the normal stack and the shadow stack. Upon
 function return, the processor pops the shadow stack copy and compares it
 to the normal stack copy. If the two differ, the processor raises a
 control-protection fault. Indirect branch tracking verifies indirect
-CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
-opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking
-and only Shadow Stack is currently supported in the kernel.
+``CALL``/``JMP`` targets are intended as marked by the compiler with ``ENDBR``
+opcodes. Not all CPUs have both Shadow Stack and Indirect Branch Tracking
+and only Shadow Stack is currently supported by the kernel.
 
-The Kconfig options is X86_SHADOW_STACK, and it can be disabled with
-the kernel parameter clearcpuid, like this: "clearcpuid=shstk".
+The Kconfig options is ``X86_SHADOW_STACK`` and it can be overridden with
+the kernel command-line parameter ``clearcpuid`` (for example
+``clearcpuid=shstk``).
 
 To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1
-or later are required. To build a CET-enabled application, GLIBC v2.28 or
+or later are required. To build a CET-enabled application, glibc v2.28 or
 later is also required.
 
-At run time, /proc/cpuinfo shows CET features if the processor supports
-CET.
+At run time, ``/proc/cpuinfo`` shows CET features if the processor supports
+them
 
-Application Enabling
-====================
+Enabling CET in applications
+============================
 
-An application's CET capability is marked in its ELF header and can be
-verified from readelf/llvm-readelf output:
+The CET capability of an application is marked in its ELF header and can be
+verified from ``readelf``/``llvm-readelf`` output::
 
     readelf -n <application> | grep -a SHSTK
         properties: x86 feature: SHSTK
 
 The kernel does not process these applications directly. Applications must
-enable them using the interface descriped in section 4. Typically this
+enable them using :ref:`cet-arch_prctl`. Typically this
 would be done in dynamic loader or static runtime objects, as is the case
 in glibc.
 
 Backward Compatibility
 ======================
 
-GLIBC provides a few CET tunables via the GLIBC_TUNABLES environment
+glibc provides a few CET tunables via the ``GLIBC_TUNABLES`` environment
 variable:
 
-GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-WRSS
+  * ``GLIBC_TUNABLES=glibc.tune.hwcaps=-SHSTK,-WRSS``
+
     Turn off SHSTK/WRSS.
 
-GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>
-    This controls how dlopen() handles SHSTK legacy libraries::
+  * ``GLIBC_TUNABLES=glibc.tune.x86_shstk=<on, permissive>``
 
-        on         - continue with SHSTK enabled;
-        permissive - continue with SHSTK off.
+    This controls how :manpage:`dlopen(3)` handles SHSTK legacy libraries.
+    Possible values are:
 
-Details can be found in the GLIBC manual pages.
+    * ``on``         - continue with SHSTK enabled;
+    * ``permissive`` - continue with SHSTK off.
 
-CET arch_prctl()'s
-==================
+.. _cet-arch_prctl:
 
-Elf features should be enabled by the loader using the below arch_prctl's.
+CET arch_prctl() interface
+==========================
 
-arch_prctl(ARCH_CET_ENABLE, unsigned int feature)
-    Enable a single feature specified in 'feature'. Can only operate on
+ELF features should be enabled by the loader using the following
+:manpage:`arch_prctl(2)` subfunctions:
+
+  * ``arch_prctl(ARCH_CET_ENABLE, unsigned int feature)``
+
+    Enable a single feature specified in ``feature``. Can only operate on
     one feature at a time.
 
-arch_prctl(ARCH_CET_DISABLE, unsigned int feature)
-    Disable features specified in 'feature'. Can only operate on
+  * ``arch_prctl(ARCH_CET_DISABLE, unsigned int feature)``
+
+    Disable features specified in ``feature``. Can only operate on
     one feature at a time.
 
-arch_prctl(ARCH_CET_LOCK, unsigned int features)
-    Lock in features at their current enabled or disabled status.
+  * ``arch_prctl(ARCH_CET_LOCK, unsigned int features)``
+
+    Lock in features at their current status.
+
+  * ``arch_prctl(ARCH_CET_UNLOCK, unsigned int features)``
 
-arch_prctl(ARCH_CET_UNLOCK, unsigned int features)
     Unlock features.
 
-The return values are as following:
-    On success, return 0. On error, errno can be::
+On success, :manpage:`arch_prctl(2)` returns 0, otherwise the errno
+can be:
 
-        -EPERM if any of the passed feature are locked.
-        -EOPNOTSUPP if the feature is not supported by the hardware or
-         disabled by kernel parameter.
-        -EINVAL arguments (non existing feature, etc)
+  - ``EPERM`` if any of the passed feature are locked.
+  - ``EOPNOTSUPP`` if the feature is not supported by the hardware or
+    disabled by the kernel command-line parameter.
+  - ``EINVAL`` if the arguments are invalid (non existing feature, etc).
 
 Currently shadow stack and WRSS are supported via this interface. WRSS
 can only be enabled with shadow stack, and is automatically disabled
 if shadow stack is disabled.
 
-Proc status
+proc status
 ===========
-To check if an application is actually running with shadow stack, the
-user can read the /proc/$PID/arch_status. It will report "wrss" or
-"shstk" depending on what is enabled.
+To check if an application is actually running with shadow stack, users can
+read ``/proc/$PID/arch_status``. It will report ``wrss`` or
+``shstk`` depending on what is enabled.
 
 The implementation of the Shadow Stack
 ======================================
@@ -108,11 +117,11 @@ The implementation of the Shadow Stack
 Shadow Stack size
 -----------------
 
-A task's shadow stack is allocated from memory to a fixed size of
-MIN(RLIMIT_STACK, 4 GB). In other words, the shadow stack is allocated to
+The shadow stack of a task is allocated from memory to a fixed size of
+``MIN(RLIMIT_STACK, 4 GB)``. In other words, the shadow stack is allocated to
 the maximum size of the normal stack, but capped to 4 GB. However,
-a compat-mode application's address space is smaller, each of its thread's
-shadow stack size is MIN(1/4 RLIMIT_STACK, 4 GB).
+the address space of a compat-mode application is smaller; the shadow stack
+size of each of its thread is ``MIN(1/4 RLIMIT_STACK, 4 GB)``.
 
 Signal
 ------
@@ -123,19 +132,19 @@ shadow stack covers the condition that both the program stack and the
 signal alternate stack run out.
 
 The kernel creates a restore token for the shadow stack and pushes the
-restorer address to the shadow stack. Then verifies that token when
-restoring from the signal handler.
+restorer address to it. Then the kernel verifies that token when restoring
+from the signal handler.
 
 Fork
 ----
 
-The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required
-to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a
+The shadow stack vma has ``VM_SHADOW_STACK`` flag set; its PTEs are required
+to be read-only and dirty. When a shadow stack PTE is read-write and dirty, a
 shadow access triggers a page fault with the shadow stack access bit set
 in the page fault error code.
 
 When a task forks a child, its shadow stack PTEs are copied and both the
-parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+shadow stack PTEs of parent and child are cleared of the dirty bit.
 Upon the next shadow stack access, the resulting shadow stack page fault
 is handled by page copy/re-use.
 

Thanks.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
maintainer
#3 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 9 lines · Bagas Sanjaya

The documentation above can be improved (both grammar and formatting):

---- >8 ----

Documentation/x86/cet.rst +0 -0
index 6b270a24ebc3a2..f691f7995cf088 100644

Documentation/x86/cet.rst in the 64-bit kernel. · line 18

Quoted 5 lines
Documentation/x86/cet.rst +1 -1
@@ -15,92 +15,101 @@ in the 64-bit kernel.
 
 CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
 a secondary stack allocated from memory and cannot be directly modified by
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the processor pushes the

Just to be clear, not everybody is fond of sprinkling lots of ``literal text`` throughout the documentation in this way. Heavy use of it will certainly clutter the plain-text file and can be a net negative overall.

Thanks,

jon

#4 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 9 lines · Jonathan Corbet

Documentation/x86/cet.rst in the 64-bit kernel. · line 18

Quoted 4 lines

CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is a secondary stack allocated from memory and cannot be directly modified by

Documentation/x86/cet.rst +1 -1
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the processor pushes the

Just to be clear, not everybody is fond of sprinkling lots of ``literal text`` throughout the documentation in this way. Heavy use of it will certainly clutter the plain-text file and can be a net negative overall.

Actually there is a trade-off between semantic correctness and plain-text clarity. With regards to inline code samples (like identifiers), I fall into the former camp. But when I'm reviewing patches for which the surrounding documentation go latter camp (leave code samples alone without markup), I can adapt to that style as long as it causes no warnings whatsover.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
#5 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 18 lines · Bagas Sanjaya
Quoted 7 lines · Jonathan Corbet
Quoted 7 lines

CET introduces Shadow Stack and Indirect Branch Tracking.
Shadow stack is
a secondary stack allocated from memory and cannot be directly
modified by
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the

Quoted 9 lines
Quoted 1 line

processor pushes the

Just to be clear, not everybody is fond of sprinkling lots of ``literal
text`` throughout the documentation in this way. Heavy use of it will
certainly clutter the plain-text file and can be a net negative overall.

Quoted 9 lines

Actually there is a trade-off between semantic correctness and plain- text
clarity. With regards to inline code samples (like identifiers), I fall
into the former camp. But when I'm reviewing patches for which the surrounding documentation go latter camp (leave code samples alone without
markup), I can adapt to that style as long as it causes no warnings whatsover.

Thanks. Unless anyone has any objections, I think I'll take all these changes, except for the literal-izing of the instructions. They are not really being used as code samples in this case.

Bagas, can you reply with your sign-off and I'll just apply it?

#6 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 29 lines · Rick P
Quoted 18 lines · Bagas Sanjaya
Quoted 7 lines · Jonathan Corbet
Quoted 7 lines

CET introduces Shadow Stack and Indirect Branch Tracking.
Shadow stack is
a secondary stack allocated from memory and cannot be directly
modified by
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the

Quoted 9 lines
Quoted 1 line

processor pushes the

Just to be clear, not everybody is fond of sprinkling lots of ``literal
text`` throughout the documentation in this way. Heavy use of it will
certainly clutter the plain-text file and can be a net negative overall.

Quoted 9 lines

Actually there is a trade-off between semantic correctness and plain- text
clarity. With regards to inline code samples (like identifiers), I fall
into the former camp. But when I'm reviewing patches for which the surrounding documentation go latter camp (leave code samples alone without
markup), I can adapt to that style as long as it causes no warnings whatsover.

Quoted 5 lines

Thanks. Unless anyone has any objections, I think I'll take all these changes, except for the literal-izing of the instructions. They are not really being used as code samples in this case.

Bagas, can you reply with your sign-off and I'll just apply it?

OK, here goes...

Signature · 1 line
An old man doll... just what I always wanted! - Clara
maintainer
#7 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Edgecombe, Rick P
Quoted 1 line · Rick P

Thanks. Unless anyone has any objections

Well, I'll object. I still feel rst should burn in hell. Plain text FTW.

#8 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Rick P

Thanks. Unless anyone has any objections

Well, I'll object. I still feel rst should burn in hell. Plain text FTW.

.txt maybe?

Signature · 1 line
An old man doll... just what I always wanted! - Clara
maintainer
#9 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 9 lines · Bagas Sanjaya
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Rick P

Thanks. Unless anyone has any objections

Well, I'll object. I still feel rst should burn in hell. Plain text FTW.

.txt maybe?

We had that, but some idiots went and converted the lot to .rst :-(

reviewer
#10 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Bagas Sanjaya

On 9/29/22 20:41, Bagas Sanjaya wrote:
...

Quoted 9 lines

The documentation above can be improved (both grammar and formatting):

---- >8 ----

Documentation/x86/cet.rst +0 -0
index 6b270a24ebc3a2..f691f7995cf088 100644

Documentation/x86/cet.rst in the 64-bit kernel. · line 18

Quoted 5 lines
Documentation/x86/cet.rst +1 -1
@@ -15,92 +15,101 @@ in the 64-bit kernel.
 
  CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
  a secondary stack allocated from memory and cannot be directly modified by
-applications. When executing a CALL instruction, the processor pushes the
+applications. When executing a ``CALL`` instruction, the processor pushes the

It's always a judgment call, as to whether to use something like ``CALL` or just plain CALL. Here, I'd like to opine that that the benefits of ``CALL`` are very small, whereas plain text in cet.rst has been made significantly worse. So the result is, "this is not worth it".

The same is true of pretty much all of the other literalizing changes below, IMHO.

Just so you have some additional input on this. I tend to spend time fussing a lot (too much, yes) over readability issues, so this jumps right out at me. :)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA
#11 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 4 lines · John Hubbard

It's always a judgment call, as to whether to use something like ``CALL` or just plain CALL. Here, I'd like to opine that that the benefits of ``CALL`` are very small, whereas plain text in cet.rst has been made significantly worse. So the result is, "this is not worth it".

I'm definitely in this camp as well. Unless the markup *really* adds to readability, just leave it alone.

#12 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to John Hubbard
Quoted 5 lines · John Hubbard

It's always a judgment call, as to whether to use something like ``CALL` or just plain CALL. Here, I'd like to opine that that the benefits of ``CALL`` are very small, whereas plain text in cet.rst has been made significantly worse. So the result is, "this is not worth it".

Hmm, seems like neither CALL nor ``CALL`` is better, right?

Signature · 1 line
An old man doll... just what I always wanted! - Clara
#13 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Rick Edgecombe

Documentation/x86/cet.rst line 14

Quoted 9 lines · Rick Edgecombe

[...]

Documentation/x86/cet.rst new file +8 -0
+Overview
+========
+
+Control-flow Enforcement Technology (CET) is term referring to several
+related x86 processor features that provides protection against control
+flow hijacking attacks. The HW feature itself can be set up to protect
+both applications and the kernel. Only user-mode protection is implemented
+in the 64-bit kernel.

This likely needs rewording, since it's not strictly true any more:
IBT is supported in kernel-mode now (CONFIG_X86_IBT).

Quoted 10 lines
Documentation/x86/cet.rst new file +10 -0
+CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
+a secondary stack allocated from memory and cannot be directly modified by
+applications. When executing a CALL instruction, the processor pushes the
+return address to both the normal stack and the shadow stack. Upon
+function return, the processor pops the shadow stack copy and compares it
+to the normal stack copy. If the two differ, the processor raises a
+control-protection fault. Indirect branch tracking verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking
+and only Shadow Stack is currently supported in the kernel.

Documentation/x86/cet.rst line 35

Quoted 10 lines
Documentation/x86/cet.rst new file +10 -0
+
+The Kconfig options is X86_SHADOW_STACK, and it can be disabled with
+the kernel parameter clearcpuid, like this: "clearcpuid=shstk".
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1
+or later are required. To build a CET-enabled application, GLIBC v2.28 or
+later is also required.
+
+At run time, /proc/cpuinfo shows CET features if the processor supports
+CET.

Maybe call them out by name: shstk ibt

Documentation/x86/cet.rst line 75

Quoted 8 lines
Documentation/x86/cet.rst new file +8 -0
+CET arch_prctl()'s
+==================
+
+Elf features should be enabled by the loader using the below arch_prctl's.
+
+arch_prctl(ARCH_CET_ENABLE, unsigned int feature)
+    Enable a single feature specified in 'feature'. Can only operate on
+    one feature at a time.

Does this mean only 1 bit out of the 32 may be specified?

Documentation/x86/cet.rst line 82

Quoted 7 lines
Documentation/x86/cet.rst new file +7 -0
+
+arch_prctl(ARCH_CET_DISABLE, unsigned int feature)
+    Disable features specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_LOCK, unsigned int features)
+    Lock in features at their current enabled or disabled status.

How is the "features" argument processed here?

Documentation/x86/cet.rst line 100

Quoted 6 lines

[...]

Documentation/x86/cet.rst new file +5 -0
+Proc status
+===========
+To check if an application is actually running with shadow stack, the
+user can read the /proc/$PID/arch_status. It will report "wrss" or
+"shstk" depending on what is enabled.

TIL about "arch_status". :) Why is this a separate file? "status" is already has unique field names.

Quoted 5 lines
Documentation/x86/cet.rst new file +5 -0
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a

Documentation/x86/cet.rst line 140

Quoted 10 lines
Documentation/x86/cet.rst new file +10 -0
+shadow access triggers a page fault with the shadow stack access bit set
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page fault
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow stack
+for the new thread.

Perhaps speak to the ASLR characteristics of the shstk here?

Also, it seems if there is a "Fork" section, there should be an "Exec" section? I suspect it would be short: shstk is disabled when execve() is called and must be re-enabled from userspace, yes?

-Kees

Signature · 1 line
Kees Cook
#14 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 17 lines · Kees Cook
Quoted 5 lines · Rick Edgecombe

[...]

Documentation/x86/cet.rst new file +4 -0
+Overview
+========
+
+Control-flow Enforcement Technology (CET) is term referring to
Quoted 8 lines

several
+related x86 processor features that provides protection against control
+flow hijacking attacks. The HW feature itself can be set up to protect
+both applications and the kernel. Only user-mode protection is implemented
+in the 64-bit kernel.

This likely needs rewording, since it's not strictly true any more:
IBT is supported in kernel-mode now (CONFIG_X86_IBT).

Yep, thanks.

Quoted 27 lines
Quoted 27 lines

+CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
+a secondary stack allocated from memory and cannot be directly modified by
+applications. When executing a CALL instruction, the processor pushes the

(quoted diff) +2 -0
+return address to both the normal stack and the shadow stack. Upon
+function return, the processor pops the shadow stack copy and

compares it
+to the normal stack copy. If the two differ, the processor raises a
+control-protection fault. Indirect branch tracking verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking

Documentation/x86/cet.rst new file +3 -0
+and only Shadow Stack is currently supported in the kernel.
+
+The Kconfig options is X86_SHADOW_STACK, and it can be disabled

with

Documentation/x86/cet.rst new file +3 -0
+the kernel parameter clearcpuid, like this: "clearcpuid=shstk".
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM

v10.0.1
+or later are required. To build a CET-enabled application, GLIBC

Quoted 8 lines
Quoted 6 lines

v2.28 or

Documentation/x86/cet.rst new file +3 -0
+later is also required.
+
+At run time, /proc/cpuinfo shows CET features if the processor

supports
+CET.

Maybe call them out by name: shstk ibt

Ok.

Quoted 13 lines
Quoted 11 lines
Documentation/x86/cet.rst new file +4 -0
+CET arch_prctl()'s
+==================
+
+Elf features should be enabled by the loader using the below

arch_prctl's.

Documentation/x86/cet.rst new file +3 -0
+
+arch_prctl(ARCH_CET_ENABLE, unsigned int feature)
+    Enable a single feature specified in 'feature'. Can only

operate on
+ one feature at a time.

Does this mean only 1 bit out of the 32 may be specified?

Yes, exactly.

Quoted 10 lines

Documentation/x86/cet.rst line 82

Quoted 8 lines
Documentation/x86/cet.rst new file +7 -0
+
+arch_prctl(ARCH_CET_DISABLE, unsigned int feature)
+    Disable features specified in 'feature'. Can only operate on
+    one feature at a time.
+
+arch_prctl(ARCH_CET_LOCK, unsigned int features)
+    Lock in features at their current enabled or disabled status.

How is the "features" argument processed here?

Yes, this should have more info. The kernel keeps a mask of features that are "locked". The mask is ORed with the existing value. So any bits set here cannot be enabled or disabled afterwards. Bit's unset in the mask passed are ignored.

Quoted 11 lines
Quoted 8 lines

[...]

Documentation/x86/cet.rst new file +3 -0
+Proc status
+===========
+To check if an application is actually running with shadow stack,

the
+user can read the /proc/$PID/arch_status. It will report "wrss" or +"shstk" depending on what is enabled.

TIL about "arch_status". :) Why is this a separate file? "status" is already has unique field names.

It looks like "status" only has arch-agnostic feature status today. Maybe that's the reason? CET seems to fit there though.

Quoted 17 lines
Quoted 17 lines
Documentation/x86/cet.rst new file +4 -0
+Fork
+----
+
+The shadow stack's vma has VM_SHADOW_STACK flag set; its PTEs are

required
+to be read-only and dirty. When a shadow stack PTE is not RO and dirty, a
+shadow access triggers a page fault with the shadow stack access bit set

Documentation/x86/cet.rst new file +3 -0
+in the page fault error code.
+
+When a task forks a child, its shadow stack PTEs are copied and

both the
+parent's and the child's shadow stack PTEs are cleared of the dirty bit.
+Upon the next shadow stack access, the resulting shadow stack page

Quoted 8 lines
Quoted 6 lines

fault

Documentation/x86/cet.rst new file +3 -0
+is handled by page copy/re-use.
+
+When a pthread child is created, the kernel allocates a new shadow

stack
+for the new thread.

Perhaps speak to the ASLR characteristics of the shstk here?

It behaves just like mmap(). I can add some info.

Quoted 6 lines

Also, it seems if there is a "Fork" section, there should be an "Exec"
section? I suspect it would be short: shstk is disabled when execve() is
called and must be re-enabled from userspace, yes?

Sure, I can add some info.

#15 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Rick Edgecombe
Quoted 21 lines · Rick Edgecombe
Documentation/x86/cet.rst new file +15 -0
new file mode 100644
index 000000000000..4a0dfb6830f9
@@ -0,0 +1,140 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=========================================
+Control-flow Enforcement Technology (CET)
+=========================================
+
+Overview
+========
+
+Control-flow Enforcement Technology (CET) is term referring to several
+related x86 processor features that provides protection against control
+flow hijacking attacks. The HW feature itself can be set up to protect
+both applications and the kernel. Only user-mode protection is implemented
+in the 64-bit kernel.
+

Documentation/x86/cet.rst line 25

Quoted 10 lines
Documentation/x86/cet.rst new file +10 -0
+CET introduces Shadow Stack and Indirect Branch Tracking. Shadow stack is
+a secondary stack allocated from memory and cannot be directly modified by
+applications. When executing a CALL instruction, the processor pushes the
+return address to both the normal stack and the shadow stack. Upon
+function return, the processor pops the shadow stack copy and compares it
+to the normal stack copy. If the two differ, the processor raises a
+control-protection fault. Indirect branch tracking verifies indirect
+CALL/JMP targets are intended as marked by the compiler with 'ENDBR'
+opcodes. Not all CPU's have both Shadow Stack and Indirect Branch Tracking
+and only Shadow Stack is currently supported in the kernel.

This paragraph is stale, isn't it?

AIUI, by the end of this series, what is supported is in-kernel self-protection using CET-IBT, and userspace shadow stacks.

It is probably worth keeping the implementation-agnostic bits separate from the "what is currently supported" matrix.  I'm not certain if its worth splitting into cet.rst, cet-kernel.rst and cet-user.rst at this point, but it's something to consider.

Documentation/x86/cet.rst line 28

Quoted 2 lines
Documentation/x86/cet.rst new file +2 -0
+The Kconfig options is X86_SHADOW_STACK, and it can be disabled with
+the kernel parameter clearcpuid, like this: "clearcpuid=shstk".

What about namespacing?  For the CPUID features themselves, yes they're shstk and ibt.

But for the Kconfig options, the user and kernel implementations are wildly different for both shstk and ibt.  Are they going to want to share the same Kconfig option from the getgo?

Independent of the Kconfig symbol, user and kernel have separate enablement criteria.  e.g. kernel shstk is likely going to be dependent on the FRED feature, and simply looking at shstk in /proc/cpuinfo doesn't necessarily tell you all you want to know.

Documentation/x86/cet.rst line 29

Quoted 2 lines
Documentation/x86/cet.rst new file +2 -0
+
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1

What are the other dependences here?

In principle shstk only needs assembler support for the new instructions, and that's Binutils 2.29 / LLVM 6 from my notes.

It's IBT which needs compiler support (and then, even only kernel IBT), and that work is already done.

Documentation/x86/cet.rst line 35

Quoted 5 lines
Documentation/x86/cet.rst new file +5 -0
+or later are required. To build a CET-enabled application, GLIBC v2.28 or
+later is also required.
+
+At run time, /proc/cpuinfo shows CET features if the processor supports
+CET.

Probably helpful to state what these are.

Documentation/x86/cet.rst line 40

Quoted 5 lines
Documentation/x86/cet.rst new file +5 -0
+
+Application Enabling
+====================
+
+An application's CET capability is marked in its ELF header and can be

Technically its in an ELF note, not the ELF header.

~Andrew

#16 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Rick Edgecombe

* Rick Edgecombe:

Documentation/x86/cet.rst line 32

Quoted 3 lines
Documentation/x86/cet.rst new file +3 -0
+To build a CET-enabled kernel, Binutils v2.31 and GCC v8.1 or LLVM v10.0.1
+or later are required. To build a CET-enabled application, GLIBC v2.28 or
+later is also required.

Uhm, I think we are using binutils 2.30 with extra fixes. I hope that these binaries are still valid.

More importantly, glibc needs to be configured with --enable-cet explicitly (unless the compiler defaults to CET). The default glibc build with a default GCC will produce dynamically-linked executables that disable CET (when running on later/differently configured glibc builds). The statically linked object files are not marked up for CET in that case.

I think the goal is to support the new kernel interface for actually switching on SHSTK in glibc 2.37. But at that point, hopefully all those existing binaries can start enjoying the STSTK benefits.

Thanks,
Florian

#17 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 3 lines · Florian Weimer

Uhm, I think we are using binutils 2.30 with extra fixes. I hope that
these binaries are still valid.

Yea, you're right. Andrew Cooper pointed out it has been supported since 2.29, so 2.30 should be fine.

Quoted 8 lines

More importantly, glibc needs to be configured with --enable-cet explicitly (unless the compiler defaults to CET). The default glibc build with a default GCC will produce dynamically-linked executables that disable CET (when running on later/differently configured glibc builds). The statically linked object files are not marked up for CET
in that case.

Thanks, that's a good point. I'll add a blurb about glibc needs to be compiled with CET support.

Quoted 4 lines

I think the goal is to support the new kernel interface for actually switching on SHSTK in glibc 2.37. But at that point, hopefully all those existing binaries can start enjoying the STSTK benefits.

Can you share more about this plan? HJ was previously planning to wait until the kernel support was upstream before making any more glibc changes. Hopefully this will be in time for that, but I'd really rather not repeat what happened last time where we had to design the kernel interface around not breaking old glibc's with mismatched CET enablement.

What did you think of the proposal to disable existing binaries and start from scratch? Elaborated in the coverletter in the section "Compatibility of Existing Binaries/Enabling Interface".

#18 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 33 lines · Rick P
Quoted 3 lines · Florian Weimer

Uhm, I think we are using binutils 2.30 with extra fixes. I hope that
these binaries are still valid.

Yea, you're right. Andrew Cooper pointed out it has been supported since 2.29, so 2.30 should be fine.

Quoted 8 lines

More importantly, glibc needs to be configured with --enable-cet explicitly (unless the compiler defaults to CET). The default glibc build with a default GCC will produce dynamically-linked executables that disable CET (when running on later/differently configured glibc builds). The statically linked object files are not marked up for CET
in that case.

Thanks, that's a good point. I'll add a blurb about glibc needs to be compiled with CET support.

Quoted 4 lines

I think the goal is to support the new kernel interface for actually switching on SHSTK in glibc 2.37. But at that point, hopefully all those existing binaries can start enjoying the STSTK benefits.

Can you share more about this plan? HJ was previously planning to wait until the kernel support was upstream before making any more glibc changes. Hopefully this will be in time for that, but I'd really rather not repeat what happened last time where we had to design the kernel interface around not breaking old glibc's with mismatched CET enablement.

Quoted 3 lines

What did you think of the proposal to disable existing binaries and start from scratch? Elaborated in the coverletter in the section "Compatibility of Existing Binaries/Enabling Interface".

My current glibc plan is that kernel won't enable CET automatically
and glibc will issue syscall to enable CET at early startup time. All
existing CET enabled dynamic executables will have CET enabled under the CET kernel and the updated CET glibc.

Signature · 1 line
H.J.
#19 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Edgecombe, Rick P

* Rick P. Edgecombe:

Quoted 10 lines
Quoted 3 lines

I think the goal is to support the new kernel interface for actually switching on SHSTK in glibc 2.37. But at that point, hopefully all those existing binaries can start enjoying the STSTK benefits.

Can you share more about this plan? HJ was previously planning to wait until the kernel support was upstream before making any more glibc changes. Hopefully this will be in time for that, but I'd really rather not repeat what happened last time where we had to design the kernel interface around not breaking old glibc's with mismatched CET enablement.

You're still doing that (keeping that gap in this constant), and this appreciated and very much necessary.

Quoted 3 lines

What did you think of the proposal to disable existing binaries and start from scratch? Elaborated in the coverletter in the section "Compatibility of Existing Binaries/Enabling Interface".

The ABI was finalized around four years ago, and we have shipped several Fedora and Red Hat Enterprise Linux versions with it. Other distributions did as well. It's a bit late to make changes now, and certainly not for such trivialities. In the case of the IBT ABI, it may be tempting to start over in a less trivial way, to radically reduce the amount of ENDBR instructions. But that doesn't concern SHSTK, and there's no actual implementation anyway.

But as H.J. implied, you would have to do rather nasty things in the kernel to prevent us from achieving ABI compatibility in userspace, like parsing property notes on the main executable and disabling the new arch_prctl calls if you see something there that you don't like. 8-) Of course no one is going to implement that.

(We are fine with swapping out glibc and its dynamic loader to enable CET with the appropriate kernel mechanism, but we wouldn't want to change the way all other binaries are marked up.)

Thanks,
Florian

#20 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 7 lines · Florian Weimer
Quoted 3 lines

What did you think of the proposal to disable existing binaries and start from scratch? Elaborated in the coverletter in the section "Compatibility of Existing Binaries/Enabling Interface".

The ABI was finalized around four years ago, and we have shipped several Fedora and Red Hat Enterprise Linux versions with it. Other distributions did as well. It's a bit late to make changes now, and certainly not for such trivialities.

Just to be clear: You're saying that a user/kernel ABI was "finalized" by glibc shipping the user side of it, before there being an upstream kernel implementation?

#21 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

* Dave Hansen:

Quoted 12 lines · Florian Weimer
Quoted 7 lines
Quoted 3 lines

What did you think of the proposal to disable existing binaries and start from scratch? Elaborated in the coverletter in the section "Compatibility of Existing Binaries/Enabling Interface".

The ABI was finalized around four years ago, and we have shipped several Fedora and Red Hat Enterprise Linux versions with it. Other distributions did as well. It's a bit late to make changes now, and certainly not for such trivialities.

Just to be clear: You're saying that a user/kernel ABI was "finalized" by glibc shipping the user side of it, before there being an upstream kernel implementation?

Sorry for being unclear. I was refering to the x86-64 ELF psABI supplement for CET, not the kernel/userspace interface, which still does not exist in its final form as of today, as far as I understand it.

Thanks,
Florian

#22 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Florian Weimer
Quoted 18 lines · Florian Weimer

The ABI was finalized around four years ago, and we have shipped several
Fedora and Red Hat Enterprise Linux versions with it. Other distributions did as well. It's a bit late to make changes now, and certainly not for such trivialities. In the case of the IBT ABI, it may
be tempting to start over in a less trivial way, to radically reduce the
amount of ENDBR instructions. But that doesn't concern SHSTK, and there's no actual implementation anyway.

But as H.J. implied, you would have to do rather nasty things in the kernel to prevent us from achieving ABI compatibility in userspace, like
parsing property notes on the main executable and disabling the new arch_prctl calls if you see something there that you don't like. 8-) Of course no one is going to implement that.

Quoted 3 lines

(We are fine with swapping out glibc and its dynamic loader to enable CET with the appropriate kernel mechanism, but we wouldn't want to change the way all other binaries are marked up.)

So we have these compatibility issues with existing binaries. We know some apps are totally broken. It sounds like you are proposing to ignore this and let people who hit the issues work through it themselves. This was also proposed by other glibc developers as a solution for past CET compatibility issues that broke boot on kernel upgrade. I have to say, as the person pushing these patches, I’m uncomfortable with this approach. I don’t think users will like the results. Basically, do they want to upgrade and run a bunch of untested integration with known failures? I also don’t want to get this feature reverted and I’m not exactly sure how this scenario would be taken.

But I hear the point about it not being ideal to abandon the existing CET userspace. I think there is also a point about how userspace chose to do this optimistic and early wide enabling, even if it was a bad idea, and so how much should the kernel try to save userspace from itself. So what do you think about this instead:

The current psABI spec talks about the binary being compatible with shadow stack. It doesn’t say much about what should happen after the loader. Since the greater ecosystem has used this bit with a more cavalier attitude, glibc could treat it as a request for a warn and continue mode. In the meantime we could have a new bit shstk_strict, that requests behavior like these patches implement, and kills the process on violation. Glibc/tools could add support for this strict bit and anyone that wants to more carefully compile with it could finally get shadow stack today. Then the implementation of the warn and continue mode could follow that, and glibc could map the original shstk bit to that kernel mode. So the old binaries would get there eventually, which is better than the continuing nothing they have today.

And speaking of having nothing today, there are people that really want to use shadow stack and do not care at all about having CET support for existing binaries. Neither glibc or elf bits are required to use kernel shadow stack support. So if it comes to it, I don’t want to hold support back for other users because the elf note bit enabling path grew some issues.

Please let me know about what you think of that plan.

#23 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

Quoted 55 lines · Rick P
Quoted 18 lines · Florian Weimer

The ABI was finalized around four years ago, and we have shipped several
Fedora and Red Hat Enterprise Linux versions with it. Other distributions did as well. It's a bit late to make changes now, and certainly not for such trivialities. In the case of the IBT ABI, it may
be tempting to start over in a less trivial way, to radically reduce the
amount of ENDBR instructions. But that doesn't concern SHSTK, and there's no actual implementation anyway.

But as H.J. implied, you would have to do rather nasty things in the kernel to prevent us from achieving ABI compatibility in userspace, like
parsing property notes on the main executable and disabling the new arch_prctl calls if you see something there that you don't like. 8-) Of course no one is going to implement that.

Quoted 3 lines

(We are fine with swapping out glibc and its dynamic loader to enable CET with the appropriate kernel mechanism, but we wouldn't want to change the way all other binaries are marked up.)

So we have these compatibility issues with existing binaries. We know some apps are totally broken. It sounds like you are proposing to ignore this and let people who hit the issues work through it themselves. This was also proposed by other glibc developers as a solution for past CET compatibility issues that broke boot on kernel upgrade. I have to say, as the person pushing these patches, I’m uncomfortable with this approach. I don’t think users will like the results. Basically, do they want to upgrade and run a bunch of untested integration with known failures? I also don’t want to get this feature reverted and I’m not exactly sure how this scenario would be taken.

But I hear the point about it not being ideal to abandon the existing CET userspace. I think there is also a point about how userspace chose to do this optimistic and early wide enabling, even if it was a bad idea, and so how much should the kernel try to save userspace from itself. So what do you think about this instead:

The current psABI spec talks about the binary being compatible with shadow stack. It doesn’t say much about what should happen after the loader. Since the greater ecosystem has used this bit with a more cavalier attitude, glibc could treat it as a request for a warn and continue mode. In the meantime we could have a new bit shstk_strict, that requests behavior like these patches implement, and kills the process on violation. Glibc/tools could add support for this strict bit and anyone that wants to more carefully compile with it could finally get shadow stack today. Then the implementation of the warn and continue mode could follow that, and glibc could map the original shstk bit to that kernel mode. So the old binaries would get there eventually, which is better than the continuing nothing they have today.

Quoted 8 lines

And speaking of having nothing today, there are people that really want to use shadow stack and do not care at all about having CET support for existing binaries. Neither glibc or elf bits are required to use kernel shadow stack support. So if it comes to it, I don’t want to hold support back for other users because the elf note bit enabling path grew some issues.

Please let me know about what you think of that plan.

The kernel CET description

+The kernel does not process these applications directly. Applications must +enable them using the interface descriped in section 4. Typically this +would be done in dynamic loader or static runtime objects, as is the case +in glibc.

may leave an impression that each application needs to use the kernel interface to enable CET itself. This is an option. But the updated glibc will enable CET automatically on behalf of the CET enabled application. If the glibc isn't updated to use the new CET kernel interface, the existing CET enabled binaries will run correctly under the new CET enabled kernel without CET enabled.

Signature · 1 line
H.J.
#24 lore raw

Re: [PATCH v2 01/39] Documentation/x86: Add CET description

In reply to Edgecombe, Rick P
Quoted 11 lines · Rick Edgecombe

In the meantime we could have a new bit shstk_strict,
that requests behavior like these patches implement, and kills the process on violation. Glibc/tools could add support for this strict bit
and anyone that wants to more carefully compile with it could finally get shadow stack today. Then the implementation of the warn and continue mode could follow that, and glibc could map the original shstk
bit to that kernel mode. So the old binaries would get there eventually, which is better than the continuing nothing they have today.

Hi,

Any thoughts on this proposal?

Thanks,

Rick

#25 lore raw

PATCH v2 2/39 x86/cet/shstk: Add Kconfig option for Shadow Stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow Stack provides protection against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-Shadow Stack applications continue to work, but without protection.

---

v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)

Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.

Yu-cheng v24:
- Update for the splitting X86_CET to X86_SHADOW_STACK and X86_IBT.

 arch/x86/Kconfig           | 18 ++++++++++++++++++
 arch/x86/Kconfig.assembler |  5 +++++
 2 files changed, 23 insertions(+)
arch/x86/Kconfig +18 -0
index f9920f1341c8..b68eb75887b8 100644
@@ -26,6 +26,7 @@ config X86_64
 	depends on 64BIT
 	# Options that are inherently 64-bit kernel only:
 	select ARCH_HAS_GIGANTIC_PAGE
+	select ARCH_HAS_SHADOW_STACK
 	select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
 	select ARCH_USE_CMPXCHG_LOCKREF
 	select HAVE_ARCH_SOFT_DIRTY
@@ -1936,6 +1937,23 @@ config X86_SGX
 
 	  If unsure, say N.
 
+config ARCH_HAS_SHADOW_STACK
+	def_bool n
+
+config X86_SHADOW_STACK
+	prompt "X86 Shadow Stack"
+	def_bool n
+	depends on ARCH_HAS_SHADOW_STACK
+	select ARCH_USES_HIGH_VMA_FLAGS
+	help
+	  Shadow Stack protection is a hardware feature that detects function
+	  return address corruption. Today the kernel's support is limited to
+	  virtualizing it in KVM guests.
+
+	  CPUs supporting shadow stacks were first released in 2020.
+
+	  If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
arch/x86/Kconfig.assembler +5 -0
index 26b8c08e2fc4..00c79dd93651 100644
@@ -19,3 +19,8 @@ config AS_TPAUSE
 	def_bool $(as-instr,tpause %ecx)
 	help
 	  Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7
+
+config AS_WRUSS
+	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
+	help
+	  Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.17.1
#26 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 44 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow Stack provides protection against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-Shadow Stack applications continue to work, but without protection.

---

v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)

Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.

Yu-cheng v24:
- Update for the splitting X86_CET to X86_SHADOW_STACK and X86_IBT.

 arch/x86/Kconfig           | 18 ++++++++++++++++++
 arch/x86/Kconfig.assembler |  5 +++++
 2 files changed, 23 insertions(+)
arch/x86/Kconfig +1 -0
index f9920f1341c8..b68eb75887b8 100644
@@ -26,6 +26,7 @@ config X86_64
 	depends on 64BIT
 	# Options that are inherently 64-bit kernel only:
 	select ARCH_HAS_GIGANTIC_PAGE
+	select ARCH_HAS_SHADOW_STACK
 	select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
 	select ARCH_USE_CMPXCHG_LOCKREF
 	select HAVE_ARCH_SOFT_DIRTY

arch/x86/Kconfig config X86_SGX · line 1941

Quoted 5 lines
arch/x86/Kconfig +2 -0
@@ -1936,6 +1937,23 @@ config X86_SGX
 
 	  If unsure, say N.
 
+config ARCH_HAS_SHADOW_STACK
+	def_bool n

Hm. Shouldn't ARCH_HAS_SHADOW_STACK definition be in arch/Kconfig, not under arch/x86?

Also, I think "def_bool n" has the same meaning as just "bool", no?

arch/x86/Kconfig config X86_SGX · line 1945

Quoted 4 lines
arch/x86/Kconfig +4 -0
+
+config X86_SHADOW_STACK
+	prompt "X86 Shadow Stack"
+	def_bool n

Maybe just

bool "X86 Shadow Stack"

?

Quoted 30 lines
arch/x86/Kconfig +11 -0
+	depends on ARCH_HAS_SHADOW_STACK
+	select ARCH_USES_HIGH_VMA_FLAGS
+	help
+	  Shadow Stack protection is a hardware feature that detects function
+	  return address corruption. Today the kernel's support is limited to
+	  virtualizing it in KVM guests.
+
+	  CPUs supporting shadow stacks were first released in 2020.
+
+	  If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
arch/x86/Kconfig.assembler +5 -0
index 26b8c08e2fc4..00c79dd93651 100644
@@ -19,3 +19,8 @@ config AS_TPAUSE
 	def_bool $(as-instr,tpause %ecx)
 	help
 	  Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7
+
+config AS_WRUSS
+	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
+	help
+	  Supported by binutils >= 2.31 and LLVM integrated assembler
Signature · 1 line
2.17.1
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#27 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 13 lines · Kirill A . Shutemov

Hm. Shouldn't ARCH_HAS_SHADOW_STACK definition be in arch/Kconfig, not
under arch/x86?

Also, I think "def_bool n" has the same meaning as just "bool", no?

arch/x86/Kconfig config X86_SGX · line 1945

Quoted 4 lines
arch/x86/Kconfig +4 -0
+
+config X86_SHADOW_STACK
+     prompt "X86 Shadow Stack"
+     def_bool n

Maybe just

Quoted 3 lines

bool "X86 Shadow Stack"

?

Yep, will change it. Thanks.

#28 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

In reply to Rick Edgecombe
Quoted 44 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow Stack provides protection against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-Shadow Stack applications continue to work, but without protection.

---

v2:
- Remove already wrong kernel size increase info (tlgx)
- Change prompt to remove "Intel" (tglx)
- Update line about what CPUs are supported (Dave)

Yu-cheng v25:
- Remove X86_CET and use X86_SHADOW_STACK directly.

Yu-cheng v24:
- Update for the splitting X86_CET to X86_SHADOW_STACK and X86_IBT.

 arch/x86/Kconfig           | 18 ++++++++++++++++++
 arch/x86/Kconfig.assembler |  5 +++++
 2 files changed, 23 insertions(+)
arch/x86/Kconfig +1 -0
index f9920f1341c8..b68eb75887b8 100644
@@ -26,6 +26,7 @@ config X86_64
 	depends on 64BIT
 	# Options that are inherently 64-bit kernel only:
 	select ARCH_HAS_GIGANTIC_PAGE
+	select ARCH_HAS_SHADOW_STACK
 	select ARCH_SUPPORTS_INT128 if CC_HAS_INT128
 	select ARCH_USE_CMPXCHG_LOCKREF
 	select HAVE_ARCH_SOFT_DIRTY

arch/x86/Kconfig config X86_SGX · line 1945

Quoted 9 lines
arch/x86/Kconfig +6 -0
@@ -1936,6 +1937,23 @@ config X86_SGX
 
 	  If unsure, say N.
 
+config ARCH_HAS_SHADOW_STACK
+	def_bool n
+
+config X86_SHADOW_STACK
+	prompt "X86 Shadow Stack"
+	def_bool n

I hope we can switch this to "default y" soon, given it's a hardware feature that is disabled at runtime when not available.

arch/x86/Kconfig config X86_SGX · line 1946

Quoted 1 line
arch/x86/Kconfig +1 -0
+	depends on ARCH_HAS_SHADOW_STACK

Doesn't this depend on AS_WRUSS too?

Quoted 18 lines
arch/x86/Kconfig +10 -0
+	select ARCH_USES_HIGH_VMA_FLAGS
+	help
+	  Shadow Stack protection is a hardware feature that detects function
+	  return address corruption. Today the kernel's support is limited to
+	  virtualizing it in KVM guests.
+
+	  CPUs supporting shadow stacks were first released in 2020.
+
+	  If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
arch/x86/Kconfig.assembler +0 -0
index 26b8c08e2fc4..00c79dd93651 100644

arch/x86/Kconfig.assembler config AS_TPAUSE · line 26

Quoted 8 lines
arch/x86/Kconfig.assembler +5 -0
@@ -19,3 +19,8 @@ config AS_TPAUSE
 	def_bool $(as-instr,tpause %ecx)
 	help
 	  Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7
+
+config AS_WRUSS
+	def_bool $(as-instr,wrussq %rax$(comma)(%rbx))
+	help
+	  Supported by binutils >= 2.31 and LLVM integrated assembler

Otherwise, I don't see anything else using OCNFIG_AS_WRUSS:

$ git grep AS_WRUSS
arch/x86/Kconfig.assembler:config AS_WRUSS

-Kees

Signature · 1 line
Kees Cook
#29 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 6 lines · Kees Cook

arch/x86/Kconfig config X86_SGX · line 1945

Quoted 3 lines
arch/x86/Kconfig +3 -0
+config X86_SHADOW_STACK
+     prompt "X86 Shadow Stack"
+     def_bool n

I hope we can switch this to "default y" soon, given it's a hardware feature that is disabled at runtime when not available.

Hmm, yes. Not sure on this. I'm inclined to leave it as is for now.

Quoted 4 lines
Quoted 2 lines

+ depends on ARCH_HAS_SHADOW_STACK

Doesn't this depend on AS_WRUSS too?

Yes, this got messed up when this patch went to and from the CET KVM series.

Thanks!

#30 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

In reply to Rick Edgecombe

arch/x86/Kconfig config X86_SGX · line 1952

Quoted 10 lines · Rick Edgecombe
arch/x86/Kconfig +10 -0
+config X86_SHADOW_STACK
+	prompt "X86 Shadow Stack"
+	def_bool n
+	depends on ARCH_HAS_SHADOW_STACK
+	select ARCH_USES_HIGH_VMA_FLAGS
+	help
+	  Shadow Stack protection is a hardware feature that detects function
+	  return address corruption. Today the kernel's support is limited to
+	  virtualizing it in KVM guests.
+

Is this help text up to date? It seems a bit at odds with the series title.

#31 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 16 lines · Dave Hansen

Review

Quoted 12 lines · Rick Edgecombe
arch/x86/Kconfig +7 -0
+config X86_SHADOW_STACK
+     prompt "X86 Shadow Stack"
+     def_bool n
+     depends on ARCH_HAS_SHADOW_STACK
+     select ARCH_USES_HIGH_VMA_FLAGS
+     help
+       Shadow Stack protection is a hardware feature that detects

function
+ return address corruption. Today the kernel's support is
limited to

(quoted diff) +2 -0
+       virtualizing it in KVM guests.
+

Is this help text up to date? It seems a bit at odds with the series title.

Arg, yes. This patch got screwed up when I converted it back and forth for the KVM series.

maintainer
#32 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

In reply to Rick Edgecombe
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Subject: Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Please remove all "CET", "cet", etc strings from the text as that is confusing. We should use either shadow stack or IBT and not CET.

Quoted 1 line

+config ARCH_HAS_SHADOW_STACK

Do I see it correctly that this thing is needed only once in show_smap_vma_flags()?

If so, can we do a arch_show_smap_vma_flags(), call it at the end of former function and avoid adding yet another Kconfig symbol?

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#33 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>
Subject: Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Please remove all "CET", "cet", etc strings from the text as that is confusing. We should use either shadow stack or IBT and not CET.

Good point, I'll remove it. Thanks.

Quoted 8 lines
Quoted 2 lines

+config ARCH_HAS_SHADOW_STACK

Do I see it correctly that this thing is needed only once in show_smap_vma_flags()?

If so, can we do a arch_show_smap_vma_flags(), call it at the end of former function and avoid adding yet another Kconfig symbol?

Yea, I was thinking to maybe just change it to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags(). In that function there is already CONFIG_ARM64_BTI and CONFIG_ARM64_MTE.

I'm not sure if there is any aversion to having arch CONFIGs in core code, but it's kind of nice to have all of the potentially conflicting strings in once place.

maintainer
#34 lore raw

Re: [PATCH v2 02/39] x86/cet/shstk: Add Kconfig option for Shadow Stack

Quoted 3 lines · Rick P

Yea, I was thinking to maybe just change it to CONFIG_X86_USER_SHADOW_STACK in show_smap_vma_flags(). In that function there is already CONFIG_ARM64_BTI and CONFIG_ARM64_MTE.

I was thinking exactly the same thing. :-)

Quoted 3 lines

I'm not sure if there is any aversion to having arch CONFIGs in core code, but it's kind of nice to have all of the potentially conflicting strings in once place.

Yeah, ok.

I guess you can do the CONFIG_X86_USER_SHADOW_STACK thing for the sake of simplicity. We have *waaay* too many Kconfig symbols and we should introduce only the least amount of new ones.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#35 lore raw

PATCH v2 3/39 x86/cpufeatures: Add CPU feature flags for shadow stacks

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).

To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.

---

v2:
- Remove IBT reference in commit log (Kees)
- Describe xsaves dependency using text from (Dave)

v1:
- Remove IBT, can be added in a follow on IBT series.

Yu-cheng v25:
- Make X86_FEATURE_IBT depend on X86_FEATURE_SHSTK.

Yu-cheng v24:
- Update for splitting CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK and CONFIG_X86_IBT.
- Move DISABLE_IBT definition to the IBT series.

 arch/x86/include/asm/cpufeatures.h       | 1 +
 arch/x86/include/asm/disabled-features.h | 8 +++++++-
 arch/x86/kernel/cpu/cpuid-deps.c         | 1 +
 3 files changed, 9 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index ef4775c6db01..d0b49da95c70 100644
@@ -365,6 +365,7 @@
 #define X86_FEATURE_OSPKE		(16*32+ 4) /* OS Protection Keys Enable */
 #define X86_FEATURE_WAITPKG		(16*32+ 5) /* UMONITOR/UMWAIT/TPAUSE Instructions */
 #define X86_FEATURE_AVX512_VBMI2	(16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */
+#define X86_FEATURE_SHSTK		(16*32+ 7) /* Shadow Stack */
 #define X86_FEATURE_GFNI		(16*32+ 8) /* Galois Field New Instructions */
 #define X86_FEATURE_VAES		(16*32+ 9) /* Vector AES */
 #define X86_FEATURE_VPCLMULQDQ		(16*32+10) /* Carry-Less Multiplication Double Quadword */
arch/x86/include/asm/disabled-features.h +7 -1
index 33d2cd04d254..00fe41eee92d 100644
@@ -87,6 +87,12 @@
 # define DISABLE_TDX_GUEST	(1 << (X86_FEATURE_TDX_GUEST & 31))
 #endif
 
+#ifdef CONFIG_X86_SHADOW_STACK
+#define DISABLE_SHSTK	0
+#else
+#define DISABLE_SHSTK	(1 << (X86_FEATURE_SHSTK & 31))
+#endif
+
 /*
  * Make sure to add features to the correct mask
  */
@@ -107,7 +113,7 @@
 #define DISABLED_MASK14	0
 #define DISABLED_MASK15	0
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \
-			 DISABLE_ENQCMD)
+			 DISABLE_ENQCMD|DISABLE_SHSTK)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
 #define DISABLED_MASK19	0
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
index c881bcafba7d..bf1b55a1ba21 100644
@@ -78,6 +78,7 @@ static const struct cpuid_dep cpuid_deps[] = {
 	{ X86_FEATURE_XFD,			X86_FEATURE_XSAVES    },
 	{ X86_FEATURE_XFD,			X86_FEATURE_XGETBV1   },
 	{ X86_FEATURE_AMX_TILE,			X86_FEATURE_XFD       },
+	{ X86_FEATURE_SHSTK,			X86_FEATURE_XSAVES    },
 	{}
 };
 
Signature · 1 line
2.17.1
Reviewed-by on v2 3/39
#36 lore raw

Re: [PATCH v2 03/39] x86/cpufeatures: Add CPU feature flags for shadow stacks

Quoted 7 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).

Quoted 6 lines

To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.

Signature · 1 line
Kees Cook
maintainer Reviewed-by on v2 3/39
#37 lore raw

Re: [PATCH v2 03/39] x86/cpufeatures: Add CPU feature flags for shadow stacks

In reply to Rick Edgecombe
Quoted 12 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).

To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES.

Quoted 4 lines
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#38 lore raw

Re: [PATCH v2 03/39] x86/cpufeatures: Add CPU feature flags for shadow stacks

Quoted 18 lines · Borislav Petkov
Quoted 17 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Control-Flow Enforcement Technology contains two related features,
one of which is Shadow Stacks. Future patches will utilize this feature
for shadow stack support in KVM, so add a CPU feature flags for Shadow
Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]).

To protect shadow stack state from malicious modification, the registers
are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend
on XSAVES.

Quoted 6 lines
Quoted 4 lines

Thanks!

#39 lore raw

PATCH v2 4/39 x86/cpufeatures: Enable CET CR4 bit for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Utilizing CET features requires a CR4 bit to be enabled as well as bits to be set in CET MSRs. Setting the CR4 bit does two things:
1. Enables the usage of WRUSS instruction, which the kernel can use to write to userspace shadow stacks.
2. Allows those individual aspects of CET to be enabled later via the MSR.
3. Allows CET to be enabled in guests

While future patches will allow the MSR values to be saved and restored per task, the CR4 bit will allow for WRUSS to be used regardless of if a tasks CET MSRs have been restored.

Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.

Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.

---

v2:
- In the shadow stack case, go back to only setting CR4.CET if the kernel is compiled with user shadow stack support.
- Clear MSR_IA32_U_CET as well. (PeterZ)

KVM refresh:
- Set CR4.CET if SHSTK or IBT are supported by HW, so that KVM can support CET even if IBT is disabled.
- Drop no_user_shstk (Dave Hansen)
- Elaborate on what the CR4 bit does in the commit log
- Integrate with Kernel IBT logic

v1:
- Moved kernel-parameters.txt changes here from patch 1.

Yu-cheng v25:
- Remove software-defined X86_FEATURE_CET.

 arch/x86/kernel/cpu/common.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/common.c +16 -6
index 3e508f239098..d7415bb556b2 100644
@@ -598,16 +598,21 @@ __noendbr void ibt_restore(u64 save)
 
 static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 {
-	u64 msr = CET_ENDBR_EN;
+	bool kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT);
+	bool user_shstk = IS_ENABLED(CONFIG_X86_SHADOW_STACK) &&
+			  cpu_feature_enabled(X86_FEATURE_SHSTK);
+	u64 msr = 0;
 
-	if (!HAS_KERNEL_IBT ||
-	    !cpu_feature_enabled(X86_FEATURE_IBT))
+	if (!kernel_ibt && !user_shstk)
 		return;
 
+	if (kernel_ibt)
+		msr = CET_ENDBR_EN;
+
 	wrmsrl(MSR_IA32_S_CET, msr);
 	cr4_set_bits(X86_CR4_CET);
 
-	if (!ibt_selftest()) {
+	if (kernel_ibt && !ibt_selftest()) {
 		pr_err("IBT selftest: Failed!\n");
 		setup_clear_cpu_cap(X86_FEATURE_IBT);
 		return;
@@ -616,10 +621,15 @@ static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 
 __noendbr void cet_disable(void)
 {
-	if (cpu_feature_enabled(X86_FEATURE_IBT))
-		wrmsrl(MSR_IA32_S_CET, 0);
+	if (!(cpu_feature_enabled(X86_FEATURE_IBT) ||
+	      cpu_feature_enabled(X86_FEATURE_SHSTK)))
+		return;
+
+	wrmsrl(MSR_IA32_S_CET, 0);
+	wrmsrl(MSR_IA32_U_CET, 0);
 }
 
+
 /*
  * Some CPU features depend on higher CPUID levels, which may not always
  * be available due to CPUID level capping or broken virtualization
Signature · 1 line
2.17.1
Reviewed-by on v2 4/39
#40 lore raw

Re: [PATCH v2 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack

Quoted 18 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Utilizing CET features requires a CR4 bit to be enabled as well as bits to be set in CET MSRs. Setting the CR4 bit does two things:
1. Enables the usage of WRUSS instruction, which the kernel can use to write to userspace shadow stacks.
2. Allows those individual aspects of CET to be enabled later via the MSR.
3. Allows CET to be enabled in guests

While future patches will allow the MSR values to be saved and restored per task, the CR4 bit will allow for WRUSS to be used regardless of if a tasks CET MSRs have been restored.

Kernel IBT already enables the CET CR4 bit when it detects IBT HW support and is configured with kernel IBT. However future patches that enable userspace shadow stack support will need the bit set as well. So change the logic to enable it in either case.

Quoted 7 lines

Clear MSR_IA32_U_CET in cet_disable() so that it can't live to see userspace in a new kexec-ed kernel that has CR4.CET set from kernel IBT.

Signature · 1 line
Kees Cook
#41 lore raw

Re: [PATCH v2 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack

In reply to Rick Edgecombe
Quoted 8 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Utilizing CET features requires a CR4 bit to be enabled as well as bits to be set in CET MSRs. Setting the CR4 bit does two things:
1. Enables the usage of WRUSS instruction, which the kernel can use to write to userspace shadow stacks.
2. Allows those individual aspects of CET to be enabled later via the MSR.
3. Allows CET to be enabled in guests

Point 1, yes, but the others, not really.  Guests aren't interesting because host CR4 != guest CR4.

CET is a tangled mess of control bits.  The MSRs can be configured and context switched independently CR4.

The 4 main sub-feature enablement conditions are CR4.CET && MSR_{U,S}_CET.{SHSTK,ENDBR}_EN.

The WRUSS instruction is keyed on CR4.CET alone.  This is because CR4.CET is the paging control which changes the interpretation of R/O+Dirty, and is a prerequisite for any shstk memory accesses.  Most other shstk instructions have finer grain enablement conditions.

I'd suggest simplifying the commit message massively, to say that CR4.CET is a prerequisite for all CET operation, so extend setup_cet() to enable it for user shadow stacks.

It hopefully goes without saying that you cannot do an equivalent piece of code for supervisor shadow stacks.  If you try, you'll discover that everything works fine until you try returning from the function which activated the second of CR4.CET and MSR_S_CET.SHSTK_EN, and the valid content on the shadow stack underflows.

~Andrew

P.S. There's a fun infoleak.

Userspace can probe for kernel shstk enablement using fault analysis on the SETSSBUSY instruction.  It takes #UD for !CR4.CET || !MSR_S_CET.SHSTK_EN, and then #GP for CPL !=0.

maintainer
#42 lore raw

Re: [PATCH v2 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack

In reply to Rick Edgecombe

arch/x86/kernel/cpu/common.c __noendbr void ibt_restore(u64 save) · line 601

Quoted 4 lines · Rick Edgecombe

static __always_inline void setup_cet(struct cpuinfo_x86 *c) {

arch/x86/kernel/cpu/common.c +1 -1
-	u64 msr = CET_ENDBR_EN;
+	bool kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT);

So I'd love it if we can get rid of that HAS_KERNEL_IBT thing and use the usual ifdeffery with Kconfig symbols. I wouldn't like for yet another HAS_XXX feature checking method to proliferate as this is the only one:

$ git grep -E "\WHAS_" arch/x86/
arch/x86/include/asm/ibt.h:18: * When all the above are satisfied, HAS_KERNEL_IBT will be 1, otherwise 0.
arch/x86/include/asm/ibt.h:22:#define HAS_KERNEL_IBT    1
arch/x86/include/asm/ibt.h:92:#define HAS_KERNEL_IBT    0
arch/x86/include/asm/ibt.h:114:#define ENDBR_INSN_SIZE          (4*HAS_KERNEL_IBT)
arch/x86/include/asm/idtentry.h:8:#define IDT_ALIGN     (8 * (1 + HAS_KERNEL_IBT))
arch/x86/kernel/cpu/common.c:601:       bool kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT);
arch/x86/kernel/cpu/common.c:1942:      if (HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT))

arch/x86/kernel/cpu/common.c static __always_inline void setup_cet(struct cpuinfo_x86 *c) · line 632

Quoted 13 lines

__noendbr void cet_disable(void)
{

arch/x86/kernel/cpu/common.c +7 -2
-	if (cpu_feature_enabled(X86_FEATURE_IBT))
-		wrmsrl(MSR_IA32_S_CET, 0);
+	if (!(cpu_feature_enabled(X86_FEATURE_IBT) ||
+	      cpu_feature_enabled(X86_FEATURE_SHSTK)))
+		return;
+
+	wrmsrl(MSR_IA32_S_CET, 0);
+	wrmsrl(MSR_IA32_U_CET, 0);
 }
 
+

Stray newline.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#43 lore raw

Re: [PATCH v2 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack

Quoted 11 lines · Borislav Petkov
Quoted 5 lines · Rick Edgecombe
  static __always_inline void setup_cet(struct cpuinfo_x86 *c)
  {
(quoted diff) +1 -1
-     u64 msr = CET_ENDBR_EN;
+     bool kernel_ibt = HAS_KERNEL_IBT &&

cpu_feature_enabled(X86_FEATURE_IBT);

So I'd love it if we can get rid of that HAS_KERNEL_IBT thing and use the usual ifdeffery with Kconfig symbols. I wouldn't like for yet another HAS_XXX feature checking method to proliferate as this is the only one:

Andrew Cooper has suggested to create some software cpu features to differentiate user/supervisor CET feature use. It could replace HAS_KERNEL_IBT. Any objections to that versus Kconfig symbols?

[snip]

Quoted 14 lines

cpu_feature_enabled(X86_FEATURE_IBT))

Quoted 12 lines
  __noendbr void cet_disable(void)
  {
arch/x86/kernel/cpu/common.c +6 -2
-     if (cpu_feature_enabled(X86_FEATURE_IBT))
-             wrmsrl(MSR_IA32_S_CET, 0);
+     if (!(cpu_feature_enabled(X86_FEATURE_IBT) ||
+           cpu_feature_enabled(X86_FEATURE_SHSTK)))
+             return;
+
+     wrmsrl(MSR_IA32_S_CET, 0);
+     wrmsrl(MSR_IA32_U_CET, 0);
  }
Quoted 3 lines
Quoted 1 line

+

Stray newline.

Oops, will clean that up. Thanks.

maintainer
#44 lore raw

Re: [PATCH v2 04/39] x86/cpufeatures: Enable CET CR4 bit for shadow stack

Quoted 3 lines · Rick P

Andrew Cooper has suggested to create some software cpu features to differentiate user/supervisor CET feature use. It could replace HAS_KERNEL_IBT. Any objections to that versus Kconfig symbols?

Sure, except you can't use them in

arch/x86/include/asm/idtentry.h:8:#define IDT_ALIGN (8 * (1 + HAS_KERNEL_IBT))

as that gets used in asm code.

But you don't have to do this in this patchset - it is huge already anyway. I have this thing on my todo so I'll get to it eventually. Unless you're itching to remove it yourself - then I won't stay in the way.

:-)

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#45 lore raw

PATCH v2 5/39 x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups:
* Registers controlling user-mode operation
* Registers controlling kernel-mode operation

The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size.

Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required.

Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable.

While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks.

Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET.

To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches.

---

v2:
- Change name to XFEATURE_CET_KERNEL_UNUSED (peterz)

KVM refresh:
- Reword commit log using some verbiage posted by Dave Hansen
- Remove unlikely to be used supervisor cet xsave struct
- Clarify that supervisor cet state is not saved by xsave
- Remove unused supervisor MSRs

v1:
- Remove outdated reference to sigreturn checks on msr's.

Yu-cheng v29:
- Move CET MSR definition up in msr-index.h.

 arch/x86/include/asm/fpu/types.h  | 14 ++++-
 arch/x86/include/asm/fpu/xstate.h |  6 +-
 arch/x86/kernel/fpu/xstate.c      | 93 ++++++++++++++++---------------
 3 files changed, 63 insertions(+), 50 deletions(-)
arch/x86/include/asm/fpu/types.h +12 -2
index eb7cd1139d97..344baad02b97 100644
@@ -115,8 +115,8 @@ enum xfeature {
 	XFEATURE_PT_UNIMPLEMENTED_SO_FAR,
 	XFEATURE_PKRU,
 	XFEATURE_PASID,
-	XFEATURE_RSRVD_COMP_11,
-	XFEATURE_RSRVD_COMP_12,
+	XFEATURE_CET_USER,
+	XFEATURE_CET_KERNEL_UNUSED,
 	XFEATURE_RSRVD_COMP_13,
 	XFEATURE_RSRVD_COMP_14,
 	XFEATURE_LBR,
@@ -138,6 +138,8 @@ enum xfeature {
 #define XFEATURE_MASK_PT		(1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR)
 #define XFEATURE_MASK_PKRU		(1 << XFEATURE_PKRU)
 #define XFEATURE_MASK_PASID		(1 << XFEATURE_PASID)
+#define XFEATURE_MASK_CET_USER		(1 << XFEATURE_CET_USER)
+#define XFEATURE_MASK_CET_KERNEL	(1 << XFEATURE_CET_KERNEL_UNUSED)
 #define XFEATURE_MASK_LBR		(1 << XFEATURE_LBR)
 #define XFEATURE_MASK_XTILE_CFG		(1 << XFEATURE_XTILE_CFG)
 #define XFEATURE_MASK_XTILE_DATA	(1 << XFEATURE_XTILE_DATA)
@@ -252,6 +254,14 @@ struct pkru_state {
 	u32				pad;
 } __packed;
 
+/*
+ * State component 11 is Control-flow Enforcement user states
+ */
+struct cet_user_state {
+	u64 user_cet;			/* user control-flow settings */
+	u64 user_ssp;			/* user shadow stack pointer */
+};
+
 /*
  * State component 15: Architectural LBR configuration state.
  * The size of Arch LBR state depends on the number of LBRs (lbr_depth).
arch/x86/include/asm/fpu/xstate.h +4 -2
index cd3dd170e23a..d4427b88ee12 100644
@@ -50,7 +50,8 @@
 #define XFEATURE_MASK_USER_DYNAMIC	XFEATURE_MASK_XTILE_DATA
 
 /* All currently supported supervisor features */
-#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID)
+#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \
+					    XFEATURE_MASK_CET_USER)
 
 /*
  * A supervisor state component may not always contain valuable information,
@@ -77,7 +78,8 @@
  * Unsupported supervisor features. When a supervisor feature in this mask is
  * supported in the future, move it to the supported supervisor feature mask.
  */
-#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT)
+#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \
+					      XFEATURE_MASK_CET_KERNEL)
 
 /* All supervisor states including supported and unsupported states. */
 #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \
arch/x86/kernel/fpu/xstate.c +47 -46

137 lines not on this page · raw message

Signature · 1 line
2.17.1
Reviewed-by on v2 5/39
#46 lore raw

Re: [PATCH v2 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

Quoted 3 lines · Rick Edgecombe

[...]
xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was

Spelling nit through-out all patches: possessive used for plurals. E.g. the above "check's" instances should be "checks". Please review all the documentation and commit logs; it shows up a lot. :)

Quoted 39 lines

[...]

arch/x86/kernel/fpu/xstate.c +10 -20
index c8340156bfd2..5e6a4867fd05 100644
@@ -39,26 +39,26 @@
  */
 static const char *xfeature_names[] =
 {
-	"x87 floating point registers"	,
-	"SSE registers"			,
-	"AVX registers"			,
-	"MPX bounds registers"		,
-	"MPX CSR"			,
-	"AVX-512 opmask"		,
-	"AVX-512 Hi256"			,
-	"AVX-512 ZMM_Hi256"		,
-	"Processor Trace (unused)"	,
-	"Protection Keys User registers",
-	"PASID state",
-	"unknown xstate feature"	,
-	"unknown xstate feature"	,
-	"unknown xstate feature"	,
-	"unknown xstate feature"	,
-	"unknown xstate feature"	,
-	"unknown xstate feature"	,
-	"AMX Tile config"		,
-	"AMX Tile data"			,
-	"unknown xstate feature"	,
+	"x87 floating point registers"			,
+	"SSE registers"					,
+	"AVX registers"					,
+	"MPX bounds registers"				,
+	"MPX CSR"					,
+	"AVX-512 opmask"				,
+	"AVX-512 Hi256"					,
+	"AVX-512 ZMM_Hi256"				,
+	"Processor Trace (unused)"			,
+	"Protection Keys User registers"		,

arch/x86/kernel/fpu/xstate.c line 61

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+	"PASID state"					,
+	"Control-flow User registers"			,
+	"Control-flow Kernel registers (unused)"	,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"AMX Tile config"				,
+	"AMX Tile data"					,
+	"unknown xstate feature"			,

What a strange style. Why not just leave the commas after the " ? Then these kinds of multi-line updates aren't needed in the future.

Quoted 4 lines

[...]

arch/x86/kernel/fpu/xstate.c +0 -3
-	/*
-	 * Make *SURE* to add any feature numbers in below if
-	 * there are "holes" in the xsave state component

arch/x86/kernel/fpu/xstate.c static bool __init check_xstate_against_struct(int nr) · line 560

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +1 -6
-	 * numbers.
-	 */
-	if ((nr < XFEATURE_YMM) ||
-	    (nr >= XFEATURE_MAX) ||
-	    (nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) ||
-	    ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) {
+	if (!chked) {
 		WARN_ONCE(1, "no structure for xstate: %d\n", nr);
 		XSTATE_WARN_ON(1);
 		return false;

This clean-up feels like it should be part of a separate patch, but okay. :)

Signature · 1 line
Kees Cook
maintainer
#47 lore raw

Re: [PATCH v2 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

In reply to Rick Edgecombe
Quoted 3 lines · Rick Edgecombe

Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state a normal user (non-supervisor)

^^^^^

A verb is missing in that sentence.

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+	"x87 floating point registers"			,
+	"SSE registers"					,
+	"AVX registers"					,
+	"MPX bounds registers"				,
+	"MPX CSR"					,
+	"AVX-512 opmask"				,
+	"AVX-512 Hi256"					,
+	"AVX-512 ZMM_Hi256"				,
+	"Processor Trace (unused)"			,
+	"Protection Keys User registers"		,

arch/x86/kernel/fpu/xstate.c line 61

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+	"PASID state"					,
+	"Control-flow User registers"			,
+	"Control-flow Kernel registers (unused)"	,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"AMX Tile config"				,
+	"AMX Tile data"					,
+	"unknown xstate feature"			,

What Kees said. :)

arch/x86/kernel/fpu/xstate.c static bool __init check_xstate_against_struct(int nr) · line 549

Quoted 10 lines
arch/x86/kernel/fpu/xstate.c +10 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_YMM,       struct ymmh_struct);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDREGS,   struct mpx_bndreg_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDCSR,    struct mpx_bndcsr_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_OPMASK,    struct avx_512_opmask_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_Hi16_ZMM,  struct avx_512_hi16_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_PKRU,      struct pkru_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_PASID,     struct ia32_pasid_state);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg);
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_CET_USER,  struct cet_user_state);

That looks silly. I wonder if you could do:

	switch (nr) {
	case XFEATURE_YMM:	XCHECK_SZ(sz, XFEATURE_YMM, struct ymmh_struct);	  return;
	case XFEATURE_BNDREGS:	XCHECK_SZ(sz, XFEATURE_BNDREGS, struct mpx_bndreg_state); return;
	case ...
	...
	default:
		/* that falls into the WARN etc */

and then you get rid of the if check in the macro itself and leave the macro be a dumb, unconditional one.

Hmmm.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#48 lore raw

Re: [PATCH v2 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

Quoted 10 lines · Borislav Petkov
Quoted 5 lines · Rick Edgecombe

Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features
like protection keys where the PKRU state a normal user (non- supervisor)

^^^^^

A verb is missing in that sentence.

Oops yes.

Quoted 15 lines
Quoted 15 lines
arch/x86/kernel/fpu/xstate.c +14 -0
+	"x87 floating point registers"			,
+	"SSE registers"					,
+	"AVX registers"					,
+	"MPX bounds registers"				,
+	"MPX CSR"					,
+	"AVX-512 opmask"				,
+	"AVX-512 Hi256"					,
+	"AVX-512 ZMM_Hi256"				,
+	"Processor Trace (unused)"			,
+	"Protection Keys User registers"		,
+	"PASID state"					,
+	"Control-flow User registers"			,
+	"Control-flow Kernel registers (unused)"	,
+	"unknown xstate feature"			,
Quoted 8 lines

arch/x86/kernel/fpu/xstate.c line 61

Quoted 6 lines
arch/x86/kernel/fpu/xstate.c +6 -0
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"unknown xstate feature"			,
+	"AMX Tile config"				,
+	"AMX Tile data"					,
+	"unknown xstate feature"			,

What Kees said. :)

Sure, I'll adjust the comma.

Quoted 34 lines
Quoted 20 lines
(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_YMM,       struct

ymmh_struct);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDREGS,   struct

mpx_bndreg_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDCSR,    struct

mpx_bndcsr_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_OPMASK,    struct

avx_512_opmask_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_ZMM_Hi256, struct

avx_512_zmm_uppers_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_Hi16_ZMM,  struct

avx_512_hi16_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_PKRU,      struct

pkru_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_PASID,     struct

ia32_pasid_state);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_XTILE_CFG, struct

xtile_cfg);

(quoted diff) +1 -0
+	XCHECK_SZ(&chked, sz, nr, XFEATURE_CET_USER,  struct
Quoted 1 line

cet_user_state);

That looks silly. I wonder if you could do:

	switch (nr) {
	case XFEATURE_YMM:	XCHECK_SZ(sz, XFEATURE_YMM, struct
ymmh_struct);	  return;
	case XFEATURE_BNDREGS:	XCHECK_SZ(sz, XFEATURE_BNDREGS,
struct mpx_bndreg_state); return;
	case ...
	...
	default:
		/* that falls into the WARN etc */
Quoted 6 lines

and then you get rid of the if check in the macro itself and leave the
macro be a dumb, unconditional one.

Hmmm.

Hmm yea. Another reason the actual define is passed in is that the macro want's to stringify the XFEATURE define in order to generate the message like this:
XFEATURE_YMM: struct is 123 bytes, cpu state is 456 bytes

The exact format of the message is probably not too critical though. If instead it used xfeature_names[], it could be:
[AVX registers]: struct is 123 bytes, cpu state is 456 bytes

The full block looks like (like you had):
switch (nr) {
case XFEATURE_YMM: return XCHECK_SZ(sz, nr, struct ymmh_struct);
case XFEATURE_BNDREGS: return XCHECK_SZ(sz, nr, struct
mpx_bndreg_state);
case XFEATURE_BNDCSR: return XCHECK_SZ(sz, nr, struct
mpx_bndcsr_state);
case XFEATURE_OPMASK: return XCHECK_SZ(sz, nr, struct
avx_512_opmask_state);
case XFEATURE_ZMM_Hi256: return XCHECK_SZ(sz, nr, struct avx_512_zmm_uppers_state);
case XFEATURE_Hi16_ZMM: return XCHECK_SZ(sz, nr, struct
avx_512_hi16_state);
case XFEATURE_PKRU: return XCHECK_SZ(sz, nr, struct pkru_state);
case XFEATURE_PASID: return XCHECK_SZ(sz, nr, struct
ia32_pasid_state);
case XFEATURE_XTILE_CFG: return XCHECK_SZ(sz, nr, struct xtile_cfg);
case XFEATURE_CET_USER: return XCHECK_SZ(sz, nr, struct
cet_user_state);
case XFEATURE_XTILE_DATA: check_xtile_data_against_struct(sz); return true;
default:

	WARN_ONCE(1, "no structure for xstate: %d\n", nr);
	XSTATE_WARN_ON(1);
	return false;
}

I like how it fits the XFEATURE_XTILE_DATA check in with the rest.

Thanks!

maintainer
#49 lore raw

Re: [PATCH v2 05/39] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states

Quoted 8 lines · Rick P

Hmm yea. Another reason the actual define is passed in is that the macro want's to stringify the XFEATURE define in order to generate the message like this:
XFEATURE_YMM: struct is 123 bytes, cpu state is 456 bytes

The exact format of the message is probably not too critical though. If instead it used xfeature_names[], it could be:
[AVX registers]: struct is 123 bytes, cpu state is 456 bytes

Bah, "registers", that made me look at the thing. Yeah, not sure if all those "registers" strings even matter there.

[AVX]: struct is 123 bytes, cpu state is 456 bytes

looks good enough to me too. But WTH do I know.

Quoted 22 lines

The full block looks like (like you had):
switch (nr) {
case XFEATURE_YMM: return XCHECK_SZ(sz, nr, struct ymmh_struct);
case XFEATURE_BNDREGS: return XCHECK_SZ(sz, nr, struct
mpx_bndreg_state);
case XFEATURE_BNDCSR: return XCHECK_SZ(sz, nr, struct
mpx_bndcsr_state);
case XFEATURE_OPMASK: return XCHECK_SZ(sz, nr, struct
avx_512_opmask_state);
case XFEATURE_ZMM_Hi256: return XCHECK_SZ(sz, nr, struct avx_512_zmm_uppers_state);
case XFEATURE_Hi16_ZMM: return XCHECK_SZ(sz, nr, struct
avx_512_hi16_state);
case XFEATURE_PKRU: return XCHECK_SZ(sz, nr, struct pkru_state);
case XFEATURE_PASID: return XCHECK_SZ(sz, nr, struct
ia32_pasid_state);
case XFEATURE_XTILE_CFG: return XCHECK_SZ(sz, nr, struct xtile_cfg);
case XFEATURE_CET_USER: return XCHECK_SZ(sz, nr, struct
cet_user_state);
case XFEATURE_XTILE_DATA: check_xtile_data_against_struct(sz); return true;
default:

Quoted 6 lines
	WARN_ONCE(1, "no structure for xstate: %d\n", nr);
	XSTATE_WARN_ON(1);
	return false;
}

I like how it fits the XFEATURE_XTILE_DATA check in with the rest.

Yap, nice and straight-forward pattern. :)

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#50 lore raw

PATCH v2 6/39 x86/fpu: Add helper for modifying xstate

In reply to Rick Edgecombe

Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.

When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.

The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.

Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.

---

v2:
- Drop optimization of writing directly the buffer, and change API accordingly.
- fpregs_lock_and_load() suggested by tglx
- Some commit log verbiage from dhansen

v1:
- New patch.

 arch/x86/include/asm/fpu/api.h |  6 ++++++
 arch/x86/kernel/fpu/core.c     | 19 +++++++++++++++++++
 2 files changed, 25 insertions(+)
arch/x86/include/asm/fpu/api.h +6 -0
index 503a577814b2..3a86ee18ae99 100644
@@ -82,6 +82,12 @@ static inline void fpregs_unlock(void)
 		preempt_enable();
 }
 
+/*
+ * Lock and load the fpu state into the registers, if they are not already
+ * loaded.
+ */
+void fpu_lock_and_load(void);
+
 #ifdef CONFIG_X86_DEBUG_FPU
 extern void fpregs_assert_state_consistent(void);
 #else
arch/x86/kernel/fpu/core.c +19 -0
index 3b28c5b25e12..778d3054ccc7 100644
@@ -756,6 +756,25 @@ void switch_fpu_return(void)
 }
 EXPORT_SYMBOL_GPL(switch_fpu_return);
 
+void fpu_lock_and_load(void)
+{
+	/*
+	 * fpregs_lock() only disables preemption (mostly). So modifing state
+	 * in an interrupt could screw up some in progress fpregs operation,
+	 * but appear to work. Warn about it.
+	 */
+	WARN_ON_ONCE(!irq_fpu_usable());
+	WARN_ON_ONCE(current->flags & PF_KTHREAD);
+
+	fpregs_lock();
+
+	fpregs_assert_state_consistent();
+
+	if (test_thread_flag(TIF_NEED_FPU_LOAD))
+		fpregs_restore_userregs();
+}
+EXPORT_SYMBOL_GPL(fpu_lock_and_load);
+
 #ifdef CONFIG_X86_DEBUG_FPU
 /*
  * If current FPU state according to its tracking (loaded FPU context on this
Signature · 1 line
2.17.1
Reviewed-by on v2 6/39
#51 lore raw

Re: [PATCH v2 06/39] x86/fpu: Add helper for modifying xstate

Quoted 12 lines · Rick Edgecombe

Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer.

When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is.

Quoted 4 lines

The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future.

Just for my own clarity, does this mean lock/load _needs_ to happen before MSR access, or is it just a convenient place to do it? From later patches it seems it's a requirement during MSR access, which might be a good idea to detail here. It answers the question "when is this function needed?"

Quoted 4 lines

Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.

Nit: this is called "fpu_lock_and_load" in the patch itself.

Quoted 3 lines
Signature · 1 line
Kees Cook
#52 lore raw

Re: [PATCH v2 06/39] x86/fpu: Add helper for modifying xstate

Quoted 9 lines · Kees Cook
Quoted 8 lines

The easiest way to modify supervisor xfeature data is to force restore
the registers and write directly to the MSRs. Often times this is just fine
anyway as the registers need to be restored before returning to userspace.
Do this for now, leaving buffer writing optimizations for the future.

Quoted 8 lines

Just for my own clarity, does this mean lock/load _needs_ to happen before MSR access, or is it just a convenient place to do it? From later
patches it seems it's a requirement during MSR access, which might be a
good idea to detail here. It answers the question "when is this function
needed?"

The CET state is xsaves managed. It gets lazily restored before returning to userspace with the rest of the fpu stuff. This function will force restore all the fpu state to the registers early and lock them from being automatically saved/restored. Then the tasks CET state can be modified in the MSRs, before unlocking the fpregs. Last time I tried to modify the state directly in the xsave buffer when it was efficient, but it had issues and Thomas suggested this.

Quoted 9 lines
Quoted 7 lines

Add a new function fpregs_lock_and_load() that can simultaneously call
fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code.

Nit: this is called "fpu_lock_and_load" in the patch itself.

Oops, thanks.

#53 lore raw

Re: [PATCH v2 06/39] x86/fpu: Add helper for modifying xstate

Quoted 19 lines · Rick P
Quoted 9 lines · Kees Cook
Quoted 8 lines

The easiest way to modify supervisor xfeature data is to force restore
the registers and write directly to the MSRs. Often times this is just fine
anyway as the registers need to be restored before returning to userspace.
Do this for now, leaving buffer writing optimizations for the future.

Quoted 8 lines

Just for my own clarity, does this mean lock/load _needs_ to happen before MSR access, or is it just a convenient place to do it? From later
patches it seems it's a requirement during MSR access, which might be a
good idea to detail here. It answers the question "when is this function
needed?"

Quoted 7 lines

The CET state is xsaves managed. It gets lazily restored before returning to userspace with the rest of the fpu stuff. This function will force restore all the fpu state to the registers early and lock them from being automatically saved/restored. Then the tasks CET state can be modified in the MSRs, before unlocking the fpregs. Last time I tried to modify the state directly in the xsave buffer when it was efficient, but it had issues and Thomas suggested this.

Okay, gotcha. Thanks!

Signature · 1 line
Kees Cook
#54 lore raw

Re: [PATCH v2 06/39] x86/fpu: Add helper for modifying xstate

In reply to Edgecombe, Rick P
Quoted 7 lines · Rick P

The CET state is xsaves managed. It gets lazily restored before returning to userspace with the rest of the fpu stuff. This function will force restore all the fpu state to the registers early and lock them from being automatically saved/restored. Then the tasks CET state can be modified in the MSRs, before unlocking the fpregs. Last time I tried to modify the state directly in the xsave buffer when it was efficient, but it had issues and Thomas suggested this.

Can you get the gist of this in a comment, please?

#55 lore raw

Re: [PATCH v2 06/39] x86/fpu: Add helper for modifying xstate

Quoted 13 lines · Dave Hansen
Quoted 11 lines · Rick P

The CET state is xsaves managed. It gets lazily restored before returning to userspace with the rest of the fpu stuff. This function
will force restore all the fpu state to the registers early and lock
them from being automatically saved/restored. Then the tasks CET state
can be modified in the MSRs, before unlocking the fpregs. Last time I
tried to modify the state directly in the xsave buffer when it was efficient, but it had issues and Thomas suggested this.

Quoted 1 line

Can you get the gist of this in a comment, please?

Sure.

#56 lore raw

PATCH v2 7/39 x86/cet: Add user control-protection fault handler

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

A control-protection fault is triggered when a control-flow transfer attempt violates Shadow Stack or Indirect Branch Tracking constraints. For example, the return address for a RET instruction differs from the copy on the shadow stack.

There already exists a control-protection fault handler for handling kernel IBT. Refactor this fault handler into sparate user and kernel handlers, like the page fault handler. Add a control-protection handler for usermode.

The control-protection fault handler works in a similar way as the general protection fault handler. It provides the si_code SEGV_CPERR to the signal handler.

---

v2:
- Integrate with kernel IBT fault handler
- Update printed messages. (Dave)
- Remove array_index_nospec() usage. (Dave)
- Remove IBT messages. (Dave)
- Add enclave error code bit processing it case it can get triggered somehow.
- Add extra "unknown" in control_protection_err.

v1:
- Update static asserts for NSIGSEGV

Yu-cheng v29:
- Remove pr_emerg() since it is followed by die().
- Change boot_cpu_has() to cpu_feature_enabled().

Yu-cheng v25:
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.

 arch/arm/kernel/signal.c           |  2 +-
 arch/arm64/kernel/signal.c         |  2 +-
 arch/arm64/kernel/signal32.c       |  2 +-
 arch/sparc/kernel/signal32.c       |  2 +-
 arch/sparc/kernel/signal_64.c      |  2 +-
 arch/x86/include/asm/idtentry.h    |  2 +-
 arch/x86/kernel/idt.c              |  2 +-
 arch/x86/kernel/signal_compat.c    |  2 +-
 arch/x86/kernel/traps.c            | 98 ++++++++++++++++++++++++++----
 arch/x86/xen/enlighten_pv.c        |  2 +-
 arch/x86/xen/xen-asm.S             |  2 +-
 include/uapi/asm-generic/siginfo.h |  3 +-
 12 files changed, 97 insertions(+), 24 deletions(-)
arch/arm/kernel/signal.c +1 -1
index ea128e32e8ca..fa47b8754624 100644
@@ -681,7 +681,7 @@ asmlinkage void do_rseq_syscall(struct pt_regs *regs)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/arm64/kernel/signal.c +1 -1
index 9ad911f1647c..81b13a21046e 100644
@@ -1166,7 +1166,7 @@ void __init minsigstksz_setup(void)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/arm64/kernel/signal32.c +1 -1
index 4700f8522d27..bbd542704730 100644
@@ -460,7 +460,7 @@ void compat_setup_restart_syscall(struct pt_regs *regs)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/sparc/kernel/signal32.c +1 -1
index dad38960d1a8..82da8a2d769d 100644
@@ -751,7 +751,7 @@ asmlinkage int do_sys32_sigstack(u32 u_ssptr, u32 u_ossptr, unsigned long sp)
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/sparc/kernel/signal_64.c +1 -1
index 570e43e6fda5..b4e410976e0d 100644
@@ -562,7 +562,7 @@ void do_notify_resume(struct pt_regs *regs, unsigned long orig_i0, unsigned long
  */
 static_assert(NSIGILL	== 11);
 static_assert(NSIGFPE	== 15);
-static_assert(NSIGSEGV	== 9);
+static_assert(NSIGSEGV	== 10);
 static_assert(NSIGBUS	== 5);
 static_assert(NSIGTRAP	== 6);
 static_assert(NSIGCHLD	== 6);
arch/x86/include/asm/idtentry.h +1 -1
index 72184b0b2219..6768c9d4468c 100644
@@ -618,7 +618,7 @@ DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_DF,	xenpv_exc_double_fault);
 #endif
 
 /* #CP */
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
 DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_CP,	exc_control_protection);
 #endif
 
arch/x86/kernel/idt.c +1 -1
index a58c6bc1cd68..90cce3614ead 100644
@@ -107,7 +107,7 @@ static const __initconst struct idt_data def_idts[] = {
 	ISTG(X86_TRAP_MC,		asm_exc_machine_check, IST_INDEX_MCE),
 #endif
 
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
 	INTG(X86_TRAP_CP,		asm_exc_control_protection),
 #endif
 
arch/x86/kernel/signal_compat.c +1 -1
index 879ef8c72f5c..d441804443d5 100644
@@ -27,7 +27,7 @@ static inline void signal_compat_build_tests(void)
 	 */
 	BUILD_BUG_ON(NSIGILL  != 11);
 	BUILD_BUG_ON(NSIGFPE  != 15);
-	BUILD_BUG_ON(NSIGSEGV != 9);
+	BUILD_BUG_ON(NSIGSEGV != 10);
 	BUILD_BUG_ON(NSIGBUS  != 5);
 	BUILD_BUG_ON(NSIGTRAP != 6);
 	BUILD_BUG_ON(NSIGCHLD != 6);
arch/x86/kernel/traps.c +85 -13

122 lines not on this page · raw message

arch/x86/xen/enlighten_pv.c +1 -1
index 0ed2e487a693..57faa287163f 100644
@@ -628,7 +628,7 @@ static struct trap_array_entry trap_array[] = {
 	TRAP_ENTRY(exc_coprocessor_error,		false ),
 	TRAP_ENTRY(exc_alignment_check,			false ),
 	TRAP_ENTRY(exc_simd_coprocessor_error,		false ),
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
 	TRAP_ENTRY(exc_control_protection,		false ),
 #endif
 };
arch/x86/xen/xen-asm.S +1 -1
index 6b4fdf6b9542..e45ff6300c7d 100644
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault
 xen_pv_trap asm_exc_spurious_interrupt_bug
 xen_pv_trap asm_exc_coprocessor_error
 xen_pv_trap asm_exc_alignment_check
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
 xen_pv_trap asm_exc_control_protection
 #endif
 #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +2 -1
index ffbe4cec9f32..0f52d0ac47c5 100644
@@ -242,7 +242,8 @@ typedef struct siginfo {
 #define SEGV_ADIPERR	7	/* Precise MCD exception */
 #define SEGV_MTEAERR	8	/* Asynchronous ARM MTE error */
 #define SEGV_MTESERR	9	/* Synchronous ARM MTE exception */
-#define NSIGSEGV	9
+#define SEGV_CPERR	10	/* Control protection fault */
+#define NSIGSEGV	10
 
 /*
  * SIGBUS si_codes
Signature · 1 line
2.17.1
#57 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

arch/x86/kernel/traps.c enum cp_error_code { · line 282

Quoted 5 lines · Rick Edgecombe
arch/x86/kernel/traps.c +5 -0
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
+{
+	WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");

Why is this a warning, but runtime check for !X86_FEATURE_IBT and !X86_FEATURE_SHSTK below is fatal?

Quoted 15 lines
arch/x86/kernel/traps.c +10 -0
+}
+#endif
+
+#ifdef CONFIG_X86_KERNEL_IBT
+
+static __ro_after_init bool ibt_fatal = true;
+
+extern void ibt_selftest_ip(void); /* code label defined in asm below */
 
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
 	if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) {
 		regs->ax = 0;
 		return;

arch/x86/kernel/traps.c static int __init ibt_setup(char *str) · line 341

Quoted 8 lines
arch/x86/kernel/traps.c +4 -1
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
 }
 
 __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT disabled\n");

Ditto.

Quoted 22 lines
arch/x86/kernel/traps.c +17 -0
+}
 #endif /* CONFIG_X86_KERNEL_IBT */
 
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}
+
+	if (user_mode(regs))
+		do_user_control_protection_fault(regs, error_code);
+	else
+		do_kernel_control_protection_fault(regs);
+}
+#endif /* defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK) */
+
 #ifdef CONFIG_X86_F00F_BUG
 void handle_invalid_op(struct pt_regs *regs)
 #else
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#58 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

Quoted 11 lines · Kirill A . Shutemov
Quoted 7 lines · Rick Edgecombe
arch/x86/kernel/traps.c +3 -0
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+                                          unsigned long

error_code)

(quoted diff) +2 -0
+{
+     WARN_ONCE(1, "User-mode control protection fault with shadow

support disabled\n");

Why is this a warning, but runtime check for !X86_FEATURE_IBT and !X86_FEATURE_SHSTK below is fatal?

It was a BUG() in the original KERNEL_IBT focused handler IIRC. There seems to be some renewed effort to stop doing those:

https://lore.kernel.org/all/[email protected]/T/#u

...so I'll change it to a WARN for this. In the kernel specific portion of the handler, it also does a BUG on endbranch violation. I'll leave that one for this change.

Reviewed-by on v2 7/39
#59 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

In reply to Rick Edgecombe

Review

Quoted 3 lines · Rick Edgecombe

[...]

(quoted diff) +1 -1
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)

This pattern is repeated several times. Perhaps there needs to be a CONFIG_X86_CET to make this more readable? Really just a style question.

arch/x86/Kconfig +7 -0
index b68eb75887b8..6cb52616e0cf 100644
@@ -1836,6 +1836,11 @@ config CC_HAS_IBT
 		  (CC_IS_CLANG && CLANG_VERSION >= 140000)) && \
 		  $(as-instr,endbr64)
 
+config X86_CET
+	def_bool n
+	help
+	  CET features are enabled (IBT and/or Shadow Stack)
+
 config X86_KERNEL_IBT
 	prompt "Indirect Branch Tracking"
 	bool
@@ -1843,6 +1848,7 @@ config X86_KERNEL_IBT
 	# https://github.com/llvm/llvm-project/commit/9d7001eba9c4cb311e03cd8cdc231f9e579f2d0f
 	depends on !LD_IS_LLD || LLD_VERSION >= 140000
 	select OBJTOOL
+	select X86_CET
 	help
 	  Build the kernel with support for Indirect Branch Tracking, a
 	  hardware support course-grain forward-edge Control Flow Integrity
@@ -1945,6 +1951,7 @@ config X86_SHADOW_STACK
 	def_bool n
 	depends on ARCH_HAS_SHADOW_STACK
 	select ARCH_USES_HIGH_VMA_FLAGS
+	select X86_CET
 	help
 	  Shadow Stack protection is a hardware feature that detects function
 	  return address corruption. Today the kernel's support is limited to

Review

Quoted 9 lines

[...]

(quoted diff) +8 -0
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}

I second Kirill's question here. This seems an entirely survivable (but highly unexpected) state. I think this whole "if" could just be replaced with:

	WARN_ON_ONCE(!cpu_feature_enabled(X86_FEATURE_IBT) &&
		     !cpu_feature_enabled(X86_FEATURE_SHSTK),
		     "Unexpected #CP\n");

Otherwise this looks good to me.

Signature · 1 line
Kees Cook
#60 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

Quoted 9 lines · Kees Cook
Quoted 4 lines · Rick Edgecombe

[...]

(quoted diff) +1 -1
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) ||

defined(CONFIG_X86_SHADOW_STACK)

This pattern is repeated several times. Perhaps there needs to be a CONFIG_X86_CET to make this more readable? Really just a style question.

Hmm, good idea. Thanks.

maintainer
#61 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

In reply to Rick Edgecombe
Quoted 5 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

arch/x86/kernel/traps.c +2 -0
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)

arch/x86/kernel/traps.c enum cp_error_code { · line 248

Quoted 10 lines
arch/x86/kernel/traps.c +6 -3
  {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();
+	struct task_struct *tsk;
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp); > +
+	cond_local_irq_enable(regs);

I feel like I'm missing something. Either PL3_SSL is context switched correctly and reading it with IRQs off is useless, or it's not context switched, and I'm very confused.

Please either improve the comment or move it after the cond_local_irq_enable().

--Andy

Quoted 121 lines
arch/x86/kernel/traps.c +63 -3
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+
+	tsk = current;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_nr = X86_TRAP_CP;
+
+	/* Ratelimit to prevent log spamming. */
+	if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+	    __ratelimit(&cpf_rate)) {
+		unsigned int cpec;
+
+		cpec = error_code & CP_EC;
+		if (cpec >= ARRAY_SIZE(control_protection_err))
+			cpec = 0;
+
+		pr_emerg("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s",
+			 tsk->comm, task_pid_nr(tsk),
+			 regs->ip, regs->sp, ssp, error_code,
+			 control_protection_err[cpec],
+			 error_code & CP_ENCL ? " in enclave" : "");
+		print_vma_addr(KERN_CONT " in ", regs->ip);
+		pr_cont("\n");
  	}
 
-	if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR))
-		return;
+	force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0);
+	cond_local_irq_disable(regs);
+}
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
+{
+	WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+}
+#endif
+
+#ifdef CONFIG_X86_KERNEL_IBT
+
+static __ro_after_init bool ibt_fatal = true;
+
+extern void ibt_selftest_ip(void); /* code label defined in asm below */
 
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
  	if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) {
  		regs->ax = 0;
  		return;
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
  }
 
  __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT disabled\n");
+}
  #endif /* CONFIG_X86_KERNEL_IBT */
 
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}
+
+	if (user_mode(regs))
+		do_user_control_protection_fault(regs, error_code);
+	else
+		do_kernel_control_protection_fault(regs);
+}
+#endif /* defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK) */
+
  #ifdef CONFIG_X86_F00F_BUG
  void handle_invalid_op(struct pt_regs *regs)
  #else
arch/x86/xen/enlighten_pv.c +1 -1
index 0ed2e487a693..57faa287163f 100644
@@ -628,7 +628,7 @@ static struct trap_array_entry trap_array[] = {
  	TRAP_ENTRY(exc_coprocessor_error,		false ),
  	TRAP_ENTRY(exc_alignment_check,			false ),
  	TRAP_ENTRY(exc_simd_coprocessor_error,		false ),
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
  	TRAP_ENTRY(exc_control_protection,		false ),
  #endif
  };
arch/x86/xen/xen-asm.S +1 -1
index 6b4fdf6b9542..e45ff6300c7d 100644
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault
  xen_pv_trap asm_exc_spurious_interrupt_bug
  xen_pv_trap asm_exc_coprocessor_error
  xen_pv_trap asm_exc_alignment_check
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
  xen_pv_trap asm_exc_control_protection
  #endif
  #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +2 -1
index ffbe4cec9f32..0f52d0ac47c5 100644
@@ -242,7 +242,8 @@ typedef struct siginfo {
  #define SEGV_ADIPERR	7	/* Precise MCD exception */
  #define SEGV_MTEAERR	8	/* Asynchronous ARM MTE error */
  #define SEGV_MTESERR	9	/* Synchronous ARM MTE exception */
-#define NSIGSEGV	9
+#define SEGV_CPERR	10	/* Control protection fault */
+#define NSIGSEGV	10
 
  /*
   * SIGBUS si_codes
#62 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

Quoted 131 lines · Andy Lutomirski
Quoted 5 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

arch/x86/kernel/traps.c +2 -0
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)

arch/x86/kernel/traps.c enum cp_error_code { · line 248

Quoted 10 lines
arch/x86/kernel/traps.c +6 -3
  {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();
+	struct task_struct *tsk;
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp); > +
+	cond_local_irq_enable(regs);

I feel like I'm missing something. Either PL3_SSL is context switched correctly and reading it with IRQs off is useless, or it's not context switched, and I'm very confused.

Please either improve the comment or move it after the cond_local_irq_enable().

--Andy

Quoted 108 lines
arch/x86/kernel/traps.c +61 -2
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+
+	tsk = current;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_nr = X86_TRAP_CP;
+
+	/* Ratelimit to prevent log spamming. */
+	if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+	    __ratelimit(&cpf_rate)) {
+		unsigned int cpec;
+
+		cpec = error_code & CP_EC;
+		if (cpec >= ARRAY_SIZE(control_protection_err))
+			cpec = 0;
+
+		pr_emerg("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s",
+			 tsk->comm, task_pid_nr(tsk),
+			 regs->ip, regs->sp, ssp, error_code,
+			 control_protection_err[cpec],
+			 error_code & CP_ENCL ? " in enclave" : "");
+		print_vma_addr(KERN_CONT " in ", regs->ip);
+		pr_cont("\n");
  	}
  -	if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR))
-		return;
+	force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0);
+	cond_local_irq_disable(regs);
+}
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
+{
+	WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+}
+#endif
+
+#ifdef CONFIG_X86_KERNEL_IBT
+
+static __ro_after_init bool ibt_fatal = true;
+
+extern void ibt_selftest_ip(void); /* code label defined in asm below */
  +static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
  	if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) {
  		regs->ax = 0;
  		return;
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
  }
    __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT disabled\n");
+}
  #endif /* CONFIG_X86_KERNEL_IBT */
  +#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}
+
+	if (user_mode(regs))
+		do_user_control_protection_fault(regs, error_code);
+	else
+		do_kernel_control_protection_fault(regs);
+}
+#endif /* defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK) */
+
  #ifdef CONFIG_X86_F00F_BUG
  void handle_invalid_op(struct pt_regs *regs)
  #else
arch/x86/xen/enlighten_pv.c +1 -1
index 0ed2e487a693..57faa287163f 100644
@@ -628,7 +628,7 @@ static struct trap_array_entry trap_array[] = {
  	TRAP_ENTRY(exc_coprocessor_error,		false ),
  	TRAP_ENTRY(exc_alignment_check,			false ),
  	TRAP_ENTRY(exc_simd_coprocessor_error,		false ),
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
  	TRAP_ENTRY(exc_control_protection,		false ),
  #endif
  };
arch/x86/xen/xen-asm.S +1 -1
index 6b4fdf6b9542..e45ff6300c7d 100644
@@ -148,7 +148,7 @@ xen_pv_trap asm_exc_page_fault
  xen_pv_trap asm_exc_spurious_interrupt_bug
  xen_pv_trap asm_exc_coprocessor_error
  xen_pv_trap asm_exc_alignment_check
-#ifdef CONFIG_X86_KERNEL_IBT
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
  xen_pv_trap asm_exc_control_protection
  #endif
  #ifdef CONFIG_X86_MCE
include/uapi/asm-generic/siginfo.h +0 -0
index ffbe4cec9f32..0f52d0ac47c5 100644
@@ -242,7 +242,8 @@ typedef struct siginfo {
Quoted 9 lines

include/uapi/asm-generic/siginfo.h typedef struct siginfo { · line 249

Quoted 8 lines
include/uapi/asm-generic/siginfo.h +2 -1
  #define SEGV_ADIPERR	7	/* Precise MCD exception */
  #define SEGV_MTEAERR	8	/* Asynchronous ARM MTE error */
  #define SEGV_MTESERR	9	/* Synchronous ARM MTE exception */
-#define NSIGSEGV	9
+#define SEGV_CPERR	10	/* Control protection fault */
+#define NSIGSEGV	10
    /*
   * SIGBUS si_codes

Could something change the value under a switched-out thread, though?

#63 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

In reply to Andy Lutomirski
Quoted 23 lines · Andy Lutomirski
Quoted 6 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

(quoted diff) +2 -0
+static void do_user_control_protection_fault(struct pt_regs *regs,
+                                          unsigned long

error_code)
{

arch/x86/kernel/traps.c enum cp_error_code { · line 248

Quoted 9 lines
arch/x86/kernel/traps.c +6 -3
-     if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-             pr_err("Unexpected #CP\n");
-             BUG();
+     struct task_struct *tsk;
+     unsigned long ssp;
+
+     /* Read SSP before enabling interrupts. */
+     rdmsrl(MSR_IA32_PL3_SSP, ssp); > +
+     cond_local_irq_enable(regs);

I feel like I'm missing something. Either PL3_SSL is context switched
correctly and reading it with IRQs off is useless, or it's not context
switched, and I'm very confused.

Quoted 2 lines

Please either improve the comment or move it after the cond_local_irq_enable().

The thinking was, we were just in userspace and we took a #CP. Since we were in userspace, we had a live SSP. After we re-enable interrupts we could get scheduled and it would be in the xsave buffer. So we can grab it for free now, otherwise we would have to force restore it and read it after we re-enable interrupts.

I can clarify the comments, unless there is something wrong with that reasoning.

#64 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

In reply to Rick Edgecombe
Quoted 8 lines · Rick Edgecombe
arch/x86/kernel/traps.c +0 -0
index d62b2cb85cea..b7dde8730236 100644
@@ -229,16 +223,74 @@ enum cp_error_code {
 	CP_ENCL	     = 1 << 15,
 };
 

arch/x86/kernel/traps.c enum cp_error_code { · line 234

Quoted 10 lines
arch/x86/kernel/traps.c +9 -1
-DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+#ifdef CONFIG_X86_SHADOW_STACK
+static const char * const control_protection_err[] = {
+	"unknown",
+	"near-ret",
+	"far-ret/iret",
+	"endbranch",
+	"rstorssp",
+	"setssbsy",
+};

These are a mix of SHSTK and IBT errors.  They should be inside CONFIG_X86_CET using Kees' suggestion.

Also, if you express this as

static const char errors[][10] = {
    [0] = "unknown",
    [1] = "near ret",
    [2] = "far/iret",
    [3] = "endbranch",
    [4] = "rstorssp",
    [5] = "setssbsy",
};

then you can encode all the strings in roughly the space it takes to lay out the pointers above.

Quoted 10 lines
arch/x86/kernel/traps.c +6 -3
+
+static DEFINE_RATELIMIT_STATE(cpf_rate, DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();

arch/x86/kernel/traps.c enum cp_error_code { · line 251

Quoted 10 lines
arch/x86/kernel/traps.c +10 -0
+	struct task_struct *tsk;
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	cond_local_irq_enable(regs);
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");

So it's ok to get an unexpected #CP on CET-capable hardware, but not on CET-incapable hardware?

The conditions for this WARN() (and others) probably want adjusting to what the kernel has enabled, not what hardware is capable of.

Quoted 8 lines
arch/x86/kernel/traps.c +3 -1
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
 }
 
 __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{

arch/x86/kernel/traps.c static int __init ibt_setup(char *str) · line 350

Quoted 10 lines
arch/x86/kernel/traps.c +8 -0
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT disabled\n");
+}
 #endif /* CONFIG_X86_KERNEL_IBT */
 
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");

Do some future poor sole a favour and render the numeric error code too.  Without it, the error is ambiguous between SHSTK and IBT when %rip points at a call/ret instruction.

~Andrew

#65 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

Quoted 34 lines · Andrew Cooper
Quoted 8 lines · Rick Edgecombe
arch/x86/kernel/traps.c +0 -0
index d62b2cb85cea..b7dde8730236 100644
@@ -229,16 +223,74 @@ enum cp_error_code {
 	CP_ENCL	     = 1 << 15,
 };
 

arch/x86/kernel/traps.c enum cp_error_code { · line 234

Quoted 10 lines
arch/x86/kernel/traps.c +9 -1
-DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+#ifdef CONFIG_X86_SHADOW_STACK
+static const char * const control_protection_err[] = {
+	"unknown",
+	"near-ret",
+	"far-ret/iret",
+	"endbranch",
+	"rstorssp",
+	"setssbsy",
+};

These are a mix of SHSTK and IBT errors. They should be inside CONFIG_X86_CET using Kees' suggestion.

Also, if you express this as

static const char errors[][10] = {
    [0] = "unknown",
    [1] = "near ret",
    [2] = "far/iret",
    [3] = "endbranch",
    [4] = "rstorssp",
    [5] = "setssbsy",
};
Quoted 3 lines

then you can encode all the strings in roughly the space it takes to lay
out the pointers above.

It is only used in the user shadow stack side of the handler. I guess the kernel IBT side of the handler could print these out too.

Can you explain more about why this array is better than the other one?

Quoted 28 lines
Quoted 13 lines
arch/x86/kernel/traps.c +7 -3
+
+static DEFINE_RATELIMIT_STATE(cpf_rate,
 DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();
+	struct task_struct *tsk;

arch/x86/kernel/traps.c enum cp_error_code {

Quoted 10 lines
arch/x86/kernel/traps.c +9 -0
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	cond_local_irq_enable(regs);
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with
 shadow support disabled\n");

So it's ok to get an unexpected #CP on CET-capable hardware, but not on
CET-incapable hardware?

Quoted 3 lines

The conditions for this WARN() (and others) probably want adjusting to
what the kernel has enabled, not what hardware is capable of.

Sorry, I don't follow. This code is only compiled in if the kernel has been compiled for userspace shadow stacks. If the HW supports it and the kernel is configured for it, it should be enabled. If you clear it with the clearcpuid command line it should be as if the HW doesn't support it. So I think it should not be too unexpected, in situations where it gets passed this check.

Quoted 15 lines
Quoted 15 lines
arch/x86/kernel/traps.c +5 -1
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
 }
 
 __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs
 *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT
 disabled\n");
+}
 #endif /* CONFIG_X86_KERNEL_IBT */
 
Quoted 13 lines

arch/x86/kernel/traps.c static int __init ibt_setup(char *str) · line 350

Quoted 7 lines
arch/x86/kernel/traps.c +6 -0
+#if defined(CONFIG_X86_KERNEL_IBT) ||
 defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");

Do some future poor sole a favour and render the numeric error code too. Without it, the error is ambiguous between SHSTK and IBT when %rip
points at a call/ret instruction.

This was from the original kernel IBT handler. Yes, all these messages should probably be unified too. Thanks.

maintainer
#66 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

In reply to Rick Edgecombe
Quoted 54 lines · Rick Edgecombe
arch/x86/kernel/traps.c +40 -4
index d62b2cb85cea..b7dde8730236 100644
@@ -229,16 +223,74 @@ enum cp_error_code {
 	CP_ENCL	     = 1 << 15,
 };
 
-DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+#ifdef CONFIG_X86_SHADOW_STACK
+static const char * const control_protection_err[] = {
+	"unknown",
+	"near-ret",
+	"far-ret/iret",
+	"endbranch",
+	"rstorssp",
+	"setssbsy",
+};
+
+static DEFINE_RATELIMIT_STATE(cpf_rate, DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();
+	struct task_struct *tsk;
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	cond_local_irq_enable(regs);
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+
+	tsk = current;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_nr = X86_TRAP_CP;
+
+	/* Ratelimit to prevent log spamming. */
+	if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+	    __ratelimit(&cpf_rate)) {
+		unsigned int cpec;
+
+		cpec = error_code & CP_EC;
+		if (cpec >= ARRAY_SIZE(control_protection_err))
+			cpec = 0;
+
+		pr_emerg("%s[%d] control protection ip:%lx sp:%lx ssp:%lx error:%lx(%s)%s",

arch/x86/kernel/traps.c enum cp_error_code { · line 240

Quoted 10 lines
arch/x86/kernel/traps.c +6 -2
+			 tsk->comm, task_pid_nr(tsk),
+			 regs->ip, regs->sp, ssp, error_code,
+			 control_protection_err[cpec],
+			 error_code & CP_ENCL ? " in enclave" : "");
+		print_vma_addr(KERN_CONT " in ", regs->ip);
+		pr_cont("\n");
 	}
 
-	if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) != CP_ENDBR))
-		return;

Why are you removing the (error_code & CP_EC) != CP_ENDBR check from the kernel handler?

Quoted 37 lines
arch/x86/kernel/traps.c +26 -1
+	force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0);
+	cond_local_irq_disable(regs);
+}
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
+{
+	WARN_ONCE(1, "User-mode control protection fault with shadow support disabled\n");
+}
+#endif
+
+#ifdef CONFIG_X86_KERNEL_IBT
+
+static __ro_after_init bool ibt_fatal = true;
+
+extern void ibt_selftest_ip(void); /* code label defined in asm below */
 
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
 	if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) {
 		regs->ax = 0;
 		return;
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
 }
 
 __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT disabled\n");
+}
 #endif /* CONFIG_X86_KERNEL_IBT */
 
+#if defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{

arch/x86/kernel/traps.c static int __init ibt_setup(char *str) · line 357

Quoted 10 lines
arch/x86/kernel/traps.c +10 -0
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}
+
+	if (user_mode(regs))
+		do_user_control_protection_fault(regs, error_code);
+	else
+		do_kernel_control_protection_fault(regs);

These function names are weirdly long, surely they can do without the _fault part at the very least. And as stated above, I would really like the kernel thing to retain the error_code argument.

Quoted 2 lines
arch/x86/kernel/traps.c +2 -0
+}
+#endif /* defined(CONFIG_X86_KERNEL_IBT) || defined(CONFIG_X86_SHADOW_STACK) */
#67 lore raw

Re: [PATCH v2 07/39] x86/cet: Add user control-protection fault handler

Quoted 66 lines · Peter Zijlstra
Quoted 64 lines · Rick Edgecombe
arch/x86/kernel/traps.c +46 -4
index d62b2cb85cea..b7dde8730236 100644
@@ -229,16 +223,74 @@ enum cp_error_code {
 	CP_ENCL	     = 1 << 15,
 };
 
-DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+#ifdef CONFIG_X86_SHADOW_STACK
+static const char * const control_protection_err[] = {
+	"unknown",
+	"near-ret",
+	"far-ret/iret",
+	"endbranch",
+	"rstorssp",
+	"setssbsy",
+};
+
+static DEFINE_RATELIMIT_STATE(cpf_rate,
 DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
 {
-	if (!cpu_feature_enabled(X86_FEATURE_IBT)) {
-		pr_err("Unexpected #CP\n");
-		BUG();
+	struct task_struct *tsk;
+	unsigned long ssp;
+
+	/* Read SSP before enabling interrupts. */
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	cond_local_irq_enable(regs);
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		WARN_ONCE(1, "User-mode control protection fault with
 shadow support disabled\n");
+
+	tsk = current;
+	tsk->thread.error_code = error_code;
+	tsk->thread.trap_nr = X86_TRAP_CP;
+
+	/* Ratelimit to prevent log spamming. */
+	if (show_unhandled_signals && unhandled_signal(tsk, SIGSEGV) &&
+	    __ratelimit(&cpf_rate)) {
+		unsigned int cpec;
+
+		cpec = error_code & CP_EC;
+		if (cpec >= ARRAY_SIZE(control_protection_err))
+			cpec = 0;
+
+		pr_emerg("%s[%d] control protection ip:%lx sp:%lx
 ssp:%lx error:%lx(%s)%s",
+			 tsk->comm, task_pid_nr(tsk),
+			 regs->ip, regs->sp, ssp, error_code,
+			 control_protection_err[cpec],
+			 error_code & CP_ENCL ? " in enclave" : "");
+		print_vma_addr(KERN_CONT " in ", regs->ip);
+		pr_cont("\n");
 	}
 
Quoted 7 lines

arch/x86/kernel/traps.c enum cp_error_code { · line 244

Quoted 3 lines
arch/x86/kernel/traps.c +0 -2
-	if (WARN_ON_ONCE(user_mode(regs) || (error_code & CP_EC) !=
 CP_ENDBR))
-		return;

Why are you removing the (error_code & CP_EC) != CP_ENDBR check from the
kernel handler?

Argh. It was accidentally removed with the user_mode() check. I'll fix it.

Quoted 51 lines
Quoted 51 lines
arch/x86/kernel/traps.c +8 -0
+	force_sig_fault(SIGSEGV, SEGV_CPERR, (void __user *)0);
+	cond_local_irq_disable(regs);
+}
+#else
+static void do_user_control_protection_fault(struct pt_regs *regs,
+					     unsigned long error_code)
+{
+	WARN_ONCE(1, "User-mode control protection fault with shadow

support disabled\n");

arch/x86/kernel/traps.c +8 -0
+}
+#endif
+
+#ifdef CONFIG_X86_KERNEL_IBT
+
+static __ro_after_init bool ibt_fatal = true;
+
+extern void ibt_selftest_ip(void); /* code label defined in asm

below */

+static void do_kernel_control_protection_fault(struct pt_regs *regs)

arch/x86/kernel/traps.c +16 -1
+{
 	if (unlikely(regs->ip == (unsigned long)&ibt_selftest_ip)) {
 		regs->ax = 0;
 		return;
@@ -283,9 +335,29 @@ static int __init ibt_setup(char *str)
 }
 
 __setup("ibt=", ibt_setup);
-
+#else
+static void do_kernel_control_protection_fault(struct pt_regs
 *regs)
+{
+	WARN_ONCE(1, "Kernel-mode control protection fault with IBT
 disabled\n");
+}
 #endif /* CONFIG_X86_KERNEL_IBT */
 
+#if defined(CONFIG_X86_KERNEL_IBT) ||
 defined(CONFIG_X86_SHADOW_STACK)
+DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_IBT) &&
+	    !cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pr_err("Unexpected #CP\n");
+		BUG();
+	}
+
+	if (user_mode(regs))
Quoted 9 lines

arch/x86/kernel/traps.c static int __init ibt_setup(char *str) · line 357

Quoted 3 lines
arch/x86/kernel/traps.c +3 -0
+		do_user_control_protection_fault(regs, error_code);
+	else
+		do_kernel_control_protection_fault(regs);

These function names are weirdly long, surely they can do without the _fault part at the very least. And as stated above, I would really like
the kernel thing to retain the error_code argument.

I can shorten them. Thanks.

Quoted 5 lines
Quoted 3 lines
arch/x86/kernel/traps.c +2 -0
+}
+#endif /* defined(CONFIG_X86_KERNEL_IBT) ||

defined(CONFIG_X86_SHADOW_STACK) */

#68 lore raw

PATCH v2 8/39 x86/mm: Remove _PAGE_DIRTY from kernel RO pages

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Processors sometimes directly create Write=0,Dirty=1 PTEs. These PTEs are created by software. One such case is that kernel read-only pages are historically set up as Dirty.

New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages. When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, but to reduce ambiguity between shadow stack and regular Write=0 pages, removed Dirty=1 from any kernel Write=0 PTEs.

Cc "H. Peter Anvin" <[email protected]>
Cc Kees Cook <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Peter Zijlstra <[email protected]>

---

v2:
- Normalize PTE bit descriptions between patches

 arch/x86/include/asm/pgtable_types.h | 6 +++---
 arch/x86/mm/pat/set_memory.c         | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
index aa174fed3a71..ff82237e7b6b 100644
@@ -192,10 +192,10 @@ enum page_cache_mode {
 #define _KERNPG_TABLE		 (__PP|__RW|   0|___A|   0|___D|   0|   0| _ENC)
 #define _PAGE_TABLE_NOENC	 (__PP|__RW|_USR|___A|   0|___D|   0|   0)
 #define _PAGE_TABLE		 (__PP|__RW|_USR|___A|   0|___D|   0|   0| _ENC)
-#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|___D|   0|___G)
-#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|___D|   0|___G)
+#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|   0|   0|___G)
+#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|   0|   0|___G)
 #define __PAGE_KERNEL_NOCACHE	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __NC)
-#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|___D|   0|___G)
+#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|   0|   0|___G)
 #define __PAGE_KERNEL_LARGE	 (__PP|__RW|   0|___A|__NX|___D|_PSE|___G)
 #define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW|   0|___A|   0|___D|_PSE|___G)
 #define __PAGE_KERNEL_WP	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __WP)
arch/x86/mm/pat/set_memory.c +1 -1
index 1abd5438f126..ed9193b469ba 100644
@@ -1977,7 +1977,7 @@ int set_memory_nx(unsigned long addr, int numpages)
 
 int set_memory_ro(unsigned long addr, int numpages)
 {
-	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0);
+	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW | _PAGE_DIRTY), 0);
 }
 
 int set_memory_rw(unsigned long addr, int numpages)
Signature · 1 line
2.17.1
#69 lore raw

Re: [PATCH v2 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages

Quoted 57 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Processors sometimes directly create Write=0,Dirty=1 PTEs. These PTEs are created by software. One such case is that kernel read-only pages are historically set up as Dirty.

New processors that support Shadow Stack regard Write=0,Dirty=1 PTEs as shadow stack pages. When CR4.CET=1 and IA32_S_CET.SH_STK_EN=1, some instructions can write to such supervisor memory. The kernel does not set IA32_S_CET.SH_STK_EN, but to reduce ambiguity between shadow stack and regular Write=0 pages, removed Dirty=1 from any kernel Write=0 PTEs.

Cc "H. Peter Anvin" <[email protected]>
Cc Kees Cook <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Peter Zijlstra <[email protected]>

---

v2:
- Normalize PTE bit descriptions between patches

 arch/x86/include/asm/pgtable_types.h | 6 +++---
 arch/x86/mm/pat/set_memory.c         | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable_types.h +3 -3
index aa174fed3a71..ff82237e7b6b 100644
@@ -192,10 +192,10 @@ enum page_cache_mode {
 #define _KERNPG_TABLE		 (__PP|__RW|   0|___A|   0|___D|   0|   0| _ENC)
 #define _PAGE_TABLE_NOENC	 (__PP|__RW|_USR|___A|   0|___D|   0|   0)
 #define _PAGE_TABLE		 (__PP|__RW|_USR|___A|   0|___D|   0|   0| _ENC)
-#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|___D|   0|___G)
-#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|___D|   0|___G)
+#define __PAGE_KERNEL_RO	 (__PP|   0|   0|___A|__NX|   0|   0|___G)
+#define __PAGE_KERNEL_ROX	 (__PP|   0|   0|___A|   0|   0|   0|___G)
 #define __PAGE_KERNEL_NOCACHE	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __NC)
-#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|___D|   0|___G)
+#define __PAGE_KERNEL_VVAR	 (__PP|   0|_USR|___A|__NX|   0|   0|___G)
 #define __PAGE_KERNEL_LARGE	 (__PP|__RW|   0|___A|__NX|___D|_PSE|___G)
 #define __PAGE_KERNEL_LARGE_EXEC (__PP|__RW|   0|___A|   0|___D|_PSE|___G)
 #define __PAGE_KERNEL_WP	 (__PP|__RW|   0|___A|__NX|___D|   0|___G| __WP)
arch/x86/mm/pat/set_memory.c +0 -0
index 1abd5438f126..ed9193b469ba 100644

arch/x86/mm/pat/set_memory.c int set_memory_nx(unsigned long addr, int numpages) · line 1981

Quoted 6 lines
arch/x86/mm/pat/set_memory.c +1 -1
@@ -1977,7 +1977,7 @@ int set_memory_nx(unsigned long addr, int numpages)
 
 int set_memory_ro(unsigned long addr, int numpages)
 {
-	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0);
+	return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW | _PAGE_DIRTY), 0);
 }

Hm. Do we need to modify also *_wrprotect() helpers to clear dirty bit?

I guess not (at least without a lot of audit), as we risk loosing dirty bit on page cache pages. But why is it safe? Do we only care about about kernel PTEs here? Userspace Write=0,Dirty=1 PTEs handled as before?

Quoted 4 lines
arch/x86/mm/pat/set_memory.c +0 -0
 int set_memory_rw(unsigned long addr, int numpages)
Signature · 1 line
2.17.1
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#70 lore raw

Re: [PATCH v2 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages

In reply to Rick Edgecombe
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Processors sometimes directly create Write=0,Dirty=1 PTEs.

Do they? (Rhetorical)

Yes, this is a relevant anecdote for why CET isn't available on pre-TGL parts, but it one of the more wrong things to have as the first sentence of this commit message.

The point you want to express is that under the CET-SS spec, R/O+Dirty has a new meaning as type=shstk, so stop using this bit combination for existing mappings.

I'm not even sure it's relevant to note that CET capable processors can set D on a R/O mapping, because that depends on !CR0.WP which in turn prohibits CR4.CET being enabled.

~Andrew

maintainer
#71 lore raw

Re: [PATCH v2 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages

Quoted 11 lines · Andrew Cooper
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Processors sometimes directly create Write=0,Dirty=1 PTEs.

Do they? (Rhetorical)

Yes, this is a relevant anecdote for why CET isn't available on pre-TGL parts, but it one of the more wrong things to have as the first sentence of this commit message.

Quoted 7 lines

The point you want to express is that under the CET-SS spec, R/O+Dirty has a new meaning as type=shstk, so stop using this bit combination for existing mappings.

I'm not even sure it's relevant to note that CET capable processors can set D on a R/O mapping, because that depends on !CR0.WP which in turn prohibits CR4.CET being enabled.

Whilst I agree that the Changelog is 'suboptimal' -- I do think it might be good to mention how we ended up at the current state where we explicitly set this non-sensical W=0,D=1 state.

Looking at the git history this seems to be a bit of a hysterical accident, not something done on purpose to 'optimize' for these weird CPUs.

#72 lore raw

Re: [PATCH v2 08/39] x86/mm: Remove _PAGE_DIRTY from kernel RO pages

Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Andrew Cooper
Quoted 3 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Processors sometimes directly create Write=0,Dirty=1 PTEs.

Do they? (Rhetorical)

Yes, this is a relevant anecdote for why CET isn't available on pre-TGL parts, but it one of the more wrong things to have as the first sentence of this commit message.

The point you want to express is that under the CET-SS spec, R/O+Dirty has a new meaning as type=shstk, so stop using this bit combination for existing mappings.

Quoted 6 lines
Quoted 3 lines

I'm not even sure it's relevant to note that CET capable processors can set D on a R/O mapping, because that depends on !CR0.WP which in turn prohibits CR4.CET being enabled.

Whilst I agree that the Changelog is 'suboptimal' -- I do think it might be good to mention how we ended up at the current state where we explicitly set this non-sensical W=0,D=1 state.

Sure, but that's got nothing to do with hardware errata.

Having hardware set A/D bits is expensive.  Being a locked operation, it's roughly a smp_mb() behind the scenes.

Therefore, when A/D tracking doesn't matter, traditional wisdom says set both of them when creating the PTE.

It's only now that R/O+Dirty has a meaning (other than being a slightly weird but safe bit combination), and we've got to be more careful about using it.

~Andrew

#73 lore raw

PATCH v2 9/39 x86/mm: Move pmd_write(), pud_write() up in the file

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

To prepare the introduction of _PAGE_COW, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes.

---
 arch/x86/include/asm/pgtable.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)
arch/x86/include/asm/pgtable.h +12 -12
index 44e2d6f1dbaa..6496ec84b953 100644
@@ -159,6 +159,18 @@ static inline int pte_write(pte_t pte)
 	return pte_flags(pte) & _PAGE_RW;
 }
 
+#define pmd_write pmd_write
+static inline int pmd_write(pmd_t pmd)
+{
+	return pmd_flags(pmd) & _PAGE_RW;
+}
+
+#define pud_write pud_write
+static inline int pud_write(pud_t pud)
+{
+	return pud_flags(pud) & _PAGE_RW;
+}
+
 static inline int pte_huge(pte_t pte)
 {
 	return pte_flags(pte) & _PAGE_PSE;
@@ -1102,12 +1114,6 @@ extern int pmdp_clear_flush_young(struct vm_area_struct *vma,
 				  unsigned long address, pmd_t *pmdp);
 
 
-#define pmd_write pmd_write
-static inline int pmd_write(pmd_t pmd)
-{
-	return pmd_flags(pmd) & _PAGE_RW;
-}
-
 #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR
 static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr,
 				       pmd_t *pmdp)
@@ -1137,12 +1143,6 @@ static inline void pmdp_set_wrprotect(struct mm_struct *mm,
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp);
 }
 
-#define pud_write pud_write
-static inline int pud_write(pud_t pud)
-{
-	return pud_flags(pud) & _PAGE_RW;
-}
-
 #ifndef pmdp_establish
 #define pmdp_establish pmdp_establish
 static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
Signature · 1 line
2.17.1
Reviewed-by on v2 9/39
#74 lore raw

Re: [PATCH v2 09/39] x86/mm: Move pmd_write(), pud_write() up in the file

Quoted 7 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

To prepare the introduction of _PAGE_COW, move pmd_write() and pud_write() up in the file, so that they can be used by other helpers below. No functional changes.

Hey, a PTE patch I'm able to review! ;)

Signature · 1 line
Kees Cook
#75 lore raw

PATCH v2 10/39 x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes (not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1.

The reason it's lightly used is that Dirty=1 is normally set _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.

The kernel should avoid inadvertently creating shadow stack memory because it is security sensitive. So given the above, all it needs to do is avoid manually crating Write=0,Dirty=1 PTEs in software.

In places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. This clearly separates shadow stack from other data, and results in the following:

(a) (Write=0,Cow=1,Dirty=0) A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW via fork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit.
(b) (Write=0,Cow=1,Dirty=0) A R/O page that has been COW'ed. The user page is in a R/O VMA, and get_user_pages() needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
(c) (Write=0,Cow=0,Dirty=1) A shadow stack PTE.
(d) (Write=0,Cow=1,Dirty=0) A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.
(e) (Write=0,Cow=0,Dirty=1) A Cow PTE created when a processor without shadow stack support set Dirty=1.

Define _PAGE_COW and update pte_*() helpers and apply the same changes to pmd and pud.

There are six bits left available to software in the 64-bit PTE after consuming a bit for _PAGE_COW. No space is consumed in 32-bit kernels because shadow stacks are not enabled there.

---

v2:
- Update commit log with comments (Dave Hansen)
- Add comments in code to explain pte modification code better (Dave)
- Clarify info on the meaning of various Write,Cow,Dirty combinations

 arch/x86/include/asm/pgtable.h       | 210 ++++++++++++++++++++++++---
 arch/x86/include/asm/pgtable_types.h |  42 +++++-
 2 files changed, 231 insertions(+), 21 deletions(-)
arch/x86/include/asm/pgtable.h +190 -20
index 6496ec84b953..ad201dae7316 100644
@@ -124,9 +124,17 @@ extern pmdval_t early_pmd_flags;
  * The following only work if pte_present() is true.
  * Undefined behaviour if not..
  */
-static inline int pte_dirty(pte_t pte)
+static inline bool pte_dirty(pte_t pte)
 {
-	return pte_flags(pte) & _PAGE_DIRTY;
+	return pte_flags(pte) & _PAGE_DIRTY_BITS;
+}
+
+static inline bool pte_shstk(pte_t pte)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return false;
+
+	return (pte_flags(pte) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY;
 }
 
 static inline int pte_young(pte_t pte)
@@ -134,9 +142,17 @@ static inline int pte_young(pte_t pte)
 	return pte_flags(pte) & _PAGE_ACCESSED;
 }
 
-static inline int pmd_dirty(pmd_t pmd)
+static inline bool pmd_dirty(pmd_t pmd)
 {
-	return pmd_flags(pmd) & _PAGE_DIRTY;
+	return pmd_flags(pmd) & _PAGE_DIRTY_BITS;
+}
+
+static inline bool pmd_shstk(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return false;
+
+	return (pmd_flags(pmd) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY;
 }
 
 static inline int pmd_young(pmd_t pmd)
@@ -144,9 +160,9 @@ static inline int pmd_young(pmd_t pmd)
 	return pmd_flags(pmd) & _PAGE_ACCESSED;
 }
 
-static inline int pud_dirty(pud_t pud)
+static inline bool pud_dirty(pud_t pud)
 {
-	return pud_flags(pud) & _PAGE_DIRTY;
+	return pud_flags(pud) & _PAGE_DIRTY_BITS;
 }
 
 static inline int pud_young(pud_t pud)
@@ -156,13 +172,21 @@ static inline int pud_young(pud_t pud)
 
 static inline int pte_write(pte_t pte)
 {
-	return pte_flags(pte) & _PAGE_RW;
+	/*
+	 * Shadow stack pages are logically writable, but do not have
+	 * _PAGE_RW.  Check for them separately from _PAGE_RW itself.
+	 */
+	return (pte_flags(pte) & _PAGE_RW) || pte_shstk(pte);
 }
 
 #define pmd_write pmd_write
 static inline int pmd_write(pmd_t pmd)
 {
-	return pmd_flags(pmd) & _PAGE_RW;
+	/*
+	 * Shadow stack pages are logically writable, but do not have
+	 * _PAGE_RW.  Check for them separately from _PAGE_RW itself.
+	 */
+	return (pmd_flags(pmd) & _PAGE_RW) || pmd_shstk(pmd);
 }
 
 #define pud_write pud_write
@@ -300,6 +324,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)
 	return native_make_pte(v & ~clear);
 }
 
+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But
+ * in the case of X86_FEATURE_SHSTK, the software COW bit is used,
+ * since the Dirty=1,Write=0 will result in the memory being treated
+ * as shaodw stack by the HW. So when creating COW memory, a software
+ * bit is used _PAGE_BIT_COW. The following functions pte_mkcow() and
+ * pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
+ * transition it to the shadow stack compatible version of COW (Cow=1).
+ */
+
+static inline pte_t pte_mkcow(pte_t pte)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pte;
+
+	pte = pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_set_flags(pte, _PAGE_COW);
+}
+
+static inline pte_t pte_clear_cow(pte_t pte)
+{
+	/*
+	 * _PAGE_COW is unnecessary on !X86_FEATURE_SHSTK kernels.
+	 * See the _PAGE_COW definition for more details.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pte;
+
+	/*
+	 * PTE is getting copied-on-write, so it will be dirtied
+	 * if writable, or made shadow stack if shadow stack and
+	 * being copied on access. Set they dirty bit for both
+	 * cases.
+	 */
+	pte = pte_set_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_COW);
+}
+
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
 static inline int pte_uffd_wp(pte_t pte)
 {
@@ -319,7 +381,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)
 
 static inline pte_t pte_mkclean(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_DIRTY_BITS);
 }
 
 static inline pte_t pte_mkold(pte_t pte)
@@ -329,7 +391,16 @@ static inline pte_t pte_mkold(pte_t pte)
 
 static inline pte_t pte_wrprotect(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_RW);
+	pte = pte_clear_flags(pte, _PAGE_RW);
+
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PTE (Write=0,Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pte_dirty(pte))
+		pte = pte_mkcow(pte);
+	return pte;
 }
 
 static inline pte_t pte_mkexec(pte_t pte)
@@ -339,7 +410,19 @@ static inline pte_t pte_mkexec(pte_t pte)
 
 static inline pte_t pte_mkdirty(pte_t pte)
 {
-	return pte_set_flags(pte, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pteval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating Dirty=1,Write=0 PTEs */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK) && !pte_write(pte))
+		dirty = _PAGE_COW;
+
+	return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY);
+}
+
+static inline pte_t pte_mkwrite_shstk(pte_t pte)
+{
+	/* pte_clear_cow() also sets Dirty=1 */
+	return pte_clear_cow(pte);
 }
 
 static inline pte_t pte_mkyoung(pte_t pte)
@@ -349,7 +432,12 @@ static inline pte_t pte_mkyoung(pte_t pte)
 
 static inline pte_t pte_mkwrite(pte_t pte)
 {
-	return pte_set_flags(pte, _PAGE_RW);
+	pte = pte_set_flags(pte, _PAGE_RW);
+
+	if (pte_dirty(pte))
+		pte = pte_clear_cow(pte);
+
+	return pte;
 }
 
 static inline pte_t pte_mkhuge(pte_t pte)
@@ -396,6 +484,26 @@ static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear)
 	return native_make_pmd(v & ~clear);
 }
 
+/* See comments above pte_mkcow() */
+static inline pmd_t pmd_mkcow(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pmd;
+
+	pmd = pmd_clear_flags(pmd, _PAGE_DIRTY);
+	return pmd_set_flags(pmd, _PAGE_COW);
+}
+
+/* See comments above pte_mkcow() */
+static inline pmd_t pmd_clear_cow(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pmd;
+
+	pmd = pmd_set_flags(pmd, _PAGE_DIRTY);
+	return pmd_clear_flags(pmd, _PAGE_COW);
+}
+
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
 static inline int pmd_uffd_wp(pmd_t pmd)
 {
@@ -420,17 +528,36 @@ static inline pmd_t pmd_mkold(pmd_t pmd)
 
 static inline pmd_t pmd_mkclean(pmd_t pmd)
 {
-	return pmd_clear_flags(pmd, _PAGE_DIRTY);
+	return pmd_clear_flags(pmd, _PAGE_DIRTY_BITS);
 }
 
 static inline pmd_t pmd_wrprotect(pmd_t pmd)
 {
-	return pmd_clear_flags(pmd, _PAGE_RW);
+	pmd = pmd_clear_flags(pmd, _PAGE_RW);
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PMD (RW=0, Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pmd_dirty(pmd))
+		pmd = pmd_mkcow(pmd);
+	return pmd;
 }
 
 static inline pmd_t pmd_mkdirty(pmd_t pmd)
 {
-	return pmd_set_flags(pmd, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pmdval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating (HW)Dirty=1, Write=0 PMDs */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK) && !pmd_write(pmd))
+		dirty = _PAGE_COW;
+
+	return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY);
+}
+
+static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd)
+{
+	return pmd_clear_cow(pmd);
 }
 
 static inline pmd_t pmd_mkdevmap(pmd_t pmd)
@@ -450,7 +577,11 @@ static inline pmd_t pmd_mkyoung(pmd_t pmd)
 
 static inline pmd_t pmd_mkwrite(pmd_t pmd)
 {
-	return pmd_set_flags(pmd, _PAGE_RW);
+	pmd = pmd_set_flags(pmd, _PAGE_RW);
+
+	if (pmd_dirty(pmd))
+		pmd = pmd_clear_cow(pmd);
+	return pmd;
 }
 
 static inline pud_t pud_set_flags(pud_t pud, pudval_t set)
@@ -467,6 +598,26 @@ static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)
 	return native_make_pud(v & ~clear);
 }
 
+/* See comments above pte_mkcow() */
+static inline pud_t pud_mkcow(pud_t pud)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pud;
+
+	pud = pud_clear_flags(pud, _PAGE_DIRTY);
+	return pud_set_flags(pud, _PAGE_COW);
+}
+
+/* See comments above pte_mkcow() */
+static inline pud_t pud_clear_cow(pud_t pud)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pud;
+
+	pud = pud_set_flags(pud, _PAGE_DIRTY);
+	return pud_clear_flags(pud, _PAGE_COW);
+}
+
 static inline pud_t pud_mkold(pud_t pud)
 {
 	return pud_clear_flags(pud, _PAGE_ACCESSED);
@@ -474,17 +625,32 @@ static inline pud_t pud_mkold(pud_t pud)
 
 static inline pud_t pud_mkclean(pud_t pud)
 {
-	return pud_clear_flags(pud, _PAGE_DIRTY);
+	return pud_clear_flags(pud, _PAGE_DIRTY_BITS);
 }
 
 static inline pud_t pud_wrprotect(pud_t pud)
 {
-	return pud_clear_flags(pud, _PAGE_RW);
+	pud = pud_clear_flags(pud, _PAGE_RW);
+
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PUD (RW=0, Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pud_dirty(pud))
+		pud = pud_mkcow(pud);
+	return pud;
 }
 
 static inline pud_t pud_mkdirty(pud_t pud)
 {
-	return pud_set_flags(pud, _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
+	pudval_t dirty = _PAGE_DIRTY;
+
+	/* Avoid creating (HW)Dirty=1, Write=0 PUDs */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK) && !pud_write(pud))
+		dirty = _PAGE_COW;
+
+	return pud_set_flags(pud, dirty | _PAGE_SOFT_DIRTY);
 }
 
 static inline pud_t pud_mkdevmap(pud_t pud)
@@ -504,7 +670,11 @@ static inline pud_t pud_mkyoung(pud_t pud)
 
 static inline pud_t pud_mkwrite(pud_t pud)
 {
-	return pud_set_flags(pud, _PAGE_RW);
+	pud = pud_set_flags(pud, _PAGE_RW);
+
+	if (pud_dirty(pud))
+		pud = pud_clear_cow(pud);
+	return pud;
 }
 
 #ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
arch/x86/include/asm/pgtable_types.h +41 -1

60 lines not on this page · raw message

Signature · 1 line
2.17.1
reviewer
#76 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 5 lines · Rick Edgecombe

The reason it's lightly used is that Dirty=1 is normally set _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.

Stupid question, since I just recently learned that IOMMUv2 is a
thing: I assume this also holds for IOMMUs that implement IOMMUv2/SVA, where the IOMMU directly walks the userspace page tables, and not just for the CPU core?

#77 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 12 lines · Jann Horn
Quoted 9 lines · Rick Edgecombe

The reason it's lightly used is that Dirty=1 is normally set _before_ a
write. A write with a Write=0 PTE would typically only generate a fault,
not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the
fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow
stacks will no longer exhibit this oddity.

Quoted 6 lines

Stupid question, since I just recently learned that IOMMUv2 is a
thing: I assume this also holds for IOMMUs that implement IOMMUv2/SVA,
where the IOMMU directly walks the userspace page tables, and not just
for the CPU core?

Sorry for the delay, I had to go find out. IOMMU behaves similar to the CET CPUs in this regard. Thanks for the question.

#78 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe
Quoted 27 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +27 -0
+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But
+ * in the case of X86_FEATURE_SHSTK, the software COW bit is used,
+ * since the Dirty=1,Write=0 will result in the memory being treated
+ * as shaodw stack by the HW. So when creating COW memory, a software
+ * bit is used _PAGE_BIT_COW. The following functions pte_mkcow() and
+ * pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
+ * transition it to the shadow stack compatible version of COW (Cow=1).
+ */
+
+static inline pte_t pte_mkcow(pte_t pte)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pte;
+
+	pte = pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_set_flags(pte, _PAGE_COW);
+}
+
+static inline pte_t pte_clear_cow(pte_t pte)
+{
+	/*
+	 * _PAGE_COW is unnecessary on !X86_FEATURE_SHSTK kernels.
+	 * See the _PAGE_COW definition for more details.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return pte;

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 363

Quoted 10 lines
arch/x86/include/asm/pgtable.h +10 -0
+
+	/*
+	 * PTE is getting copied-on-write, so it will be dirtied
+	 * if writable, or made shadow stack if shadow stack and
+	 * being copied on access. Set they dirty bit for both
+	 * cases.
+	 */
+	pte = pte_set_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_COW);
+}

These X86_FEATURE_SHSTK checks make me uneasy. Maybe use the _PAGE_COW logic for all machines with 64-bit entries. It will get you much more coverage and more universal rules.

Quoted 19 lines
arch/x86/include/asm/pgtable.h +3 -2
+
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
 static inline int pte_uffd_wp(pte_t pte)
 {
@@ -319,7 +381,7 @@ static inline pte_t pte_clear_uffd_wp(pte_t pte)
 
 static inline pte_t pte_mkclean(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_DIRTY);
+	return pte_clear_flags(pte, _PAGE_DIRTY_BITS);
 }
 
 static inline pte_t pte_mkold(pte_t pte)
@@ -329,7 +391,16 @@ static inline pte_t pte_mkold(pte_t pte)
 
 static inline pte_t pte_wrprotect(pte_t pte)
 {
-	return pte_clear_flags(pte, _PAGE_RW);
+	pte = pte_clear_flags(pte, _PAGE_RW);

arch/x86/include/asm/pgtable.h static inline pte_t pte_mkold(pte_t pte) · line 404

Quoted 10 lines
arch/x86/include/asm/pgtable.h +9 -0
+
+	/*
+	 * Blindly clearing _PAGE_RW might accidentally create
+	 * a shadow stack PTE (Write=0,Dirty=1). Move the hardware
+	 * dirty value to the software bit.
+	 */
+	if (pte_dirty(pte))
+		pte = pte_mkcow(pte);
+	return pte;
 }

Hm. What about ptep/pmdp_set_wrprotect()? They clear _PAGE_RW blindly.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#79 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 40 lines · Kirill A . Shutemov
Quoted 39 lines · Rick Edgecombe

+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But
+ * in the case of X86_FEATURE_SHSTK, the software COW bit is used,
+ * since the Dirty=1,Write=0 will result in the memory being treated
+ * as shaodw stack by the HW. So when creating COW memory, a software
+ * bit is used _PAGE_BIT_COW. The following functions pte_mkcow() and
+ * pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
+ * transition it to the shadow stack compatible version of COW (Cow=1).
+ */

arch/x86/include/asm/pgtable.h +25 -0
+
+static inline pte_t pte_mkcow(pte_t pte)
+{
+     if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+             return pte;
+
+     pte = pte_clear_flags(pte, _PAGE_DIRTY);
+     return pte_set_flags(pte, _PAGE_COW);
+}
+
+static inline pte_t pte_clear_cow(pte_t pte)
+{
+     /*
+      * _PAGE_COW is unnecessary on !X86_FEATURE_SHSTK kernels.
+      * See the _PAGE_COW definition for more details.
+      */
+     if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+             return pte;
+
+     /*
+      * PTE is getting copied-on-write, so it will be dirtied
+      * if writable, or made shadow stack if shadow stack and
+      * being copied on access. Set they dirty bit for both
+      * cases.
+      */
Quoted 8 lines

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 363

Quoted 3 lines
arch/x86/include/asm/pgtable.h +3 -0
+     pte = pte_set_flags(pte, _PAGE_DIRTY);
+     return pte_clear_flags(pte, _PAGE_COW);
+}

These X86_FEATURE_SHSTK checks make me uneasy. Maybe use the _PAGE_COW
logic for all machines with 64-bit entries. It will get you much more coverage and more universal rules.

Yes, I didn't like them either at first. The reasoning originally was that _PAGE_COW is a bit more work and it might show up for some benchmark.

Looking at this again though, it is just a few more operations on memory that is already getting touched either way. It must be a very tiny amount of impact if any. I'm fine removing them. Having just one set of logic around this would make it easier to reason about.

Dave, any thoughts on this?

reviewer
#80 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 54 lines · Rick P
Quoted 40 lines · Kirill A . Shutemov
Quoted 39 lines · Rick Edgecombe

+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But
+ * in the case of X86_FEATURE_SHSTK, the software COW bit is used,
+ * since the Dirty=1,Write=0 will result in the memory being treated
+ * as shaodw stack by the HW. So when creating COW memory, a software
+ * bit is used _PAGE_BIT_COW. The following functions pte_mkcow() and
+ * pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
+ * transition it to the shadow stack compatible version of COW (Cow=1).
+ */

arch/x86/include/asm/pgtable.h +25 -0
+
+static inline pte_t pte_mkcow(pte_t pte)
+{
+     if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+             return pte;
+
+     pte = pte_clear_flags(pte, _PAGE_DIRTY);
+     return pte_set_flags(pte, _PAGE_COW);
+}
+
+static inline pte_t pte_clear_cow(pte_t pte)
+{
+     /*
+      * _PAGE_COW is unnecessary on !X86_FEATURE_SHSTK kernels.
+      * See the _PAGE_COW definition for more details.
+      */
+     if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+             return pte;
+
+     /*
+      * PTE is getting copied-on-write, so it will be dirtied
+      * if writable, or made shadow stack if shadow stack and
+      * being copied on access. Set they dirty bit for both
+      * cases.
+      */
Quoted 8 lines

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 363

Quoted 3 lines
arch/x86/include/asm/pgtable.h +3 -0
+     pte = pte_set_flags(pte, _PAGE_DIRTY);
+     return pte_clear_flags(pte, _PAGE_COW);
+}

These X86_FEATURE_SHSTK checks make me uneasy. Maybe use the _PAGE_COW
logic for all machines with 64-bit entries. It will get you much more coverage and more universal rules.

Yes, I didn't like them either at first. The reasoning originally was that _PAGE_COW is a bit more work and it might show up for some benchmark.

Quoted 6 lines

Looking at this again though, it is just a few more operations on memory that is already getting touched either way. It must be a very tiny amount of impact if any. I'm fine removing them. Having just one set of logic around this would make it easier to reason about.

Dave, any thoughts on this?

But the rules wouldn't actually be universal - you'd still have to look at X86_FEATURE_SHSTK in code that wants to figure out whether a PTE is shadow stack (on a newer CPU) or readonly dirty (on an older CPU that can set dirty bits on non-present PTEs), right?

#81 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 18 lines · Jann Horn
Quoted 18 lines
Quoted 5 lines

These X86_FEATURE_SHSTK checks make me uneasy. Maybe use the _PAGE_COW
logic for all machines with 64-bit entries. It will get you much more
coverage and more universal rules.

Yes, I didn't like them either at first. The reasoning originally was
that _PAGE_COW is a bit more work and it might show up for some benchmark.

Looking at this again though, it is just a few more operations on memory that is already getting touched either way. It must be a very
tiny amount of impact if any. I'm fine removing them. Having just one
set of logic around this would make it easier to reason about.

Quoted 6 lines
Quoted 1 line

Dave, any thoughts on this?

But the rules wouldn't actually be universal - you'd still have to look at X86_FEATURE_SHSTK in code that wants to figure out whether a PTE is shadow stack (on a newer CPU) or readonly dirty (on an older CPU that can set dirty bits on non-present PTEs), right?

Good point. It still would need a check in pte_shstk() or pte_write(), so pte_write() doesn't think CPU created Write=0,Dirty=1 memory is writable. Which then percolates to most of the other checks anyway.

#82 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Edgecombe, Rick P
Quoted 26 lines · Rick P
Quoted 14 lines
Quoted 14 lines
arch/x86/include/asm/pgtable.h +14 -0
+static inline pte_t pte_clear_cow(pte_t pte)
+{
+     /*
+      * _PAGE_COW is unnecessary on !X86_FEATURE_SHSTK kernels.
+      * See the _PAGE_COW definition for more details.
+      */
+     if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+             return pte;
+
+     /*
+      * PTE is getting copied-on-write, so it will be dirtied
+      * if writable, or made shadow stack if shadow stack and
+      * being copied on access. Set they dirty bit for both
+      * cases.
Quoted 8 lines

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 363

Quoted 4 lines
arch/x86/include/asm/pgtable.h +4 -0
+      */
+     pte = pte_set_flags(pte, _PAGE_DIRTY);
+     return pte_clear_flags(pte, _PAGE_COW);
+}

These X86_FEATURE_SHSTK checks make me uneasy. Maybe use the _PAGE_COW
logic for all machines with 64-bit entries. It will get you much more coverage and more universal rules.

Yes, I didn't like them either at first. The reasoning originally was that _PAGE_COW is a bit more work and it might show up for some benchmark.

Quoted 6 lines

Looking at this again though, it is just a few more operations on memory that is already getting touched either way. It must be a very tiny amount of impact if any. I'm fine removing them. Having just one set of logic around this would make it easier to reason about.

Dave, any thoughts on this?

The cpu_feature_enabled(X86_FEATURE_SHSTK) checks enable both compile-time and runtime optimization. What makes this even more fun is:

+#ifdef CONFIG_X86_SHADOW_STACK
+#define _PAGE_COW (_AT(pteval_t, 1) << _PAGE_BIT_COW)
+#else
+#define _PAGE_COW (_AT(pteval_t, 0))
+#endif

which I think means that the pte_clear_flags() goes away if CONFIG_X86_SHADOW_STACK is disabled. So, what Rick posted here ends up doing the following with:

	  | X86_FEATURE_SHSTK=1	|  X86_FEATURE_SHSTK=0
==========+=====================+========================
CONFIG=n  |  compiled out	|  compiled out
CONFIG=y  |  set/clear		|  boot-time patched out

If we pull the cpu_feature_enabled() out, I think we end up getting behavior like this:

	  | X86_FEATURE_SHSTK=1	|  X86_FEATURE_SHSTK=0
==========+=====================+========================
CONFIG=n  |  set _PAGE_DIRTY	|  set _PAGE_DIRTY
CONFIG=y  |  set/clear		|  set/clear

It ends up adding instruction overhead (set _PAGE_DIRTY) to two cases where it completely compiled out before. It also adds runtime overhead (the "tiny amount of impact" you mentioned) to set/clear where it would have runtime patched out before.

None of this is a deal breaker in terms of runtime overhead. But, I do think the benefits of the cpu_feature_enabled() are worth it, even if it's just an optimization. You could move it to the end of the series and we can debate it on its own merits if you want.

#83 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe
Quoted 6 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes (not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1.

How does "Some OSes have a greater dependence on software available bits in PTEs than Linux" sound?

Quoted 5 lines

The reason it's lightly used is that Dirty=1 is normally set _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.

Again, an interesting anecdote but not salient information here.

Quoted 11 lines
arch/x86/include/asm/pgtable.h +1 -1
index 6496ec84b953..ad201dae7316 100644
@@ -134,9 +142,17 @@ static inline int pte_young(pte_t pte)
 	return pte_flags(pte) & _PAGE_ACCESSED;
 }
 
-static inline int pmd_dirty(pmd_t pmd)
+static inline bool pmd_dirty(pmd_t pmd)
 {

arch/x86/include/asm/pgtable.h static inline int pte_young(pte_t pte) · line 155

Quoted 10 lines
arch/x86/include/asm/pgtable.h +9 -1
-	return pmd_flags(pmd) & _PAGE_DIRTY;
+	return pmd_flags(pmd) & _PAGE_DIRTY_BITS;
+}
+
+static inline bool pmd_shstk(pmd_t pmd)
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return false;
+
+	return (pmd_flags(pmd) & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY;

(flags & PSE|RW|D) == PSE|D;

R/O+D can exist higher in the paging structures and does not convey type=shstk-ness to later stages of the walk.

However, there is a further complication which is bound rear its head sooner or later, and warrants discussing.

type=shstk isn't actually only R/O+D on the leaf PTE; its also R/W on the accumulated access rights on non-leaf PTEs.

Specifically, if you clear the RW bit on any higher level in the pagetable, then everything mapped by that PTE ceases to be of type shstk, even if the leaf has the R/O+D bit combination.

This is allegedly a feature for the database folks, where they can create R/O and R/W aliases of the same memory, sharing intermediate pagetables, where the R/W alias will set D bits per usual and the R/O alias needs not to transmogrify itself into a shadow stack.

~Andrew

#84 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 10 lines · Andrew Cooper
Quoted 6 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes (not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1.

How does "Some OSes have a greater dependence on software available bits in PTEs than Linux" sound?

Quoted 6 lines
Quoted 5 lines

The reason it's lightly used is that Dirty=1 is normally set _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow stacks will no longer exhibit this oddity.

Again, an interesting anecdote but not salient information here.

As much as I like the sound of my own voice (and anecdotes), I agree that this is a bit oblique for the patch. Maybe this anecdote should get banished elsewhere.

The changelog here could definitely get to the point faster.

#85 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 26 lines · Dave Hansen
Quoted 25 lines · Andrew Cooper
Quoted 9 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes
(not Linux). That left the hardware architects looking for a way to
represent a new memory type (shadow stack) within the existing bits.
They chose to repurpose a lightly-used state: Write=0,Dirty=1.

How does "Some OSes have a greater dependence on software available bits
in PTEs than Linux" sound?

Quoted 9 lines

The reason it's lightly used is that Dirty=1 is normally set _before_ a
write. A write with a Write=0 PTE would typically only generate a fault,
not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the
fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow
stacks will no longer exhibit this oddity.

Quoted 7 lines
Quoted 1 line

Again, an interesting anecdote but not salient information here.

As much as I like the sound of my own voice (and anecdotes), I agree that this is a bit oblique for the patch. Maybe this anecdote should get banished elsewhere.

The changelog here could definitely get to the point faster.

Although this text was inherited, I thought it was useful to disperse any "huh, I wonder why" thoughts that may be lingering in the readers head as they try to grok the rest of the text. I'll shorten it as suggested. Thanks all.

#86 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Andrew Cooper
Quoted 4 lines · Andrew Cooper

(flags & PSE|RW|D) == PSE|D;

R/O+D can exist higher in the paging structures and does not convey type=shstk-ness to later stages of the walk.

Hmm, yes. I guess it would be more correct to check if it's a leaf as well.

Quoted 8 lines

However, there is a further complication which is bound rear its head sooner or later, and warrants discussing.

type=shstk isn't actually only R/O+D on the leaf PTE; its also R/W on the accumulated access rights on non-leaf PTEs.

Quoted 8 lines

Specifically, if you clear the RW bit on any higher level in the pagetable, then everything mapped by that PTE ceases to be of type shstk, even if the leaf has the R/O+D bit combination.

This is allegedly a feature for the database folks, where they can create R/O and R/W aliases of the same memory, sharing intermediate pagetables, where the R/W alias will set D bits per usual and the R/O alias needs not to transmogrify itself into a shadow stack.

Thanks, I somehow missed this corner of the architecture. It looks like this is not an issue for Linux at the moment because non-leaf PTEs should have Write=1. I guess we need to keep this in mind if we ever have Write=0 upper level PTEs though. Maybe a comment around _PAGE_TABLE would be useful.

maintainer
#87 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe

On Thu, Sep 29, 2022 at 03:29:07PM -0700, Rick Edgecombe wrote:

Mucho confusion here:

Quoted 3 lines

(a) (Write=0,Cow=1,Dirty=0) A modified, copy-on-write (COW) page.
(b) (Write=0,Cow=1,Dirty=0) A R/O page that has been COW'ed. The user page
(d) (Write=0,Cow=1,Dirty=0) A shared shadow stack PTE. When a shadow stack

are all identical cases;

Quoted 2 lines

(c) (Write=0,Cow=0,Dirty=1) A shadow stack PTE.
(e) (Write=0,Cow=0,Dirty=1) A Cow PTE created when a processor without

as are these.

maintainer
#88 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe
Quoted 10 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes (not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits. They chose to repurpose a lightly-used state: Write=0,Dirty=1.

The reason it's lightly used is that Dirty=1 is normally set _before_ a write. A write with a Write=0 PTE would typically only generate a fault, not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the

s/Write/Dirty/

Quoted 1 line

fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow

s/Dirty=0,Write=1/Write=0,Dirty=1/

Quoted 5 lines

stacks will no longer exhibit this oddity.

The kernel should avoid inadvertently creating shadow stack memory because it is security sensitive. So given the above, all it needs to do is avoid manually crating Write=0,Dirty=1 PTEs in software.

Whichever way around you choose, please be consistent.

Quoted 16 lines

In places where Linux normally creates Write=0,Dirty=1, it can use the software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other words, whenever Linux needs to create Write=0,Dirty=1, it instead creates Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. This clearly separates shadow stack from other data, and results in the following:

(a) (Write=0,Cow=1,Dirty=0) A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made COW via fork(), the kernel would mark it Write=0,Dirty=1. Now it will instead use the Cow bit.
(b) (Write=0,Cow=1,Dirty=0) A R/O page that has been COW'ed. The user page is in a R/O VMA, and get_user_pages() needs a writable copy. The page fault handler creates a copy of the page and sets the new copy's PTE as Write=0 and Cow=1.
(c) (Write=0,Cow=0,Dirty=1) A shadow stack PTE.

Quoted 8 lines

(d) (Write=0,Cow=1,Dirty=0) A shared shadow stack PTE. When a shadow stack page is being shared among processes (this happens at fork()), its PTE is made Dirty=0, so the next shadow stack access causes a fault, and the page is duplicated and Dirty=1 is set again. This is the COW equivalent for shadow stack pages, even though it's copy-on-access rather than copy-on-write.
(e) (Write=0,Cow=0,Dirty=1) A Cow PTE created when a processor without shadow stack support set Dirty=1.

Please restureture this (and the comment) something like:

(Write=0,Dirty=0,Cow=1):

- copy_present_pte(): A modified copy-on-write page.
- ...

(Write=0,Dirty=1,Cow=0):

- FEATURE_CET: Shadow Stack entry
- !FEATURE_CET: see the above Cow=1 cases

#89 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There is essentially no room left in the x86 hardware PTEs on some OSes
(not Linux). That left the hardware architects looking for a way to represent a new memory type (shadow stack) within the existing bits.
They chose to repurpose a lightly-used state: Write=0,Dirty=1.

Quoted 8 lines
Quoted 6 lines

The reason it's lightly used is that Dirty=1 is normally set _before_ a
write. A write with a Write=0 PTE would typically only generate a fault,
not set Dirty=1. Hardware can (rarely) both set Write=1 *and* generate the

s/Write/Dirty/

Oops, yes.

Quoted 5 lines
Quoted 3 lines

fault, resulting in a Dirty=0,Write=1 PTE. Hardware which supports shadow

s/Dirty=0,Write=1/Write=0,Dirty=1/

Ok, I'll scrub the series for the order.

Quoted 56 lines
Quoted 8 lines

stacks will no longer exhibit this oddity.

The kernel should avoid inadvertently creating shadow stack memory because
it is security sensitive. So given the above, all it needs to do is avoid
manually crating Write=0,Dirty=1 PTEs in software.

Whichever way around you choose, please be consistent.

Quoted 31 lines

In places where Linux normally creates Write=0,Dirty=1, it can use the
software-defined _PAGE_COW in place of the hardware _PAGE_DIRTY. In other
words, whenever Linux needs to create Write=0,Dirty=1, it instead creates
Write=0,Cow=1 except for shadow stack, which is Write=0,Dirty=1. This
clearly separates shadow stack from other data, and results in the following:

(a) (Write=0,Cow=1,Dirty=0) A modified, copy-on-write (COW) page. Previously when a typical anonymous writable mapping was made
COW via
fork(), the kernel would mark it Write=0,Dirty=1. Now it will
instead
use the Cow bit.
(b) (Write=0,Cow=1,Dirty=0) A R/O page that has been COW'ed. The user page
is in a R/O VMA, and get_user_pages() needs a writable copy.
The page
fault handler creates a copy of the page and sets the new
copy's PTE
as Write=0 and Cow=1.
(c) (Write=0,Cow=0,Dirty=1) A shadow stack PTE.
(d) (Write=0,Cow=1,Dirty=0) A shared shadow stack PTE. When a shadow stack
page is being shared among processes (this happens at fork()),
its PTE
is made Dirty=0, so the next shadow stack access causes a
fault, and

Quoted 8 lines

the page is duplicated and Dirty=1 is set again. This is the
COW
equivalent for shadow stack pages, even though it's copy-on-
access
rather than copy-on-write.
(e) (Write=0,Cow=0,Dirty=1) A Cow PTE created when a processor without
shadow stack support set Dirty=1.

Please restureture this (and the comment) something like:

(Write=0,Dirty=0,Cow=1):

Quoted 8 lines

- copy_present_pte(): A modified copy-on-write page.
- ...

(Write=0,Dirty=1,Cow=0):

- FEATURE_CET: Shadow Stack entry
- !FEATURE_CET: see the above Cow=1 cases

Yes, I incorporated feedback from your earlier comment. Sorry for bad communication.

maintainer
#90 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

In reply to Rick Edgecombe

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 328

Quoted 6 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +2 -0
@@ -300,6 +324,44 @@ static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear)
 	return native_make_pte(v & ~clear);
 }
 
+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But

This is misleading; this isn't an x86 specific thing. The core-mm code does this.

Quoted 7 lines
arch/x86/include/asm/pgtable.h +7 -0
+ * in the case of X86_FEATURE_SHSTK, the software COW bit is used,
+ * since the Dirty=1,Write=0 will result in the memory being treated
+ * as shaodw stack by the HW. So when creating COW memory, a software
+ * bit is used _PAGE_BIT_COW. The following functions pte_mkcow() and
+ * pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
+ * transition it to the shadow stack compatible version of COW (Cow=1).
+ */
#91 lore raw

Re: [PATCH v2 10/39] x86/mm: Introduce _PAGE_COW

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +0 -0
@@ -300,6 +324,44 @@ static inline pte_t pte_clear_flags(pte_t pte,
 pteval_t clear)
       return native_make_pte(v & ~clear);
  }
 
Quoted 6 lines

arch/x86/include/asm/pgtable.h static inline pte_t pte_clear_flags(pte_t pte, pteval_t clear) · line 328

Quoted 2 lines
arch/x86/include/asm/pgtable.h +2 -0
+/*
+ * Normally the Dirty bit is used to denote COW memory on x86. But

This is misleading; this isn't an x86 specific thing. The core-mm code
does this.

Well pte_mkdirty() does map to other HW bits on different architectures. But yea, it's confusing.

Hmm, is this comment a bit stale either way now though? In the past it was probably more accurate to say core MM code used it to "detect" cowed memory. But the GUP pte_dirty() check was changed recently:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5535be3099717646781ce1540cf725965d680e7b

I don't think any code is looking specifically for COWed memory using the PTE dirty bit anymore, it just happens to coincide with it. Double checking my understanding...

Maybe this would be more accurate?

/*
* Normally COW memory can result in Dirty=1,Write=0 PTEs. But in the
* case of X86_FEATURE_SHSTK, the software COW bit is used, since the
* Dirty=1,Write=0 will result in the memory being treated as shaodw
* stack by the HW. So when creating COW memory, a software bit is used
* _PAGE_BIT_COW. The following functions pte_mkcow() and
* pte_clear_cow() take a PTE marked conventially COW (Dirty=1) and
* transition it to the shadow stack compatible version of COW (Cow=1). */

#92 lore raw

PATCH v2 11/39 x86/mm: Update pte_modify for _PAGE_COW

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The Write=0,Dirty=1 PTE has been used to indicate copy-on-write pages. However, newer x86 processors also regard a Write=0,Dirty=1 PTE as a shadow stack page. In order to separate the two, the software-defined _PAGE_DIRTY is changed to _PAGE_COW for the copy-on-write case, and pte_*() are updated to do this.

pte_modify() takes a "raw" pgprot_t which was not necessarily created with any of the existing PTE bit helpers. That means that it can return a pte_t with Write=0,Dirty=1, a shadow stack PTE, when it did not intend to create one.

However pte_modify() changes a PTE to 'newprot', but it doesn't use the pte_*(). Modify it to also move _PAGE_DIRTY to _PAGE_COW. Apply the same changes to pmd_modify().

---

v2:
- Update commit log with text and suggestions from (Dave Hansen)
- Drop fixup_dirty_pte() in favor of clearing the HW dirty bit along with the _PAGE_CHG_MASK masking, then calling pte_mkdirty() (Dave Hansen)

 arch/x86/include/asm/pgtable.h | 41 +++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 6 deletions(-)
arch/x86/include/asm/pgtable.h +35 -6
index ad201dae7316..2f2963429f48 100644
@@ -790,26 +790,55 @@ static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask);
 
 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 {
+	pteval_t _page_chg_mask_no_dirty = _PAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pteval_t val = pte_val(pte), oldval = val;
+	pte_t pte_result;
 
 	/*
 	 * Chop off the NX bit (if present), and add the NX portion of
 	 * the newprot (if present):
 	 */
-	val &= _PAGE_CHG_MASK;
-	val |= check_pgprot(newprot) & ~_PAGE_CHG_MASK;
+	val &= _page_chg_mask_no_dirty;
+	val |= check_pgprot(newprot) & ~_page_chg_mask_no_dirty;
 	val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);
-	return __pte(val);
+
+	pte_result = __pte(val);
+
+	/*
+	 * Dirty bit is not preserved above so it can be done
+	 * in a special way for the shadow stack case, where it
+	 * needs to set _PAGE_COW. pte_mkdirty() will do this in
+	 * the case of shadow stack.
+	 */
+	if (pte_dirty(pte))
+		pte_result = pte_mkdirty(pte_result);
+
+	return pte_result;
 }
 
 static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
 {
+	pteval_t _hpage_chg_mask_no_dirty = _HPAGE_CHG_MASK & ~_PAGE_DIRTY;
 	pmdval_t val = pmd_val(pmd), oldval = val;
+	pmd_t pmd_result;
 
-	val &= _HPAGE_CHG_MASK;
-	val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK;
+	val &= _hpage_chg_mask_no_dirty;
+	val |= check_pgprot(newprot) & ~_hpage_chg_mask_no_dirty;
 	val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
-	return __pmd(val);
+
+
+	pmd_result = __pmd(val);
+
+	/*
+	 * Dirty bit is not preserved above so it can be done
+	 * specially for the shadow stack case. It needs to move
+	 * the HW dirty bit to the software COW bit. Set in the
+	 * result if it was set in the original value.
+	 */
+	if (pmd_dirty(pmd))
+		pmd_result = pmd_mkdirty(pmd_result);
+
+	return pmd_result;
 }
 
 /*
Signature · 1 line
2.17.1
#93 lore raw

PATCH v2 12/39 x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When Shadow Stack is in use, Write=0,Dirty=1 PTE are reserved for shadow stack. Copy-on-write PTes then have Write=0,Cow=1.

When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could become a transient shadow stack PTE in two cases:

The first case is that some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting Shadow Stack, and a TLB flush is not necessary.

The second case is that when _PAGE_DIRTY is replaced with _PAGE_COW non- atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.

Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.

---

v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)

Yu-cheng v30:
- Replace (pmdval_t) cast with CONFIG_PGTABLE_LEVELES > 2 (Borislav Petkov).

 arch/x86/include/asm/pgtable.h | 36 ++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
arch/x86/include/asm/pgtable.h +36 -0
index 2f2963429f48..58c7bf9d7392 100644
@@ -1287,6 +1287,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 static inline void ptep_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pte_t *ptep)
 {
+#ifdef CONFIG_X86_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pte_t old_pte, new_pte;
+
+		old_pte = READ_ONCE(*ptep);
+		do {
+			new_pte = pte_wrprotect(old_pte);
+		} while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte));
+
+		return;
+	}
+#endif
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte);
 }
 
@@ -1339,6 +1356,25 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,
 static inline void pmdp_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pmd_t *pmdp)
 {
+#ifdef CONFIG_X86_SHADOW_STACK
+	/*
+	 * If Shadow Stack is enabled, pmd_wrprotect() moves _PAGE_DIRTY
+	 * to _PAGE_COW (see comments at pmd_wrprotect()).
+	 * When a thread reads a RW=1, Dirty=0 PMD and before changing it
+	 * to RW=0, Dirty=0, another thread could have written to the page
+	 * and the PMD is RW=1, Dirty=1 now.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pmd_t old_pmd, new_pmd;
+
+		old_pmd = READ_ONCE(*pmdp);
+		do {
+			new_pmd = pmd_wrprotect(old_pmd);
+		} while (!try_cmpxchg(&pmdp->pmd, &old_pmd.pmd, new_pmd.pmd));
+
+		return;
+	}
+#endif
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)pmdp);
 }
 
Signature · 1 line
2.17.1
#94 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 17 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +9 -0
index 2f2963429f48..58c7bf9d7392 100644
@@ -1287,6 +1287,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 static inline void ptep_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pte_t *ptep)
 {
+#ifdef CONFIG_X86_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pte_t old_pte, new_pte;
+

arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1308

Quoted 10 lines
arch/x86/include/asm/pgtable.h +8 -0
+		old_pte = READ_ONCE(*ptep);
+		do {
+			new_pte = pte_wrprotect(old_pte);
+		} while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte));
+
+		return;
+	}
+#endif
 	clear_bit(_PAGE_BIT_RW, (unsigned long *)&ptep->pte);
 }

Okay, this addresses my previous question. The need in cmpxchg is unfortunate, but well.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#95 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

In reply to Rick Edgecombe
Quoted 53 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When Shadow Stack is in use, Write=0,Dirty=1 PTE are reserved for shadow stack. Copy-on-write PTes then have Write=0,Cow=1.

When a PTE goes from Write=1,Dirty=1 to Write=0,Cow=1, it could become a transient shadow stack PTE in two cases:

The first case is that some processors can start a write but end up seeing a Write=0 PTE by the time they get to the Dirty bit, creating a transient shadow stack PTE. However, this will not occur on processors supporting Shadow Stack, and a TLB flush is not necessary.

The second case is that when _PAGE_DIRTY is replaced with _PAGE_COW non- atomically, a transient shadow stack PTE can be created as a result. Thus, prevent that with cmpxchg.

Dave Hansen, Jann Horn, Andy Lutomirski, and Peter Zijlstra provided many insights to the issue. Jann Horn provided the cmpxchg solution.

---

v2:
- Compile out some code due to clang build error
- Clarify commit log (dhansen)
- Normalize PTE bit descriptions between patches (dhansen)
- Update comment with text from (dhansen)

Yu-cheng v30:
- Replace (pmdval_t) cast with CONFIG_PGTABLE_LEVELES > 2 (Borislav Petkov).

arch/x86/include/asm/pgtable.h | 36 ++++++++++++++++++++++++++++++++++

1 file changed, 36 insertions(+)
arch/x86/include/asm/pgtable.h +7 -0
index 2f2963429f48..58c7bf9d7392 100644
@@ -1287,6 +1287,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 static inline void ptep_set_wrprotect(struct mm_struct *mm,
 				      unsigned long addr, pte_t *ptep)
 {
+#ifdef CONFIG_X86_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK)) {

arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1306

Quoted 10 lines
arch/x86/include/asm/pgtable.h +10 -0
+		pte_t old_pte, new_pte;
+
+		old_pte = READ_ONCE(*ptep);
+		do {
+			new_pte = pte_wrprotect(old_pte);
+		} while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte));
+
+		return;
+	}
+#endif

There is no way of using IS_ENABLED() here instead of these ifdefs?

Did you have a look at ptep_set_access_flags() and friends and checked they do not need to be changed too? Perhaps you should at least add some assertion just to ensure nothing breaks.

#96 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 10 lines · Nadav Amit
Quoted 10 lines
arch/x86/include/asm/pgtable.h +10 -0
+#ifdef CONFIG_X86_SHADOW_STACK
+	/*
+	 * Avoid accidentally creating shadow stack PTEs
+	 * (Write=0,Dirty=1).  Use cmpxchg() to prevent races with
+	 * the hardware setting Dirty=1.
+	 */
+	if (cpu_feature_enabled(X86_FEATURE_SHSTK)) {
+		pte_t old_pte, new_pte;
+
+		old_pte = READ_ONCE(*ptep);
Quoted 8 lines

arch/x86/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1306

Quoted 7 lines
arch/x86/include/asm/pgtable.h +7 -0
+		do {
+			new_pte = pte_wrprotect(old_pte);
+		} while (!try_cmpxchg(&ptep->pte, &old_pte.pte, new_pte.pte));
+
+		return;
+	}
+#endif

There is no way of using IS_ENABLED() here instead of these ifdefs?

Actually, both the existing #ifdef and an IS_ENABLED() check would be is superfluous as-is.

Adding X86_FEATURE_SHSTK disabled-features.h gives cpu_feature_enabled() compile-time optimizations for free. No need for *any* additional CONFIG_* checks.

The only issue would be if the #ifdef'd code won't even compile with X86_FEATURE_SHSTK disabled.

#97 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

In reply to Nadav Amit
Quoted 3 lines · Nadav Amit

Did you have a look at ptep_set_access_flags() and friends and checked they
do not need to be changed too?

ptep_set_access_flags() doesn't actually set any additional dirty bits on x86, so I think it's ok.

Quoted 2 lines

Perhaps you should at least add some
assertion just to ensure nothing breaks.

You mean in ptep_set_access_flags()? I think some assertions would be really great, I'm just not sure where.

#98 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 7 lines · Rick P
Quoted 3 lines · Nadav Amit

Did you have a look at ptep_set_access_flags() and friends and checked they
do not need to be changed too?

ptep_set_access_flags() doesn't actually set any additional dirty bits on x86, so I think it's ok.

Are you sure about that? (lost my confidence today so I am hesitant).

Looking on insert_pfn(), I see:

                        entry = maybe_mkwrite(pte_mkdirty(entry), vma);
                        if (ptep_set_access_flags(vma, addr, pte, entry, 1)) ...

This appears to set the dirty bit while potentially leaving the write-bit clear. This is the scenario you want to avoid, no?

Quoted 5 lines
Quoted 2 lines

Perhaps you should at least add some
assertion just to ensure nothing breaks.

You mean in ptep_set_access_flags()? I think some assertions would be really great, I'm just not sure where.

Yes, on x86’s version of the function.

#99 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 12 lines · Nadav Amit
Quoted 7 lines · Rick P
Quoted 3 lines · Nadav Amit

Did you have a look at ptep_set_access_flags() and friends and checked they
do not need to be changed too?

ptep_set_access_flags() doesn't actually set any additional dirty bits on x86, so I think it's ok.

Are you sure about that? (lost my confidence today so I am hesitant).

Quoted 7 lines

Looking on insert_pfn(), I see:

                       entry = maybe_mkwrite(pte_mkdirty(entry), vma);
                       if (ptep_set_access_flags(vma, addr, pte, entry, 1)) ...

This appears to set the dirty bit while potentially leaving the write-bit clear. This is the scenario you want to avoid, no?

No. I am not paying attention. Ignore.

#100 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 16 lines · Nadav Amit
Quoted 14 lines · Nadav Amit
Quoted 7 lines · Rick P
Quoted 3 lines · Nadav Amit

Did you have a look at ptep_set_access_flags() and friends and checked they
do not need to be changed too?

ptep_set_access_flags() doesn't actually set any additional dirty bits on x86, so I think it's ok.

Are you sure about that? (lost my confidence today so I am hesitant).

Looking on insert_pfn(), I see:

Quoted 7 lines
Quoted 5 lines
                      entry = maybe_mkwrite(pte_mkdirty(entry), vma);
                      if (ptep_set_access_flags(vma, addr, pte, entry, 1)) ...

This appears to set the dirty bit while potentially leaving the write-bit clear. This is the scenario you want to avoid, no?

No. I am not paying attention. Ignore.

Sorry for the spam. Just this “dirty” argument is confusing. This indeed seems like a flow that can set the dirty bit. I think.

#101 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 32 lines · Nadav Amit
Quoted 30 lines · Nadav Amit
Quoted 26 lines · Nadav Amit
Quoted 9 lines · Rick P
Quoted 4 lines · Nadav Amit

Did you have a look at ptep_set_access_flags() and friends and
checked they
do not need to be changed too?

ptep_set_access_flags() doesn't actually set any additional dirty bits
on x86, so I think it's ok.

Are you sure about that? (lost my confidence today so I am hesitant).

Looking on insert_pfn(), I see:

entry = maybe_mkwrite(pte_mkdirty(entry),
vma);
if (ptep_set_access_flags(vma, addr, pte,
entry, 1)) ...

This appears to set the dirty bit while potentially leaving the write-bit
clear. This is the scenario you want to avoid, no?

Quoted 5 lines
Quoted 1 line

No. I am not paying attention. Ignore.

Sorry for the spam. Just this “dirty” argument is confusing. This indeed
seems like a flow that can set the dirty bit. I think.

I think the HW dirty bit will not be set here. How it works is, pte_mkdirty() will not actually set the HW dirty bit, but instead the software COW bit. Here is the relevant snippet:

static inline pte_t pte_mkdirty(pte_t pte)

{
	pteval_t dirty = _PAGE_DIRTY;

	/* Avoid creating Dirty=1,Write=0 PTEs */
	if (cpu_feature_enabled(X86_FEATURE_SHSTK) && !pte_write(pte))
		dirty = _PAGE_COW;

	return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY);
}

So for a !VM_WRITE vma, you end up with Write=0,Cow=1 PTE passed into ptep_set_access_flags(). Does it make sense?

#102 lore raw

Re: [PATCH v2 12/39] x86/mm: Update ptep_set_wrprotect() and pmdp_set_wrprotect() for transition from _PAGE_DIRTY to _PAGE_COW

Quoted 15 lines · Rick P

I think the HW dirty bit will not be set here. How it works is, pte_mkdirty() will not actually set the HW dirty bit, but instead the software COW bit. Here is the relevant snippet:

static inline pte_t pte_mkdirty(pte_t pte)

{
	pteval_t dirty = _PAGE_DIRTY;

	/* Avoid creating Dirty=1,Write=0 PTEs */
	if (cpu_feature_enabled(X86_FEATURE_SHSTK) && !pte_write(pte))
		dirty = _PAGE_COW;

	return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY);
}
Quoted 2 lines

So for a !VM_WRITE vma, you end up with Write=0,Cow=1 PTE passed into ptep_set_access_flags(). Does it make sense?

Thanks for your patience with me. I should have read the series in order.

#103 lore raw

PATCH v2 13/39 mm: Move VM_UFFD_MINOR_BIT from 37 to 38

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

To introduce VM_SHADOW_STACK as VM_HIGH_ARCH_BIT (37), and make all VM_HIGH_ARCH_BITs stay together, move VM_UFFD_MINOR_BIT from 37 to 38.

---
 include/linux/mm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/mm.h +1 -1
index 21f8b27bd9fd..be80fc827212 100644
@@ -365,7 +365,7 @@ extern unsigned int kobjsize(const void *objp);
 #endif
 
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
-# define VM_UFFD_MINOR_BIT	37
+# define VM_UFFD_MINOR_BIT	38
 # define VM_UFFD_MINOR		BIT(VM_UFFD_MINOR_BIT)	/* UFFD minor faults */
 #else /* !CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
 # define VM_UFFD_MINOR		VM_NONE
Signature · 1 line
2.17.1
reviewer Acked-by on v2 13/39
#105 lore raw

Re: [PATCH v2 13/39] mm: Move VM_UFFD_MINOR_BIT from 37 to 38

In reply to Rick Edgecombe
Quoted 10 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

To introduce VM_SHADOW_STACK as VM_HIGH_ARCH_BIT (37), and make all VM_HIGH_ARCH_BITs stay together, move VM_UFFD_MINOR_BIT from 37 to 38.

Signature · 1 line
Peter Xu
#106 lore raw

PATCH v2 14/39 mm: Introduce VM_SHADOW_STACK for shadow stack memory

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.

---
 Documentation/filesystems/proc.rst | 1 +
 arch/x86/mm/mmap.c                 | 2 ++
 fs/proc/task_mmu.c                 | 3 +++
 include/linux/mm.h                 | 8 ++++++++
 4 files changed, 14 insertions(+)
Documentation/filesystems/proc.rst +1 -0
index e7aafc82be99..d54ff397947a 100644
@@ -560,6 +560,7 @@ encoded manner. The codes are the following:
     mt    arm64 MTE allocation tags are enabled
     um    userfaultfd missing tracking
     uw    userfaultfd wr-protect tracking
+    ss    shadow stack page
     ==    =======================================
 
 Note that there is no guarantee that every flag and associated mnemonic will
arch/x86/mm/mmap.c +2 -0
index c90c20904a60..f3f52c5e2fd6 100644
@@ -165,6 +165,8 @@ unsigned long get_mmap_base(int is_legacy)
 
 const char *arch_vma_name(struct vm_area_struct *vma)
 {
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return "[shadow stack]";
 	return NULL;
 }
 
fs/proc/task_mmu.c +3 -0
index 4e0023643f8b..a20899392c8d 100644
@@ -700,6 +700,9 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma)
 #ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
 		[ilog2(VM_UFFD_MINOR)]	= "ui",
 #endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */
+#ifdef CONFIG_ARCH_HAS_SHADOW_STACK
+		[ilog2(VM_SHADOW_STACK)] = "ss",
+#endif
 	};
 	size_t i;
 
include/linux/mm.h +8 -0
index be80fc827212..8cd413c5a329 100644
@@ -314,11 +314,13 @@ extern unsigned int kobjsize(const void *objp);
 #define VM_HIGH_ARCH_BIT_2	34	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_3	35	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_BIT_4	36	/* bit only usable on 64-bit architectures */
+#define VM_HIGH_ARCH_BIT_5	37	/* bit only usable on 64-bit architectures */
 #define VM_HIGH_ARCH_0	BIT(VM_HIGH_ARCH_BIT_0)
 #define VM_HIGH_ARCH_1	BIT(VM_HIGH_ARCH_BIT_1)
 #define VM_HIGH_ARCH_2	BIT(VM_HIGH_ARCH_BIT_2)
 #define VM_HIGH_ARCH_3	BIT(VM_HIGH_ARCH_BIT_3)
 #define VM_HIGH_ARCH_4	BIT(VM_HIGH_ARCH_BIT_4)
+#define VM_HIGH_ARCH_5	BIT(VM_HIGH_ARCH_BIT_5)
 #endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */
 
 #ifdef CONFIG_ARCH_HAS_PKEYS
@@ -334,6 +336,12 @@ extern unsigned int kobjsize(const void *objp);
 #endif
 #endif /* CONFIG_ARCH_HAS_PKEYS */
 
+#ifdef CONFIG_X86_SHADOW_STACK
+# define VM_SHADOW_STACK	VM_HIGH_ARCH_5
+#else
+# define VM_SHADOW_STACK	VM_NONE
+#endif
+
 #if defined(CONFIG_X86)
 # define VM_PAT		VM_ARCH_1	/* PAT reserves whole VMA at once (x86) */
 #elif defined(CONFIG_PPC)
Signature · 1 line
2.17.1
#107 lore raw

Re: [PATCH v2 14/39] mm: Introduce VM_SHADOW_STACK for shadow stack memory

Quoted 35 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.

---
 Documentation/filesystems/proc.rst | 1 +
 arch/x86/mm/mmap.c                 | 2 ++
 fs/proc/task_mmu.c                 | 3 +++
 include/linux/mm.h                 | 8 ++++++++
 4 files changed, 14 insertions(+)
Documentation/filesystems/proc.rst +1 -0
index e7aafc82be99..d54ff397947a 100644
@@ -560,6 +560,7 @@ encoded manner. The codes are the following:
     mt    arm64 MTE allocation tags are enabled
     um    userfaultfd missing tracking
     uw    userfaultfd wr-protect tracking
+    ss    shadow stack page
     ==    =======================================
 
 Note that there is no guarantee that every flag and associated mnemonic will
arch/x86/mm/mmap.c +0 -0
index c90c20904a60..f3f52c5e2fd6 100644

arch/x86/mm/mmap.c unsigned long get_mmap_base(int is_legacy) · line 172

Quoted 8 lines
arch/x86/mm/mmap.c +2 -0
@@ -165,6 +165,8 @@ unsigned long get_mmap_base(int is_legacy)
 
 const char *arch_vma_name(struct vm_area_struct *vma)
 {
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return "[shadow stack]";
 	return NULL;
 }
 

But why here?

CONFIG_ARCH_HAS_SHADOW_STACK implies that there will be more than one arch that supports shadow stack. The name has to come from generic code too, no?

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#108 lore raw

Re: [PATCH v2 14/39] mm: Introduce VM_SHADOW_STACK for shadow stack memory

Quoted 12 lines · Kirill A . Shutemov

arch/x86/mm/mmap.c unsigned long get_mmap_base(int is_legacy) · line 172

Quoted 9 lines
arch/x86/mm/mmap.c +2 -0
@@ -165,6 +165,8 @@ unsigned long get_mmap_base(int is_legacy)
 
  const char *arch_vma_name(struct vm_area_struct *vma)
  {
+     if (vma->vm_flags & VM_SHADOW_STACK)
+             return "[shadow stack]";
       return NULL;
  }
 

But why here?

Quoted 4 lines

CONFIG_ARCH_HAS_SHADOW_STACK implies that there will be more than one arch
that supports shadow stack. The name has to come from generic code too, no?

I'm not aware of any other arch that will, so I wonder if I should just remove ARCH_HAS_SHADOW_STACK actually.

#109 lore raw

Re: [PATCH v2 14/39] mm: Introduce VM_SHADOW_STACK for shadow stack memory

In reply to Rick Edgecombe
Quoted 35 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

A shadow stack PTE must be read-only and have _PAGE_DIRTY set. However, read-only and Dirty PTEs also exist for copy-on-write (COW) pages. These two cases are handled differently for page faults. Introduce VM_SHADOW_STACK to track shadow stack VMAs.

---
 Documentation/filesystems/proc.rst | 1 +
 arch/x86/mm/mmap.c                 | 2 ++
 fs/proc/task_mmu.c                 | 3 +++
 include/linux/mm.h                 | 8 ++++++++
 4 files changed, 14 insertions(+)
Documentation/filesystems/proc.rst +1 -0
index e7aafc82be99..d54ff397947a 100644
@@ -560,6 +560,7 @@ encoded manner. The codes are the following:
     mt    arm64 MTE allocation tags are enabled
     um    userfaultfd missing tracking
     uw    userfaultfd wr-protect tracking
+    ss    shadow stack page
     ==    =======================================
 
 Note that there is no guarantee that every flag and associated mnemonic will
arch/x86/mm/mmap.c +0 -0
index c90c20904a60..f3f52c5e2fd6 100644

arch/x86/mm/mmap.c unsigned long get_mmap_base(int is_legacy) · line 171

Quoted 7 lines
arch/x86/mm/mmap.c +2 -0
@@ -165,6 +165,8 @@ unsigned long get_mmap_base(int is_legacy)
 
 const char *arch_vma_name(struct vm_area_struct *vma)
 {
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return "[shadow stack]";
 	return NULL;
 }

I agree with Kirill: this should be in the arch-agnostic code.

Signature · 1 line
Kees Cook
#110 lore raw

PATCH v2 15/39 x86/mm: Check Shadow Stack page fault errors

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).

Shadow stacks accesses to shadow-stack mappings can see faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.

The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.

This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.

Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.

---

v2:
- Update commit log with verbiage/feedback from Dave Hansen
- Clarify reasoning for FAULT_FLAG_WRITE for all shadow stack accesses
- Update comments with some verbiage from Dave Hansen

Yu-cheng v30:
- Update Subject line and add a verb

 arch/x86/include/asm/trap_pf.h |  2 ++
 arch/x86/mm/fault.c            | 21 +++++++++++++++++++++
 2 files changed, 23 insertions(+)
arch/x86/include/asm/trap_pf.h +2 -0
index 10b1de500ab1..afa524325e55 100644
@@ -11,6 +11,7 @@
  *   bit 3 ==				1: use of reserved bit detected
  *   bit 4 ==				1: fault was an instruction fetch
  *   bit 5 ==				1: protection keys block access
+ *   bit 6 ==				1: shadow stack access fault
  *   bit 15 ==				1: SGX MMU page-fault
  */
 enum x86_pf_error_code {
@@ -20,6 +21,7 @@ enum x86_pf_error_code {
 	X86_PF_RSVD	=		1 << 3,
 	X86_PF_INSTR	=		1 << 4,
 	X86_PF_PK	=		1 << 5,
+	X86_PF_SHSTK	=		1 << 6,
 	X86_PF_SGX	=		1 << 15,
 };
 
arch/x86/mm/fault.c +21 -0
index fa71a5d12e87..e5697b393069 100644
@@ -1107,8 +1107,22 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)
 				       (error_code & X86_PF_INSTR), foreign))
 		return 1;
 
+	/*
+	 * Shadow stack accesses (PF_SHSTK=1) are only permitted to
+	 * shadow stack VMAs. All other accesses result in an error.
+	 */
+	if (error_code & X86_PF_SHSTK) {
+		if (unlikely(!(vma->vm_flags & VM_SHADOW_STACK)))
+			return 1;
+		if (unlikely(!(vma->vm_flags & VM_WRITE)))
+			return 1;
+		return 0;
+	}
+
 	if (error_code & X86_PF_WRITE) {
 		/* write, present and write, not present: */
+		if (unlikely(vma->vm_flags & VM_SHADOW_STACK))
+			return 1;
 		if (unlikely(!(vma->vm_flags & VM_WRITE)))
 			return 1;
 		return 0;
@@ -1300,6 +1314,13 @@ void do_user_addr_fault(struct pt_regs *regs,
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
 
+	/*
+	 * In order to fullfull a shadow stack access, the page needs
+	 * to be made (shadow stack) writable. So treat all shadow stack
+	 * accesses as writes.
+	 */
+	if (error_code & X86_PF_SHSTK)
+		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_WRITE)
 		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_INSTR)
Signature · 1 line
2.17.1
Reviewed-by on v2 15/39
#111 lore raw

Re: [PATCH v2 15/39] x86/mm: Check Shadow Stack page fault errors

Quoted 23 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The CPU performs "shadow stack accesses" when it expects to encounter shadow stack mappings. These accesses can be implicit (via CALL/RET instructions) or explicit (instructions like WRSS).

Shadow stacks accesses to shadow-stack mappings can see faults in normal, valid operation just like regular accesses to regular mappings. Shadow stacks need some of the same features like delayed allocation, swap and copy-on-write. The kernel needs to use faults to implement those features.

The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.

This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.

Quoted 5 lines

Shadow stack accesses can also result in errors, such as when a shadow stack overflows, or if a shadow stack access occurs to a non-shadow-stack mapping. Also, generate the errors for invalid shadow stack accesses.

Signature · 1 line
Kees Cook
maintainer
#112 lore raw

Re: [PATCH v2 15/39] x86/mm: Check Shadow Stack page fault errors

In reply to Rick Edgecombe
Quoted 11 lines · Rick Edgecombe

The architecture has concepts of both shadow stack reads and shadow stack writes. Any shadow stack access to non-shadow stack memory will generate a fault with the shadow stack error code bit set.

This means that, unlike normal write protection, the fault handler needs to create a type of memory that can be written to (with instructions that generate shadow stack writes), even to fulfill a read access. So in the case of COW memory, the COW needs to take place even with a shadow stack read. Otherwise the page will be left (shadow stack) writable in userspace. So to trigger the appropriate behavior, set FAULT_FLAG_WRITE for shadow stack accesses, even if the access was a shadow stack read.

That ^ should be moved into the comment below

arch/x86/mm/fault.c void do_user_addr_fault(struct pt_regs *regs, · line 1321

Quoted 11 lines

- Clarify reasoning for FAULT_FLAG_WRITE for all shadow stack accesses

arch/x86/mm/fault.c +5 -0
@@ -1300,6 +1314,13 @@ void do_user_addr_fault(struct pt_regs *regs,
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
 
+	/*
+	 * In order to fullfull a shadow stack access, the page needs
+	 * to be made (shadow stack) writable. So treat all shadow stack
+	 * accesses as writes.
+	 */

Because that's impenetrable.

Quoted 8 lines
arch/x86/mm/fault.c +2 -0
+	if (error_code & X86_PF_SHSTK)
+		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_WRITE)
 		flags |= FAULT_FLAG_WRITE;
 	if (error_code & X86_PF_INSTR)
Signature · 1 line
2.17.1
#113 lore raw

Re: [PATCH v2 15/39] x86/mm: Check Shadow Stack page fault errors

Quoted 1 line · Peter Zijlstra

That ^ should be moved into the comment below

Ok, good idea. Thanks.

#114 lore raw

PATCH v2 16/39 x86/mm: Update maybe_mkwrite() for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When serving a page fault, maybe_mkwrite() makes a PTE writable if there is a write access to it, and its vma has VM_WRITE. Shadow stack accesses to shadow stack vma's are also treated as write accesses by the fault handler. This is because setting shadow stack memory makes it writable via some instructions, so COW has to happen even for shadow stack reads.

So maybe_mkwrite() should continue to set VM_WRITE vma's as normally writable, but also set VM_WRITE|VM_SHADOW_STACK vma's as shadow stack.

Do this by adding a pte_mkwrite_shstk() and a cross-arch stub. Check for VM_SHADOW_STACK in maybe_mkwrite() and call pte_mkwrite_shstk() accordingly.

Apply the same changes to maybe_pmd_mkwrite().

---

v2:
- Change to handle shadow stacks that are VM_WRITE|VM_SHADOW_STACK
- Ditch arch specific maybe_mkwrite(), and make the code generic

Yu-cheng v29:
- Remove likely()'s.

 arch/x86/include/asm/pgtable.h |  2 ++
 include/linux/mm.h             | 14 +++++++++++++-
 include/linux/pgtable.h        | 14 ++++++++++++++
 mm/huge_memory.c               |  9 ++++++++-
 mm/memory.c                    |  3 +--
 5 files changed, 38 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgtable.h +2 -0
index 58c7bf9d7392..7a769c4dbc1c 100644
@@ -419,6 +419,7 @@ static inline pte_t pte_mkdirty(pte_t pte)
 	return pte_set_flags(pte, dirty | _PAGE_SOFT_DIRTY);
 }
 
+#define pte_mkwrite_shstk pte_mkwrite_shstk
 static inline pte_t pte_mkwrite_shstk(pte_t pte)
 {
 	/* pte_clear_cow() also sets Dirty=1 */
@@ -555,6 +556,7 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)
 	return pmd_set_flags(pmd, dirty | _PAGE_SOFT_DIRTY);
 }
 
+#define pmd_mkwrite_shstk pmd_mkwrite_shstk
 static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd)
 {
 	return pmd_clear_cow(pmd);
include/linux/mm.h +13 -1
index 8cd413c5a329..fef14ab3abcb 100644
@@ -981,13 +981,25 @@ void free_compound_page(struct page *page);
  * servicing faults for write access.  In the normal case, do always want
  * pte_mkwrite.  But get_user_pages can cause write faults for mappings
  * that do not have writing enabled, when used by access_process_vm.
+ *
+ * If a vma is shadow stack (a type of writable memory), mark the pte shadow
+ * stack.
  */
+#ifndef maybe_mkwrite
 static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	if (likely(vma->vm_flags & VM_WRITE))
+	if (!(vma->vm_flags & VM_WRITE))
+		goto out;
+
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		pte = pte_mkwrite_shstk(pte);
+	else
 		pte = pte_mkwrite(pte);
+
+out:
 	return pte;
 }
+#endif
 
 vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page);
 void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr);
include/linux/pgtable.h +14 -0
index 014ee8f0fbaa..21115b4895ca 100644
@@ -480,6 +480,13 @@ static inline pte_t pte_sw_mkyoung(pte_t pte)
 #define pte_mk_savedwrite pte_mkwrite
 #endif
 
+#ifndef pte_mkwrite_shstk
+static inline pte_t pte_mkwrite_shstk(pte_t pte)
+{
+	return pte;
+}
+#endif
+
 #ifndef pte_clear_savedwrite
 #define pte_clear_savedwrite pte_wrprotect
 #endif
@@ -488,6 +495,13 @@ static inline pte_t pte_sw_mkyoung(pte_t pte)
 #define pmd_savedwrite pmd_write
 #endif
 
+#ifndef pmd_mkwrite_shstk
+static inline pmd_t pmd_mkwrite_shstk(pmd_t pmd)
+{
+	return pmd;
+}
+#endif
+
 #ifndef pmd_mk_savedwrite
 #define pmd_mk_savedwrite pmd_mkwrite
 #endif
mm/huge_memory.c +8 -1
index e9414ee57c5b..11fc69eb4717 100644
@@ -554,8 +554,15 @@ __setup("transparent_hugepage=", setup_transparent_hugepage);
 
 pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
 {
-	if (likely(vma->vm_flags & VM_WRITE))
+	if (!(vma->vm_flags & VM_WRITE))
+		goto out;
+
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		pmd = pmd_mkwrite_shstk(pmd);
+	else
 		pmd = pmd_mkwrite(pmd);
+
+out:
 	return pmd;
 }
 
mm/memory.c +1 -2
index 4ba73f5aa8bb..6e8379f6793c 100644
@@ -4098,8 +4098,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
 
 	entry = mk_pte(page, vma->vm_page_prot);
 	entry = pte_sw_mkyoung(entry);
-	if (vma->vm_flags & VM_WRITE)
-		entry = pte_mkwrite(pte_mkdirty(entry));
+	entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 
 	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
 			&vmf->ptl);
Signature · 1 line
2.17.1
Reviewed-by on v2 16/39
#115 lore raw

Re: [PATCH v2 16/39] x86/mm: Update maybe_mkwrite() for shadow stack

Quoted 11 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When serving a page fault, maybe_mkwrite() makes a PTE writable if there is a write access to it, and its vma has VM_WRITE. Shadow stack accesses to shadow stack vma's are also treated as write accesses by the fault handler. This is because setting shadow stack memory makes it writable via some instructions, so COW has to happen even for shadow stack reads.

So maybe_mkwrite() should continue to set VM_WRITE vma's as normally writable, but also set VM_WRITE|VM_SHADOW_STACK vma's as shadow stack.

Quoted 7 lines

Do this by adding a pte_mkwrite_shstk() and a cross-arch stub. Check for VM_SHADOW_STACK in maybe_mkwrite() and call pte_mkwrite_shstk() accordingly.

Apply the same changes to maybe_pmd_mkwrite().

Signature · 1 line
Kees Cook
#116 lore raw

Re: [PATCH v2 16/39] x86/mm: Update maybe_mkwrite() for shadow stack

In reply to Rick Edgecombe
Quoted 18 lines · Rick Edgecombe
include/linux/mm.h +6 -1
index 8cd413c5a329..fef14ab3abcb 100644
@@ -981,13 +981,25 @@ void free_compound_page(struct page *page);
  * servicing faults for write access.  In the normal case, do always want
  * pte_mkwrite.  But get_user_pages can cause write faults for mappings
  * that do not have writing enabled, when used by access_process_vm.
+ *
+ * If a vma is shadow stack (a type of writable memory), mark the pte shadow
+ * stack.
  */
+#ifndef maybe_mkwrite
 static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	if (likely(vma->vm_flags & VM_WRITE))
+	if (!(vma->vm_flags & VM_WRITE))
+		goto out;

include/linux/mm.h void free_compound_page(struct page *page); · line 1002

Quoted 10 lines
include/linux/mm.h +7 -0
+
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		pte = pte_mkwrite_shstk(pte);
+	else
 		pte = pte_mkwrite(pte);
+
+out:
 	return pte;
 }
+#endif

Maybe take opportunity to move it to <linux/pgtable.h>? It is really not a place for the helper.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
maintainer
#117 lore raw

Re: [PATCH v2 16/39] x86/mm: Update maybe_mkwrite() for shadow stack

In reply to Rick Edgecombe
Quoted 18 lines · Rick Edgecombe
include/linux/mm.h +6 -1
index 8cd413c5a329..fef14ab3abcb 100644
@@ -981,13 +981,25 @@ void free_compound_page(struct page *page);
  * servicing faults for write access.  In the normal case, do always want
  * pte_mkwrite.  But get_user_pages can cause write faults for mappings
  * that do not have writing enabled, when used by access_process_vm.
+ *
+ * If a vma is shadow stack (a type of writable memory), mark the pte shadow
+ * stack.
  */
+#ifndef maybe_mkwrite
 static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
 {
-	if (likely(vma->vm_flags & VM_WRITE))
+	if (!(vma->vm_flags & VM_WRITE))
+		goto out;

include/linux/mm.h void free_compound_page(struct page *page); · line 1002

Quoted 10 lines
include/linux/mm.h +7 -0
+
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		pte = pte_mkwrite_shstk(pte);
+	else
 		pte = pte_mkwrite(pte);
+
+out:
 	return pte;
 }
+#endif

Why the #ifndef guard? There is no other implementation, nor does this patch introduce one.

Also, wouldn't it be simpler to write it like:

static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)

{
	if (!(vma->vm_flags & VM_WRITE))
		return pte;

	if (vma->vm_flags & VM_SHADOW_STACK)
		return pte_mkwrite_shstk(pte);

	return pte_mkwrite(pte);
}

? (idem for the pmd version etc..)

#118 lore raw

Re: [PATCH v2 16/39] x86/mm: Update maybe_mkwrite() for shadow stack

Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Rick Edgecombe
include/linux/mm.h +0 -0
index 8cd413c5a329..fef14ab3abcb 100644
@@ -981,13 +981,25 @@ void free_compound_page(struct page *page);
   * servicing faults for write access.  In the normal case, do

always want
* pte_mkwrite. But get_user_pages can cause write faults for
mappings
* that do not have writing enabled, when used by
access_process_vm.
+ *
+ * If a vma is shadow stack (a type of writable memory), mark the pte shadow
+ * stack.
*/
+#ifndef maybe_mkwrite
static inline pte_t maybe_mkwrite(pte_t pte, struct
vm_area_struct *vma)
{

include/linux/mm.h +7 -1
-     if (likely(vma->vm_flags & VM_WRITE))
+     if (!(vma->vm_flags & VM_WRITE))
+             goto out;
+
+     if (vma->vm_flags & VM_SHADOW_STACK)
+             pte = pte_mkwrite_shstk(pte);
+     else
               pte = pte_mkwrite(pte);
+
Quoted 8 lines

include/linux/mm.h void free_compound_page(struct page *page); · line 1002

Quoted 4 lines
include/linux/mm.h +2 -0
+out:
       return pte;
  }
+#endif

Why the #ifndef guard? There is no other implementation, nor does this
patch introduce one.

Oh yea, this series used to add another one, but I forgot to remove the guards. Thanks.

Quoted 5 lines

Also, wouldn't it be simpler to write it like:

static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)

Quoted 9 lines
{
        if (!(vma->vm_flags & VM_WRITE))
                return pte;

        if (vma->vm_flags & VM_SHADOW_STACK)
                return pte_mkwrite_shstk(pte);

        return pte_mkwrite(pte);
}

Yep, that looks better. Thanks.

#119 lore raw

PATCH v2 17/39 mm: Fixup places that call pte_mkwrite() directly

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.

In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite() or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places where pte_mkwrite() is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.

- do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(), which is the same as maybe_mkwrite() in logic and intention. Just change them to maybe_mkwrite().

- When userfaultfd is creating a PTE after userspace handles the fault it calls pte_mkwrite() directly. Teach it about pte_mkwrite_shstk()

In other cases where pte_mkwrite() is called directly, the VMA will not be VM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case of pte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization in mprotect(), shadow stack VMA's won't be VM_SHARED, so it is not nessary.

---

v2:
- Updated commit log with comment's from Dave Hansen
- Dave also suggested (I understood) to maybe tweak vm_get_page_prot() to avoid having to call maybe_mkwrite(). After playing around with this I opted to *not* do this. Shadow stack memory memory is effectively writable, so having the default permissions be writable ended up mapping the zero page as writable and other surprises. So creating shadow stack memory needs to be done with manual logic like pte_mkwrite().
- Drop change in change_pte_range() because it couldn't actually trigger for shadow stack VMAs.
- Clarify reasoning for skipped cases of pte_mkwrite().

Yu-cheng v25:
- Apply same changes to do_huge_pmd_numa_page() as to do_numa_page().

 mm/migrate_device.c |  3 +--
 mm/userfaultfd.c    | 10 +++++++---
 2 files changed, 8 insertions(+), 5 deletions(-)
mm/migrate_device.c +1 -2
index 27fb37d65476..eba3164736b3 100644
@@ -606,8 +606,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
 			goto abort;
 		}
 		entry = mk_pte(page, vma->vm_page_prot);
-		if (vma->vm_flags & VM_WRITE)
-			entry = pte_mkwrite(pte_mkdirty(entry));
+		entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 	}
 
 	ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl);
mm/userfaultfd.c +7 -3
index 7327b2573f7c..b49372c7de41 100644
@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
 	int ret;
 	pte_t _dst_pte, *dst_pte;
 	bool writable = dst_vma->vm_flags & VM_WRITE;
+	bool shstk = dst_vma->vm_flags & VM_SHADOW_STACK;
 	bool vm_shared = dst_vma->vm_flags & VM_SHARED;
 	bool page_in_cache = page->mapping;
 	spinlock_t *ptl;
@@ -83,9 +84,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
 		writable = false;
 	}
 
-	if (writable)
-		_dst_pte = pte_mkwrite(_dst_pte);
-	else
+	if (writable) {
+		if (shstk)
+			_dst_pte = pte_mkwrite_shstk(_dst_pte);
+		else
+			_dst_pte = pte_mkwrite(_dst_pte);
+	} else
 		/*
 		 * We need this to make sure write bit removed; as mk_pte()
 		 * could return a pte with write bit set.
Signature · 1 line
2.17.1
Reviewed-by on v2 17/39
#120 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

Quoted 18 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.

In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite() or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places where pte_mkwrite() is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.

- do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(), which is the same as maybe_mkwrite() in logic and intention. Just change them to maybe_mkwrite().

- When userfaultfd is creating a PTE after userspace handles the fault it calls pte_mkwrite() directly. Teach it about pte_mkwrite_shstk()

Quoted 7 lines

In other cases where pte_mkwrite() is called directly, the VMA will not be VM_SHADOW_STACK, and so shadow stack memory should not be created.
- In the case of pte_savedwrite(), shadow stack VMA's are excluded.
- In the case of the "dirty_accountable" optimization in mprotect(), shadow stack VMA's won't be VM_SHARED, so it is not nessary.

Signature · 1 line
Kees Cook
#121 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

In reply to Rick Edgecombe
Quoted 11 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

With the introduction of shadow stack memory there are two ways a pte can be writable: regular writable memory and shadow stack memory.

In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite() or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases where a PTE is made writable. However, there are places where pte_mkwrite() is called directly and the logic should now also create a shadow stack PTE in the case of a shadow stack VMA.

Quoted 3 lines

- do_anonymous_page() and migrate_vma_insert_page() check VM_WRITE directly and call pte_mkwrite(), which is the same as maybe_mkwrite() in logic and intention. Just change them to maybe_mkwrite().

Looks like you folded change for do_anonymous_page() into the wrong patch. I see the relevant change in the previous patch.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#122 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

Quoted 23 lines · Kirill A . Shutemov
Quoted 16 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

With the introduction of shadow stack memory there are two ways a pte can
be writable: regular writable memory and shadow stack memory.

In past patches, maybe_mkwrite() has been updated to apply pte_mkwrite()
or pte_mkwrite_shstk() depending on the VMA flag. This covers most cases
where a PTE is made writable. However, there are places where pte_mkwrite()
is called directly and the logic should now also create a shadow stack PTE
in the case of a shadow stack VMA.

Quoted 5 lines

- do_anonymous_page() and migrate_vma_insert_page() check
VM_WRITE
directly and call pte_mkwrite(), which is the same as
maybe_mkwrite()
in logic and intention. Just change them to maybe_mkwrite().

Quoted 3 lines

Looks like you folded change for do_anonymous_page() into the wrong patch.
I see the relevant change in the previous patch.

Arg, yep thanks. It got moved accidentally.

#123 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

In reply to Rick Edgecombe

Hopefully I will not waste your time again… If it has been discussed in the last 26 iterations, just tell me and ignore.

mm/migrate_device.c static void migrate_vma_insert_page(struct migrate_vma *migrate, · line 610

Quoted 10 lines · Rick Edgecombe
mm/migrate_device.c +1 -2
@@ -606,8 +606,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
 			goto abort;
 		}
 		entry = mk_pte(page, vma->vm_page_prot);
-		if (vma->vm_flags & VM_WRITE)
-			entry = pte_mkwrite(pte_mkdirty(entry));
+		entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 	}

This is not exactly the same logic. You might dirty read-only pages since you call pte_mkdirty() unconditionally. It has been known not to be very robust (e.g., dirty-COW and friends). Perhaps it is not dangerous following some recent enhancements, but why do you want to take the risk?

Instead, although it might seem redundant, the compiler will hopefully would make it efficient:

		if (vma->vm_flags & VM_WRITE) {
			entry = pte_mkdirty(entry);
			entry = maybe_mkwrite(entry, vma);
		}
#124 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

Quoted 19 lines · Nadav Amit

Hopefully I will not waste your time again… If it has been discussed in the
last 26 iterations, just tell me and ignore.

mm/migrate_device.c static void migrate_vma_insert_page(struct · line 611

Quoted 11 lines · Rick Edgecombe
mm/migrate_device.c +1 -2
@@ -606,8 +606,7 @@ static void migrate_vma_insert_page(struct
 migrate_vma *migrate,
 			goto abort;
 		}
 		entry = mk_pte(page, vma->vm_page_prot);
-		if (vma->vm_flags & VM_WRITE)
-			entry = pte_mkwrite(pte_mkdirty(entry));
+		entry = maybe_mkwrite(pte_mkdirty(entry), vma);
 	}
Quoted 7 lines

This is not exactly the same logic. You might dirty read-only pages since
you call pte_mkdirty() unconditionally. It has been known not to be very
robust (e.g., dirty-COW and friends). Perhaps it is not dangerous following
some recent enhancements, but why do you want to take the risk?

Yea those changes let me drop a patch. But, it's a good point.

Quoted 10 lines

Instead, although it might seem redundant, the compiler will hopefully would
make it efficient:

		if (vma->vm_flags & VM_WRITE) {
			entry = pte_mkdirty(entry);
			entry = maybe_mkwrite(entry, vma);
		}

Thanks Nadav. I think you're right, it should have the open coded logic here and in the do_anonymous_page() chunk that got moved to the previous patch on accident.

maintainer
#125 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

In reply to Rick Edgecombe
Quoted 18 lines · Rick Edgecombe
mm/userfaultfd.c +1 -2
index 7327b2573f7c..b49372c7de41 100644
@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
 	int ret;
 	pte_t _dst_pte, *dst_pte;
 	bool writable = dst_vma->vm_flags & VM_WRITE;
+	bool shstk = dst_vma->vm_flags & VM_SHADOW_STACK;
 	bool vm_shared = dst_vma->vm_flags & VM_SHARED;
 	bool page_in_cache = page->mapping;
 	spinlock_t *ptl;
@@ -83,9 +84,12 @@ int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd,
 		writable = false;
 	}
 
-	if (writable)
-		_dst_pte = pte_mkwrite(_dst_pte);

mm/userfaultfd.c int mfill_atomic_install_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, · line 95

Quoted 10 lines
mm/userfaultfd.c +6 -1
-	else
+	if (writable) {
+		if (shstk)
+			_dst_pte = pte_mkwrite_shstk(_dst_pte);
+		else
+			_dst_pte = pte_mkwrite(_dst_pte);
+	} else
 		/*
 		 * We need this to make sure write bit removed; as mk_pte()
 		 * could return a pte with write bit set.

Urgh.. that's unfortunate. But yeah, I don't see a way to make that pretty either.

#126 lore raw

Re: [PATCH v2 17/39] mm: Fixup places that call pte_mkwrite() directly

Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Rick Edgecombe
mm/userfaultfd.c +1 -0
index 7327b2573f7c..b49372c7de41 100644
@@ -63,6 +63,7 @@ int mfill_atomic_install_pte(struct mm_struct
 *dst_mm, pmd_t *dst_pmd,
       int ret;
       pte_t _dst_pte, *dst_pte;
       bool writable = dst_vma->vm_flags & VM_WRITE;
+     bool shstk = dst_vma->vm_flags & VM_SHADOW_STACK;
       bool vm_shared = dst_vma->vm_flags & VM_SHARED;
       bool page_in_cache = page->mapping;
       spinlock_t *ptl;
mm/userfaultfd.c +0 -0
@@ -83,9 +84,12 @@ int mfill_atomic_install_pte(struct mm_struct
 *dst_mm, pmd_t *dst_pmd,
               writable = false;
       }
 
Quoted 16 lines
Quoted 11 lines
mm/userfaultfd.c +6 -3
-     if (writable)
-             _dst_pte = pte_mkwrite(_dst_pte);
-     else
+     if (writable) {
+             if (shstk)
+                     _dst_pte = pte_mkwrite_shstk(_dst_pte);
+             else
+                     _dst_pte = pte_mkwrite(_dst_pte);
+     } else
               /*
                * We need this to make sure write bit removed; as
Quoted 2 lines

mk_pte()
* could return a pte with write bit set.

Urgh.. that's unfortunate. But yeah, I don't see a way to make that pretty either.

Nadav pointed out that:
entry = maybe_mkwrite(pte_mkdirty(entry), vma);

and:
if (vma->vm_flags & VM_WRITE)

	entry = pte_mkwrite(pte_mkdirty(entry));

Are not actually the same, because in the former the non-writable PTE gets marked dirty. So I was actually going to add two more cases like the ugly case.

#127 lore raw

PATCH v2 18/39 mm: Add guard pages around a shadow stack.

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

The architecture of shadow stack constrains the ability of userspace to move the shadow stack pointer (SSP) in order to prevent corrupting or switching to other shadow stacks. The RSTORSSP can move the spp to different shadow stacks, but it requires a specially placed token in order to do this. However, the architecture does not prevent incrementing the stack pointer to wander onto an adjacent shadow stack. To prevent this in software, enforce guard pages at the beginning of shadow stack vmas, such that there will always be a gap between adjacent shadow stacks.

Make the gap big enough so that no userspace SSP changing operations (besides RSTORSSP), can move the SSP from one stack to the next. The SSP can increment or decrement by CALL, RET and INCSSP. CALL and RET can move the SSP by a maximum of 8 bytes, at which point the shadow stack would be accessed.

The INCSSP instruction can also increment the shadow stack pointer. It is the shadow stack analog of an instruction like:

	addq    $0x80, %rsp

However, there is one important difference between an ADD on %rsp and INCSSP. In addition to modifying SSP, INCSSP also reads from the memory of the first and last elements that were "popped". It can be thought of as acting like this:

READ_ONCE(ssp); // read+discard top element on stack
ssp += nr_to_pop * 8; // move the shadow stack
READ_ONCE(ssp-8); // read+discard last popped stack element

The maximum distance INCSSP can move the SSP is 2040 bytes, before it would read the memory. Therefore a single page gap will be enough to prevent any operation from shifting the SSP to an adjacent stack, since it would have to land in the gap at least once, causing a fault.

This could be accomplished by using VM_GROWSDOWN, but this has a downside. The behavior would allow shadow stack's to grow, which is unneeded and adds a strange difference to how most regular stacks work.

---

v2:
- Use __weak instead of #ifdef (Dave Hansen)
- Only have start gap on shadow stack (Andy Luto)
- Create stack_guard_start_gap() to not duplicate code in an arch version of vm_start_gap() (Dave Hansen)
- Improve commit log partly with verbiage from (Dave Hansen)

Yu-cheng v25:
- Move SHADOW_STACK_GUARD_GAP to arch/x86/mm/mmap.c.

Yu-cheng v24:
- Instead changing vm_*_gap(), create x86-specific versions.

 arch/x86/mm/mmap.c | 23 +++++++++++++++++++++++
 include/linux/mm.h | 11 ++++++-----
 mm/mmap.c          |  7 +++++++
 3 files changed, 36 insertions(+), 5 deletions(-)
arch/x86/mm/mmap.c +23 -0
index f3f52c5e2fd6..b0427bd2da30 100644
@@ -250,3 +250,26 @@ bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
 		return false;
 	return true;
 }
+
+unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+
+	/*
+	 * Shadow stack pointer is moved by CALL, RET, and INCSSP(Q/D).
+	 * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB
+	 * (~1KB for INCSSPD) and touches the first and the last element
+	 * in the range, which triggers a page fault if the range is not
+	 * in a shadow stack. Because of this, creating 4-KB guard pages
+	 * around a shadow stack prevents these instructions from going
+	 * beyond.
+	 *
+	 * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+	 * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+	 */
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		return PAGE_SIZE;
+
+	return 0;
+}
include/linux/mm.h +6 -5
index fef14ab3abcb..09458e77bf52 100644
@@ -2775,15 +2775,16 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
 	return vma;
 }
 
+unsigned long stack_guard_start_gap(struct vm_area_struct *vma);
+
 static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
 {
+	unsigned long gap = stack_guard_start_gap(vma);
 	unsigned long vm_start = vma->vm_start;
 
-	if (vma->vm_flags & VM_GROWSDOWN) {
-		vm_start -= stack_guard_gap;
-		if (vm_start > vma->vm_start)
-			vm_start = 0;
-	}
+	vm_start -= gap;
+	if (vm_start > vma->vm_start)
+		vm_start = 0;
 	return vm_start;
 }
 
mm/mmap.c +7 -0
index 9d780f415be3..f0d2e9143bd0 100644
@@ -247,6 +247,13 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
 	return origbrk;
 }
 
+unsigned long __weak stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+	return 0;
+}
+
 static inline unsigned long vma_compute_gap(struct vm_area_struct *vma)
 {
 	unsigned long gap, prev_end;
Signature · 1 line
2.17.1
Reviewed-by on v2 18/39
#128 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

Quoted 8 lines · Rick Edgecombe

[...]

arch/x86/mm/mmap.c +7 -0
+unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+
+	/*
+	 * Shadow stack pointer is moved by CALL, RET, and INCSSP(Q/D).

arch/x86/mm/mmap.c bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot) · line 270

Quoted 10 lines
arch/x86/mm/mmap.c +10 -0
+	 * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB
+	 * (~1KB for INCSSPD) and touches the first and the last element
+	 * in the range, which triggers a page fault if the range is not
+	 * in a shadow stack. Because of this, creating 4-KB guard pages
+	 * around a shadow stack prevents these instructions from going
+	 * beyond.
+	 *
+	 * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+	 * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+	 */

Thank you for the details on how the size choice is made here! :)

Quoted 31 lines
include/linux/mm.h +6 -5
index fef14ab3abcb..09458e77bf52 100644
@@ -2775,15 +2775,16 @@ struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
 	return vma;
 }
 
+unsigned long stack_guard_start_gap(struct vm_area_struct *vma);
+
 static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
 {
+	unsigned long gap = stack_guard_start_gap(vma);
 	unsigned long vm_start = vma->vm_start;
 
-	if (vma->vm_flags & VM_GROWSDOWN) {
-		vm_start -= stack_guard_gap;
-		if (vm_start > vma->vm_start)
-			vm_start = 0;
-	}
+	vm_start -= gap;
+	if (vm_start > vma->vm_start)
+		vm_start = 0;
 	return vm_start;
 }
 
mm/mmap.c +0 -0
index 9d780f415be3..f0d2e9143bd0 100644

mm/mmap.c SYSCALL_DEFINE1(brk, unsigned long, brk) · line 249

Quoted 3 lines
mm/mmap.c +0 -0
@@ -247,6 +247,13 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
 	return origbrk;
 }
 

I feel like something could be done with this definitions to make them inline, instead of __weak:

#ifndef stack_guard_start_gap

mm/mmap.c SYSCALL_DEFINE1(brk, unsigned long, brk) · line 255

Quoted 6 lines
mm/mmap.c +6 -0
+unsigned long __weak stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+	return 0;
+}

#endif

And then move the x86 stack_guard_start_gap to a header?

It's not exactly fast-path, but it feels a little weird. Regardlesss:

Signature · 1 line
Kees Cook
#129 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

Quoted 12 lines · Kees Cook
Quoted 11 lines · Rick Edgecombe

[...]

arch/x86/mm/mmap.c +10 -0
+unsigned long stack_guard_start_gap(struct vm_area_struct *vma)
+{
+	if (vma->vm_flags & VM_GROWSDOWN)
+		return stack_guard_gap;
+
+	/*
+	 * Shadow stack pointer is moved by CALL, RET, and INCSSP(Q/D).
+	 * INCSSPQ moves shadow stack pointer up to 255 * 8 = ~2 KB
+	 * (~1KB for INCSSPD) and touches the first and the last element
+	 * in the range, which triggers a page fault if the range is not
Quoted 8 lines

arch/x86/mm/mmap.c bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot) · line 270

Quoted 7 lines
arch/x86/mm/mmap.c +7 -0
+	 * in a shadow stack. Because of this, creating 4-KB guard pages
+	 * around a shadow stack prevents these instructions from going
+	 * beyond.
+	 *
+	 * Creation of VM_SHADOW_STACK is tightly controlled, so a vma
+	 * can't be both VM_GROWSDOWN and VM_SHADOW_STACK
+	 */

Thank you for the details on how the size choice is made here! :)

(In case anyone is hankering for some premature optimisation...)

You don't actually need a hole to create a guard.  Any mapping of type != shstk will do.

If you've got a load of threads, you can tightly pack stack / shstk / stack / shstk with no holes, and they each act as each other guard pages.

~Andrew

#130 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

* Andrew Cooper:

Quoted 5 lines

You don't actually need a hole to create a guard.  Any mapping of type != shstk will do.

If you've got a load of threads, you can tightly pack stack / shstk / stack / shstk with no holes, and they each act as each other guard pages.

Can userspace read the shadow stack directly? Writing is obviously blocked, but reading?

GCC's stack-clash probing uses OR instructions, so it would be fine with a readable mapping. POSIX does not appear to require PROT_NONE mappings for the stack guard region, either. However, the pthread_attr_setguardsize manual page pretty clearly says that it's got to be unreadable and unwriteable. Hence my question.

Thanks,
Florian

#131 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

Quoted 9 lines · Florian Weimer

* Andrew Cooper:

Quoted 5 lines

You don't actually need a hole to create a guard.  Any mapping of type != shstk will do.

If you've got a load of threads, you can tightly pack stack / shstk / stack / shstk with no holes, and they each act as each other guard pages.

Can userspace read the shadow stack directly? Writing is obviously blocked, but reading?

Yes - regular reads are permitted to shstk memory.

It's actually a great way to get backtraces with no extra metadata needed.

Quoted 2 lines

GCC's stack-clash probing uses OR instructions, so it would be fine with a readable mapping.

It's or $0, (%rsp) which is a read/modify/write and will fault when hitting a shstk mapping.

Quoted 4 lines

POSIX does not appear to require PROT_NONE mappings
for the stack guard region, either. However, the pthread_attr_setguardsize manual page pretty clearly says that it's got to be unreadable and unwriteable. Hence my question.

Hmm.  If that's what the manuals say, then fine.

But honestly, you don't get very far at all without faulting on a read-only stack.

~Andrew

#132 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

* Andrew Cooper:

Quoted 8 lines · Florian Weimer
Quoted 7 lines

* Andrew Cooper:

Quoted 5 lines

You don't actually need a hole to create a guard.  Any mapping of type != shstk will do.

If you've got a load of threads, you can tightly pack stack / shstk / stack / shstk with no holes, and they each act as each other guard pages.

Quoted 7 lines
Quoted 2 lines

Can userspace read the shadow stack directly? Writing is obviously blocked, but reading?

Yes - regular reads are permitted to shstk memory.

It's actually a great way to get backtraces with no extra metadata needed.

Indeed, I hope shadow stacks can be used to put the discussion around frame pointers to a rest, at least when it comes to profiling. 8-)

Quoted 9 lines
Quoted 4 lines

POSIX does not appear to require PROT_NONE mappings
for the stack guard region, either. However, the pthread_attr_setguardsize manual page pretty clearly says that it's got to be unreadable and unwriteable. Hence my question.

Hmm.  If that's what the manuals say, then fine.

But honestly, you don't get very far at all without faulting on a read-only stack.

I guess we can update the manual page proactively. It does look like a tempting optimization.

Thanks,
Florian

#133 lore raw

Re: [PATCH v2 18/39] mm: Add guard pages around a shadow stack.

Quoted 6 lines · Florian Weimer

* Andrew Cooper:

Quoted 4 lines
Quoted 4 lines

POSIX does not appear to require PROT_NONE mappings
for the stack guard region, either. However, the pthread_attr_setguardsize manual page pretty clearly says that it's got to be unreadable and unwriteable. Hence my question.

Quoted 6 lines
Quoted 4 lines

Hmm.  If that's what the manuals say, then fine.

But honestly, you don't get very far at all without faulting on a read-only stack.

I guess we can update the manual page proactively. It does look like a tempting optimization.

Here's one I prepared earlier, discussing getting supervisor shadow stacks working in Xen.

http://xenbits.xen.org/people/andrewcoop/Xen-CET-SS.pdf

This optimisation turned out to be very helpful by being able to put the shadow stacks in what were previously the guard holes, meaning we didn't actually need to allocate any more memory for the stacks.

~Andrew

#134 lore raw

PATCH v2 19/39 mm/mmap: Add shadow stack pages to memory accounting

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Account shadow stack pages to stack memory.

---

v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.

Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.

Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().

 mm/mmap.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/mmap.c +5 -0
index f0d2e9143bd0..8569ef09614c 100644
@@ -1682,6 +1682,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags)
 	if (file && is_file_hugepages(file))
 		return 0;
 
+	if (vm_flags & VM_SHADOW_STACK)
+		return 1;
+
 	return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE;
 }
 
@@ -3289,6 +3292,8 @@ void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages)
 		mm->exec_vm += npages;
 	else if (is_stack_mapping(flags))
 		mm->stack_vm += npages;
+	else if (flags & VM_SHADOW_STACK)
+		mm->stack_vm += npages;
 	else if (is_data_mapping(flags))
 		mm->data_vm += npages;
 }
Signature · 1 line
2.17.1
#136 lore raw

Re: [PATCH v2 19/39] mm/mmap: Add shadow stack pages to memory accounting

In reply to Rick Edgecombe
Quoted 29 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Account shadow stack pages to stack memory.

---

v2:
- Remove is_shadow_stack_mapping() and just change it to directly bitwise and VM_SHADOW_STACK.

Yu-cheng v26:
- Remove redundant #ifdef CONFIG_MMU.

Yu-cheng v25:
- Remove #ifdef CONFIG_ARCH_HAS_SHADOW_STACK for is_shadow_stack_mapping().

 mm/mmap.c | 5 +++++
 1 file changed, 5 insertions(+)
mm/mmap.c +0 -0
index f0d2e9143bd0..8569ef09614c 100644

mm/mmap.c static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags) · line 1688

Quoted 7 lines
mm/mmap.c +3 -0
@@ -1682,6 +1682,9 @@ static inline int accountable_mapping(struct file *file, vm_flags_t vm_flags)
 	if (file && is_file_hugepages(file))
 		return 0;
 
+	if (vm_flags & VM_SHADOW_STACK)
+		return 1;
+
 	return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) == VM_WRITE;

Hm. Isn't the last check true for shadow stack too? IIUC, shadow stack has VM_WRITE set, so accountable_mapping() should work correctly as is.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#137 lore raw

Re: [PATCH v2 19/39] mm/mmap: Add shadow stack pages to memory accounting

Quoted 9 lines · Kirill A . Shutemov
Quoted 5 lines
mm/mmap.c +3 -0
+     if (vm_flags & VM_SHADOW_STACK)
+             return 1;
+
       return (vm_flags & (VM_NORESERVE | VM_SHARED | VM_WRITE)) ==

VM_WRITE;

Hm. Isn't the last check true for shadow stack too? IIUC, shadow stack has
VM_WRITE set, so accountable_mapping() should work correctly as is.

They are not always VM_WRITE, that can have it removed via mprotect(). But in that case it is just specially tagged read only memory, so probably isn't accountable. So, yea, I'll remove it. Thanks.

#138 lore raw

PATCH v2 20/39 mm/mprotect: Exclude shadow stack from preserve_write

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

In change_pte_range(), when a PTE is changed for prot_numa, _PAGE_RW is preserved to avoid the additional write fault after the NUMA hinting fault. However, pte_write() now includes both normal writable and shadow stack (Write=0, Dirty=1) PTEs, but the latter does not have _PAGE_RW and has no need to preserve it.

Exclude shadow stack from preserve_write test, and apply the same change to change_huge_pmd().

---

Yu-cheng v25:
- Move is_shadow_stack_mapping() to a separate line.

Yu-cheng v24:
- Change arch_shadow_stack_mapping() to is_shadow_stack_mapping().

 mm/huge_memory.c | 7 +++++++
 mm/mprotect.c    | 7 +++++++
 2 files changed, 14 insertions(+)
mm/huge_memory.c +7 -0
index 11fc69eb4717..492c4f190f55 100644
@@ -1800,6 +1800,13 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		return 0;
 
 	preserve_write = prot_numa && pmd_write(*pmd);
+
+	/*
+	 * Preserve only normal writable huge PMD, but not shadow
+	 * stack (RW=0, Dirty=1).
+	 */
+	if (vma->vm_flags & VM_SHADOW_STACK)
+		preserve_write = false;
 	ret = 1;
 
 #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
mm/mprotect.c +7 -0
index bc6bddd156ca..983206529dce 100644
@@ -114,6 +114,13 @@ static unsigned long change_pte_range(struct mmu_gather *tlb,
 			pte_t ptent;
 			bool preserve_write = prot_numa && pte_write(oldpte);
 
+			/*
+			 * Preserve only normal writable PTE, but not shadow
+			 * stack (RW=0, Dirty=1).
+			 */
+			if (vma->vm_flags & VM_SHADOW_STACK)
+				preserve_write = false;
+
 			/*
 			 * Avoid trapping faults against the zero or KSM
 			 * pages. See similar comment in change_huge_pmd.
Signature · 1 line
2.17.1
#139 lore raw

PATCH v2 21/39 mm: Re-introduce vm_flags to do_mmap()

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

There was no more caller passing vm_flags to do_mmap(), and vm_flags was removed from the function's input by:

commit 45e55300f114 ("mm: remove unnecessary wrapper function do_mmap_pgoff()").

There is a new user now. Shadow stack allocation passes VM_SHADOW_STACK to do_mmap(). Thus, re-introduce vm_flags to do_mmap().

---
 fs/aio.c           |  2 +-
 include/linux/mm.h |  3 ++-
 ipc/shm.c          |  2 +-
 mm/mmap.c          | 10 +++++-----
 mm/nommu.c         |  4 ++--
 mm/util.c          |  2 +-
 6 files changed, 12 insertions(+), 11 deletions(-)
fs/aio.c +1 -1
index 606613e9d1f4..a54b5ee72f1c 100644
@@ -554,7 +554,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)
 
 	ctx->mmap_base = do_mmap(ctx->aio_ring_file, 0, ctx->mmap_size,
 				 PROT_READ | PROT_WRITE,
-				 MAP_SHARED, 0, &unused, NULL);
+				 MAP_SHARED, 0, 0, &unused, NULL);
 	mmap_write_unlock(mm);
 	if (IS_ERR((void *)ctx->mmap_base)) {
 		ctx->mmap_size = 0;
include/linux/mm.h +2 -1
index 09458e77bf52..6aa0ffe3666c 100644
@@ -2667,7 +2667,8 @@ extern unsigned long mmap_region(struct file *file, unsigned long addr,
 	struct list_head *uf);
 extern unsigned long do_mmap(struct file *file, unsigned long addr,
 	unsigned long len, unsigned long prot, unsigned long flags,
-	unsigned long pgoff, unsigned long *populate, struct list_head *uf);
+	vm_flags_t vm_flags, unsigned long pgoff, unsigned long *populate,
+	struct list_head *uf);
 extern int __do_munmap(struct mm_struct *, unsigned long, size_t,
 		       struct list_head *uf, bool downgrade);
 extern int do_munmap(struct mm_struct *, unsigned long, size_t,
ipc/shm.c +1 -1
index b3048ebd5c31..f236b3e14ec4 100644
@@ -1646,7 +1646,7 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,
 			goto invalid;
 	}
 
-	addr = do_mmap(file, addr, size, prot, flags, 0, &populate, NULL);
+	addr = do_mmap(file, addr, size, prot, flags, 0, 0, &populate, NULL);
 	*raddr = addr;
 	err = 0;
 	if (IS_ERR_VALUE(addr))
mm/mmap.c +5 -5
index 8569ef09614c..e1006c41b1cc 100644
@@ -1375,11 +1375,11 @@ static inline bool file_mmap_ok(struct file *file, struct inode *inode,
  */
 unsigned long do_mmap(struct file *file, unsigned long addr,
 			unsigned long len, unsigned long prot,
-			unsigned long flags, unsigned long pgoff,
-			unsigned long *populate, struct list_head *uf)
+			unsigned long flags, vm_flags_t vm_flags,
+			unsigned long pgoff, unsigned long *populate,
+			struct list_head *uf)
 {
 	struct mm_struct *mm = current->mm;
-	vm_flags_t vm_flags;
 	int pkey = 0;
 
 	*populate = 0;
@@ -1439,7 +1439,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
 	 * to. we assume access permissions have been handled by the open
 	 * of the memory object, so we don't do any here.
 	 */
-	vm_flags = calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) |
+	vm_flags |= calc_vm_prot_bits(prot, pkey) | calc_vm_flag_bits(flags) |
 			mm->def_flags | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC;
 
 	if (flags & MAP_LOCKED)
@@ -2964,7 +2964,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
 
 	file = get_file(vma->vm_file);
 	ret = do_mmap(vma->vm_file, start, size,
-			prot, flags, pgoff, &populate, NULL);
+			prot, flags, 0, pgoff, &populate, NULL);
 	fput(file);
 out:
 	mmap_write_unlock(mm);
mm/nommu.c +2 -2
index e819cbc21b39..85b41107a192 100644
@@ -1059,6 +1059,7 @@ unsigned long do_mmap(struct file *file,
 			unsigned long len,
 			unsigned long prot,
 			unsigned long flags,
+			vm_flags_t vm_flags,
 			unsigned long pgoff,
 			unsigned long *populate,
 			struct list_head *uf)
@@ -1066,7 +1067,6 @@ unsigned long do_mmap(struct file *file,
 	struct vm_area_struct *vma;
 	struct vm_region *region;
 	struct rb_node *rb;
-	vm_flags_t vm_flags;
 	unsigned long capabilities, result;
 	int ret;
 
@@ -1085,7 +1085,7 @@ unsigned long do_mmap(struct file *file,
 
 	/* we've determined that we can make the mapping, now translate what we
 	 * now know into VMA flags */
-	vm_flags = determine_vm_flags(file, prot, flags, capabilities);
+	vm_flags |= determine_vm_flags(file, prot, flags, capabilities);
 
 	/* we're going to need to record the mapping */
 	region = kmem_cache_zalloc(vm_region_jar, GFP_KERNEL);
mm/util.c +1 -1
index c9439c66d8cf..f15929f2c5bd 100644
@@ -549,7 +549,7 @@ unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr,
 	if (!ret) {
 		if (mmap_write_lock_killable(mm))
 			return -EINTR;
-		ret = do_mmap(file, addr, len, prot, flag, pgoff, &populate,
+		ret = do_mmap(file, addr, len, prot, flag, 0, pgoff, &populate,
 			      &uf);
 		mmap_write_unlock(mm);
 		userfaultfd_unmap_complete(mm, &uf);
Signature · 1 line
2.17.1
#140 lore raw

PATCH v2 22/39 mm: Don't allow write GUPs to shadow stack memory

In reply to Rick Edgecombe

Shadow stack memory is writable only in very specific, controlled ways. However, since it is writable, the kernel treats it as such. As a result there remain many ways for userspace to trigger the kernel to write to shadow stack's via get_user_pages(, FOLL_WRITE) operations. To make this a little less exposed, block writable GUPs for shadow stack VMAs.

Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.

---

v2:
- New patch

 arch/x86/include/asm/pgtable.h | 3 +++
 mm/gup.c                       | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/include/asm/pgtable.h +3 -0
index 7a769c4dbc1c..2e6a5ee70034 100644
@@ -1633,6 +1633,9 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write)
 {
 	unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
+	if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY)
+		return 0;
+
 	if (write)
 		need_pte_bits |= _PAGE_RW;
 
mm/gup.c +1 -1
index 5abdaf487460..56da98f3335c 100644
@@ -1043,7 +1043,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 		return -EFAULT;
 
 	if (write) {
-		if (!(vm_flags & VM_WRITE)) {
+		if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {
 			if (!(gup_flags & FOLL_FORCE))
 				return -EFAULT;
 			/*
Signature · 1 line
2.17.1
#141 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

arch/x86/include/asm/pgtable.h static inline bool __pte_access_permitted(unsigned long pteval, bool write) · line 1637

Quoted 6 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +2 -0
@@ -1633,6 +1633,9 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write)
 {
 	unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
+	if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY)
+		return 0;

Do we not have a helper for this? Seems a bit messy to open-code these shadow-stack permissions. Definitely at least needs a comment.

#142 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

Quoted 6 lines · Dave Hansen
Quoted 5 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +0 -0
@@ -1633,6 +1633,9 @@ static inline bool
 __pte_access_permitted(unsigned long pteval, bool write)
  {
       unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
Quoted 7 lines

Review static inline bool · line 1639

Quoted 3 lines
(quoted diff) +2 -0
+     if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) ==
 _PAGE_DIRTY)
+             return 0;

Do we not have a helper for this? Seems a bit messy to open-code these
shadow-stack permissions. Definitely at least needs a comment.

It's because pteval is an unsigned long. We could create a pte_t, and use the helpers, but then we would be using pte_foo() on pmd's, etc. So probably comment is the better option?

#143 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

Quoted 13 lines · Rick P
Quoted 6 lines · Dave Hansen
Quoted 5 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +0 -0
@@ -1633,6 +1633,9 @@ static inline bool
 __pte_access_permitted(unsigned long pteval, bool write)
  {
       unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
Quoted 6 lines

Review static inline bool · line 1639

Quoted 3 lines
(quoted diff) +2 -0
+     if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) ==
 _PAGE_DIRTY)
+             return 0;

Do we not have a helper for this? Seems a bit messy to open-code these
shadow-stack permissions. Definitely at least needs a comment.

Quoted 3 lines

It's because pteval is an unsigned long. We could create a pte_t, and use the helpers, but then we would be using pte_foo() on pmd's, etc. So probably comment is the better option?

Yeah, a comment is probably best.

This is one of those "generic" page table functions that doesn't work well with the p{te,md,ud}_* types. It's either this or cast over to a pteval_t for pmd/pud and pretend this is a pte-only function.

reviewer
#144 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

In reply to Dave Hansen
Quoted 10 lines · Dave Hansen

arch/x86/include/asm/pgtable.h static inline bool __pte_access_permitted(unsigned long pteval, bool write) · line 1637

Quoted 6 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +2 -0
@@ -1633,6 +1633,9 @@ static inline bool __pte_access_permitted(unsigned long pteval, bool write)
 {
      unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
+     if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY)
+             return 0;

Do we not have a helper for this? Seems a bit messy to open-code these shadow-stack permissions. Definitely at least needs a comment.

FWIW, if you look at more context around this diff, the function looks like this:

static inline bool __pte_access_permitted(unsigned long pteval, bool write)

 {
        unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;

+ if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) == _PAGE_DIRTY)
+ return 0;
+

        if (write)
                need_pte_bits |= _PAGE_RW;

        if ((pteval & need_pte_bits) != need_pte_bits)
                return 0;

        return __pkru_allows_pkey(pte_flags_pkey(pteval), write);

}

So I think this change is actually a no-op - the only thing it does is to return 0 if write==1, !_PAGE_RW, and _PAGE_DIRTY. But the check below will always return 0 if !_PAGE_RW, unless I'm misreading it? And this is the only patch in the series that touches this function, so it's not like this becomes necessary with a later patch in the series either.

Should this check go in anyway for clarity reasons, or should this instead be a comment explaining that __pte_access_permitted() behaves just like the hardware access check, which means shadow pages are treated as readonly?

reviewer
#145 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

Quoted 3 lines · Jann Horn

So I think this change is actually a no-op - the only thing it does is to return 0 if write==1, !_PAGE_RW, and _PAGE_DIRTY. But the check below will always return 0 if !_PAGE_RW, unless I'm misreading it?

Er, to be precise, it will always return 0 if write==1 and !_PAGE_RW.

#146 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

In reply to Jann Horn
Quoted 47 lines · Jann Horn
Quoted 6 lines · Dave Hansen
Quoted 5 lines · Rick Edgecombe
arch/x86/include/asm/pgtable.h +0 -0
@@ -1633,6 +1633,9 @@ static inline bool
 __pte_access_permitted(unsigned long pteval, bool write)
  {
       unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;
 
Quoted 7 lines

Review static inline bool · line 1639

Quoted 3 lines
(quoted diff) +2 -0
+     if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) ==
 _PAGE_DIRTY)
+             return 0;

Do we not have a helper for this? Seems a bit messy to open-code these
shadow-stack permissions. Definitely at least needs a comment.

FWIW, if you look at more context around this diff, the function looks
like this:

static inline bool __pte_access_permitted(unsigned long pteval, bool
write)

 {
        unsigned long need_pte_bits = _PAGE_PRESENT|_PAGE_USER;

+ if (write && (pteval & (_PAGE_RW | _PAGE_DIRTY)) ==
_PAGE_DIRTY)

arch/x86/include/asm/pgtable.h +2 -0
+               return 0;
+
        if (write)
                need_pte_bits |= _PAGE_RW;
 
        if ((pteval & need_pte_bits) != need_pte_bits)
                return 0;
 
        return __pkru_allows_pkey(pte_flags_pkey(pteval), write);
 }

So I think this change is actually a no-op - the only thing it does is
to return 0 if write==1, !_PAGE_RW, and _PAGE_DIRTY. But the check below will always return 0 if !_PAGE_RW, unless I'm misreading it? And
this is the only patch in the series that touches this function, so it's not like this becomes necessary with a later patch in the series either.

Quoted 4 lines

Should this check go in anyway for clarity reasons, or should this instead be a comment explaining that __pte_access_permitted() behaves just like the hardware access check, which means shadow pages are treated as readonly?

Thanks Jann, I was just realizing the same thing. Yes, I think it doesn't do anything. I can add a comment of why there is no check, but otherwise the check seems like unnecessary work.

#147 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

In reply to Rick Edgecombe
Quoted 3 lines · Rick Edgecombe

[...]
Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.

As I asked in the cover letter: why do we need to add this for shstk? It was a mistake for general memory. :P

Quoted 6 lines

[...]

mm/gup.c +0 -0
index 5abdaf487460..56da98f3335c 100644

mm/gup.c static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) · line 1049

Quoted 8 lines
mm/gup.c +1 -1
@@ -1043,7 +1043,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
 		return -EFAULT;
 
 	if (write) {
-		if (!(vm_flags & VM_WRITE)) {
+		if (!(vm_flags & VM_WRITE) || (vm_flags & VM_SHADOW_STACK)) {
 			if (!(gup_flags & FOLL_FORCE))
 				return -EFAULT;
 			/*

How about this instead:

return -EFAULT;

if (write) {
+ if (vm_flags & VM_SHADOW_STACK)
+ return -EFAULT;
if (!(vm_flags & VM_WRITE)) {
if (!(gup_flags & FOLL_FORCE))
return -EFAULT;

Signature · 1 line
Kees Cook
maintainer
#148 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

Quoted 7 lines · Kees Cook
Quoted 3 lines · Rick Edgecombe

[...]
Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.

As I asked in the cover letter: why do we need to add this for shstk? It was a mistake for general memory. :P

For debuggers, which use FOLL_FORCE, quite intentionally, to modify text. And once a debugger has ptrace write access to a target, shadow stacks provide exactly no protection -- ptrace can modify text and all registers.

But /proc/.../mem may be a different story, and I'd be okay with having FOLL_PROC_MEM for legacy compatibility via /proc/.../mem and not allowing that to access shadow stacks. This does seem like it may not be very useful, though.

#149 lore raw

Re: [PATCH v2 22/39] mm: Don't allow write GUPs to shadow stack memory

Quoted 6 lines · Andy Lutomirski
Quoted 5 lines · Kees Cook
Quoted 3 lines · Rick Edgecombe

[...]
Still allow FOLL_FORCE to write through shadow stack protections, as it does for read-only protections.

Quoted 6 lines
Quoted 2 lines

As I asked in the cover letter: why do we need to add this for shstk? It was a mistake for general memory. :P

For debuggers, which use FOLL_FORCE, quite intentionally, to modify text. And once a debugger has ptrace write access to a target, shadow stacks provide exactly no protection -- ptrace can modify text and all registers.

i.e. via ptrace? Yeah, I grudgingly accept the ptrace need for FOLL_FORCE.

Quoted 4 lines

But /proc/.../mem may be a different story, and I'd be okay with having FOLL_PROC_MEM for legacy compatibility via /proc/.../mem and not allowing that to access shadow stacks. This does seem like it may not be very useful, though.

I *really* don't like the /mem use of FOLL_FORCE, though. I think the rationale has been "using PTRACE_POKE is too slow". Again, I can live with it, I was just hoping we could avoid expanding that questionable behavior, especially since it's a bypass of WRSS.

Signature · 1 line
Kees Cook
#150 lore raw

PATCH v2 23/39 x86: Introduce userspace API for CET enabling

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_CET_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

- ARCH_CET_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().

This is preparation patch. It does not impelement any features.

[tweaked with feedback from tglx]

---

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)

 arch/x86/include/asm/cet.h        | 20 ++++++++++++++++
 arch/x86/include/asm/processor.h  |  3 +++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/process.c         |  4 ++++
 arch/x86/kernel/process_64.c      |  5 +++-
 arch/x86/kernel/shstk.c           | 38 +++++++++++++++++++++++++++++++
 6 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/cet.h
 create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/cet.h new file +20 -0
new file mode 100644
index 000000000000..0fa4dbc98c49
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_CET_H
+#define _ASM_X86_CET_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+struct task_struct;
+
+#ifdef CONFIG_X86_SHADOW_STACK
+long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features);
+#else
+static inline long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features) { return -EINVAL; }
+#endif /* CONFIG_X86_SHADOW_STACK */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_CET_H */
arch/x86/include/asm/processor.h +3 -0
index 356308c73951..a92bf76edafe 100644
@@ -530,6 +530,9 @@ struct thread_struct {
 	 */
 	u32			pkru;
 
+	unsigned long		features;
+	unsigned long		features_locked;
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/uapi/asm/prctl.h +6 -0
index 500b96e71f18..028158e35269 100644
@@ -20,4 +20,10 @@
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
 
+/* Don't use 0x3001-0x3004 because of old glibcs */
+
+#define ARCH_CET_ENABLE			0x4001
+#define ARCH_CET_DISABLE		0x4002
+#define ARCH_CET_LOCK			0x4003
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process.c +4 -0
index 58a6ea472db9..034880311e6b 100644
@@ -367,6 +367,10 @@ void arch_setup_new_exec(void)
 		task_clear_spec_ssb_noexec(current);
 		speculation_ctrl_update(read_thread_flags());
 	}
+
+	/* Reset thread features on exec */
+	current->thread.features = 0;
+	current->thread.features_locked = 0;
 }
 
 #ifdef CONFIG_X86_IOPL_IOPERM
arch/x86/kernel/process_64.c +4 -1
index 1962008fe743..8fa2c2b7de65 100644
@@ -829,7 +829,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 	case ARCH_MAP_VDSO_64:
 		return prctl_map_vdso(&vdso_image_64, arg2);
 #endif
-
+	case ARCH_CET_ENABLE:
+	case ARCH_CET_DISABLE:
+	case ARCH_CET_LOCK:
+		return cet_prctl(task, option, arg2);
 	default:
 		ret = -EINVAL;
 		break;
arch/x86/kernel/shstk.c new file +38 -0
new file mode 100644
index 000000000000..e3276ac9e9b9
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * shstk.c - Intel shadow stack support
+ *
+ * Copyright (c) 2021, Intel Corporation.
+ * Yu-cheng Yu <[email protected]>
+ */
+
+#include <linux/sched.h>
+#include <linux/bitops.h>
+#include <asm/prctl.h>
+
+long cet_prctl(struct task_struct *task, int option, unsigned long features)
+{
+	if (option == ARCH_CET_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;
+
+	/* Do not allow to change locked features */
+	if (features & task->thread.features_locked)
+		return -EPERM;
+
+	/* Only support enabling/disabling one feature at a time. */
+	if (hweight_long(features) > 1)
+		return -EINVAL;
+
+	if (option == ARCH_CET_DISABLE) {
+		return -EINVAL;
+	}
+
+	/* Handle ARCH_CET_ENABLE */
+	return -EINVAL;
+}
Signature · 1 line
2.17.1
#151 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

Quoted 7 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_CET_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

Quoted 7 lines

- ARCH_CET_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

The features are handled per-thread and inherited over fork(2)/clone(2), but reset on exec().

This is preparation patch. It does not impelement any features.

typo: "implement"

Quoted 54 lines
[tweaked with feedback from tglx]

---

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander Potapenko)
- Make CET only (tglx)

 arch/x86/include/asm/cet.h        | 20 ++++++++++++++++
 arch/x86/include/asm/processor.h  |  3 +++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/process.c         |  4 ++++
 arch/x86/kernel/process_64.c      |  5 +++-
 arch/x86/kernel/shstk.c           | 38 +++++++++++++++++++++++++++++++
 6 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/cet.h
 create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/cet.h new file +20 -0
new file mode 100644
index 000000000000..0fa4dbc98c49
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_CET_H
+#define _ASM_X86_CET_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+struct task_struct;
+
+#ifdef CONFIG_X86_SHADOW_STACK
+long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features);
+#else
+static inline long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features) { return -EINVAL; }
+#endif /* CONFIG_X86_SHADOW_STACK */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_CET_H */
arch/x86/include/asm/processor.h +0 -0
index 356308c73951..a92bf76edafe 100644

arch/x86/include/asm/processor.h struct thread_struct { · line 534

Quoted 5 lines
arch/x86/include/asm/processor.h +2 -0
@@ -530,6 +530,9 @@ struct thread_struct {
 	 */
 	u32			pkru;
 
+	unsigned long		features;
+	unsigned long		features_locked;

Should these be wrapped in #ifdef CONFIG_X86_SHADOW_STACK (or CONFIG_X86_CET) ?

Also, just named "features"? Is this expected to be more than CET?

Quoted 24 lines
arch/x86/include/asm/processor.h +1 -0
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/uapi/asm/prctl.h +6 -0
index 500b96e71f18..028158e35269 100644
@@ -20,4 +20,10 @@
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
 
+/* Don't use 0x3001-0x3004 because of old glibcs */
+
+#define ARCH_CET_ENABLE			0x4001
+#define ARCH_CET_DISABLE		0x4002
+#define ARCH_CET_LOCK			0x4003
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process.c +0 -0
index 58a6ea472db9..034880311e6b 100644

arch/x86/kernel/process.c void arch_setup_new_exec(void) · line 373

Quoted 7 lines
arch/x86/kernel/process.c +4 -0
@@ -367,6 +367,10 @@ void arch_setup_new_exec(void)
 		task_clear_spec_ssb_noexec(current);
 		speculation_ctrl_update(read_thread_flags());
 	}
+
+	/* Reset thread features on exec */
+	current->thread.features = 0;
+	current->thread.features_locked = 0;

Same ifdef question here.

Quoted 9 lines
arch/x86/kernel/process.c +0 -0
 }
 
 #ifdef CONFIG_X86_IOPL_IOPERM
arch/x86/kernel/process_64.c +0 -0
index 1962008fe743..8fa2c2b7de65 100644
@@ -829,7 +829,10 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
 	case ARCH_MAP_VDSO_64:

arch/x86/kernel/process_64.c long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) · line 838

Quoted 10 lines
arch/x86/kernel/process_64.c +4 -1
 		return prctl_map_vdso(&vdso_image_64, arg2);
 #endif
-
+	case ARCH_CET_ENABLE:
+	case ARCH_CET_DISABLE:
+	case ARCH_CET_LOCK:
+		return cet_prctl(task, option, arg2);
 	default:
 		ret = -EINVAL;
 		break;

I remain annoyed that prctl interfaces didn't use -ENOTSUP for "unknown option". :P

arch/x86/kernel/shstk.c

Quoted 5 lines
arch/x86/kernel/shstk.c new file +0 -0
new file mode 100644
index 000000000000..e3276ac9e9b9

I think the Makefile addition should be moved from "x86/cet/shstk:
Add user-mode shadow stack support" to here, yes? Otherwise, there is a bisectability randconfig-with-CONFIG_X86_SHADOW_STACK risk here (nothing will implement "cet_prctl").

Quoted 13 lines
arch/x86/kernel/shstk.c new file +12 -0
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * shstk.c - Intel shadow stack support
+ *
+ * Copyright (c) 2021, Intel Corporation.
+ * Yu-cheng Yu <[email protected]>
+ */
+
+#include <linux/sched.h>
+#include <linux/bitops.h>
+#include <asm/prctl.h>
+

arch/x86/kernel/shstk.c line 22

Quoted 10 lines
arch/x86/kernel/shstk.c new file +10 -0
+long cet_prctl(struct task_struct *task, int option, unsigned long features)
+{
+	if (option == ARCH_CET_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;

... but locking _is_ allowed via ptrace? If that intended, it should be explicitly mentioned in the commit log and in a comment here.

Also, perhaps -ESRCH ?

arch/x86/kernel/shstk.c line 30

Quoted 8 lines
arch/x86/kernel/shstk.c new file +8 -0
+
+	/* Do not allow to change locked features */
+	if (features & task->thread.features_locked)
+		return -EPERM;
+
+	/* Only support enabling/disabling one feature at a time. */
+	if (hweight_long(features) > 1)
+		return -EINVAL;

Perhaps -E2BIG ?

Quoted 10 lines
arch/x86/kernel/shstk.c new file +7 -0
+	if (option == ARCH_CET_DISABLE) {
+		return -EINVAL;
+	}
+
+	/* Handle ARCH_CET_ENABLE */
+	return -EINVAL;
+}
Signature · 1 line
2.17.1
Signature · 1 line
Kees Cook
#152 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

CC Mike about ptrace/CRIU question.

Quoted 11 lines · Kees Cook
Quoted 10 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_CET_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

- ARCH_CET_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

Quoted 7 lines
Quoted 5 lines

The features are handled per-thread and inherited over fork(2)/clone(2),
but reset on exec().

This is preparation patch. It does not impelement any features.

typo: "implement"

Oops, thanks.

Quoted 62 lines
Quoted 62 lines
[tweaked with feedback from tglx]

---

v2:
- Only allow one enable/disable per call (tglx)
- Return error code like a normal arch_prctl() (Alexander
Potapenko)
- Make CET only (tglx)

 arch/x86/include/asm/cet.h        | 20 ++++++++++++++++
 arch/x86/include/asm/processor.h  |  3 +++
 arch/x86/include/uapi/asm/prctl.h |  6 +++++
 arch/x86/kernel/process.c         |  4 ++++
 arch/x86/kernel/process_64.c      |  5 +++-
 arch/x86/kernel/shstk.c           | 38
+++++++++++++++++++++++++++++++
 6 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/cet.h
 create mode 100644 arch/x86/kernel/shstk.c
arch/x86/include/asm/cet.h new file +20 -0
new file mode 100644
index 000000000000..0fa4dbc98c49
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_CET_H
+#define _ASM_X86_CET_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+struct task_struct;
+
+#ifdef CONFIG_X86_SHADOW_STACK
+long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features);
+#else
+static inline long cet_prctl(struct task_struct *task, int option,
+		      unsigned long features) { return -EINVAL; }
+#endif /* CONFIG_X86_SHADOW_STACK */
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_CET_H */
arch/x86/include/asm/processor.h +0 -0
index 356308c73951..a92bf76edafe 100644
@@ -530,6 +530,9 @@ struct thread_struct {
 	 */
 	u32			pkru;
 
Quoted 7 lines

arch/x86/include/asm/processor.h struct thread_struct { · line 534

Quoted 2 lines
arch/x86/include/asm/processor.h +2 -0
+	unsigned long		features;
+	unsigned long		features_locked;

Should these be wrapped in #ifdef CONFIG_X86_SHADOW_STACK (or CONFIG_X86_CET) ?

Also, just named "features"? Is this expected to be more than CET?

Sigh, there have been many ideas about how this API and features tracking could be shared with LAM. At some point there was some discussion about LAM using the features as well, even if it had a separate arch_prctl() interface. Just checking the last LAM posting, I'm not sure it needs it. So yes, this could go back to being CET only for the time being.

Quoted 62 lines
Quoted 26 lines
arch/x86/include/asm/processor.h +1 -0
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/uapi/asm/prctl.h +6 -0
index 500b96e71f18..028158e35269 100644
@@ -20,4 +20,10 @@
 #define ARCH_MAP_VDSO_32		0x2002
 #define ARCH_MAP_VDSO_64		0x2003
 
+/* Don't use 0x3001-0x3004 because of old glibcs */
+
+#define ARCH_CET_ENABLE			0x4001
+#define ARCH_CET_DISABLE		0x4002
+#define ARCH_CET_LOCK			0x4003
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/process.c +0 -0
index 58a6ea472db9..034880311e6b 100644

arch/x86/kernel/process.c void arch_setup_new_exec(void) · line 373

Quoted 7 lines
arch/x86/kernel/process.c +4 -0
@@ -367,6 +367,10 @@ void arch_setup_new_exec(void)
 		task_clear_spec_ssb_noexec(current);
 		speculation_ctrl_update(read_thread_flags());
 	}
+
+	/* Reset thread features on exec */
+	current->thread.features = 0;
+	current->thread.features_locked = 0;

Same ifdef question here.

Quoted 20 lines
arch/x86/kernel/process.c +0 -0
 }
 
 #ifdef CONFIG_X86_IOPL_IOPERM
arch/x86/kernel/process_64.c +4 -1
index 1962008fe743..8fa2c2b7de65 100644
@@ -829,7 +829,10 @@ long do_arch_prctl_64(struct task_struct
 *task, int option, unsigned long arg2)
 	case ARCH_MAP_VDSO_64:
 		return prctl_map_vdso(&vdso_image_64, arg2);
 #endif
-
+	case ARCH_CET_ENABLE:
+	case ARCH_CET_DISABLE:
+	case ARCH_CET_LOCK:
+		return cet_prctl(task, option, arg2);
 	default:
 		ret = -EINVAL;
Quoted 1 line

break;

I remain annoyed that prctl interfaces didn't use -ENOTSUP for "unknown
option". :P

Quoted 12 lines

arch/x86/kernel/shstk.c

Quoted 5 lines
arch/x86/kernel/shstk.c new file +0 -0
new file mode 100644
index 000000000000..e3276ac9e9b9

I think the Makefile addition should be moved from "x86/cet/shstk:
Add user-mode shadow stack support" to here, yes? Otherwise, there is a
bisectability randconfig-with-CONFIG_X86_SHADOW_STACK risk here (nothing
will implement "cet_prctl").

Oh, yep, good point.

Quoted 21 lines
Quoted 21 lines
arch/x86/kernel/shstk.c new file +13 -0
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * shstk.c - Intel shadow stack support
+ *
+ * Copyright (c) 2021, Intel Corporation.
+ * Yu-cheng Yu <[email protected]>
+ */
+
+#include <linux/sched.h>
+#include <linux/bitops.h>
+#include <asm/prctl.h>
+
+long cet_prctl(struct task_struct *task, int option, unsigned long

features)

arch/x86/kernel/shstk.c new file +5 -0
+{
+	if (option == ARCH_CET_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
Quoted 8 lines

arch/x86/kernel/shstk.c line 22

Quoted 4 lines
arch/x86/kernel/shstk.c new file +4 -0
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;

... but locking _is_ allowed via ptrace? If that intended, it should be
explicitly mentioned in the commit log and in a comment here.

I believe CRIU needs to lock via ptrace as well. Maybe Mike can confirm.

I can mention it.

Quoted 13 lines

Also, perhaps -ESRCH ?

arch/x86/kernel/shstk.c line 30

Quoted 8 lines
arch/x86/kernel/shstk.c new file +8 -0
+
+	/* Do not allow to change locked features */
+	if (features & task->thread.features_locked)
+		return -EPERM;
+
+	/* Only support enabling/disabling one feature at a time. */
+	if (hweight_long(features) > 1)
+		return -EINVAL;

Perhaps -E2BIG ?

Ehh, I don't know. E2MUCH maybe. It's not necessarily too big. Like if a third bit was added for IBT some day, you could set SHSTK and WRSS, it would be invalid, but still be "less" than the valid passing of just the (hypothetical IBT bit).

Quoted 13 lines
Quoted 11 lines
arch/x86/kernel/shstk.c new file +7 -0
+	if (option == ARCH_CET_DISABLE) {
+		return -EINVAL;
+	}
+
+	/* Handle ARCH_CET_ENABLE */
+	return -EINVAL;
+}
Signature · 1 line
2.17.1
maintainer
#153 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

Quoted 17 lines · Rick P

CC Mike about ptrace/CRIU question.

Quoted 14 lines · Kees Cook
Quoted 13 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add three new arch_prctl() handles:

- ARCH_CET_ENABLE/DISABLE enables or disables the specified feature. Returns 0 on success or an error.

- ARCH_CET_LOCK prevents future disabling or enabling of the specified feature. Returns 0 on success or an error

The features are handled per-thread and inherited over fork(2)/clone(2),
but reset on exec().

...

Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/shstk.c new file +5 -0
+#include <linux/sched.h>
+#include <linux/bitops.h>
+#include <asm/prctl.h>
+
+long cet_prctl(struct task_struct *task, int option, unsigned long

features)

arch/x86/kernel/shstk.c new file +5 -0
+{
+	if (option == ARCH_CET_LOCK) {
+		task->thread.features_locked |= features;
+		return 0;
+	}
Quoted 8 lines

arch/x86/kernel/shstk.c line 22

Quoted 4 lines
arch/x86/kernel/shstk.c new file +4 -0
+
+	/* Don't allow via ptrace */
+	if (task != current)
+		return -EINVAL;

... but locking _is_ allowed via ptrace? If that intended, it should be
explicitly mentioned in the commit log and in a comment here.

Quoted 2 lines

I believe CRIU needs to lock via ptrace as well. Maybe Mike can confirm.

Actually, I didn't use ptrace for locking, I did it with "plain" arch_prctl().

I still can't say for sure CRIU won't need this, I didn't have time yet to have a closer look at this set.

Signature · 2 lines
Sincerely yours,
Mike.
#154 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

In reply to Rick Edgecombe

* Rick Edgecombe:

arch/x86/kernel/shstk.c line 30

Quoted 3 lines
arch/x86/kernel/shstk.c new file +3 -0
+	/* Only support enabling/disabling one feature at a time. */
+	if (hweight_long(features) > 1)
+		return -EINVAL;

This means we'll soon need three extra system calls for x86-64 process
start: SHSTK, IBT, and switching off vsyscall emulation. (The latter does not need any special CPU support.)

Maybe we can do something else instead to make the strace output a little bit cleaner?

Thanks,
Florian

#155 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

Quoted 11 lines · Florian Weimer

arch/x86/kernel/shstk.c line 30

Quoted 3 lines
arch/x86/kernel/shstk.c new file +3 -0
+     /* Only support enabling/disabling one feature at a time. */
+     if (hweight_long(features) > 1)
+             return -EINVAL;

This means we'll soon need three extra system calls for x86-64 process
start: SHSTK, IBT, and switching off vsyscall emulation. (The latter does not need any special CPU support.)

Maybe we can do something else instead to make the strace output a little bit cleaner?

In previous versions it supported enabling multiple features in a single syscall. Thomas Gleixner pointed out that (this was on the LAM patchset that shared the interface at the time) it makes the behavior of what to do when one feature fails to enable complicated:

https://lore.kernel.org/lkml/87zgjjqico.ffs@tglx/

#156 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

* Rick P. Edgecombe:

Quoted 13 lines · Florian Weimer
Quoted 11 lines

arch/x86/kernel/shstk.c line 30

Quoted 3 lines
arch/x86/kernel/shstk.c new file +3 -0
+     /* Only support enabling/disabling one feature at a time. */
+     if (hweight_long(features) > 1)
+             return -EINVAL;

This means we'll soon need three extra system calls for x86-64 process
start: SHSTK, IBT, and switching off vsyscall emulation. (The latter does not need any special CPU support.)

Maybe we can do something else instead to make the strace output a little bit cleaner?

Quoted 6 lines

In previous versions it supported enabling multiple features in a single syscall. Thomas Gleixner pointed out that (this was on the LAM patchset that shared the interface at the time) it makes the behavior of what to do when one feature fails to enable complicated:

https://lore.kernel.org/lkml/87zgjjqico.ffs@tglx/

Can we return the bits for the features that were actually enabled? Those three don't have cross-dependencies in the sense that you would only use X & Y together, but not X or Y alone.

Thanks,
Florian

#157 lore raw

Re: [PATCH v2 23/39] x86: Introduce userspace API for CET enabling

Quoted 25 lines · Florian Weimer

* Rick P. Edgecombe:

Quoted 23 lines · Florian Weimer
Quoted 11 lines

Review

Quoted 4 lines

+ /* Only support enabling/disabling one feature at a time.
*/

(quoted diff) +2 -0
+     if (hweight_long(features) > 1)
+             return -EINVAL;

This means we'll soon need three extra system calls for x86-64 process
start: SHSTK, IBT, and switching off vsyscall emulation. (The latter
does not need any special CPU support.)

Quoted 3 lines

Maybe we can do something else instead to make the strace output a
little bit cleaner?

In previous versions it supported enabling multiple features in a single syscall. Thomas Gleixner pointed out that (this was on the LAM
patchset that shared the interface at the time) it makes the behavior
of what to do when one feature fails to enable complicated:

Quoted 3 lines
Quoted 1 line

https://lore.kernel.org/lkml/87zgjjqico.ffs@tglx/

Can we return the bits for the features that were actually enabled?

Actually that specific option is covered in that thread as well. I was thinking we would need to pass a struct in an out to do a batch operation. Thomas suggested it could be added later and to start with a simpler option. Is an extra syscall or two at startup really a big problem?

Quoted 2 lines

Those three don't have cross-dependencies in the sense that you would only use X & Y together, but not X or Y alone.

I don't fully follow this, but WRSS does actually depend on SHSTK.

#158 lore raw

PATCH v2 24/39 x86/cet/shstk: Add user-mode shadow stack support

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).

Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.

Do not support IA32 emulation.

---

v2:
- Get rid of unnessary shstk->base checks
- Don't support IA32 emulation

v1:
- Switch to xsave helpers.
- Expand commit log.

Yu-cheng v30:
- Remove superfluous comments for struct thread_shstk.
- Replace 'populate' with 'unused'.

Yu-cheng v28:
- Update shstk_setup() with wrmsrl_safe(), returns success when shadow stack feature is not present (since this is a setup function).

 arch/x86/include/asm/cet.h        |  13 +++
 arch/x86/include/asm/msr.h        |  11 +++
 arch/x86/include/asm/processor.h  |   5 ++
 arch/x86/include/uapi/asm/prctl.h |   2 +
 arch/x86/kernel/Makefile          |   2 +
 arch/x86/kernel/process_64.c      |   2 +
 arch/x86/kernel/shstk.c           | 143 ++++++++++++++++++++++++++++++
 7 files changed, 178 insertions(+)
arch/x86/include/asm/cet.h +13 -0
index 0fa4dbc98c49..a4a1f4c0089b 100644
@@ -7,12 +7,25 @@
 
 struct task_struct;
 
+struct thread_shstk {
+	u64	base;
+	u64	size;
+};
+
 #ifdef CONFIG_X86_SHADOW_STACK
 long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features);
+int shstk_setup(void);
+void shstk_free(struct task_struct *p);
+int shstk_disable(void);
+void reset_thread_shstk(void);
 #else
 static inline long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features) { return -EINVAL; }
+static inline int shstk_setup(void) { return -EOPNOTSUPP; }
+static inline void shstk_free(struct task_struct *p) {}
+static inline int shstk_disable(void) { return -EOPNOTSUPP; }
+static inline void reset_thread_shstk(void) {}
 #endif /* CONFIG_X86_SHADOW_STACK */
 
 #endif /* __ASSEMBLY__ */
arch/x86/include/asm/msr.h +11 -0
index 65ec1965cd28..a9cb4c434e60 100644
@@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs);
 int msr_set_bit(u32 msr, u8 bit);
 int msr_clear_bit(u32 msr, u8 bit);
 
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
+}
+
 #ifdef CONFIG_SMP
 int rdmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 *l, u32 *h);
 int wrmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 l, u32 h);
arch/x86/include/asm/processor.h +5 -0
index a92bf76edafe..3a0c9d9d4d1d 100644
@@ -27,6 +27,7 @@ struct vm86;
 #include <asm/unwind_hints.h>
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
+#include <asm/cet.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -533,6 +534,10 @@ struct thread_struct {
 	unsigned long		features;
 	unsigned long		features_locked;
 
+#ifdef CONFIG_X86_SHADOW_STACK
+	struct thread_shstk	shstk;
+#endif
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/uapi/asm/prctl.h +2 -0
index 028158e35269..41af3a8c4fa4 100644
@@ -26,4 +26,6 @@
 #define ARCH_CET_DISABLE		0x4002
 #define ARCH_CET_LOCK			0x4003
 
+#define CET_SHSTK			0x1
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +2 -0
index a20a5ebfacd7..8950d1f71226 100644
@@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_GUESS)		+= unwind_guess.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)		+= sev.o
 
+obj-$(CONFIG_X86_SHADOW_STACK)		+= shstk.o
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/process_64.c +2 -0
index 8fa2c2b7de65..be544b4b4c8b 100644
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,
 		load_gs_index(__USER_DS);
 	}
 
+	reset_thread_shstk();
+
 	loadsegment(fs, 0);
 	loadsegment(es, _ds);
 	loadsegment(ds, _ds);
arch/x86/kernel/shstk.c +143 -0

151 lines not on this page · raw message

Signature · 1 line
2.17.1
#159 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 62 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Introduce basic shadow stack enabling/disabling/allocation routines. A task's shadow stack is allocated from memory with VM_SHADOW_STACK flag and has a fixed size of min(RLIMIT_STACK, 4GB).

Keep the task's shadow stack address and size in thread_struct. This will be copied when cloning new threads, but needs to be cleared during exec, so add a function to do this.

Do not support IA32 emulation.

---

v2:
- Get rid of unnessary shstk->base checks
- Don't support IA32 emulation

v1:
- Switch to xsave helpers.
- Expand commit log.

Yu-cheng v30:
- Remove superfluous comments for struct thread_shstk.
- Replace 'populate' with 'unused'.

Yu-cheng v28:
- Update shstk_setup() with wrmsrl_safe(), returns success when shadow stack feature is not present (since this is a setup function).

 arch/x86/include/asm/cet.h        |  13 +++
 arch/x86/include/asm/msr.h        |  11 +++
 arch/x86/include/asm/processor.h  |   5 ++
 arch/x86/include/uapi/asm/prctl.h |   2 +
 arch/x86/kernel/Makefile          |   2 +
 arch/x86/kernel/process_64.c      |   2 +
 arch/x86/kernel/shstk.c           | 143 ++++++++++++++++++++++++++++++
 7 files changed, 178 insertions(+)
arch/x86/include/asm/cet.h +7 -0
index 0fa4dbc98c49..a4a1f4c0089b 100644
@@ -7,12 +7,25 @@
 
 struct task_struct;
 
+struct thread_shstk {
+	u64	base;
+	u64	size;
+};
+
 #ifdef CONFIG_X86_SHADOW_STACK
 long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features);
+int shstk_setup(void);
+void shstk_free(struct task_struct *p);

arch/x86/include/asm/cet.h line 29

Quoted 10 lines
arch/x86/include/asm/cet.h +6 -0
+int shstk_disable(void);
+void reset_thread_shstk(void);
 #else
 static inline long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features) { return -EINVAL; }
+static inline int shstk_setup(void) { return -EOPNOTSUPP; }
+static inline void shstk_free(struct task_struct *p) {}
+static inline int shstk_disable(void) { return -EOPNOTSUPP; }
+static inline void reset_thread_shstk(void) {}
 #endif /* CONFIG_X86_SHADOW_STACK */

shstk_setup() and shstk_disable() are not called outside of shstk.c, so they can be removed from this header entirely.

Quoted 10 lines
arch/x86/include/asm/cet.h +0 -0
 #endif /* __ASSEMBLY__ */
arch/x86/include/asm/msr.h +0 -0
index 65ec1965cd28..a9cb4c434e60 100644
@@ -310,6 +310,17 @@ void msrs_free(struct msr *msrs);
 int msr_set_bit(u32 msr, u8 bit);
 int msr_clear_bit(u32 msr, u8 bit);
 

arch/x86/include/asm/msr.h void msrs_free(struct msr *msrs); · line 322

Quoted 10 lines
arch/x86/include/asm/msr.h +10 -0
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
+}

I always get uncomfortable when I see these kinds of generalized helper functions for touching cpu bits, etc. It just begs for future attacker abuse to muck with arbitrary bits -- even marked inline there is a risk the compiler will ignore that in some circumstances (not as currently used in the code, but I'm imagining future changes leading to such a condition). Will you humor me and change this to a macro instead? That'll force it always inline (even __always_inline isn't always inline):

/* Helper that can never get accidentally un-inlined. */

#define set_clr_bits_msrl(msr, set, clear)	do {	\
	u64 __val, __new_val;				\
							\
	rdmsrl(msr, __val);				\
	__new_val = (__val & ~(clear)) | (set);		\
							\
	if (__new_val != __val)				\
		wrmsrl(msr, __new_val);			\
} while (0)
Quoted 32 lines
arch/x86/include/asm/msr.h +1 -0
+
 #ifdef CONFIG_SMP
 int rdmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 *l, u32 *h);
 int wrmsr_on_cpu(unsigned int cpu, u32 msr_no, u32 l, u32 h);
arch/x86/include/asm/processor.h +5 -0
index a92bf76edafe..3a0c9d9d4d1d 100644
@@ -27,6 +27,7 @@ struct vm86;
 #include <asm/unwind_hints.h>
 #include <asm/vmxfeatures.h>
 #include <asm/vdso/processor.h>
+#include <asm/cet.h>
 
 #include <linux/personality.h>
 #include <linux/cache.h>
@@ -533,6 +534,10 @@ struct thread_struct {
 	unsigned long		features;
 	unsigned long		features_locked;
 
+#ifdef CONFIG_X86_SHADOW_STACK
+	struct thread_shstk	shstk;
+#endif
+
 	/* Floating point and extended processor state */
 	struct fpu		fpu;
 	/*
arch/x86/include/uapi/asm/prctl.h +0 -0
index 028158e35269..41af3a8c4fa4 100644

arch/x86/include/uapi/asm/prctl.h line 28

Quoted 3 lines
arch/x86/include/uapi/asm/prctl.h +0 -0
@@ -26,4 +26,6 @@
 #define ARCH_CET_DISABLE		0x4002
 #define ARCH_CET_LOCK			0x4003
 

For readability, maybe add: /* ARCH_CET_* "features" bits */

arch/x86/include/uapi/asm/prctl.h line 29

Quoted 1 line
arch/x86/include/uapi/asm/prctl.h +1 -0
+#define CET_SHSTK			0x1

This is UAPI, so the BIT() macro isn't available, but since this is unsigned long, please use the form: (1ULL << 0) etc...

Quoted 57 lines
arch/x86/include/uapi/asm/prctl.h +1 -0
+
 #endif /* _ASM_X86_PRCTL_H */
arch/x86/kernel/Makefile +2 -0
index a20a5ebfacd7..8950d1f71226 100644
@@ -139,6 +139,8 @@ obj-$(CONFIG_UNWINDER_GUESS)		+= unwind_guess.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)		+= sev.o
 
+obj-$(CONFIG_X86_SHADOW_STACK)		+= shstk.o
+
 ###
 # 64 bit specific files
 ifeq ($(CONFIG_X86_64),y)
arch/x86/kernel/process_64.c +2 -0
index 8fa2c2b7de65..be544b4b4c8b 100644
@@ -514,6 +514,8 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,
 		load_gs_index(__USER_DS);
 	}
 
+	reset_thread_shstk();
+
 	loadsegment(fs, 0);
 	loadsegment(es, _ds);
 	loadsegment(ds, _ds);
arch/x86/kernel/shstk.c +19 -0
index e3276ac9e9b9..a0b8d4adb2bf 100644
@@ -8,8 +8,151 @@
 
 #include <linux/sched.h>
 #include <linux/bitops.h>
+#include <linux/types.h>
+#include <linux/mm.h>
+#include <linux/mman.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/sched/signal.h>
+#include <linux/compat.h>
+#include <linux/sizes.h>
+#include <linux/user.h>
+#include <asm/msr.h>
+#include <asm/fpu/xstate.h>
+#include <asm/fpu/types.h>
+#include <asm/cet.h>
+#include <asm/special_insns.h>
+#include <asm/fpu/api.h>
 #include <asm/prctl.h>
 
+static bool feature_enabled(unsigned long features)
+{
+	return current->thread.features & features;
+}

arch/x86/kernel/shstk.c line 41

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+
+static void feature_set(unsigned long features)
+{
+	current->thread.features |= features;
+}
+
+static void feature_clr(unsigned long features)
+{
+	current->thread.features &= ~features;
+}

"feature" vs "features" here is confusing. Should these helpers enforce the single-bit-set requirements? If so, please switch to a bit number instead of a mask. If not, please rename these to "features_{enabled,set,clr}", and fix "features_enabled" to check them all:

	return (current->thread.features & features) == features;

arch/x86/kernel/shstk.c line 47

Quoted 5 lines
arch/x86/kernel/shstk.c +5 -0
+static unsigned long alloc_shstk(unsigned long size)
+{
+	int flags = MAP_ANONYMOUS | MAP_PRIVATE;
+	struct mm_struct *mm = current->mm;
+	unsigned long addr, unused;

WARN_ON + clamp on "size" here, or perhaps move the bounds check from shstk_setup() into here?

arch/x86/kernel/shstk.c line 51

Quoted 4 lines
arch/x86/kernel/shstk.c +4 -0
+
+	mmap_write_lock(mm);
+	addr = do_mmap(NULL, addr, size, PROT_READ, flags,
+		       VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL);

This will use the mmap base address offset randomization, I guess?

Quoted 23 lines
arch/x86/kernel/shstk.c +23 -0
+
+	mmap_write_unlock(mm);
+
+	return addr;
+}
+
+static void unmap_shadow_stack(u64 base, u64 size)
+{
+	while (1) {
+		int r;
+
+		r = vm_munmap(base, size);
+
+		/*
+		 * vm_munmap() returns -EINTR when mmap_lock is held by
+		 * something else, and that lock should not be held for a
+		 * long time.  Retry it for the case.
+		 */
+		if (r == -EINTR) {
+			cond_resched();
+			continue;
+		}
+

arch/x86/kernel/shstk.c line 84

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+		/*
+		 * For all other types of vm_munmap() failure, either the
+		 * system is out of memory or there is bug.
+		 */
+		WARN_ON_ONCE(r);
+		break;
+	}
+}
+
+int shstk_setup(void)

Only called local. Make static?

Quoted 25 lines
arch/x86/kernel/shstk.c +25 -0
+{
+	struct thread_shstk *shstk = &current->thread.shstk;
+	unsigned long addr, size;
+
+	/* Already enabled */
+	if (feature_enabled(CET_SHSTK))
+		return 0;
+
+	/* Also not supported for 32 bit */
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK) || in_ia32_syscall())
+		return -EOPNOTSUPP;
+
+	size = PAGE_ALIGN(min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G));
+	addr = alloc_shstk(size);
+	if (IS_ERR_VALUE(addr))
+		return PTR_ERR((void *)addr);
+
+	fpu_lock_and_load();
+	wrmsrl(MSR_IA32_PL3_SSP, addr + size);
+	wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN);
+	fpregs_unlock();
+
+	shstk->base = addr;
+	shstk->size = size;
+	feature_set(CET_SHSTK);

arch/x86/kernel/shstk.c line 119

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+
+	return 0;
+}
+
+void reset_thread_shstk(void)
+{
+	memset(&current->thread.shstk, 0, sizeof(struct thread_shstk));
+	current->thread.features = 0;
+	current->thread.features_locked = 0;
+}

If features is always going to be tied to shstk, why not put them in the shstk struct?

Also, shouldn't this also be called from arch_setup_new_exec() instead of the open-coded wipe of features there?

arch/x86/kernel/shstk.c line 132

Quoted 13 lines
arch/x86/kernel/shstk.c +13 -0
+
+void shstk_free(struct task_struct *tsk)
+{
+	struct thread_shstk *shstk = &tsk->thread.shstk;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK) ||
+	    !feature_enabled(CET_SHSTK))
+		return;
+
+	if (!tsk->mm)
+		return;
+
+	unmap_shadow_stack(shstk->base, shstk->size);

I feel like base and size should be zeroed here?

arch/x86/kernel/shstk.c line 135

Quoted 3 lines
arch/x86/kernel/shstk.c +3 -0
+}
+
+int shstk_disable(void)

This is only called locally. static?

Quoted 26 lines
arch/x86/kernel/shstk.c +20 -0
+{
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK))
+		return -EOPNOTSUPP;
+
+	/* Already disabled? */
+	if (!feature_enabled(CET_SHSTK))
+		return 0;
+
+	fpu_lock_and_load();
+	/* Disable WRSS too when disabling shadow stack */
+	set_clr_bits_msrl(MSR_IA32_U_CET, 0, CET_SHSTK_EN);
+	wrmsrl(MSR_IA32_PL3_SSP, 0);
+	fpregs_unlock();
+
+	shstk_free(current);
+	feature_clr(CET_SHSTK);
+
+	return 0;
+}
+
 long cet_prctl(struct task_struct *task, int option, unsigned long features)
 {
 	if (option == ARCH_CET_LOCK) {
Signature · 1 line
2.17.1
Signature · 1 line
Kees Cook
#160 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 9 lines · Kees Cook
Quoted 9 lines
arch/x86/include/asm/msr.h +9 -0
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
Quoted 8 lines
Quoted 1 line

+}

I always get uncomfortable when I see these kinds of generalized helper functions for touching cpu bits, etc. It just begs for future attacker abuse to muck with arbitrary bits -- even marked inline there is a risk the compiler will ignore that in some circumstances (not as currently used in the code, but I'm imagining future changes leading to such a condition). Will you humor me and change this to a macro instead? That'll force it always inline (even __always_inline isn't always inline):

Oh, are you thinking that this is dangerous because it's so surgical and non-intrusive? It's even more powerful to an attacker than, say wrmsrl(), because there they actually have to know what the existing value is to update it. With this helper, it's quite easy to flip an individual bit without disturbing the neighboring bits.

Is that it?

I don't _like_ the #defines, but doing one here doesn't seem too onerous considering how critical MSRs are.

#161 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 19 lines · Dave Hansen
Quoted 9 lines · Kees Cook
Quoted 9 lines
arch/x86/include/asm/msr.h +9 -0
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
Quoted 8 lines
Quoted 1 line

+}

I always get uncomfortable when I see these kinds of generalized helper functions for touching cpu bits, etc. It just begs for future attacker abuse to muck with arbitrary bits -- even marked inline there is a risk the compiler will ignore that in some circumstances (not as currently used in the code, but I'm imagining future changes leading to such a condition). Will you humor me and change this to a macro instead? That'll force it always inline (even __always_inline isn't always inline):

Quoted 7 lines

Oh, are you thinking that this is dangerous because it's so surgical and non-intrusive? It's even more powerful to an attacker than, say wrmsrl(), because there they actually have to know what the existing value is to update it. With this helper, it's quite easy to flip an individual bit without disturbing the neighboring bits.

Is that it?

Yeah, it was kind of the combo: both a potential entry point to wrmsrl for arbitrary values, but also one where all the work is done to mask stuff out.

Quoted 2 lines

I don't _like_ the #defines, but doing one here doesn't seem too onerous considering how critical MSRs are.

I bet there are others, but this just weirded me out. I'll live with a macro, especially since the chance of it mutating in a non-inline is very small, but I figured I'd mention the idea.

Signature · 1 line
Kees Cook
#162 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 8 lines · Kees Cook
Quoted 3 lines

I don't _like_ the #defines, but doing one here doesn't seem too onerous
considering how critical MSRs are.

I bet there are others, but this just weirded me out. I'll live with a
macro, especially since the chance of it mutating in a non-inline is very small, but I figured I'd mention the idea.

Makes sense. I'll change it to a define.

#163 lore raw

RE: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

In reply to Dave Hansen

From: Dave Hansen

Quoted 27 lines

Sent: 03 October 2022 21:05

Quoted 9 lines · Kees Cook
Quoted 9 lines
arch/x86/include/asm/msr.h +9 -0
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
Quoted 8 lines
Quoted 1 line

+}

I always get uncomfortable when I see these kinds of generalized helper functions for touching cpu bits, etc. It just begs for future attacker abuse to muck with arbitrary bits -- even marked inline there is a risk the compiler will ignore that in some circumstances (not as currently used in the code, but I'm imagining future changes leading to such a condition). Will you humor me and change this to a macro instead? That'll force it always inline (even __always_inline isn't always inline):

Oh, are you thinking that this is dangerous because it's so surgical and non-intrusive? It's even more powerful to an attacker than, say wrmsrl(), because there they actually have to know what the existing value is to update it. With this helper, it's quite easy to flip an individual bit without disturbing the neighboring bits.

Quoted 4 lines

Is that it?

I don't _like_ the #defines, but doing one here doesn't seem too onerous considering how critical MSRs are.

How often is the 'msr' number not a compile-time constant? Adding rd/wrmsr variants that verify this would reduce the attack surface as well.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#164 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 28 lines · David Laight

From: Dave Hansen

Quoted 27 lines

Sent: 03 October 2022 21:05

Quoted 9 lines · Kees Cook
Quoted 9 lines
arch/x86/include/asm/msr.h +9 -0
+static inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)
+{
+	u64 val, new_val;
+
+	rdmsrl(msr, val);
+	new_val = (val & ~clear) | set;
+
+	if (new_val != val)
+		wrmsrl(msr, new_val);
Quoted 8 lines
Quoted 1 line

+}

I always get uncomfortable when I see these kinds of generalized helper functions for touching cpu bits, etc. It just begs for future attacker abuse to muck with arbitrary bits -- even marked inline there is a risk the compiler will ignore that in some circumstances (not as currently used in the code, but I'm imagining future changes leading to such a condition). Will you humor me and change this to a macro instead? That'll force it always inline (even __always_inline isn't always inline):

Oh, are you thinking that this is dangerous because it's so surgical and non-intrusive? It's even more powerful to an attacker than, say wrmsrl(), because there they actually have to know what the existing value is to update it. With this helper, it's quite easy to flip an individual bit without disturbing the neighboring bits.

Quoted 8 lines
Quoted 4 lines

Is that it?

I don't _like_ the #defines, but doing one here doesn't seem too onerous considering how critical MSRs are.

How often is the 'msr' number not a compile-time constant? Adding rd/wrmsr variants that verify this would reduce the attack surface as well.

Oh, yes! I do this all the time with FORTIFY shenanigans. Right, so, instead of a macro, the "cannot be un-inlined" could be enforced with this (untested):

static __always_inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)

{
	u64 val, new_val;

	BUILD_BUG_ON(!__builtin_constant_p(msr) ||
		     !__builtin_constant_p(set) ||
		     !__builtin_constant_p(clear));

	rdmsrl(msr, val);
	new_val = (val & ~clear) | set;

	if (new_val != val)
		wrmsrl(msr, new_val);
}
Signature · 1 line
Kees Cook
#165 lore raw

RE: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

From: Kees Cook

Quoted 1 line

Sent: 04 October 2022 20:32

...

Quoted 11 lines

Oh, yes! I do this all the time with FORTIFY shenanigans. Right, so, instead of a macro, the "cannot be un-inlined" could be enforced with this (untested):

static __always_inline void set_clr_bits_msrl(u32 msr, u64 set, u64 clear)

{
	u64 val, new_val;

	BUILD_BUG_ON(!__builtin_constant_p(msr) ||
		     !__builtin_constant_p(set) ||
		     !__builtin_constant_p(clear));

You can reduce the amount of text the brain has to parse by using:

	BUILD_BUG_ON(!__builtin_constant_p(msr + set + clear));

Just requires the brain to do a quick 'oh yes'...

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#166 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

In reply to Kees Cook

Just now realizing, I never responded to most of this feedback as the later conversation focused in on one area. All seems good (thanks!), except not sure about the below:

Quoted 7 lines · Kees Cook

arch/x86/kernel/shstk.c line 51

Quoted 4 lines · Rick Edgecombe
arch/x86/kernel/shstk.c +4 -0
+
+	mmap_write_lock(mm);
+	addr = do_mmap(NULL, addr, size, PROT_READ, flags,
+		       VM_SHADOW_STACK | VM_WRITE, 0, &unused, NULL);

This will use the mmap base address offset randomization, I guess?

Yes.

Quoted 72 lines
Quoted 28 lines
arch/x86/kernel/shstk.c +16 -0
+
+	mmap_write_unlock(mm);
+
+	return addr;
+}
+
+static void unmap_shadow_stack(u64 base, u64 size)
+{
+	while (1) {
+		int r;
+
+		r = vm_munmap(base, size);
+
+		/*
+		 * vm_munmap() returns -EINTR when mmap_lock is held by
+		 * something else, and that lock should not be held for

a

arch/x86/kernel/shstk.c +9 -0
+		 * long time.  Retry it for the case.
+		 */
+		if (r == -EINTR) {
+			cond_resched();
+			continue;
+		}
+
+		/*
+		 * For all other types of vm_munmap() failure, either

the

arch/x86/kernel/shstk.c line 84

Quoted 8 lines
arch/x86/kernel/shstk.c +8 -0
+		 * system is out of memory or there is bug.
+		 */
+		WARN_ON_ONCE(r);
+		break;
+	}
+}
+
+int shstk_setup(void)

Only called local. Make static?

Quoted 33 lines
arch/x86/kernel/shstk.c +10 -0
+{
+	struct thread_shstk *shstk = &current->thread.shstk;
+	unsigned long addr, size;
+
+	/* Already enabled */
+	if (feature_enabled(CET_SHSTK))
+		return 0;
+
+	/* Also not supported for 32 bit */
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK) ||

in_ia32_syscall())

arch/x86/kernel/shstk.c +3 -0
+		return -EOPNOTSUPP;
+
+	size = PAGE_ALIGN(min_t(unsigned long long,

rlimit(RLIMIT_STACK), SZ_4G));

arch/x86/kernel/shstk.c +18 -0
+	addr = alloc_shstk(size);
+	if (IS_ERR_VALUE(addr))
+		return PTR_ERR((void *)addr);
+
+	fpu_lock_and_load();
+	wrmsrl(MSR_IA32_PL3_SSP, addr + size);
+	wrmsrl(MSR_IA32_U_CET, CET_SHSTK_EN);
+	fpregs_unlock();
+
+	shstk->base = addr;
+	shstk->size = size;
+	feature_set(CET_SHSTK);
+
+	return 0;
+}
+
+void reset_thread_shstk(void)
+{
Quoted 8 lines

arch/x86/kernel/shstk.c line 119

Quoted 4 lines
arch/x86/kernel/shstk.c +4 -0
+	memset(&current->thread.shstk, 0, sizeof(struct thread_shstk));
+	current->thread.features = 0;
+	current->thread.features_locked = 0;
+}

If features is always going to be tied to shstk, why not put them in the
shstk struct?

CET and LAM used to share an enabling interface and also kernel side enablement state tracking. But in the end LAM got its own enabling interface. Thomas had suggested that they could share a state field on the kernel side. But then LAM already had enough state tracking for it's needs.

Shadow stack used to track enabling with the fields in the shstk struct that keep track of the threads shadow stack. But then we added WRSS which needs another field to keep track of the status. So I thought to leave the 'features' field and use it for all the CET features tracking. I left it outside of the shstk struct so it looks usable for any other features that might be looking for an status bit. I can definitely compile it out when there is no user shadow stack.

snip

Quoted 7 lines
Quoted 7 lines
arch/x86/kernel/shstk.c +7 -0
+
+void shstk_free(struct task_struct *tsk)
+{
+	struct thread_shstk *shstk = &tsk->thread.shstk;
+
+	if (!cpu_feature_enabled(X86_FEATURE_SHSTK) ||
+	    !feature_enabled(CET_SHSTK))
Quoted 9 lines

arch/x86/kernel/shstk.c line 132

Quoted 6 lines
arch/x86/kernel/shstk.c +6 -0
+		return;
+
+	if (!tsk->mm)
+		return;
+
+	unmap_shadow_stack(shstk->base, shstk->size);

I feel like base and size should be zeroed here?

The code used to use shstk->base and shstk->size to keep track of if shadow stack was enabled. I'm not sure why to zero it now. Just defensively or did you see a concrete issue?

#167 lore raw

Re: [PATCH v2 24/39] x86/cet/shstk: Add user-mode shadow stack support

Quoted 3 lines · Rick P

The code used to use shstk->base and shstk->size to keep track of if shadow stack was enabled. I'm not sure why to zero it now. Just defensively or did you see a concrete issue?

Just to be defensive. It's not fast path by any means, to better to have values that make a bit of sense there. *shrug* It just stood out as feeling "missing" while I was reading the code.

Signature · 1 line
Kees Cook
#168 lore raw

PATCH v2 25/39 x86/cet/shstk: Handle thread shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways.

With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.

For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.

For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.

Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.

For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().

Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.

Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent.

During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner.

---

v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx)

v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.

Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)

Yu-cheng v29:
- WARN_ON_ONCE() when get_xsave_addr() returns NULL, and update comments. (Dave Hansen)

 arch/x86/include/asm/cet.h         |  7 +++++
 arch/x86/include/asm/fpu/sched.h   |  3 +-
 arch/x86/include/asm/mmu_context.h |  2 ++
 arch/x86/kernel/fpu/core.c         | 40 ++++++++++++++++++++++++-
 arch/x86/kernel/process.c          | 17 ++++++++++-
 arch/x86/kernel/shstk.c            | 48 +++++++++++++++++++++++++++++-
 6 files changed, 113 insertions(+), 4 deletions(-)
arch/x86/include/asm/cet.h +7 -0
index a4a1f4c0089b..924de99e0c61 100644
@@ -16,6 +16,9 @@ struct thread_shstk {
 long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features);
 int shstk_setup(void);
+int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
+			     unsigned long stack_size,
+			     unsigned long *shstk_addr);
 void shstk_free(struct task_struct *p);
 int shstk_disable(void);
 void reset_thread_shstk(void);
@@ -23,6 +26,10 @@ void reset_thread_shstk(void);
 static inline long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features) { return -EINVAL; }
 static inline int shstk_setup(void) { return -EOPNOTSUPP; }
+static inline int shstk_alloc_thread_stack(struct task_struct *p,
+					   unsigned long clone_flags,
+					   unsigned long stack_size,
+					   unsigned long *shstk_addr) { return 0; }
 static inline void shstk_free(struct task_struct *p) {}
 static inline int shstk_disable(void) { return -EOPNOTSUPP; }
 static inline void reset_thread_shstk(void) {}
arch/x86/include/asm/fpu/sched.h +2 -1
index b2486b2cbc6e..54c9c2fd1907 100644
@@ -11,7 +11,8 @@
 
 extern void save_fpregs_to_fpstate(struct fpu *fpu);
 extern void fpu__drop(struct fpu *fpu);
-extern int  fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal);
+extern int  fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,
+		      unsigned long shstk_addr);
 extern void fpu_flush_thread(void);
 
 /*
arch/x86/include/asm/mmu_context.h +2 -0
index b8d40ddeab00..d29988cbdf20 100644
@@ -146,6 +146,8 @@ do {						\
 #else
 #define deactivate_mm(tsk, mm)			\
 do {						\
+	if (!tsk->vfork_done)			\
+		shstk_free(tsk);		\
 	load_gs_index(0);			\
 	loadsegment(fs, 0);			\
 } while (0)
arch/x86/kernel/fpu/core.c +39 -1
index 778d3054ccc7..f332e9b42b6d 100644
@@ -555,8 +555,40 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu)
 	}
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;
+
+	xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave,
+				XFEATURE_CET_USER);
+
+	/*
+	 * If there is a non-zero ssp, then 'dst' must be configured with a shadow
+	 * stack and the fpu state should be up to date since it was just copied
+	 * from the parent in fpu_clone(). So there must be a valid non-init CET
+	 * state location in the buffer.
+	 */
+	if (WARN_ON_ONCE(!xstate))
+		return 1;
+
+	xstate->user_ssp = (u64)ssp;
+
+	return 0;
+}
+#else
+static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr)
+{
+}
+#endif
+
 /* Clone current's FPU state on fork */
-int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal)
+int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal,
+	      unsigned long ssp)
 {
 	struct fpu *src_fpu = &current->thread.fpu;
 	struct fpu *dst_fpu = &dst->thread.fpu;
@@ -616,6 +648,12 @@ int fpu_clone(struct task_struct *dst, unsigned long clone_flags, bool minimal)
 	if (use_xsave())
 		dst_fpu->fpstate->regs.xsave.header.xfeatures &= ~XFEATURE_MASK_PASID;
 
+	/*
+	 * Update shadow stack pointer, in case it changed during clone.
+	 */
+	if (update_fpu_shstk(dst, ssp))
+		return 1;
+
 	trace_x86_fpu_copy_src(src_fpu);
 	trace_x86_fpu_copy_dst(dst_fpu);
 
arch/x86/kernel/process.c +16 -1
index 034880311e6b..5e63d190becd 100644
@@ -47,6 +47,7 @@
 #include <asm/frame.h>
 #include <asm/unwind.h>
 #include <asm/tdx.h>
+#include <asm/cet.h>
 
 #include "process.h"
 
@@ -118,6 +119,7 @@ void exit_thread(struct task_struct *tsk)
 
 	free_vm86(t);
 
+	shstk_free(tsk);
 	fpu__drop(fpu);
 }
 
@@ -139,6 +141,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	struct inactive_task_frame *frame;
 	struct fork_frame *fork_frame;
 	struct pt_regs *childregs;
+	unsigned long shstk_addr = 0;
 	int ret = 0;
 
 	childregs = task_pt_regs(p);
@@ -173,7 +176,12 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	frame->flags = X86_EFLAGS_FIXED;
 #endif
 
-	fpu_clone(p, clone_flags, args->fn);
+	/* Allocate a new shadow stack for pthread if needed */
+	ret = shstk_alloc_thread_stack(p, clone_flags, args->flags, &shstk_addr);
+	if (ret)
+		return ret;
+
+	fpu_clone(p, clone_flags, args->fn, shstk_addr);
 
 	/* Kernel thread ? */
 	if (unlikely(p->flags & PF_KTHREAD)) {
@@ -219,6 +227,13 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	if (!ret && unlikely(test_tsk_thread_flag(current, TIF_IO_BITMAP)))
 		io_bitmap_share(p);
 
+	/*
+	 * If copy_thread() if failing, don't leak the shadow stack possibly
+	 * allocated in shstk_alloc_thread_stack() above.
+	 */
+	if (ret)
+		shstk_free(p);
+
 	return ret;
 }
 
arch/x86/kernel/shstk.c +47 -1

60 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#169 lore raw

Re: [PATCH v2 25/39] x86/cet/shstk: Handle thread shadow stack

Quoted 107 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a process is duplicated, but the child shares the address space with the parent, there is potential for the threads sharing a single stack to cause conflicts for each other. In the normal non-cet case this is handled in two ways.

With regular CLONE_VM a new stack is provided by userspace such that the parent and child have different stacks.

For vfork, the parent is suspended until the child exits. So as long as the child doesn't return from the vfork()/CLONE_VFORK calling function and sticks to a limited set of operations, the parent and child can share the same stack.

For shadow stack, these scenarios present similar sharing problems. For the CLONE_VM case, the child and the parent must have separate shadow stacks. Instead of changing clone to take a shadow stack, have the kernel just allocate one and switch to it.

Use stack_size passed from clone3() syscall for thread shadow stack size. A compat-mode thread shadow stack size is further reduced to 1/4. This allows more threads to run in a 32-bit address space. The clone() does not pass stack_size, which was added to clone3(). In that case, use RLIMIT_STACK size and cap to 4 GB.

For shadow stack enabled vfork(), the parent and child can share the same shadow stack, like they can share a normal stack. Since the parent is suspended until the child terminates, the child will not interfere with the parent while executing as long as it doesn't return from the vfork() and overwrite up the shadow stack. The child can safely overwrite down the shadow stack, as the parent can just overwrite this later. So CET does not add any additional limitations for vfork().

Userspace implementing posix vfork() can actually prevent the child from returning from the vfork() calling function, using CET. Glibc does this by adjusting the shadow stack pointer in the child, so that the child receives a #CP if it tries to return from vfork() calling function.

Free the shadow stack on thread exit by doing it in mm_release(). Skip this when exiting a vfork() child since the stack is shared in the parent.

During this operation, the shadow stack pointer of the new thread needs to be updated to point to the newly allocated shadow stack. Since the ability to do this is confined to the FPU subsystem, change fpu_clone() to take the new shadow stack pointer, and update it internally inside the FPU subsystem. This part was suggested by Thomas Gleixner.

---

v2:
- Have fpu_clone() take new shadow stack pointer and update SSP in xsave buffer for new task. (tglx)

v1:
- Expand commit log.
- Add more comments.
- Switch to xsave helpers.

Yu-cheng v30:
- Update comments about clone()/clone3(). (Borislav Petkov)

Yu-cheng v29:
- WARN_ON_ONCE() when get_xsave_addr() returns NULL, and update comments. (Dave Hansen)

 arch/x86/include/asm/cet.h         |  7 +++++
 arch/x86/include/asm/fpu/sched.h   |  3 +-
 arch/x86/include/asm/mmu_context.h |  2 ++
 arch/x86/kernel/fpu/core.c         | 40 ++++++++++++++++++++++++-
 arch/x86/kernel/process.c          | 17 ++++++++++-
 arch/x86/kernel/shstk.c            | 48 +++++++++++++++++++++++++++++-
 6 files changed, 113 insertions(+), 4 deletions(-)
arch/x86/kernel/fpu/core.c +18 -0
index 778d3054ccc7..f332e9b42b6d 100644
@@ -555,8 +555,40 @@ static inline void fpu_inherit_perms(struct fpu *dst_fpu)
 	}
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;
+
+	xstate = get_xsave_addr(&dst->thread.fpu.fpstate->regs.xsave,
+				XFEATURE_CET_USER);
+
+	/*
+	 * If there is a non-zero ssp, then 'dst' must be configured with a shadow
+	 * stack and the fpu state should be up to date since it was just copied
+	 * from the parent in fpu_clone(). So there must be a valid non-init CET
+	 * state location in the buffer.
+	 */

arch/x86/kernel/fpu/core.c static inline void fpu_inherit_perms(struct fpu *dst_fpu) · line 585

Quoted 10 lines
arch/x86/kernel/fpu/core.c +10 -0
+	if (WARN_ON_ONCE(!xstate))
+		return 1;
+
+	xstate->user_ssp = (u64)ssp;
+
+	return 0;
+}
+#else
+static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr)
+{

return 0; ?

Quoted 3 lines
arch/x86/kernel/fpu/core.c +3 -0
+}
+#endif
+
Signature · 2 lines
Sincerely yours,
Mike.
#170 lore raw

Re: [PATCH v2 25/39] x86/cet/shstk: Handle thread shadow stack

Quoted 9 lines · Mike Rapoport
Quoted 3 lines

+static int update_fpu_shstk(struct task_struct *dst, unsigned long shstk_addr)
+{

return 0; ?

arch/x86/kernel/fpu/core.c static inline void fpu_inherit_perms(struct fpu *dst_fpu) · line 588

Quoted 3 lines
arch/x86/kernel/fpu/core.c +3 -0
+}
+#endif
+

Oops. It was a last minute change to have update_fpu_shstk() return int. Thanks.

#171 lore raw

Re: [PATCH v2 25/39] x86/cet/shstk: Handle thread shadow stack

In reply to Rick Edgecombe

arch/x86/kernel/fpu/core.c static inline void fpu_inherit_perms(struct fpu *dst_fpu) · line 565

Quoted 9 lines · Rick Edgecombe

[...]

arch/x86/kernel/fpu/core.c +8 -0
+#ifdef CONFIG_X86_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long ssp)
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;

My brain will work to undo the collision of Shadow Stack Pointer with Stack Smashing Protection. ;)

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 122

Quoted 11 lines

[...]

arch/x86/kernel/shstk.c +2 -0
index a0b8d4adb2bf..db4e53f9fdaf 100644
@@ -118,6 +118,46 @@ void reset_thread_shstk(void)
 	current->thread.features_locked = 0;
 }
 
+int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
+			     unsigned long stack_size, unsigned long *shstk_addr)

Er, arg 3 is "stack_size". From later:

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 123

Quoted 1 line
arch/x86/kernel/shstk.c +1 -0
+     ret = shstk_alloc_thread_stack(p, clone_flags, args->flags, &shstk_addr);
                                                       ^^^^^^^^^^^

clone_flags and args->flags are identical ... this must be accidentally working. I was expecting 0 there.

Quoted 7 lines
arch/x86/kernel/shstk.c +7 -0
+{
+	struct thread_shstk *shstk = &tsk->thread.shstk;
+	unsigned long addr;
+
+	/*
+	 * If shadow stack is not enabled on the new thread, skip any
+	 * switch to a new shadow stack.

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 139

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+	 */
+	if (!feature_enabled(CET_SHSTK))
+		return 0;
+
+	/*
+	 * clone() does not pass stack_size, which was added to clone3().
+	 * Use RLIMIT_STACK and cap to 4 GB.
+	 */
+	if (!stack_size)
+		stack_size = min_t(unsigned long long, rlimit(RLIMIT_STACK), SZ_4G);

Again, perhaps the clamp should happen in alloc_shstk()?

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 149

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+
+	/*
+	 * For CLONE_VM, except vfork, the child needs a separate shadow
+	 * stack.
+	 */
+	if ((clone_flags & (CLONE_VFORK | CLONE_VM)) != CLONE_VM)
+		return 0;
+
+
+	stack_size = PAGE_ALIGN(stack_size);

Uhm, I think a line went missing here. :P

"x86/cet/shstk: Introduce map_shadow_stack syscall" adds the missing:

+ addr = alloc_shstk(0, stack_size, 0, false);

Please add back the original. :)

Quoted 32 lines
arch/x86/kernel/shstk.c +18 -1
+	if (IS_ERR_VALUE(addr))
+		return PTR_ERR((void *)addr);
+
+	shstk->base = addr;
+	shstk->size = stack_size;
+
+	*shstk_addr = addr + stack_size;
+
+	return 0;
+}
+
 void shstk_free(struct task_struct *tsk)
 {
 	struct thread_shstk *shstk = &tsk->thread.shstk;
@@ -126,7 +166,13 @@ void shstk_free(struct task_struct *tsk)
 	    !feature_enabled(CET_SHSTK))
 		return;
 
-	if (!tsk->mm)
+	/*
+	 * When fork() with CLONE_VM fails, the child (tsk) already has a
+	 * shadow stack allocated, and exit_thread() calls this function to
+	 * free it.  In this case the parent (current) and the child share
+	 * the same mm struct.
+	 */
+	if (!tsk->mm || tsk->mm != current->mm)
 		return;
 
 	unmap_shadow_stack(shstk->base, shstk->size);
Signature · 1 line
2.17.1
Signature · 1 line
Kees Cook
#172 lore raw

Re: [PATCH v2 25/39] x86/cet/shstk: Handle thread shadow stack

Quoted 31 lines · Kees Cook

arch/x86/kernel/fpu/core.c static inline void fpu_inherit_perms(struct fpu *dst_fpu) · line 565

Quoted 10 lines · Rick Edgecombe

[...]

(quoted diff) +2 -0
+#ifdef CONFIG_X86_SHADOW_STACK
+static int update_fpu_shstk(struct task_struct *dst, unsigned long

ssp)

arch/x86/kernel/fpu/core.c +6 -0
+{
+	struct cet_user_state *xstate;
+
+	/* If ssp update is not needed. */
+	if (!ssp)
+		return 0;

My brain will work to undo the collision of Shadow Stack Pointer with Stack Smashing Protection. ;)

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 124

Quoted 13 lines

[...]

arch/x86/kernel/shstk.c +2 -0
index a0b8d4adb2bf..db4e53f9fdaf 100644
@@ -118,6 +118,46 @@ void reset_thread_shstk(void)
 	current->thread.features_locked = 0;
 }
 
+int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned
 long clone_flags,
+			     unsigned long stack_size, unsigned long
 *shstk_addr)

Er, arg 3 is "stack_size". From later:

Quoted 8 lines

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 126

Quoted 2 lines
arch/x86/kernel/shstk.c +1 -0
+     ret = shstk_alloc_thread_stack(p, clone_flags, args->flags,
 &shstk_addr);
                                                       ^^^^^^^^^^^

clone_flags and args->flags are identical ... this must be accidentally
working. I was expecting 0 there.

Oh wow. A stack_size used to be passed into copy_thread(), but I messed up the rebase badly. Thanks for catching it.

Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/shstk.c +13 -0
+{
+	struct thread_shstk *shstk = &tsk->thread.shstk;
+	unsigned long addr;
+
+	/*
+	 * If shadow stack is not enabled on the new thread, skip any
+	 * switch to a new shadow stack.
+	 */
+	if (!feature_enabled(CET_SHSTK))
+		return 0;
+
+	/*
+	 * clone() does not pass stack_size, which was added to
Quoted 8 lines
Quoted 6 lines

clone3().

arch/x86/kernel/shstk.c +4 -0
+	 * Use RLIMIT_STACK and cap to 4 GB.
+	 */
+	if (!stack_size)
+		stack_size = min_t(unsigned long long,

rlimit(RLIMIT_STACK), SZ_4G);

Again, perhaps the clamp should happen in alloc_shstk()?

The map_shadow_stack() is kind of like mmap(). I think it shouldn't get the rlimit restriction. But I can pull the shared logic into a helper for the other two cases.

Quoted 19 lines

arch/x86/kernel/shstk.c void reset_thread_shstk(void) · line 149

Quoted 12 lines
arch/x86/kernel/shstk.c +3 -0
+
+	/*
+	 * For CLONE_VM, except vfork, the child needs a separate

shadow

arch/x86/kernel/shstk.c +7 -0
+	 * stack.
+	 */
+	if ((clone_flags & (CLONE_VFORK | CLONE_VM)) != CLONE_VM)
+		return 0;
+
+
+	stack_size = PAGE_ALIGN(stack_size);

Uhm, I think a line went missing here. :P

"x86/cet/shstk: Introduce map_shadow_stack syscall" adds the missing:

(quoted diff) +1 -0
+	addr = alloc_shstk(0, stack_size, 0, false);
Quoted 1 line

Please add back the original. :)

Yes, more rebase mangling. Thanks.

Quoted 38 lines
Quoted 36 lines
arch/x86/kernel/shstk.c +13 -1
+	if (IS_ERR_VALUE(addr))
+		return PTR_ERR((void *)addr);
+
+	shstk->base = addr;
+	shstk->size = stack_size;
+
+	*shstk_addr = addr + stack_size;
+
+	return 0;
+}
+
 void shstk_free(struct task_struct *tsk)
 {
 	struct thread_shstk *shstk = &tsk->thread.shstk;
@@ -126,7 +166,13 @@ void shstk_free(struct task_struct *tsk)
 	    !feature_enabled(CET_SHSTK))
 		return;
 
-	if (!tsk->mm)
+	/*
+	 * When fork() with CLONE_VM fails, the child (tsk) already has

a

arch/x86/kernel/shstk.c +1 -0
+	 * shadow stack allocated, and exit_thread() calls this

function to

arch/x86/kernel/shstk.c +1 -0
+	 * free it.  In this case the parent (current) and the child

share

arch/x86/kernel/shstk.c +3 -0
+	 * the same mm struct.
+	 */
+	if (!tsk->mm || tsk->mm != current->mm)
 		return;
 
 	unmap_shadow_stack(shstk->base, shstk->size);
Signature · 1 line
2.17.1
#173 lore raw

PATCH v2 26/39 x86/cet/shstk: Introduce routines modifying shstk

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack's are normally written to via CALL/RET or specific CET instuctions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.

To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.

---

v2:
- Add data helpers for writing to shadow stack.

v1:
- Use xsave helpers.

Yu-cheng v30:
- Update commit log, remove description about signals.
- Update various comments.
- Remove variable 'ssp' init and adjust return value accordingly.
- Check get_user_shstk_addr() return value.
- Replace 'ia32' with 'proc32'.

Yu-cheng v29:
- Update comments for the use of get_xsave_addr().

 arch/x86/include/asm/special_insns.h |  13 ++++
 arch/x86/kernel/shstk.c              | 108 +++++++++++++++++++++++++++
 2 files changed, 121 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
index 35f709f619fb..f096f52bd059 100644
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);
+	return 0;
+fail:
+	return -EFAULT;
+}
+#endif /* CONFIG_X86_SHADOW_STACK */
+
 #define nop() asm volatile ("nop")
 
 static inline void serialize(void)
arch/x86/kernel/shstk.c +108 -0

126 lines not on this page · raw message

Signature · 1 line
2.17.1
#174 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

Quoted 100 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack's are normally written to via CALL/RET or specific CET instuctions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and the address in a token must point directly above the token, which is within the same shadow stack. This is distinctively different from other pointers on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel will need something to denote the point in the stack where sigreturn may be called. This will prevent attackers calling sigreturn at arbitrary places in the stack, in order to help prevent SROP attacks.

To do this, something that can only be written by the kernel needs to be placed on the shadow stack. This can be accomplished by setting bit 63 in the frame written to the shadow stack. Userspace return addresses can't have this bit set as it is in the kernel range. It is also can't be a valid restore token.

---

v2:
- Add data helpers for writing to shadow stack.

v1:
- Use xsave helpers.

Yu-cheng v30:
- Update commit log, remove description about signals.
- Update various comments.
- Remove variable 'ssp' init and adjust return value accordingly.
- Check get_user_shstk_addr() return value.
- Replace 'ia32' with 'proc32'.

Yu-cheng v29:
- Update comments for the use of get_xsave_addr().

 arch/x86/include/asm/special_insns.h |  13 ++++
 arch/x86/kernel/shstk.c              | 108 +++++++++++++++++++++++++++
 2 files changed, 121 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
index 35f709f619fb..f096f52bd059 100644
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);
+	return 0;
+fail:
+	return -EFAULT;
+}
+#endif /* CONFIG_X86_SHADOW_STACK */
+
 #define nop() asm volatile ("nop")
 
 static inline void serialize(void)
arch/x86/kernel/shstk.c +8 -0
index db4e53f9fdaf..8904aef487bf 100644
@@ -25,6 +25,8 @@
 #include <asm/fpu/api.h>
 #include <asm/prctl.h>
 
+#define SS_FRAME_SIZE 8
+
 static bool feature_enabled(unsigned long features)
 {
 	return current->thread.features & features;
@@ -40,6 +42,31 @@ static void feature_clr(unsigned long features)
 	current->thread.features &= ~features;
 }
 
+/*
+ * Create a restore token on the shadow stack.  A token is always 8-byte
+ * and aligned to 8.
+ */
+static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)
+{

arch/x86/kernel/shstk.c static void feature_clr(unsigned long features) · line 60

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+	unsigned long addr;
+
+	/* Token must be aligned */
+	if (!IS_ALIGNED(ssp, 8))
+		return -EINVAL;
+
+	addr = ssp - SS_FRAME_SIZE;
+
+	/* Mark the token 64-bit */
+	ssp |= BIT(0);

Wow, that confused me for a moment. :) SDE says:

- Bit 63:2 – Value of shadow stack pointer when this restore point was created.
- Bit 1 – Reserved. Must be zero.
- Bit 0 – Mode bit. If 0, the token is a compatibility/legacy mode “shadow stack restore” token. If 1, then this shadow stack restore token can be used with a RSTORSSP instruction in 64-bit mode.

So shouldn't this actually be:

	ssp &= ~BIT(1);	/* Reserved */
	ssp |=  BIT(0); /* RSTORSSP instruction in 64-bit mode */
Quoted 22 lines
arch/x86/kernel/shstk.c +15 -0
+
+	if (write_user_shstk_64((u64 __user *)addr, (u64)ssp))
+		return -EFAULT;
+
+	*token_addr = addr;
+
+	return 0;
+}
+
 static unsigned long alloc_shstk(unsigned long size)
 {
 	int flags = MAP_ANONYMOUS | MAP_PRIVATE;
@@ -158,6 +185,87 @@ int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags,
 	return 0;
 }
 
+static unsigned long get_user_shstk_addr(void)
+{
+	unsigned long long ssp;
+
+	fpu_lock_and_load();
+

arch/x86/kernel/shstk.c int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, · line 203

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	fpregs_unlock();
+
+	return ssp;
+}
+
+static int put_shstk_data(u64 __user *addr, u64 data)
+{
+	WARN_ON(data & BIT(63));

Let's make this a bit more defensive:

	if (WARN_ON_ONCE(data & BIT(63)))
		return -EFAULT;
Quoted 36 lines
arch/x86/kernel/shstk.c +36 -0
+
+	/*
+	 * Mark the high bit so that the sigframe can't be processed as a
+	 * return address.
+	 */
+	if (write_user_shstk_64(addr, data | BIT(63)))
+		return -EFAULT;
+	return 0;
+}
+
+static int get_shstk_data(unsigned long *data, unsigned long __user *addr)
+{
+	unsigned long ldata;
+
+	if (unlikely(get_user(ldata, addr)))
+		return -EFAULT;
+
+	if (!(ldata & BIT(63)))
+		return -EINVAL;
+
+	*data = ldata & ~BIT(63);
+
+	return 0;
+}
+
+/*
+ * Verify the user shadow stack has a valid token on it, and then set
+ * *new_ssp according to the token.
+ */
+static int shstk_check_rstor_token(unsigned long *new_ssp)
+{
+	unsigned long token_addr;
+	unsigned long token;
+
+	token_addr = get_user_shstk_addr();
+	if (!token_addr)

arch/x86/kernel/shstk.c int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, · line 249

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+		return -EINVAL;
+
+	if (get_user(token, (unsigned long __user *)token_addr))
+		return -EFAULT;
+
+	/* Is mode flag correct? */
+	if (!(token & BIT(0)))
+		return -EINVAL;
+
+	/* Is busy flag set? */

"Busy"? Not "Reserved"?

Quoted 25 lines
arch/x86/kernel/shstk.c +19 -0
+	if (token & BIT(1))
+		return -EINVAL;
+
+	/* Mask out flags */
+	token &= ~3UL;
+
+	/* Restore address aligned? */
+	if (!IS_ALIGNED(token, 8))
+		return -EINVAL;
+
+	/* Token placed properly? */
+	if (((ALIGN_DOWN(token, 8) - 8) != token_addr) || token >= TASK_SIZE_MAX)
+		return -EINVAL;
+
+	*new_ssp = token;
+
+	return 0;
+}
+
 void shstk_free(struct task_struct *tsk)
 {
 	struct thread_shstk *shstk = &tsk->thread.shstk;
Signature · 1 line
2.17.1
Signature · 1 line
Kees Cook
#175 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

Quoted 140 lines · Kees Cook
Quoted 117 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

Shadow stack's are normally written to via CALL/RET or specific CET instuctions like RSTORSSP/SAVEPREVSSP. However during some Linux operations the kernel will need to write to directly using the ring-0 only
WRUSS instruction.

A shadow stack restore token marks a restore point of the shadow stack, and
the address in a token must point directly above the token, which is within
the same shadow stack. This is distinctively different from other pointers
on the shadow stack, since those pointers point to executable code area.

Introduce token setup and verify routines. Also introduce WRUSS, which is
a kernel-mode instruction but writes directly to user shadow stack.

In future patches that enable shadow stack to work with signals, the kernel
will need something to denote the point in the stack where sigreturn may be
called. This will prevent attackers calling sigreturn at arbitrary places
in the stack, in order to help prevent SROP attacks.

To do this, something that can only be written by the kernel needs to be
placed on the shadow stack. This can be accomplished by setting bit 63 in
the frame written to the shadow stack. Userspace return addresses can't
have this bit set as it is in the kernel range. It is also can't be a
valid restore token.

---

v2:
- Add data helpers for writing to shadow stack.

v1:
- Use xsave helpers.

Yu-cheng v30:
- Update commit log, remove description about signals.
- Update various comments.
- Remove variable 'ssp' init and adjust return value accordingly.
- Check get_user_shstk_addr() return value.
- Replace 'ia32' with 'proc32'.

Yu-cheng v29:
- Update comments for the use of get_xsave_addr().

 arch/x86/include/asm/special_insns.h |  13 ++++
 arch/x86/kernel/shstk.c              | 108
+++++++++++++++++++++++++++
 2 files changed, 121 insertions(+)
arch/x86/include/asm/special_insns.h +13 -0
index 35f709f619fb..f096f52bd059 100644
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);
+	return 0;
+fail:
+	return -EFAULT;
+}
+#endif /* CONFIG_X86_SHADOW_STACK */
+
 #define nop() asm volatile ("nop")
 
 static inline void serialize(void)
arch/x86/kernel/shstk.c +4 -0
index db4e53f9fdaf..8904aef487bf 100644
@@ -25,6 +25,8 @@
 #include <asm/fpu/api.h>
 #include <asm/prctl.h>
 
+#define SS_FRAME_SIZE 8
+
 static bool feature_enabled(unsigned long features)
 {
 	return current->thread.features & features;
@@ -40,6 +42,31 @@ static void feature_clr(unsigned long features)
 	current->thread.features &= ~features;
 }
 
+/*
+ * Create a restore token on the shadow stack.  A token is always

8-byte
+ * and aligned to 8.
+ */
+static int create_rstor_token(unsigned long ssp, unsigned long *token_addr)

arch/x86/kernel/shstk.c +1 -0
+{

arch/x86/kernel/shstk.c static void feature_clr(unsigned long features) · line 60

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+	unsigned long addr;
+
+	/* Token must be aligned */
+	if (!IS_ALIGNED(ssp, 8))
+		return -EINVAL;
+
+	addr = ssp - SS_FRAME_SIZE;
+
+	/* Mark the token 64-bit */
+	ssp |= BIT(0);

Wow, that confused me for a moment. :) SDE says:

- Bit 63:2 – Value of shadow stack pointer when this restore point was created.
- Bit 1 – Reserved. Must be zero.
- Bit 0 – Mode bit. If 0, the token is a compatibility/legacy mode “shadow stack restore” token. If 1, then this shadow stack
restore
token can be used with a RSTORSSP instruction in 64-bit
mode.

Quoted 4 lines

So shouldn't this actually be:

	ssp &= ~BIT(1);	/* Reserved */
	ssp |=  BIT(0); /* RSTORSSP instruction in 64-bit mode */

Since SSP is aligned, it should not have bits 0 to 2 set. I'll add a comment.

Quoted 37 lines
Quoted 24 lines
arch/x86/kernel/shstk.c +15 -0
+
+	if (write_user_shstk_64((u64 __user *)addr, (u64)ssp))
+		return -EFAULT;
+
+	*token_addr = addr;
+
+	return 0;
+}
+
 static unsigned long alloc_shstk(unsigned long size)
 {
 	int flags = MAP_ANONYMOUS | MAP_PRIVATE;
@@ -158,6 +185,87 @@ int shstk_alloc_thread_stack(struct
 task_struct *tsk, unsigned long clone_flags,
 	return 0;
 }
 
+static unsigned long get_user_shstk_addr(void)
+{
+	unsigned long long ssp;
+
+	fpu_lock_and_load();
+

arch/x86/kernel/shstk.c int shstk_alloc_thread_stack(struct · line 204

Quoted 10 lines
arch/x86/kernel/shstk.c +10 -0
+	rdmsrl(MSR_IA32_PL3_SSP, ssp);
+
+	fpregs_unlock();
+
+	return ssp;
+}
+
+static int put_shstk_data(u64 __user *addr, u64 data)
+{
+	WARN_ON(data & BIT(63));

Let's make this a bit more defensive:

Quoted 2 lines
	if (WARN_ON_ONCE(data & BIT(63)))
		return -EFAULT;

Hmm, sure. I'm thinking EINVAL since the failure has nothing to do with faulting.

Quoted 44 lines
Quoted 44 lines
arch/x86/kernel/shstk.c +3 -0
+
+	/*
+	 * Mark the high bit so that the sigframe can't be processed as

a

arch/x86/kernel/shstk.c +8 -0
+	 * return address.
+	 */
+	if (write_user_shstk_64(addr, data | BIT(63)))
+		return -EFAULT;
+	return 0;
+}
+
+static int get_shstk_data(unsigned long *data, unsigned long

__user *addr)

arch/x86/kernel/shstk.c +16 -0
+{
+	unsigned long ldata;
+
+	if (unlikely(get_user(ldata, addr)))
+		return -EFAULT;
+
+	if (!(ldata & BIT(63)))
+		return -EINVAL;
+
+	*data = ldata & ~BIT(63);
+
+	return 0;
+}
+
+/*
+ * Verify the user shadow stack has a valid token on it, and then

set
+ * *new_ssp according to the token.
+ */

arch/x86/kernel/shstk.c +11 -0
+static int shstk_check_rstor_token(unsigned long *new_ssp)
+{
+	unsigned long token_addr;
+	unsigned long token;
+
+	token_addr = get_user_shstk_addr();
+	if (!token_addr)
+		return -EINVAL;
+
+	if (get_user(token, (unsigned long __user *)token_addr))
+		return -EFAULT;
Quoted 8 lines

arch/x86/kernel/shstk.c int shstk_alloc_thread_stack(struct task_struct *tsk, unsigned long clone_flags, · line 249

Quoted 6 lines
arch/x86/kernel/shstk.c +6 -0
+
+	/* Is mode flag correct? */
+	if (!(token & BIT(0)))
+		return -EINVAL;
+
+	/* Is busy flag set? */

"Busy"? Not "Reserved"?

Yes reserved is more accurate, I'll change it. In a previous-ssp token it is 1, so kind of busy-like. That is probably where the comment came from.

Quoted 29 lines
Quoted 27 lines
arch/x86/kernel/shstk.c +12 -0
+	if (token & BIT(1))
+		return -EINVAL;
+
+	/* Mask out flags */
+	token &= ~3UL;
+
+	/* Restore address aligned? */
+	if (!IS_ALIGNED(token, 8))
+		return -EINVAL;
+
+	/* Token placed properly? */
+	if (((ALIGN_DOWN(token, 8) - 8) != token_addr) || token >=

TASK_SIZE_MAX)

arch/x86/kernel/shstk.c +7 -0
+		return -EINVAL;
+
+	*new_ssp = token;
+
+	return 0;
+}
+
 void shstk_free(struct task_struct *tsk)
 {
 	struct thread_shstk *shstk = &tsk->thread.shstk;
Signature · 1 line
2.17.1
#176 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

In reply to Rick Edgecombe
Quoted 5 lines · Rick Edgecombe
arch/x86/include/asm/special_insns.h +0 -0
index 35f709f619fb..f096f52bd059 100644

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 232

Quoted 10 lines
arch/x86/include/asm/special_insns.h +7 -0
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
 		: [pax] "a" (p));
 }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+	asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+			  _ASM_EXTABLE(1b, %l[fail])
+			  :: [addr] "r" (addr), [val] "r" (val)
+			  :: fail);

"1: wrssq %[val], %[addr]\n"
_ASM_EXTABLE(1b, %l[fail])
: [addr] "+m" (*addr)
: [val] "r" (val)
:: fail

Otherwise you've failed to tell the compiler that you wrote to *addr.

With that fixed, it's not volatile because there are no unexpressed side effects.

~Andrew

#177 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

Quoted 24 lines · Andrew Cooper
Quoted 6 lines · Rick Edgecombe
arch/x86/include/asm/special_insns.h +0 -0
index 35f709f619fb..f096f52bd059 100644

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 232

Quoted 10 lines
arch/x86/include/asm/special_insns.h +7 -0
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
               : [pax] "a" (p));
  }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+     asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+                       _ASM_EXTABLE(1b, %l[fail])
+                       :: [addr] "r" (addr), [val] "r" (val)
+                       :: fail);

"1: wrssq %[val], %[addr]\n"
_ASM_EXTABLE(1b, %l[fail])
: [addr] "+m" (*addr)
: [val] "r" (val)
:: fail

Quoted 5 lines

Otherwise you've failed to tell the compiler that you wrote to *addr.

With that fixed, it's not volatile because there are no unexpressed side
effects.

Ok, thanks!

#178 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

Quoted 24 lines · Rick P
Quoted 23 lines · Andrew Cooper
Quoted 6 lines · Rick Edgecombe
arch/x86/include/asm/special_insns.h +0 -0
index 35f709f619fb..f096f52bd059 100644

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void *__p) · line 232

Quoted 10 lines
arch/x86/include/asm/special_insns.h +7 -0
@@ -223,6 +223,19 @@ static inline void clwb(volatile void *__p)
               : [pax] "a" (p));
  }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64 val)
+{
+     asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+                       _ASM_EXTABLE(1b, %l[fail])
+                       :: [addr] "r" (addr), [val] "r" (val)
+                       :: fail);

"1: wrssq %[val], %[addr]\n"
_ASM_EXTABLE(1b, %l[fail])
: [addr] "+m" (*addr)
: [val] "r" (val)
:: fail

Quoted 6 lines
Quoted 5 lines

Otherwise you've failed to tell the compiler that you wrote to *addr.

With that fixed, it's not volatile because there are no unexpressed side
effects.

Ok, thanks!

On further consideration, it should be "=m" not "+m", which is even less constrained, and even easier for an enterprising optimiser to try and do something useful with.

~Andrew

#179 lore raw

Re: [PATCH v2 26/39] x86/cet/shstk: Introduce routines modifying shstk

Quoted 36 lines · Andrew Cooper
Quoted 35 lines · Rick P
Quoted 33 lines · Andrew Cooper
Quoted 8 lines · Rick Edgecombe
arch/x86/include/asm/special_insns.h +0 -0
index 35f709f619fb..f096f52bd059 100644
@@ -223,6 +223,19 @@ static inline void clwb(volatile void
 *__p)
                : [pax] "a" (p));

arch/x86/include/asm/special_insns.h static inline void clwb(volatile void · line 234

Quoted 10 lines
arch/x86/include/asm/special_insns.h +7 -0
   }
 
+#ifdef CONFIG_X86_SHADOW_STACK
+static inline int write_user_shstk_64(u64 __user *addr, u64
 val)
+{
+     asm_volatile_goto("1: wrussq %[val], (%[addr])\n"
+                       _ASM_EXTABLE(1b, %l[fail])
+                       :: [addr] "r" (addr), [val] "r" (val)
+                       :: fail);

"1: wrssq %[val], %[addr]\n"
_ASM_EXTABLE(1b, %l[fail])
: [addr] "+m" (*addr)
: [val] "r" (val)
:: fail

Otherwise you've failed to tell the compiler that you wrote to *addr.

With that fixed, it's not volatile because there are no unexpressed
side
effects.

Quoted 7 lines
Quoted 1 line

Ok, thanks!

On further consideration, it should be "=m" not "+m", which is even less
constrained, and even easier for an enterprising optimiser to try and do
something useful with.

AFAICT this won't work on all gccs. Asm goto's used to not support having outputs. They are also implicitly volatile anyway. So I think I'll have to leave it. But I can change the wrss one in the selftest to "=m".

#180 lore raw

PATCH v2 27/39 x86/cet/shstk: Handle signals for shadow stack

In reply to Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.

One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.

The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.

For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.

So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.

In sigreturn, verify SSP is stored in the data format and pop the shadow stack.

Cc Andy Lutomirski <[email protected]>
Cc Cyrill Gorcunov <[email protected]>
Cc Florian Weimer <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Kees Cook <[email protected]>

---

v2:
- Switch to new shstk signal format

v1:
- Use xsave helpers.
- Expand commit log.

Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.

Yu-cheng v25:
- Update commit log/comments for the sc_ext struct.
- Use restorer address already calculated.
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.
- Eliminate writing to MSR_IA32_U_CET for shadow stack.
- Change wrmsrl() to wrmsrl_safe() and handle error.

 arch/x86/ia32/ia32_signal.c |   1 +
 arch/x86/include/asm/cet.h  |   5 ++
 arch/x86/kernel/shstk.c     | 126 ++++++++++++++++++++++++++++++------
 arch/x86/kernel/signal.c    |  10 +++
 4 files changed, 123 insertions(+), 19 deletions(-)
arch/x86/ia32/ia32_signal.c +1 -0
index c9c3859322fa..88d71b9de616 100644
@@ -34,6 +34,7 @@
 #include <asm/sigframe.h>
 #include <asm/sighandling.h>
 #include <asm/smap.h>
+#include <asm/cet.h>
 
 static inline void reload_segments(struct sigcontext_32 *sc)
 {
arch/x86/include/asm/cet.h +5 -0
index 924de99e0c61..8c6fab9f402a 100644
@@ -6,6 +6,7 @@
 #include <linux/types.h>
 
 struct task_struct;
+struct ksignal;
 
 struct thread_shstk {
 	u64	base;
@@ -22,6 +23,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
 void shstk_free(struct task_struct *p);
 int shstk_disable(void);
 void reset_thread_shstk(void);
+int setup_signal_shadow_stack(struct ksignal *ksig);
+int restore_signal_shadow_stack(void);
 #else
 static inline long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features) { return -EINVAL; }
@@ -33,6 +36,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p,
 static inline void shstk_free(struct task_struct *p) {}
 static inline int shstk_disable(void) { return -EOPNOTSUPP; }
 static inline void reset_thread_shstk(void) {}
+static inline int setup_signal_shadow_stack(struct ksignal *ksig) { return 0; }
+static inline int restore_signal_shadow_stack(void) { return 0; }
 #endif /* CONFIG_X86_SHADOW_STACK */
 
 #endif /* __ASSEMBLY__ */
arch/x86/kernel/shstk.c +107 -19

148 lines not on this page · raw message

arch/x86/kernel/signal.c +10 -0
index 9c7265b524c7..d2081305f698 100644
@@ -47,6 +47,7 @@
 #include <asm/syscall.h>
 #include <asm/sigframe.h>
 #include <asm/signal.h>
+#include <asm/cet.h>
 
 #ifdef CONFIG_X86_64
 /*
@@ -472,6 +473,9 @@ static int __setup_rt_frame(int sig, struct ksignal *ksig,
 	frame = get_sigframe(&ksig->ka, regs, sizeof(struct rt_sigframe), &fp);
 	uc_flags = frame_uc_flags(regs);
 
+	if (setup_signal_shadow_stack(ksig))
+		return -EFAULT;
+
 	if (!user_access_begin(frame, sizeof(*frame)))
 		return -EFAULT;
 
@@ -675,6 +679,9 @@ SYSCALL_DEFINE0(rt_sigreturn)
 	if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
 		goto badframe;
 
+	if (restore_signal_shadow_stack())
+		goto badframe;
+
 	if (restore_altstack(&frame->uc.uc_stack))
 		goto badframe;
 
@@ -992,6 +999,9 @@ COMPAT_SYSCALL_DEFINE0(x32_rt_sigreturn)
 	if (!restore_sigcontext(regs, &frame->uc.uc_mcontext, uc_flags))
 		goto badframe;
 
+	if (restore_signal_shadow_stack())
+		goto badframe;
+
 	if (compat_restore_altstack(&frame->uc.uc_stack))
 		goto badframe;
 
Signature · 1 line
2.17.1
#181 lore raw

Re: [PATCH v2 27/39] x86/cet/shstk: Handle signals for shadow stack

Quoted 114 lines · Rick Edgecombe

From: Yu-cheng Yu <[email protected]>

When a signal is handled normally the context is pushed to the stack before handling it. For shadow stacks, since the shadow stack only track's return addresses, there isn't any state that needs to be pushed. However, there are still a few things that need to be done. These things are userspace visible and which will be kernel ABI for shadow stacks.

One is to make sure the restorer address is written to shadow stack, since the signal handler (if not changing ucontext) returns to the restorer, and the restorer calls sigreturn. So add the restorer on the shadow stack before handling the signal, so there is not a conflict when the signal handler returns to the restorer.

The other thing to do is to place some type of checkable token on the thread's shadow stack before handling the signal and check it during sigreturn. This is an extra layer of protection to hamper attackers calling sigreturn manually as in SROP-like attacks.

For this token we can use the shadow stack data format defined earlier. Have the data pushed be the previous SSP. In the future the sigreturn might want to return back to a different stack. Storing the SSP (instead of a restore offset or something) allows for future functionality that may want to restore to a different stack.

So, when handling a signal push
- the SSP pointing in the shadow stack data format
- the restorer address below the restore token.

In sigreturn, verify SSP is stored in the data format and pop the shadow stack.

Cc Andy Lutomirski <[email protected]>
Cc Cyrill Gorcunov <[email protected]>
Cc Florian Weimer <[email protected]>
Cc H. Peter Anvin <[email protected]>
Cc Kees Cook <[email protected]>

---

v2:
- Switch to new shstk signal format

v1:
- Use xsave helpers.
- Expand commit log.

Yu-cheng v27:
- Eliminate saving shadow stack pointer to signal context.

Yu-cheng v25:
- Update commit log/comments for the sc_ext struct.
- Use restorer address already calculated.
- Change CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK.
- Change X86_FEATURE_CET to X86_FEATURE_SHSTK.
- Eliminate writing to MSR_IA32_U_CET for shadow stack.
- Change wrmsrl() to wrmsrl_safe() and handle error.

 arch/x86/ia32/ia32_signal.c |   1 +
 arch/x86/include/asm/cet.h  |   5 ++
 arch/x86/kernel/shstk.c     | 126 ++++++++++++++++++++++++++++++------
 arch/x86/kernel/signal.c    |  10 +++
 4 files changed, 123 insertions(+), 19 deletions(-)
arch/x86/ia32/ia32_signal.c +1 -0
index c9c3859322fa..88d71b9de616 100644
@@ -34,6 +34,7 @@
 #include <asm/sigframe.h>
 #include <asm/sighandling.h>
 #include <asm/smap.h>
+#include <asm/cet.h>
 
 static inline void reload_segments(struct sigcontext_32 *sc)
 {
arch/x86/include/asm/cet.h +5 -0
index 924de99e0c61..8c6fab9f402a 100644
@@ -6,6 +6,7 @@
 #include <linux/types.h>
 
 struct task_struct;
+struct ksignal;
 
 struct thread_shstk {
 	u64	base;
@@ -22,6 +23,8 @@ int shstk_alloc_thread_stack(struct task_struct *p, unsigned long clone_flags,
 void shstk_free(struct task_struct *p);
 int shstk_disable(void);
 void reset_thread_shstk(void);
+int setup_signal_shadow_stack(struct ksignal *ksig);
+int restore_signal_shadow_stack(void);
 #else
 static inline long cet_prctl(struct task_struct *task, int option,
 		      unsigned long features) { return -EINVAL; }
@@ -33,6 +36,8 @@ static inline int shstk_alloc_thread_stack(struct task_struct *p,
 static inline void shstk_free(struct task_struct *p) {}
 static inline int shstk_disable(void) { return -EOPNOTSUPP; }
 static inline void reset_thread_shstk(void) {}
+static inline int setup_signal_shadow_stack(struct ksignal *ksig) { return 0; }
+static inline int restore_signal_shadow_stack(void) { return 0; }
 #endif /* CONFIG_X86_SHADOW_STACK */
 
 #endif /* __ASSEMBLY__ */
arch/x86/kernel/shstk.c +0 -0
index 8904aef487bf..04442134aadd 100644

arch/x86/kernel/shstk.c static int get_shstk_data(unsigned long *data, unsigned long __user *addr) · line 233

Quoted 10 lines
arch/x86/kernel/shstk.c +3 -3
@@ -227,41 +227,129 @@ static int get_shstk_data(unsigned long *data, unsigned long __user *addr)
 }
 
 /*
- * Verify the user shadow stack has a valid token on it, and then set
- * *new_ssp according to the token.
+ * Create a restore token on shadow stack, and then push the user-mode
+ * function return address.
  */
-static int shstk_check_rstor_token(unsigned long *new_ssp)
+static int shstk_setup_rstor_token(unsigned long ret_addr, unsigned long *new_ssp)

Oh, hrm. Prior patch defines shstk_check_rstor_token() and doesn't call it. This patch removes it. :P Can you please remove shstk_check_rstor_token() from the prior patch?

Quoted 52 lines
arch/x86/kernel/shstk.c +39 -7
 {
-	unsigned long token_addr;
-	unsigned long token;
+	unsigned long ssp, token_addr;
+	int err;
+
+	if (!ret_addr)
+		return -EINVAL;
+
+	ssp = get_user_shstk_addr();
+	if (!ssp)
+		return -EINVAL;
+
+	err = create_rstor_token(ssp, &token_addr);
+	if (err)
+		return err;
+
+	ssp = token_addr - sizeof(u64);
+	err = write_user_shstk_64((u64 __user *)ssp, (u64)ret_addr);
+
+	if (!err)
+		*new_ssp = ssp;
+
+	return err;
+}
+
+static int shstk_push_sigframe(unsigned long *ssp)
+{
+	unsigned long target_ssp = *ssp;
+
+	/* Token must be aligned */
+	if (!IS_ALIGNED(*ssp, 8))
+		return -EINVAL;
 
-	token_addr = get_user_shstk_addr();
-	if (!token_addr)
+	if (!IS_ALIGNED(target_ssp, 8))
 		return -EINVAL;
 
-	if (get_user(token, (unsigned long __user *)token_addr))
+	*ssp -= SS_FRAME_SIZE;
+	if (put_shstk_data((void *__user)*ssp, target_ssp))
 		return -EFAULT;
 
-	/* Is mode flag correct? */
-	if (!(token & BIT(0)))
+	return 0;
+}
+
+
+static int shstk_pop_sigframe(unsigned long *ssp)
+{

arch/x86/kernel/shstk.c static int get_shstk_data(unsigned long *data, unsigned long __user *addr) · line 288

Quoted 10 lines
arch/x86/kernel/shstk.c +9 -0
+	unsigned long token_addr;
+	int err;
+
+	err = get_shstk_data(&token_addr, (unsigned long __user *)*ssp);
+	if (unlikely(err))
+		return err;
+
+	/* Restore SSP aligned? */
+	if (unlikely(!IS_ALIGNED(token_addr, 8)))
 		return -EINVAL;

Why doesn't this always fail, given BIT(0) being set? I don't see it getting cleared until the end of this function.

arch/x86/kernel/shstk.c static int get_shstk_data(unsigned long *data, unsigned long __user *addr) · line 292

Quoted 6 lines
arch/x86/kernel/shstk.c +2 -2
-	/* Is busy flag set? */
-	if (token & BIT(1))
+	/* SSP in userspace? */
+	if (unlikely(token_addr >= TASK_SIZE_MAX))
 		return -EINVAL;

BIT(63) already got cleared by here (in get_shstk_data(), but yes, this is still a reasonable check.

Quoted 7 lines
arch/x86/kernel/shstk.c +4 -2
-	/* Mask out flags */
-	token &= ~3UL;
+	*ssp = token_addr;
+
+	return 0;
+}
Signature · 1 line
Kees Cook