add support for Clang's Shadow Call Stack
Subsystems (16)
cover add support for Clang's Shadow Call Stack
This patch series adds support for Clang's Shadow Call Stack (SCS) mitigation, which uses a separately allocated shadow stack to protect against return address overwrites. More information can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
SCS is currently supported only on arm64, where the compiler requires the x18 register to be reserved for holding the current task's shadow stack pointer. Because of this, the series includes four patches from Ard to remove x18 usage from assembly code and to reserve the register from general allocation.
With -fsanitize=shadow-call-stack, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning. As a result, SCS is incompatible with features that rely on modifying function return addresses to alter control flow, such as function graph tracing and kretprobes. A copy of the return address is still kept in the kernel stack for compatibility with stack unwinding, for example.
SCS has a minimal performance overhead, but allocating shadow stacks increases kernel memory usage. The feature is therefore mostly useful on hardware that lacks support for PAC instructions. This series adds a ROP protection choice to the kernel configuration, where other return address protection options can be selected as they are added to the kernel.
Ard Biesheuvel (4):
arm64/lib: copy_page: avoid x18 register in assembler code
arm64: kvm: stop treating register x18 as caller save
arm64: kernel: avoid x18 as an arbitrary temp register
arm64: kbuild: reserve reg x18 from general allocation by the compiler
Sami Tolvanen (14):
arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings add support for Clang's Shadow Call Stack (SCS)
scs: add accounting
scs: add support for stack usage debugging
trace: disable function graph tracing with SCS
kprobes: fix compilation without CONFIG_KRETPROBES
kprobes: disable kretprobes with SCS
arm64: reserve x18 only with Shadow Call Stack
arm64: preserve x18 when CPU is suspended
arm64: efi: restore x18 if it was corrupted
arm64: vdso: disable Shadow Call Stack
arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
arm64: disable SCS for hypervisor code
arm64: implement Shadow Call Stack
Makefile | 6 + arch/Kconfig | 41 ++++- arch/arm64/Kconfig | 1 + arch/arm64/Makefile | 4 + arch/arm64/include/asm/scs.h | 60 ++++++++ arch/arm64/include/asm/stacktrace.h | 4 + arch/arm64/include/asm/thread_info.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 + arch/arm64/kernel/cpu-reset.S | 4 +- arch/arm64/kernel/efi-rt-wrapper.S | 7 +- arch/arm64/kernel/entry.S | 23 +++ arch/arm64/kernel/head.S | 9 ++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/probes/kprobes.c | 2 + arch/arm64/kernel/process.c | 3 + arch/arm64/kernel/scs.c | 39 +++++ arch/arm64/kernel/smp.c | 4 + arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kvm/hyp/Makefile | 3 +- arch/arm64/kvm/hyp/entry.S | 12 +- arch/arm64/lib/copy_page.S | 38 ++--- arch/arm64/mm/proc.S | 69 +++++---- drivers/base/node.c | 6 + fs/proc/meminfo.c | 4 + include/linux/compiler-clang.h | 2 + include/linux/compiler_types.h | 4 + include/linux/mmzone.h | 3 + include/linux/scs.h | 88 +++++++++++ init/init_task.c | 6 + init/main.c | 3 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/kprobes.c | 38 ++--- kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 221 +++++++++++++++++++++++++++ kernel/trace/Kconfig | 1 + mm/page_alloc.c | 6 + mm/vmstat.c | 3 + 40 files changed, 656 insertions(+), 82 deletions(-) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 1/18 arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
idmap_kpti_install_ng_mappings uses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
arch/arm64/mm/proc.S +32 -31
@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */
@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*
@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 01/18] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
Quoted 6 lines · Sami Tolvanen
idmap_kpti_install_ng_mappingsuses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.Signed-off-by Sami Tolvanen <[email protected]>
TIL about .req/.unreq. Seems like a nice way of marking "variable" lifetime. Technically, only pte needed to be moved to reuse {w|x}16, but moving most the unreqs together is nicer than splitting them apart. The usage all looks correct to me.
Quoted 116 lines
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)arch/arm64/mm/proc.S +32
-32@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif --2.23.0.866.gb869b98d4c-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
PATCH 2/18 arm64/lib: copy_page: avoid x18 register in assembler code
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
Register x18 will no longer be used as a caller save register in the future, so stop using it in the copy_page() code.
--- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/arm64/lib/copy_page.S +19 -19
@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1) alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #-240] ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #-224] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #-208] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #-192] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #-176] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #-160] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #-144] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80] - stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #-240] + stnp x6, x7, [x0, #-224] + stnp x8, x9, [x0, #-208] + stnp x10, x11, [x0, #-192] + stnp x12, x13, [x0, #-176] + stnp x14, x15, [x0, #-160] + stnp x16, x17, [x0, #-144] ret ENDPROC(copy_page)
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 3/18 arm64: kvm: stop treating register x18 as caller save
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
In preparation of using x18 as a task struct pointer register when running in the kernel, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to __guest_enter(). Instead, preserve its value and restore it upon return.
--- arch/arm64/kvm/hyp/entry.S | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
arch/arm64/kvm/hyp/entry.S +5 -7
@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -38,6 +39,7 @@ ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] .endm /*
@@ -87,12 +89,9 @@ alternative_else_nop_endif ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - // Restore guest regs x19-x29, lr + // Restore guest regs x18-x29, lr restore_callee_saved_regs x18 - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] - // Do not touch any register after this! eret sb
@@ -114,7 +113,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
@@ -123,9 +122,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 03/18] arm64: kvm: stop treating register x18 as caller save
Quoted 10 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
In preparation of using x18 as a task struct pointer register when running in the kernel, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.
Quoted 3 lines
Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>
You might want to update the commit log to drop the reference to the task struct pointer.
Quoted 61 lines
--- arch/arm64/kvm/hyp/entry.S | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)arch/arm64/kvm/hyp/entry.S +5
-8@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -38,6 +39,7 @@ ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] .endm /*@@ -87,12 +89,9 @@ alternative_else_nop_endif ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - // Restore guest regs x19-x29, lr + // Restore guest regs x18-x29, lr restore_callee_saved_regs x18 - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] - // Do not touch any register after this! eret sb@@ -114,7 +113,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]@@ -123,9 +122,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3 --2.23.0.866.gb869b98d4c-goog
Marc Zyngier
maintainerRe: [PATCH 03/18] arm64: kvm: stop treating register x18 as caller save
In reply to Sami TolvanenQuoted 29 lines · On Fri, 18 Oct 2019 09:10:18 -0700 Sami Tolvanen <[email protected]>…
From: Ard Biesheuvel <[email protected]>
In preparation of using x18 as a task struct pointer register when running in the kernel, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kvm/hyp/entry.S | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)arch/arm64/kvm/hyp/entry.S +1
-0@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
arch/arm64/kvm/hyp/entry.S line 42
Quoted 4 lines
arch/arm64/kvm/hyp/entry.S +1
-0@@ -38,6 +39,7 @@ ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)]
There is now an assumption that ctxt is x18 (otherwise why would it be out of order?). Please add a comment to that effect.
arch/arm64/kvm/hyp/entry.S alternative_else_nop_endif · line 93
Quoted 10 lines
arch/arm64/kvm/hyp/entry.S +1
-1.endm /*@@ -87,12 +89,9 @@ alternative_else_nop_endif ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - // Restore guest regs x19-x29, lr + // Restore guest regs x18-x29, lr restore_callee_saved_regs x18
Or you could elect another register such as x29 as the base, and keep the above in a reasonable order.
Quoted 17 lines
arch/arm64/kvm/hyp/entry.S +1
-4- // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] - // Do not touch any register after this! eret sb@@ -114,7 +113,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
arch/arm64/kvm/hyp/entry.S ENTRY(__guest_exit) · line 129
Quoted 10 lines
arch/arm64/kvm/hyp/entry.S +1
-2@@ -123,9 +122,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH 03/18] arm64: kvm: stop treating register x18 as caller save
Quoted 14 lines · Marc Zyngier
arch/arm64/kvm/hyp/entry.S line 42
Quoted 10 lines
.macro
save_callee_saved_regsctxtarch/arm64/kvm/hyp/entry.S +2
-0+ str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -38,6 +39,7 @@ ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)] ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)] ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)] + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)]There is now an assumption that ctxt is x18 (otherwise why would it be out of order?). Please add a comment to that effect.
Quoted 6 lines
arch/arm64/kvm/hyp/entry.S alternative_else_nop_endif · line 93
Quoted 3 lines
arch/arm64/kvm/hyp/entry.S +1
-1- // Restore guest regs x19-x29, lr + // Restore guest regs x18-x29, lr restore_callee_saved_regs x18Or you could elect another register such as x29 as the base, and keep the above in a reasonable order.
I'm fine with either option. Ard, any thoughts?
Sami
PATCH 4/18 arm64: kernel: avoid x18 as an arbitrary temp register
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
The code in __cpu_soft_restart() uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/cpu-reset.S +2 -2
@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsection
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 5/18 arm64: kbuild: reserve reg x18 from general allocation by the compiler
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
Before we can start using register x18 for a special purpose (as permitted by the AAPCS64 ABI), we need to tell the compiler that it is off limits for general allocation. So tag it as 'fixed', and remove the mention from the LL/SC compiler flag override.
--- arch/arm64/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/Makefile +1 -1
@@ -55,7 +55,7 @@ endif KBUILD_CFLAGS += -mgeneral-regs-only $(lseinstr) $(brokengasinst) \ $(compat_vdso) $(cc_has_k_constraint) -KBUILD_CFLAGS += -fno-asynchronous-unwind-tables +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -ffixed-x18 KBUILD_CFLAGS += $(call cc-disable-warning, psabi) KBUILD_AFLAGS += $(lseinstr) $(brokengasinst) $(compat_vdso)
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 05/18] arm64: kbuild: reserve reg x18 from general allocation by the compiler
Quoted 6 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
Before we can start using register x18 for a special purpose (as permitted by the AAPCS64 ABI), we need to tell the compiler that it is off limits for general allocation. So tag it as 'fixed',
yep, but...
Quoted 2 lines
and remove the mention from
the LL/SC compiler flag override.
was that cut/dropped from this patch?
Quoted 2 lines
^ Looks like it. Maybe it doesn't matter, but if sending a V2, maybe the commit message to be updated?
Quoted 2 lines
Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>
If sending a V2 with the above cleaned up, you may also include:
I like how this does not conditionally reserve it based on the CONFIG for SCS. Hopefully later patches don't wrap it, but I haven't looked through all of them yet.
Quoted 20 lines
--- arch/arm64/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/Makefile +1
-2@@ -55,7 +55,7 @@ endif KBUILD_CFLAGS += -mgeneral-regs-only $(lseinstr) $(brokengasinst) \ $(compat_vdso) $(cc_has_k_constraint) -KBUILD_CFLAGS += -fno-asynchronous-unwind-tables +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -ffixed-x18 KBUILD_CFLAGS += $(call cc-disable-warning, psabi) KBUILD_AFLAGS += $(lseinstr) $(brokengasinst) $(compat_vdso) --2.23.0.866.gb869b98d4c-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH 05/18] arm64: kbuild: reserve reg x18 from general allocation by the compiler
Quoted 10 lines · Nick Desaulniers' via Clang Built Linux
Quoted 2 lines
and remove the mention from
the LL/SC compiler flag override.was that cut/dropped from this patch?
Quoted 2 lines
^ Looks like it. Maybe it doesn't matter, but if sending a V2, maybe the commit message to be updated?
True. The original patch is from 2017 and the relevant part of arm64/lib/Makefile no longer exists. I'll update this accordingly.
Quoted 3 lines
I like how this does not conditionally reserve it based on the CONFIG for SCS. Hopefully later patches don't wrap it, but I haven't looked through all of them yet.
In a later patch x18 is only reserved with SCS. I'm fine with dropping that patch and reserving it always, but wouldn't mind hearing thoughts from the maintainers about this first.
Sami
Re: [PATCH 05/18] arm64: kbuild: reserve reg x18 from general allocation by the compiler
Quoted 17 lines · Sami Tolvanen
Quoted 10 lines · Nick Desaulniers' via Clang Built Linux
Quoted 2 lines
and remove the mention from
the LL/SC compiler flag override.was that cut/dropped from this patch?
Quoted 2 lines
^ Looks like it. Maybe it doesn't matter, but if sending a V2, maybe the commit message to be updated?
True. The original patch is from 2017 and the relevant part of arm64/lib/Makefile no longer exists. I'll update this accordingly.
Quoted 8 lines
Quoted 3 lines
I like how this does not conditionally reserve it based on the CONFIG for SCS. Hopefully later patches don't wrap it, but I haven't looked through all of them yet.
In a later patch x18 is only reserved with SCS. I'm fine with dropping that patch and reserving it always, but wouldn't mind hearing thoughts from the maintainers about this first.
Why would you reserve x18 if SCS is disabled? Given that this is a choice that is made at code generation time, there is no justification for always reserving it, since it will never be used for anything. (Of course, this applies to generated code only - .S files should simply be updated to avoid x18 altogether)
Also, please combine this patch with the one that reserves it conditionally, no point in having both in the same series.
Re: [PATCH 05/18] arm64: kbuild: reserve reg x18 from general allocation by the compiler
Quoted 2 lines · Ard Biesheuvel
Also, please combine this patch with the one that reserves it conditionally, no point in having both in the same series.
Sure, I'll just drop this patch from v2 then and only reserve it with SCS.
Sami
PATCH 6/18 add support for Clang's Shadow Call Stack (SCS)
In reply to Sami TolvanenThis change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
--- Makefile | 6 ++ arch/Kconfig | 39 ++++++++ include/linux/compiler-clang.h | 2 + include/linux/compiler_types.h | 4 + include/linux/scs.h | 88 ++++++++++++++++++ init/init_task.c | 6 ++ init/main.c | 3 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 162 +++++++++++++++++++++++++++++++++ 12 files changed, 323 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Makefile +6 -0
@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +KBUILD_CFLAGS += -fsanitize=shadow-call-stack +DISABLE_SCS := -fno-sanitize=shadow-call-stack +export DISABLE_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
arch/Kconfig +39 -0
@@ -521,6 +521,45 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + def_bool n + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +choice + prompt "Return-oriented programming (ROP) protection" + default ROP_PROTECTION_NONE + help + This option controls kernel protections against return-oriented + programming (ROP) attacks. + +config ROP_PROTECTION_NONE + bool "None" + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000 + help + This option enables Clang's Shadow Call Stack, which uses a shadow + stack to protect function return addresses from being overwritten by + an attacker. More information can be found from Clang's + documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + +endchoice + config HAVE_ARCH_WITHIN_STACK_FRAMES bool help
include/linux/compiler-clang.h +2 -0
@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))
include/linux/compiler_types.h +4 -0
@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif
include/linux/scs.h new file +88 -0
88 lines not on this page · raw message
init/init_task.c +6 -0
@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h>
@@ -184,6 +185,11 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] + __init_task_data__aligned(SCS_SIZE) ; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry.
init/main.c +3 -0
@@ -93,6 +93,7 @@#include <linux/rodata_test.h> #include <linux/jump_label.h> #include <linux/mem_encrypt.h> +#include <linux/scs.h> #include <asm/io.h> #include <asm/bugs.h>
@@ -578,6 +579,8 @@ asmlinkage __visible void __init start_kernel(void)char *after_dashes;set_task_stack_end_magic (&init_task); +scs_set_init_magic (&init_task); +smp_setup_processor_id ();debug_objects_early_init ();
kernel/Makefile +1 -0
@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/
kernel/fork.c +9 -0
@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info,
@@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }
@@ -907,6 +912,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)clear_user_return_notifier (tsk);clear_tsk_need_resched (tsk);set_task_stack_end_magic (tsk); +scs_task_init (tsk);#ifdef CONFIG_STACKPROTECTOR tsk->stack_canary =get_random_canary ();
@@ -2022,6 +2028,9 @@ static __latent_entropy struct task_struct *copy_process( args->tls);if (retval) goto bad_fork_cleanup_io; + retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;stackleak_task_init (p);
kernel/sched/core.c +2 -0
@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
kernel/sched/sched.h +1 -0
@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h>
kernel/scs.c new file +162 -0
162 lines not on this page · raw message
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 3 lines · Sami Tolvanen
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker
[]
Quoted 1 line
.diff --git
a/include/linux/compiler-clang.hb/include/linux/compiler-clang.h
[]
include/linux/compiler-clang.h line 46
Quoted 6 lines
include/linux/compiler-clang.h +2
-0@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))
trivia:
This should likely use the __ prefix and suffix form:
#define __noscs __attribute__((__no_sanitize__("shadow-call-stack")))
as should the __no_sanitize_address above this
include/linux/compiler_types.h
Quoted 1 line
include/linux/compiler_types.h +0
-0
[]
Quoted 11 lines
include/linux/compiler_types.h +4
-0@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif
Nick Desaulniers
reviewerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Sami TolvanenQuoted 69 lines · Sami Tolvanen
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 39 ++++++++ include/linux/compiler-clang.h | 2 + include/linux/compiler_types.h | 4 + include/linux/scs.h | 88 ++++++++++++++++++ init/init_task.c | 6 ++ init/main.c | 3 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 162 +++++++++++++++++++++++++++++++++ 12 files changed, 323 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +6
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +KBUILD_CFLAGS += -fsanitize=shadow-call-stack +DISABLE_SCS := -fno-sanitize=shadow-call-stack +export DISABLE_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)arch/Kconfig +19
-0@@ -521,6 +521,45 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + def_bool n + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +choice + prompt "Return-oriented programming (ROP) protection" + default ROP_PROTECTION_NONE + help
arch/Kconfig config STACKPROTECTOR_STRONG · line 552
Quoted 10 lines
arch/Kconfig +10
-0+ This option controls kernel protections against return-oriented + programming (ROP) attacks. + +config ROP_PROTECTION_NONE + bool "None" + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000
Version check LGTM.
Quoted 18 lines
arch/Kconfig +10
-0+ help + This option enables Clang's Shadow Call Stack, which uses a shadow + stack to protect function return addresses from being overwritten by + an attacker. More information can be found from Clang's + documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + +endchoice + config HAVE_ARCH_WITHIN_STACK_FRAMES bool helpinclude/linux/compiler-clang.h +0
-0
include/linux/compiler-clang.h line 46
Quoted 5 lines
include/linux/compiler-clang.h +2
-0@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))
It looks like this attribute, (and thus a requirement to use this feature), didn't exist until Clang 7.0: https://godbolt.org/z/p9u1we
(as noted above)
I think it's better to put __noscs behind a __has_attribute guard in include/linux/compiler_attributes.h. Otherwise, what will happen when Clang 6.0 sees __noscs, for example? (-Wunknown-sanitizers will happen).
include/linux/compiler_types.h struct ftrace_likely_data { · line 208
Quoted 12 lines
include/linux/compiler_types.h +4
-0@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +
and then this can be removed.
Quoted 398 lines
include/linux/compiler_types.h +0
-0#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif include/linux/scs.h new file +88
-0@@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Shadow Call Stack support. +* +* Copyright (C) 2018 Google LLC +*/ + +#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include <linux/gfp.h> +#include <linux/sched.h> +#include <asm/page.h> + +#ifdef CONFIG_SHADOW_CALL_STACK + +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP +# define SCS_SIZE PAGE_SIZE +#else +# define SCS_SIZE 1024 +#endif + +#define SCS_GFP (GFP_KERNEL | __GFP_ZERO) + +extern unsigned long init_shadow_call_stack[]; + +static inline void *task_scs (struct task_struct *tsk) +{ +return task_thread_info (tsk)->shadow_call_stack; +} + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +task_thread_info (tsk)->shadow_call_stack = s; +} + +extern void scs_init (void ); +extern void scs_set_init_magic (struct task_struct *tsk); +extern void scs_task_init (struct task_struct *tsk); +extern void scs_task_reset (struct task_struct *tsk); +extern int scs_prepare (struct task_struct *tsk,int node); +extern bool scs_corrupted (struct task_struct *tsk); +extern void scs_release (struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void *task_scs (struct task_struct *tsk) +{ +return 0 ; +} + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +} + +static inline void scs_init (void ) +{ +} + +static inline void scs_set_init_magic (struct task_struct *tsk) +{ +} + +static inline void scs_task_init (struct task_struct *tsk) +{ +} + +static inline void scs_task_reset (struct task_struct *tsk) +{ +} + +static inline int scs_prepare (struct task_struct *tsk,int node) +{ +return 0 ; +} + +static inline bool scs_corrupted (struct task_struct *tsk) +{ +return false; +} + +static inline void scs_release (struct task_struct *tsk) +{ +} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ init/init_task.c +6
-0@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h> @@ -184,6 +185,11 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] + __init_task_data__aligned(SCS_SIZE) ; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry. init/main.c +3
-0@@ -93,6 +93,7 @@#include <linux/rodata_test.h> #include <linux/jump_label.h> #include <linux/mem_encrypt.h> +#include <linux/scs.h> #include <asm/io.h> #include <asm/bugs.h> @@ -578,6 +579,8 @@ asmlinkage __visible void __init start_kernel(void)char *after_dashes;set_task_stack_end_magic (&init_task); +scs_set_init_magic (&init_task); +smp_setup_processor_id ();debug_objects_early_init ();kernel/Makefile +1
-0@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/fork.c +9
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h> @@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }@@ -907,6 +912,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)clear_user_return_notifier (tsk);clear_tsk_need_resched (tsk);set_task_stack_end_magic (tsk); +scs_task_init (tsk);#ifdef CONFIG_STACKPROTECTOR tsk->stack_canary =get_random_canary ();@@ -2022,6 +2028,9 @@ static __latent_entropy struct task_struct *copy_process( args->tls);if (retval) goto bad_fork_cleanup_io; + retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;stackleak_task_init (p);kernel/sched/core.c +2
-0@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();kernel/sched/sched.h +1
-0@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h> kernel/scs.c new file +162
-1@@ -0,0 +1,162 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/cpuhotplug.h> +#include <linux/mm.h> +#include <linux/slab.h> +#include <linux/scs.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +#define SCS_END_MAGIC 0xaf0194819b1635f6UL + +static inline void *__scs_base (struct task_struct *tsk) +{ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +} + +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + +/* Keep a cache of shadow stacks */ +#define SCS_CACHE_SIZE 2 +static DEFINE_PER_CPU(void *, scs_cache[SCS_CACHE_SIZE]); + +static void *scs_alloc (int node) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +void *s; + + s =this_cpu_xchg (scs_cache[i],NULL ); +if (s) { +memset (s,0 ,SCS_SIZE ); +return s; + } + } + +return __vmalloc_node_range (SCS_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +SCS_GFP ,PAGE_KERNEL ,0 , + node,__builtin_return_address (0 )); +} + +static void scs_free (void *s) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +if (this_cpu_cmpxchg (scs_cache[i],0 , s) !=0 ) +continue ; + +return ; + } + +vfree_atomic (s); +} + +static int scs_cleanup (unsigned int cpu) +{ +int i; +void **cache =per_cpu_ptr (scs_cache, cpu); + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +vfree (cache[i]); + cache[i] =NULL ; + } + +return 0 ; +} + +void __init scs_init (void ) +{ +cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , + scs_cleanup); +} + +#else /* !CONFIG_SHADOW_CALL_STACK_VMAP */ + +static struct kmem_cache *scs_cache; + +static inline void *scs_alloc (int node) +{ +return kmem_cache_alloc_node (scs_cache,SCS_GFP , node); +} + +static inline void scs_free (void *s) +{ +kmem_cache_free (scs_cache, s); +} + +void __init scs_init (void ) +{ + scs_cache =kmem_cache_create ("scs_cache" ,SCS_SIZE ,SCS_SIZE , +0 ,NULL ); +WARN_ON (!scs_cache); +} + +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + +static inline unsigned long *scs_magic (struct task_struct *tsk) +{ +return (unsigned long *)(__scs_base (tsk) +SCS_SIZE -sizeof (long )); +} + +static inline void scs_set_magic (struct task_struct *tsk) +{ + *scs_magic (tsk) =SCS_END_MAGIC ; +} + +void scs_task_init (struct task_struct *tsk) +{ +task_set_scs (tsk,NULL ); +} + +void scs_task_reset (struct task_struct *tsk) +{ +task_set_scs (tsk,__scs_base (tsk)); +} + +void scs_set_init_magic (struct task_struct *tsk) +{ +scs_save (tsk); +scs_set_magic (tsk); +scs_load (tsk); +} + +int scs_prepare (struct task_struct *tsk,int node) +{ +void *s; + + s =scs_alloc (node); +if (!s) +return -ENOMEM ; + +task_set_scs (tsk, s); +scs_set_magic (tsk); + +return 0 ; +} + +bool scs_corrupted (struct task_struct *tsk) +{ +return *scs_magic (tsk) !=SCS_END_MAGIC ; +} + +void scs_release (struct task_struct *tsk) +{ +void *s; + + s =__scs_base (tsk); +if (!s) +return ; + +WARN_ON (scs_corrupted (tsk)); + +scs_task_init (tsk); +scs_free (s); +} --2.23.0.866.gb869b98d4c-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 15 lines · Nick Desaulniers' via Clang Built Linux
include/linux/compiler-clang.h line 46
Quoted 10 lines
include/linux/compiler-clang.h +2
-0@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))It looks like this attribute, (and thus a requirement to use this feature), didn't exist until Clang 7.0: https://godbolt.org/z/p9u1we
(as noted above)
Quoted 4 lines
I think it's better to put
__noscsbehind a__has_attributeguard ininclude/linux/compiler_attributes.h. Otherwise, what will happen when Clang 6.0 sees__noscs, for example? (-Wunknown-sanitizers will happen).
Good point, I'll fix this in v2. Thanks.
Sami
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 18 lines · Sami Tolvanen
Quoted 15 lines · Nick Desaulniers' via Clang Built Linux
include/linux/compiler-clang.h line 46
Quoted 10 lines
include/linux/compiler-clang.h +2
-0@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))It looks like this attribute, (and thus a requirement to use this feature), didn't exist until Clang 7.0: https://godbolt.org/z/p9u1we
(as noted above)
Quoted 6 lines
Quoted 4 lines
I think it's better to put
__noscsbehind a__has_attributeguard ininclude/linux/compiler_attributes.h. Otherwise, what will happen when Clang 6.0 sees__noscs, for example? (-Wunknown-sanitizers will happen).Good point, I'll fix this in v2. Thanks.
+1, please CC whenever you send it!
Cheers,
Miguel
Nick Desaulniers
reviewerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 20 lines · Miguel Ojeda
Quoted 18 lines · Sami Tolvanen
Quoted 15 lines · Nick Desaulniers' via Clang Built Linux
include/linux/compiler-clang.h line 46
Quoted 10 lines
include/linux/compiler-clang.h +2
-0@@ -42,3 +42,5 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#define __noscs __attribute__((no_sanitize("shadow-call-stack")))It looks like this attribute, (and thus a requirement to use this feature), didn't exist until Clang 7.0: https://godbolt.org/z/p9u1we
(as noted above)
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
I think it's better to put
__noscsbehind a__has_attributeguard ininclude/linux/compiler_attributes.h. Otherwise, what will happen when Clang 6.0 sees__noscs, for example? (-Wunknown-sanitizers will happen).Good point, I'll fix this in v2. Thanks.
+1, please CC whenever you send it!
Sami pointed out to me off thread that __has_attribute would only check no_sanitize, not shadow-call-stack. So maybe best to keep the definition here (include/linux/compiler-clang.h), but wrapped in a __has_feature check so that Clang 6.0 doesn't start complaining.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 5 lines · Nick Desaulniers
Sami pointed out to me off thread that
__has_attributewould only checkno_sanitize, notshadow-call-stack. So maybe best to keep the definition here (include/linux/compiler-clang.h), but wrapped in a__has_featurecheck so that Clang 6.0 doesn't start complaining.
Ah, good point -- agreed!
Cheers,
Miguel
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Sami TolvanenQuoted 5 lines · Sami Tolvanen
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
(As I mentioned in the other thread, the security documentation there doesn't fit the kernel usecase.)
[...]
arch/Kconfig config STACKPROTECTOR_STRONG · line 537
Quoted 7 lines
arch/Kconfig +7
-0+config SHADOW_CALL_STACK_VMAP + def_bool n + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack.
Without CONFIG_SHADOW_CALL_STACK_VMAP, after 128 small stack frames, you overflow into random physmap memory even if the main stack is vmapped... I guess you can't get around that without making the SCS instrumentation more verbose. :/
Could you maybe change things so that independent of whether you have vmapped SCS or slab-allocated SCS, the scs_corrupted() check looks at offset 1024-8 (where it currently is for the slab-allocated case)? That way, code won't suddenly stop working when you disable CONFIG_SHADOW_CALL_STACK_VMAP; and especially if you use CONFIG_SHADOW_CALL_STACK_VMAP for development and testing but disable it in production, that would be annoying.
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 2 lines · Jann Horn
(As I mentioned in the other thread, the security documentation there doesn't fit the kernel usecase.)
True. I'll add a note about it here too.
Quoted 4 lines
Without
CONFIG_SHADOW_CALL_STACK_VMAP, after 128 small stack frames, you overflow into random physmap memory even if the main stack is vmapped... I guess you can't get around that without making the SCS instrumentation more verbose. :/
That's correct. In our testing, 128 stack frames is nearly twice the maximum amount that's been used (on an arm64 device), and for many use cases, allocating a full page is simply too costly despite the advantages.
Quoted 7 lines
Could you maybe change things so that independent of whether you have vmapped SCS or slab-allocated SCS, the
scs_corrupted()check looks at offset 1024-8 (where it currently is for the slab-allocated case)? That way, code won't suddenly stop working when you disableCONFIG_SHADOW_CALL_STACK_VMAP; and especially if you useCONFIG_SHADOW_CALL_STACK_VMAPfor development and testing but disable it in production, that would be annoying.
Yes, that's a great idea. I'll change this in v2.
Sami
Mark Rutland
reviewerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Sami TolvanenQuoted 29 lines · Sami Tolvanen
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 39 ++++++++ include/linux/compiler-clang.h | 2 + include/linux/compiler_types.h | 4 + include/linux/scs.h | 88 ++++++++++++++++++ init/init_task.c | 6 ++ init/main.c | 3 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 162 +++++++++++++++++++++++++++++++++ 12 files changed, 323 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +0
-0
Makefile ifdef CONFIG_LIVEPATCH · line 853
Quoted 8 lines
Makefile +5
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +KBUILD_CFLAGS += -fsanitize=shadow-call-stack +DISABLE_SCS := -fno-sanitize=shadow-call-stack +export DISABLE_SCS +endif
I think it would be preferable to follow the example of CC_FLAGS_FTRACE so that this can be filtered out, e.g.
ifdef CONFIG_SHADOW_CALL_STACKCFLAGS_SCS := -fsanitize=shadow-call-stackKBUILD_CFLAGS += $(CFLAGS_SCS)
export CC_FLAGS_SCS
endif
... with removal being:
CFLAGS_REMOVE := $(CC_FLAGS_SCS)
... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
[...]
Makefile ifdef CONFIG_LIVEPATCH
Quoted 9 lines
Makefile +9
-0+config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + def_bool n
A bool is default n, so you can just say bool here.
Makefile
Quoted 12 lines
Makefile +12
-0+ depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +choice + prompt "Return-oriented programming (ROP) protection" + default ROP_PROTECTION_NONE + help + This option controls kernel protections against return-oriented + programming (ROP) attacks.
Are we expecting more options here in future?
Makefile
Quoted 2 lines
Makefile +2
-0+config ROP_PROTECTION_NONE + bool "None"
IIUC this is for the benefit of the kretprobes Kconfig.
I think it would be better to make that depend on !SHADOW_CALL_STACK, as it's plausible that we can add a different ROP protection mechanism that is compatible with kretprobes.
Makefile
Quoted 4 lines
Makefile +4
-0+config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000
Is there a reason for an explicit version check rather than a CC_HAS_<feature> check? e.g. was this available but broken in prior versions of clang?
[...]
Quoted 1 line
+#define
SCS_GFP(GFP_KERNEL|__GFP_ZERO)
Normally GFP_ is a prefix. For consistency, GFP_SCS would be preferable.
Quoted 1 line
+extern unsigned long init_shadow_call_stack[];
Do we need this exposed here? IIUC this is only assigned by assembly in arch code.
[...]
Makefile
Quoted 6 lines
Makefile +6
-0+void scs_set_init_magic(struct task_struct *tsk) +{ + scs_save(tsk); + scs_set_magic(tsk); + scs_load(tsk); +}
Can we initialize this at compile time instead?
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 42 lines · Mark Rutland
Quoted 29 lines · Sami Tolvanen
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 39 ++++++++ include/linux/compiler-clang.h | 2 + include/linux/compiler_types.h | 4 + include/linux/scs.h | 88 ++++++++++++++++++ init/init_task.c | 6 ++ init/main.c | 3 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 162 +++++++++++++++++++++++++++++++++ 12 files changed, 323 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +0
-0Makefile ifdef CONFIG_LIVEPATCH · line 853
Quoted 8 lines
Makefile +5
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +KBUILD_CFLAGS += -fsanitize=shadow-call-stack +DISABLE_SCS := -fno-sanitize=shadow-call-stack +export DISABLE_SCS +endifI think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.
Quoted 2 lines
ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stack
^^^ was this meant to be CC_FLAGS_SCS here
Quoted 1 line
KBUILD_CFLAGS+= $(CFLAGS_SCS)
^^^ and here?
Quoted 15 lines
export
CC_FLAGS_SCS
endif... with removal being:
CFLAGS_REMOVE:= $(CC_FLAGS_SCS)... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
[...]
Signature · 1 line
Kees Cook
Mark Rutland
reviewerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 11 lines · Kees Cook
Quoted 10 lines · Mark Rutland
Makefile ifdef CONFIG_LIVEPATCH · line 853
Quoted 5 lines · Sami Tolvanen
Makefile +5
-0+ifdef CONFIG_SHADOW_CALL_STACK +KBUILD_CFLAGS += -fsanitize=shadow-call-stack +DISABLE_SCS := -fno-sanitize=shadow-call-stack +export DISABLE_SCS +endifI think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.
Quoted 6 lines
Quoted 2 lines
ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stack^^^ was this meant to be
CC_FLAGS_SCShereQuoted 1 line
KBUILD_CFLAGS+= $(CFLAGS_SCS)^^^ and here?
Whoops; yes in both cases...
Quoted 5 lines
Quoted 5 lines
export
CC_FLAGS_SCS
endif... with removal being:
Quoted 8 lines
Quoted 8 lines
CFLAGS_REMOVE:= $(CC_FLAGS_SCS)... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
^^^^^ "flags" here, too.
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Mark RutlandQuoted 8 lines · Mark Rutland
I think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stackKBUILD_CFLAGS+= $(CFLAGS_SCS)
exportCC_FLAGS_SCS
endif
Sure, SGTM.
Quoted 8 lines
arch/Kconfig config STACKPROTECTOR_STRONG · line 544
Quoted 6 lines
arch/Kconfig +6
-0+choice + prompt "Return-oriented programming (ROP) protection" + default ROP_PROTECTION_NONE + help + This option controls kernel protections against return-oriented + programming (ROP) attacks.Are we expecting more options here in future?
Yes, I believe we'd be interested in seeing PAC support too once hardware is more readily available.
Quoted 3 lines
I think it would be better to ./make that depend on !SHADOW_CALL_STACK, as it's plausible that we can add a different ROP protection mechanism that is compatible with kretprobes.
OK, I can change that and remove the choice. We can always add it back when other alternatives are added.
Quoted 8 lines
arch/Kconfig config STACKPROTECTOR_STRONG · line 552
Quoted 4 lines
arch/Kconfig +4
-0+config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000Is there a reason for an explicit version check rather than a CC_HAS_<feature> check? e.g. was this available but broken in prior versions of clang?
No, this feature was added in Clang 7. However, -fsanitize=shadow-call-stack might require architecture-specific flags, so a simple $(cc-option, -fsanitize=shadow-call-stack) in arch/Kconfig is not going to work. I could add something like this to arch/arm64/Kconfig though:
select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK
...
config CC_HAVE_SHADOW_CALL_STACK
def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18)
And then drop CC_IS_CLANG and version check entirely. Thoughts?
Quoted 3 lines
Quoted 1 line
+#define
SCS_GFP(GFP_KERNEL|__GFP_ZERO)Normally
GFP_is a prefix. For consistency,GFP_SCSwould be preferable.
Ack.
Quoted 4 lines
Quoted 1 line
+extern unsigned long init_shadow_call_stack[];
Do we need this exposed here? IIUC this is only assigned by assembly in arch code.
True, it's not needed.
Quoted 10 lines
[...]
kernel/scs.c line 129
Quoted 6 lines
kernel/scs.c new file +6
-0+void scs_set_init_magic (struct task_struct *tsk) +{ +scs_save (tsk); +scs_set_magic (tsk); +scs_load (tsk); +}Can we initialize this at compile time instead?
We can. I'll change this and drop the function.
Sami
Mark Rutland
reviewerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 17 lines · Sami Tolvanen
Quoted 8 lines · Mark Rutland
arch/Kconfig config STACKPROTECTOR_STRONG · line 552
Quoted 4 lines
arch/Kconfig +4
-0+config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000Is there a reason for an explicit version check rather than a CC_HAS_<feature> check? e.g. was this available but broken in prior versions of clang?
No, this feature was added in Clang 7. However, -fsanitize=shadow-call-stack might require architecture-specific flags, so a simple $(cc-option, -fsanitize=shadow-call-stack) in
arch/Kconfigis not going to work. I could add something like this toarch/arm64/Kconfigthough:
Quoted 6 lines
select
ARCH_SUPPORTS_SHADOW_CALL_STACKifCC_HAVE_SHADOW_CALL_STACK
...
configCC_HAVE_SHADOW_CALL_STACKdef_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18)And then drop
CC_IS_CLANGand version check entirely. Thoughts?
That sounds good to me, yes!
Thanks,
Mark.
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 24 lines · Mark Rutland
Quoted 22 lines · Sami Tolvanen
Quoted 8 lines · Mark Rutland
arch/Kconfig config STACKPROTECTOR_STRONG · line 552
Quoted 4 lines
arch/Kconfig +4
-0+config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + depends on CC_IS_CLANG && CLANG_VERSION >= 70000Is there a reason for an explicit version check rather than a CC_HAS_<feature> check? e.g. was this available but broken in prior versions of clang?
No, this feature was added in Clang 7. However, -fsanitize=shadow-call-stack might require architecture-specific flags, so a simple $(cc-option, -fsanitize=shadow-call-stack) in
arch/Kconfigis not going to work. I could add something like this toarch/arm64/Kconfigthough:select
ARCH_SUPPORTS_SHADOW_CALL_STACKifCC_HAVE_SHADOW_CALL_STACK
...
configCC_HAVE_SHADOW_CALL_STACKdef_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18)
Quoted 6 lines
Quoted 1 line
And then drop
CC_IS_CLANGand version check entirely. Thoughts?That sounds good to me, yes!
Thanks,
Mark.
If you use cc-option, please add a comment like
# supported by Clang 7 or later.
I do not know the minimal supported clang version. When we bump the minimal version to clang 7, we can drop the cc-option test entirely.
--
Best Regards
Masahiro Yamada
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Mark RutlandQuoted 11 lines · Mark Rutland
I think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stackKBUILD_CFLAGS+= $(CFLAGS_SCS)
exportCC_FLAGS_SCS
endif... with removal being:
Quoted 8 lines
CFLAGS_REMOVE:= $(CC_FLAGS_SCS)... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
CFLAGS_REMOVE appears to be only implemented for objects, which means there's no convenient way to filter out flags for everything in arch/arm64/kvm/hyp, for example. I could add a CFLAGS_REMOVE separately for each object file, or we could add something like ccflags-remove-y to complement ccflags-y, which should be relatively simple. Masahiro, do you have any suggestions?
Sami
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 22 lines · Sami Tolvanen
Quoted 11 lines · Mark Rutland
I think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stackKBUILD_CFLAGS+= $(CFLAGS_SCS)
exportCC_FLAGS_SCS
endif... with removal being:
Quoted 8 lines
CFLAGS_REMOVE:= $(CC_FLAGS_SCS)... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
Quoted 6 lines
CFLAGS_REMOVEappears to be only implemented for objects, which means there's no convenient way to filter out flags for everything inarch/arm64/kvm/hyp, for example. I could add aCFLAGS_REMOVEseparately for each object file, or we could add something like ccflags-remove-y to complement ccflags-y, which should be relatively simple. Masahiro, do you have any suggestions?
I am fine with 'ccflags-remove-y'.
Thanks.
Signature · 2 lines
Best Regards Masahiro Yamada
Steven Rostedt
maintainerRe: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
In reply to Sami TolvanenQuoted 21 lines · On Wed, 23 Oct 2019 09:59:09 -0700 Sami Tolvanen <[email protected]>…
Quoted 11 lines · Mark Rutland
I think it would be preferable to follow the example of
CC_FLAGS_FTRACEso that this can be filtered out, e.g.ifdef
CONFIG_SHADOW_CALL_STACKCFLAGS_SCS:= -fsanitize=shadow-call-stackKBUILD_CFLAGS+= $(CFLAGS_SCS)
exportCC_FLAGS_SCS
endif... with removal being:
Quoted 8 lines
CFLAGS_REMOVE:= $(CC_FLAGS_SCS)... or:
CFLAGS_REMOVE_obj.o := $(CC_FLAGS_SCS)
That way you only need to define the flags once, so the enable and disable falgs remain in sync by construction.
Quoted 7 lines
CFLAGS_REMOVEappears to be only implemented for objects, which means there's no convenient way to filter out flags for everything inarch/arm64/kvm/hyp, for example. I could add aCFLAGS_REMOVEseparately for each object file, or we could add something like ccflags-remove-y to complement ccflags-y, which should be relatively simple. Masahiro, do you have any suggestions?
You can remove a CFLAGS for a whole directory. lib, kernel/trace and others do this. Look at kernel/trace/Makefile, we have:
ORIG_CFLAGS := $(KBUILD_CFLAGS)KBUILD_CFLAGS = $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))
Where it removes CC_FLAGS_FTRACE from CFLAGS for all objects in the directory.
-- Steve
Re: [PATCH 06/18] add support for Clang's Shadow Call Stack (SCS)
Quoted 5 lines · Steven Rostedt
You can remove a CFLAGS for a whole directory. lib, kernel/trace and others do this. Look at
kernel/trace/Makefile, we have:
ORIG_CFLAGS:= $(KBUILD_CFLAGS)KBUILD_CFLAGS= $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))
That definitely looks less invasive in this case than adding ccflags-remove-y, since we only really need this for one directory. I'll use this in v2. Thanks, Steven.
Sami
PATCH 7/18 scs: add accounting
In reply to Sami TolvanenThis change adds accounting for the memory allocated for shadow stacks.
--- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 20 ++++++++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 42 insertions(+)
drivers/base/node.c +6 -0
@@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif"Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n"
@@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_ANON_MAPPED )), nid,K (i.sharedram), nid,sum_zone_node_page_state (nid,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK + nid,sum_zone_node_page_state (nid,NR_KERNEL_SCS_BYTES ) /1024 , +#endif nid,K (sum_zone_node_page_state (nid,NR_PAGETABLE )), nid,K (node_page_state (pgdat,NR_UNSTABLE_NFS )), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
fs/proc/meminfo.c +4 -0
@@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)show_val_kb (m,"SUnreclaim: " , sunreclaim);seq_printf (m,"KernelStack: %8lu kB\n" ,global_zone_page_state (NR_KERNEL_STACK_KB )); +#ifdef CONFIG_SHADOW_CALL_STACK +seq_printf (m,"ShadowCallStack:%8lu kB\n" , +global_zone_page_state (NR_KERNEL_SCS_BYTES ) /1024 ); +#endif show_val_kb (m,"PageTables: " ,global_zone_page_state (NR_PAGETABLE ));
include/linux/mmzone.h +3 -0
@@ -200,6 +200,9 @@ enum zone_stat_item {NR_MLOCK ,/* mlock()ed pages found and moved off LRU */ NR_PAGETABLE ,/* used for pagetables */ NR_KERNEL_STACK_KB ,/* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +NR_KERNEL_SCS_BYTES ,/* measured in bytes */ +#endif/* Second 128 byte cacheline */ NR_BOUNCE , #if IS_ENABLED(CONFIG_ZSMALLOC)
kernel/scs.c +20 -0
@@ -7,9 +7,11 @@#include <linux/cpuhotplug.h> #include <linux/mm.h> +#include <linux/mmzone.h> #include <linux/slab.h> #include <linux/scs.h> #include <linux/vmalloc.h> +#include <linux/vmstat.h> #include <asm/scs.h> #define SCS_END_MAGIC 0xaf0194819b1635f6UL
@@ -59,6 +61,11 @@ static void scs_free(void *s)vfree_atomic (s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return vmalloc_to_page (__scs_base (tsk)); +} +static int scs_cleanup (unsigned int cpu) {int i;
@@ -92,6 +99,11 @@ static inline void scs_free(void *s)kmem_cache_free (scs_cache, s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return virt_to_page (__scs_base (tsk)); +} +void __init scs_init (void ) { scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE,
@@ -128,6 +140,12 @@ void scs_set_init_magic(struct task_struct *tsk)scs_load (tsk); } +static void scs_account (struct task_struct *tsk,int account) +{ +mod_zone_page_state (page_zone (__scs_page (tsk)),NR_KERNEL_SCS_BYTES , + account *SCS_SIZE ); +} +int scs_prepare (struct task_struct *tsk,int node) {void *s;
@@ -138,6 +156,7 @@ int scs_prepare(struct task_struct *tsk, int node)task_set_scs (tsk, s);scs_set_magic (tsk); +scs_account (tsk,1 );return 0 ; }
@@ -157,6 +176,7 @@ void scs_release(struct task_struct *tsk)WARN_ON (scs_corrupted (tsk)); +scs_account (tsk,-1 );scs_task_init (tsk);scs_free (s); }
mm/page_alloc.c +6 -0
@@ -5361,6 +5361,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif" pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB"
@@ -5382,6 +5385,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)K (zone_managed_pages (zone)),K (zone_page_state (zone,NR_MLOCK )),zone_page_state (zone,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK +zone_page_state (zone,NR_KERNEL_SCS_BYTES ) /1024 , +#endifK (zone_page_state (zone,NR_PAGETABLE )),K (zone_page_state (zone,NR_BOUNCE )), K(free_pcp),
mm/vmstat.c +3 -0
@@ -1118,6 +1118,9 @@ const char * const vmstat_text[] = {"nr_mlock" ,"nr_page_table_pages" ,"nr_kernel_stack" , +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +"nr_shadow_call_stack_bytes" , +#endif"nr_bounce" , #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages",
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 8/18 scs: add support for stack usage debugging
In reply to Sami TolvanenImplements CONFIG_DEBUG_STACK_USAGE for shadow stacks.
--- kernel/scs.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
kernel/scs.c +39 -0
@@ -161,6 +161,44 @@ int scs_prepare(struct task_struct *tsk, int node)return 0 ; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long scs_used (struct task_struct *tsk) +{ +unsigned long *p =__scs_base (tsk); +unsigned long *end =scs_magic (tsk); +uintptr_t s = (uintptr_t )p; + +while (p < end && *p) + p++; + +return (uintptr_t )p - s; +} + +static void scs_check_usage (struct task_struct *tsk) +{ +static DEFINE_SPINLOCK (lock); +static unsigned long highest; +unsigned long used =scs_used (tsk); + +if (used <= highest) +return ; + +spin_lock (&lock); + +if (used > highest) { +pr_info ("%s: highest shadow stack usage %lu bytes\n" , + __func__, used); + highest = used; + } + +spin_unlock (&lock); +} +#else +static inline void scs_check_usage (struct task_struct *tsk) +{ +} +#endif +bool scs_corrupted (struct task_struct *tsk) {return *scs_magic (tsk) !=SCS_END_MAGIC ;
@@ -175,6 +213,7 @@ void scs_release(struct task_struct *tsk)return ;WARN_ON (scs_corrupted (tsk)); +scs_check_usage (tsk);scs_account (tsk,-1 );scs_task_init (tsk);
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 9/18 trace: disable function graph tracing with SCS
In reply to Sami TolvanenWith CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified in ftrace_graph_caller and prepare_ftrace_return to redirect control flow to ftrace_return_to_handler. This is incompatible with return address protection.
--- kernel/trace/Kconfig | 1 + 1 file changed, 1 insertion(+)
kernel/trace/Kconfig +1 -0
@@ -161,6 +161,7 @@ config FUNCTION_GRAPH_TRACER depends on HAVE_FUNCTION_GRAPH_TRACER depends on FUNCTION_TRACER depends on !X86_32 || !CC_OPTIMIZE_FOR_SIZE + depends on ROP_PROTECTION_NONE default y help Enable the kernel to trace a function at both its return
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 09/18] trace: disable function graph tracing with SCS
Quoted 15 lines · On Fri, 18 Oct 2019 09:10:24 -0700 Sami Tolvanen <[email protected]>…
With
CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified inftrace_graph_callerandprepare_ftrace_returnto redirect control flow toftrace_return_to_handler. This is incompatible with return address protection.Signed-off-by Sami Tolvanen <[email protected]>--- kernel/trace/Kconfig | 1 + 1 file changed, 1 insertion(+)kernel/trace/Kconfig +0
-0
kernel/trace/Kconfig config FUNCTION_GRAPH_TRACER · line 164
Quoted 4 lines
kernel/trace/Kconfig +1
-0@@ -161,6 +161,7 @@ config FUNCTION_GRAPH_TRACER depends on HAVE_FUNCTION_GRAPH_TRACER depends on FUNCTION_TRACER depends on !X86_32 || !CC_OPTIMIZE_FOR_SIZE + depends on ROP_PROTECTION_NONE
NAK, Put this in the arch code.
kernel/trace/Kconfig config FUNCTION_GRAPH_TRACER · line 167
Quoted 3 lines
kernel/trace/Kconfig +0
-0default y help Enable the kernel to trace a function at both its return
-- Steve
arch/arm64/Kconfig +1 -1
@@ -149,7 +149,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if ROP_PROTECTION_NONE select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING
Re: [PATCH 09/18] trace: disable function graph tracing with SCS
Quoted 7 lines · Steven Rostedt
NAK, Put this in the arch code.
arch/arm64/Kconfig +0
-0
arch/arm64/Kconfig config ARM64 · line 155
Quoted 8 lines
arch/arm64/Kconfig +1
-1@@ -149,7 +149,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if ROP_PROTECTION_NONE select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING
Thanks, Steven. I'll fix this and kretprobes in v2.
Sami
Re: [PATCH 09/18] trace: disable function graph tracing with SCS
In reply to Sami TolvanenQuoted 7 lines · Sami Tolvanen
With
CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified inftrace_graph_callerandprepare_ftrace_returnto redirect control flow toftrace_return_to_handler. This is incompatible with return address protection.Signed-off-by Sami Tolvanen <[email protected]>
How difficult would it be to update the return address on the shadow call stack along with the normal one? Not having to disable infrastructure that is widely used by the distros would make this a lot more palatable in the general case (even if it is Clang only at the moment)
Quoted 19 lines
--- kernel/trace/Kconfig | 1 + 1 file changed, 1 insertion(+)kernel/trace/Kconfig +1
-1@@ -161,6 +161,7 @@ config FUNCTION_GRAPH_TRACER depends on HAVE_FUNCTION_GRAPH_TRACER depends on FUNCTION_TRACER depends on !X86_32 || !CC_OPTIMIZE_FOR_SIZE + depends on ROP_PROTECTION_NONE default y help Enable the kernel to trace a function at both its return --2.23.0.866.gb869b98d4c-goog
PATCH 10/18 kprobes: fix compilation without CONFIG_KRETPROBES
In reply to Sami Tolvanenkprobe_on_func_entry and arch_kprobe_on_func_entry need to be available even if CONFIG_KRETPROBES is not selected.
--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
kernel/kprobes.c +19 -19
@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe
@@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Steven Rostedt
maintainerRe: [PATCH 10/18] kprobes: fix compilation without CONFIG_KRETPROBES
Added Masami who's the maintainer of kprobes.
-- Steve
Quoted 64 lines · On Fri, 18 Oct 2019 09:10:25 -0700 Sami Tolvanen <[email protected]>…
kprobe_on_func_entryandarch_kprobe_on_func_entryneed to be available even ifCONFIG_KRETPROBESis not selected.Signed-off-by Sami Tolvanen <[email protected]>--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)kernel/kprobes.c +19
-19@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe @@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;
Re: [PATCH 10/18] kprobes: fix compilation without CONFIG_KRETPROBES
Quoted 11 lines · On Fri, 18 Oct 2019 13:02:57 -0400 Steven Rostedt <[email protected]> wrote:
Added Masami who's the maintainer of kprobes.
-- Steve
Quoted 2 lines · On Fri, 18 Oct 2019 09:10:25 -0700 Sami Tolvanen <[email protected]>…
kprobe_on_func_entryandarch_kprobe_on_func_entryneed to be available even ifCONFIG_KRETPROBESis not selected.
Good catch! Since nowadays all arch supports kretprobes, I've missed to test it.
Thank you,
Quoted 63 lines
Quoted 62 lines
Signed-off-by Sami Tolvanen <[email protected]>--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)kernel/kprobes.c +19
-19@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe @@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;
Signature · 6 lines
Masami Hiramatsu <[email protected]> _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 11/18 kprobes: disable kretprobes with SCS
In reply to Sami TolvanenWith CONFIG_KRETPROBES, function return addresses are modified to redirect control flow to kretprobe_trampoline. This is incompatible with return address protection.
--- arch/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/Kconfig +1 -1
@@ -171,7 +171,7 @@ config ARCH_USE_BUILTIN_BSWAP config KRETPROBES def_bool y - depends on KPROBES && HAVE_KRETPROBES + depends on KPROBES && HAVE_KRETPROBES && ROP_PROTECTION_NONE config USER_RETURN_NOTIFIER bool
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Steven Rostedt
maintainerRe: [PATCH 11/18] kprobes: disable kretprobes with SCS
[ Added Masami ]
Quoted 14 lines · On Fri, 18 Oct 2019 09:10:26 -0700 Sami Tolvanen <[email protected]>…
With
CONFIG_KRETPROBES, function return addresses are modified to redirect control flow tokretprobe_trampoline. This is incompatible with return address protection.Signed-off-by Sami Tolvanen <[email protected]>--- arch/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/Kconfig +0
-0
arch/Kconfig config ARCH_USE_BUILTIN_BSWAP · line 174
Quoted 5 lines
arch/Kconfig +1
-1@@ -171,7 +171,7 @@ config ARCH_USE_BUILTIN_BSWAP config KRETPROBES def_bool y - depends on KPROBES && HAVE_KRETPROBES + depends on KPROBES && HAVE_KRETPROBES && ROP_PROTECTION_NONE
Again, this belongs in the arch code.
-- Steve
arch/Kconfig config ARCH_USE_BUILTIN_BSWAP · line 176
Quoted 3 lines
arch/Kconfig +0
-0config USER_RETURN_NOTIFIER bool
arch/arm64/Kconfig +1 -1
@@ -166,7 +166,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if ROP_PROTECTION_NONE select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN
Masami Hiramatsu
maintainerRe: [PATCH 11/18] kprobes: disable kretprobes with SCS
Quoted 21 lines · On Fri, 18 Oct 2019 13:04:29 -0400 Steven Rostedt <[email protected]> wrote:
[ Added Masami ]
Quoted 15 lines · On Fri, 18 Oct 2019 09:10:26 -0700 Sami Tolvanen <[email protected]>…
With
CONFIG_KRETPROBES, function return addresses are modified to redirect control flow tokretprobe_trampoline. This is incompatible with return address protection.Signed-off-by Sami Tolvanen <[email protected]>--- arch/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/Kconfig +0
-0@@ -171,7 +171,7 @@ config ARCH_USE_BUILTIN_BSWAP
Quoted 6 lines
arch/Kconfig config ARCH_USE_BUILTIN_BSWAP · line 174
Quoted 4 lines
arch/Kconfig +1
-1config KRETPROBES def_bool y - depends on KPROBES && HAVE_KRETPROBES + depends on KPROBES && HAVE_KRETPROBES && ROP_PROTECTION_NONEAgain, this belongs in the arch code.
+1, below patch (from Steve) looks good to me.
Thank you,
Quoted 21 lines
-- Steve
Quoted 3 lines
arch/Kconfig +0
-0config USER_RETURN_NOTIFIER boolarch/arm64/Kconfig +1
-1@@ -166,7 +166,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if ROP_PROTECTION_NONE select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN
Signature · 1 line
Masami Hiramatsu <[email protected]>
PATCH 12/18 arm64: reserve x18 only with Shadow Call Stack
In reply to Sami TolvanenOnly reserve x18 with CONFIG_SHADOW_CALL_STACK. Note that all external kernel modules must also have x18 reserved if the kernel uses SCS.
--- arch/arm64/Makefile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/Makefile +5 -1
@@ -55,7 +55,7 @@ endif KBUILD_CFLAGS += -mgeneral-regs-only $(lseinstr) $(brokengasinst) \ $(compat_vdso) $(cc_has_k_constraint) -KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -ffixed-x18 +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_CFLAGS += $(call cc-disable-warning, psabi) KBUILD_AFLAGS += $(lseinstr) $(brokengasinst) $(compat_vdso)
@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 12/18] arm64: reserve x18 only with Shadow Call Stack
Quoted 3 lines · Sami Tolvanen' via Clang Built Linux
Only reserve x18 with
CONFIG_SHADOW_CALL_STACK. Note that all external kernel modules must also have x18 reserved if the kernel uses SCS.
Ah, ok. The tradeoff for maintainers to consider, either:
1. one less GPR for ALL kernel code or
2. remember not to use x18 in inline as lest you potentially break SCS
This patch is 2 (the earlier patch was 1). Maybe we don't write enough inline asm that this will be hard to remember, and we do have CI in Android to watch for this (on mainline, not sure about -next).
Either way,
Quoted 37 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/Makefile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)arch/arm64/Makefile +5
-2@@ -55,7 +55,7 @@ endif KBUILD_CFLAGS += -mgeneral-regs-only $(lseinstr) $(brokengasinst) \ $(compat_vdso) $(cc_has_k_constraint) -KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -ffixed-x18 +KBUILD_CFLAGS += -fno-asynchronous-unwind-tables KBUILD_CFLAGS += $(call cc-disable-warning, psabi) KBUILD_AFLAGS += $(lseinstr) $(brokengasinst) $(compat_vdso)@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ --2.23.0.866.gb869b98d4c-goog
--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20191018161033.261971-13-samitolvanen%40google.com.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Mark Rutland
reviewerRe: [PATCH 12/18] arm64: reserve x18 only with Shadow Call Stack
Quoted 9 lines · Nick Desaulniers
Quoted 3 lines · Sami Tolvanen' via Clang Built Linux
Only reserve x18 with
CONFIG_SHADOW_CALL_STACK. Note that all external kernel modules must also have x18 reserved if the kernel uses SCS.Ah, ok. The tradeoff for maintainers to consider, either:
1. one less GPR for ALL kernel code or
2. remember not to use x18 in inline as lest you potentially break SCS
This option only affects compiler-generated code, so I don't think that matters.
I think it's fine to say that we should always avoid the use of x18 in hand-written assembly (with manual register allocation), while also allowing the compiler to use x18 if we're not using SCS.
This can be folded into the earlier patch which always reserved x18.
Quoted 3 lines
This patch is 2 (the earlier patch was 1). Maybe we don't write enough inline asm that this will be hard to remember, and we do have CI in Android to watch for this (on mainline, not sure about -next).
I think that we can trust the set of people who regularly review arm64 assembly to remember this. We could also document this somewhere -- we might need to document other constraints or conventions for assembly in preparation for livepatching and so on.
If we wanted to, we could periodically grep for x18 to find any illicit usage.
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 12/18] arm64: reserve x18 only with Shadow Call Stack
Quoted 2 lines · Mark Rutland
If we wanted to, we could periodically grep for x18 to find any illicit usage.
Now we need objtool for arm64! :) (It seems CONFIG_HAVE_STACK_VALIDATION is rather a narrow description for what objtool does now...)
Signature · 1 line
Kees Cook
PATCH 13/18 arm64: preserve x18 when CPU is suspended
In reply to Sami TolvanenDon't lose the current task's shadow stack when the CPU is suspended.
--- arch/arm64/mm/proc.S | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/mm/proc.S +6 -0
@@ -73,6 +73,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + stp x18, xzr, [x0, #96] +#endif ret ENDPROC(cpu_do_suspend)
@@ -89,6 +92,9 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldp x18, x19, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Nick Desaulniers
reviewerRe: [PATCH 13/18] arm64: preserve x18 when CPU is suspended
Quoted 13 lines · Sami Tolvanen
Don't lose the current task's shadow stack when the CPU is suspended.
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/mm/proc.S | 6 ++++++ 1 file changed, 6 insertions(+)arch/arm64/mm/proc.S +0
-0
arch/arm64/mm/proc.S alternative_endif · line 77
Quoted 5 lines
arch/arm64/mm/proc.S +2
-0@@ -73,6 +73,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + stp x18, xzr, [x0, #96]
Could this be a str/ldr of just x18 rather than stp/ldp of x18 + garbage? Maybe there's no real cost difference, or some kind of alignment invariant?
Quoted 17 lines
arch/arm64/mm/proc.S +4
-1+#endif ret ENDPROC(cpu_do_suspend)@@ -89,6 +92,9 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldp x18, x19, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4 --2.23.0.866.gb869b98d4c-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH 13/18] arm64: preserve x18 when CPU is suspended
Quoted 6 lines · Nick Desaulniers
Quoted 6 lines
arch/arm64/mm/proc.S +0
-0@@ -73,6 +73,9 @@ alternative_endif stp x8, x9, [x0, #48]
Quoted 8 lines
arch/arm64/mm/proc.S alternative_endif · line 77
Quoted 4 lines
arch/arm64/mm/proc.S +2
-0stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + stp x18, xzr, [x0, #96]Could this be a str/ldr of just x18 rather than stp/ldp of x18 + garbage? Maybe there's no real cost difference, or some kind of alignment invariant?
Sure, this can be changed to str/ldr. I don't think there's a noticeable difference in cost.
Sami
Mark Rutland
reviewerRe: [PATCH 13/18] arm64: preserve x18 when CPU is suspended
In reply to Sami TolvanenQuoted 12 lines · Sami Tolvanen
Don't lose the current task's shadow stack when the CPU is suspended.
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/mm/proc.S | 6 ++++++ 1 file changed, 6 insertions(+)arch/arm64/mm/proc.S +0
-0
arch/arm64/mm/proc.S alternative_endif · line 78
Quoted 6 lines
arch/arm64/mm/proc.S +3
-0@@ -73,6 +73,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + stp x18, xzr, [x0, #96] +#endif
This should have a corresponding change to cpu_suspend_ctx in <asm/suspend.h>. Otherwise we're corrupting a portion of the stack.
Mark.
Quoted 16 lines
arch/arm64/mm/proc.S +3
-0ret ENDPROC(cpu_do_suspend)@@ -89,6 +92,9 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldp x18, x19, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4Signature · 1 line
2.23.0.866.gb869b98d4c-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 13/18] arm64: preserve x18 when CPU is suspended
Quoted 2 lines · Mark Rutland
This should have a corresponding change to
cpu_suspend_ctxin <asm/suspend.h>. Otherwise we're corrupting a portion of the stack.
Ugh, correct. I'll fix this in the next version. Thanks.
Sami
Mark Rutland
reviewerRe: [PATCH 13/18] arm64: preserve x18 when CPU is suspended
Quoted 5 lines · Sami Tolvanen
Quoted 2 lines · Mark Rutland
This should have a corresponding change to
cpu_suspend_ctxin <asm/suspend.h>. Otherwise we're corrupting a portion of the stack.Ugh, correct. I'll fix this in the next version. Thanks.
It's probably worth extending the comment above cpu_do_suspend to say:
| This must be kept in sync with struct cpu_suspend_ctx in | <asm/suspend.h>
... to match what we have above struct cpu_suspend_ctx, and make this more obvious in future.
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 14/18 arm64: efi: restore x18 if it was corrupted
In reply to Sami TolvanenIf we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code.
--- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kernel/efi-rt-wrapper.S +6 -1
@@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper)
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 14/18] arm64: efi: restore x18 if it was corrupted
Quoted 4 lines · Sami Tolvanen
If we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code.
You'll have to elaborate a bit here and explain that this is sufficient, given that we run EFI runtime services with interrupts enabled.
Quoted 24 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)arch/arm64/kernel/efi-rt-wrapper.S +6
-2@@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper) --2.23.0.866.gb869b98d4c-goog
Re: [PATCH 14/18] arm64: efi: restore x18 if it was corrupted
Quoted 3 lines · Ard Biesheuvel
You'll have to elaborate a bit here and explain that this is sufficient, given that we run EFI runtime services with interrupts enabled.
I can add a note about this in v2. This is called with preemption disabled and we have a separate interrupt shadow stack, so as far as I can tell, this should be sufficient. Did you have concerns about this?
Sami
Re: [PATCH 14/18] arm64: efi: restore x18 if it was corrupted
Quoted 11 lines · Sami Tolvanen
Quoted 3 lines · Ard Biesheuvel
You'll have to elaborate a bit here and explain that this is sufficient, given that we run EFI runtime services with interrupts enabled.
I can add a note about this in v2. This is called with preemption disabled and we have a separate interrupt shadow stack, so as far as I can tell, this should be sufficient. Did you have concerns about this?
No concerns, but we should put the above clarification in the commit log.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 15/18 arm64: vdso: disable Shadow Call Stack
In reply to Sami Tolvanen--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -19,7 +19,7 @@ obj-vdso := $(addprefix $(obj)/, $(obj-vdso)) ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ --build-id -n -T - +ccflags-y += $(DISABLE_SCS) ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 ccflags-y += -DDISABLE_BRANCH_PROFILING
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 16/18 arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
In reply to Sami TolvanenThis allows CONFIG_KRETPROBES to be disabled without disabling kprobes entirely.
--- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kernel/probes/kprobes.c +2 -0
@@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe (struct kretprobe_instance *ri,struct pt_regs *regs) {
@@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) {return 0 ; } +#endif int __init arch_init_kprobes (void ) {
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 16/18] arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
Quoted 5 lines · Sami Tolvanen
This allows
CONFIG_KRETPROBESto be disabled without disabling kprobes entirely.Signed-off-by Sami Tolvanen <[email protected]>
Can we make kretprobes work with the shadow call stack instead?
Quoted 27 lines
--- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kernel/probes/kprobes.c +2
-1@@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe (struct kretprobe_instance *ri,struct pt_regs *regs) {@@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) {return 0 ; } +#endif int __init arch_init_kprobes (void ) { --2.23.0.866.gb869b98d4c-goog
Re: [PATCH 16/18] arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
Quoted 8 lines · Ard Biesheuvel
Quoted 5 lines · Sami Tolvanen
This allows
CONFIG_KRETPROBESto be disabled without disabling kprobes entirely.Signed-off-by Sami Tolvanen <[email protected]>Can we make kretprobes work with the shadow call stack instead?
I've been viewing that as "next steps". This series is the first step:
actually gaining the feature and clearly indicating where future improvements can live.
Signature · 1 line
Kees Cook
PATCH 17/18 arm64: disable SCS for hypervisor code
In reply to Sami Tolvanen--- arch/arm64/kvm/hyp/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/Makefile +2 -1
@@ -4,7 +4,8 @@ # ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ - $(DISABLE_STACKLEAK_PLUGIN) + $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_SCS) KVM=../../../../virt/kvm
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
PATCH 18/18 arm64: implement Shadow Call Stack
In reply to Sami TolvanenThis change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
--- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/scs.h | 60 ++++++++++++++++++++++++++++ arch/arm64/include/asm/stacktrace.h | 4 ++ arch/arm64/include/asm/thread_info.h | 3 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 ++ arch/arm64/kernel/entry.S | 23 +++++++++++ arch/arm64/kernel/head.S | 9 +++++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/process.c | 3 ++ arch/arm64/kernel/scs.c | 39 ++++++++++++++++++ arch/arm64/kernel/smp.c | 4 ++ 12 files changed, 152 insertions(+) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c
arch/arm64/Kconfig +1 -0
@@ -66,6 +66,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING
arch/arm64/include/asm/scs.h new file +60 -0
60 lines not on this page · raw message
arch/arm64/include/asm/stacktrace.h +4 -0
@@ -68,6 +68,10 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif +static inline bool on_irq_stack (unsigned long sp,struct stack_info *info) {
arch/arm64/include/asm/thread_info.h +3 -0
@@ -41,6 +41,9 @@ struct thread_info {#endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK +void *shadow_call_stack; +#endif };#define thread_saved_pc (tsk) \
arch/arm64/kernel/Makefile +1 -0
@@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/
arch/arm64/kernel/asm-offsets.c +3 -0
@@ -33,6 +33,9 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit));#ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack));#endif DEFINE (TSK_STACK , offsetof(struct task_struct , stack));#ifdef CONFIG_STACKPROTECTOR
arch/arm64/kernel/entry.S +23 -0
@@ -172,6 +172,10 @@ alternative_cb_end apply_ssbd 1, x22, x23 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack + str xzr, [tsk, #TSK_TI_SCS] +#endif .else add x21, sp, #S_FRAME_SIZE get_current_task tsk
@@ -278,6 +282,12 @@ alternative_else_nop_endif ct_user_enter .endif +#ifdef CONFIG_SHADOW_CALL_STACK + .if \el == 0 + str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack + .endif +#endif + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR
@@ -383,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x20, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack.
@@ -400,6 +413,12 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + ldr_this_cpu x18, irq_shadow_call_stack_ptr, x26 +#endif + 9998: .endm
@@ -409,6 +428,10 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + /* x20 is also preserved */ + mov x18, x20 +#endif .endm /* GPRs used by entry code */
arch/arm64/kernel/head.S +9 -0
@@ -27,6 +27,7 @@ #include <asm/pgtable-hwdef.h> #include <asm/pgtable.h> #include <asm/page.h> +#include <asm/scs.h> #include <asm/smp.h> #include <asm/sysreg.h> #include <asm/thread_info.h>
@@ -424,6 +425,10 @@ __primary_switched: stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between
@@ -731,6 +736,10 @@ __secondary_switched: ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack + str xzr, [x2, #TSK_TI_SCS] +#endif mov x29, #0 mov x30, #0 b secondary_start_kernel
arch/arm64/kernel/irq.c +2 -0
@@ -21,6 +21,7 @@#include <linux/vmalloc.h> #include <asm/daifflags.h> #include <asm/vmap_stack.h> +#include <asm/scs.h> unsigned long irq_err_count;
@@ -63,6 +64,7 @@ static void init_irq_stacks(void)void __init init_IRQ (void ) {init_irq_stacks (); +scs_init_irq ();irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." );
arch/arm64/kernel/process.c +3 -0
@@ -52,6 +52,7 @@#include <asm/mmu_context.h> #include <asm/processor.h> #include <asm/pointer_auth.h> +#include <asm/scs.h> #include <asm/stacktrace.h> #if defined(CONFIG_STACKPROTECTOR ) && !defined(CONFIG_STACKPROTECTOR_PER_TASK )
@@ -508,6 +509,8 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,ptrauth_thread_switch (next);ssbs_thread_switch (next); +scs_thread_switch (prev, next); +/* * Complete any pending TLB or cache maintenance on this CPU in case * the thread migrates to a different CPU.
arch/arm64/kernel/scs.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/percpu.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); + +#ifndef CONFIG_SHADOW_CALL_STACK_VMAP +DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack) +__aligned(SCS_SIZE) ; +#endif + +void scs_init_irq (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) { +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP +unsigned long *p; + + p =__vmalloc_node_range (SCS_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +SCS_GFP ,PAGE_KERNEL , +0 ,cpu_to_node (cpu), +__builtin_return_address (0 )); + +per_cpu (irq_shadow_call_stack_ptr, cpu) = p; +#else +per_cpu (irq_shadow_call_stack_ptr, cpu) = +per_cpu (irq_shadow_call_stack, cpu); +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + } +}
arch/arm64/kernel/smp.c +4 -0
@@ -44,6 +44,7 @@#include <asm/pgtable.h> #include <asm/pgalloc.h> #include <asm/processor.h> +#include <asm/scs.h> #include <asm/smp_plat.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -357,6 +358,9 @@ void cpu_die(void) {unsigned int cpu =smp_processor_id (); +/* Save the shadow stack pointer before exiting the idle task */ +scs_save (current); +idle_task_exit ();local_daif_mask ();
Signature · 1 line
2.23.0.866.gb869b98d4c-goog
Re: [PATCH 18/18] arm64: implement Shadow Call Stack
Quoted 2 lines · Sami Tolvanen
This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
[...]
arch/arm64/include/asm/scs.h line 25
Quoted 13 lines
arch/arm64/include/asm/scs.h new file +13
-0+static inline void scs_save (struct task_struct *tsk) +{ +void *s; + + asmvolatile ("mov %0, x18" :"=r" (s)); +task_set_scs (tsk, s); +} + +static inline void scs_load (struct task_struct *tsk) +{ + asmvolatile ("mov x18, %0" : :"r" (task_scs (tsk))); +task_set_scs (tsk,NULL ); +}
These things should probably be __always_inline or something like that? If the compiler decides not to inline them (e.g. when called from scs_thread_switch()), stuff will blow up, right?
Quoted 9 lines
arch/arm64/include/asm/scs.h new file +9
-0+static inline void scs_thread_switch (struct task_struct *prev, +struct task_struct *next) +{ +scs_save (prev); +scs_load (next); + +if (unlikely (scs_corrupted (prev))) +panic ("corrupted shadow stack detected inside scheduler\n" ); +}
[...]
arch/arm64/include/asm/stacktrace.h extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); · line 73
Quoted 3 lines
arch/arm64/include/asm/stacktrace.h +3
-0+#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif
If an attacker has a leak of some random function pointer to find the ASLR base address, they'll know where irq_shadow_call_stack_ptr is. With an arbitrary read (to read irq_shadow_call_stack_ptr[sched_getcpu()]) followed by an arbitrary write, they'd be able to overwrite the shadow stack. Or with just an arbitrary write without a read, they could change irq_shadow_call_stack_ptr[sched_getcpu()] to point elsewhere. This is different from the intended protection level according to
<https://clang.llvm.org/docs/ShadowCallStack.html#security>, which talks about "a runtime that avoids exposing the address of the shadow call stack to attackers that can read arbitrary memory". Of course, that's extremely hard to implement in the context of the kernel, where you can see all the memory management data structures and all physical memory.
You might want to write something in the cover letter about what the benefits of this mechanism compared to STACKPROTECTOR are in the context of the kernel, including a specific description of which types of attacker capabilities this is supposed to defend against.
Re: [PATCH 18/18] arm64: implement Shadow Call Stack
Quoted 3 lines · Jann Horn
These things should probably be
__always_inlineor something like that? If the compiler decides not to inline them (e.g. when called fromscs_thread_switch()), stuff will blow up, right?
Correct. I'll change these to __always_inline in v2. I think there might be other places in the kernel where not inlining a static inline function would break things, but there's no need to add more.
Quoted 7 lines
This is different from the intended protection level according to
<https://clang.llvm.org/docs/ShadowCallStack.html#security>, which talks about "a runtime that avoids exposing the address of the shadow call stack to attackers that can read arbitrary memory". Of course, that's extremely hard to implement in the context of the kernel, where you can see all the memory management data structures and all physical memory.
Yes, the security guarantees in the kernel are different as hiding shadow stack pointers is more challenging.
Quoted 4 lines
You might want to write something in the cover letter about what the benefits of this mechanism compared to STACKPROTECTOR are in the context of the kernel, including a specific description of which types of attacker capabilities this is supposed to defend against.
Sure, I'll add something about that in v2. Thanks.
Sami
Mark Rutland
reviewerRe: [PATCH 18/18] arm64: implement Shadow Call Stack
In reply to Jann HornQuoted 13 lines · Jann Horn
Quoted 2 lines · Sami Tolvanen
This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
[...]
Quoted 9 lines
arch/arm64/include/asm/scs.h new file +9
-0+static inline void scs_save (struct task_struct *tsk) +{ +void *s; + + asmvolatile ("mov %0, x18" :"=r" (s)); +task_set_scs (tsk, s); +} + +static inline void scs_load (struct task_struct *tsk)
Quoted 8 lines
arch/arm64/include/asm/scs.h line 25
Quoted 4 lines
arch/arm64/include/asm/scs.h new file +4
-0+{ + asmvolatile ("mov x18, %0" : :"r" (task_scs (tsk))); +task_set_scs (tsk,NULL ); +}These things should probably be
__always_inlineor something like that? If the compiler decides not to inline them (e.g. when called fromscs_thread_switch()), stuff will blow up, right?
I think scs_save() would better live in assembly in cpu_switch_to(), where we switch the stack and current. It shouldn't matter whether scs_load() is inlined or not, since the x18 value _should_ be invariant from the PoV of the task.
We just need to add a TSK_TI_SCS to asm-offsets.c, and then insert a single LDR at the end:
mov sp, x9 msr sp_el0, x1 #ifdef CONFIG_SHADOW_CALL_STACK ldr x18, [x1, TSK_TI_SCS]
#endif
ret
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 18/18] arm64: implement Shadow Call Stack
Quoted 4 lines · Mark Rutland
I think
scs_save()would better live in assembly incpu_switch_to(), where we switch the stack and current. It shouldn't matter whetherscs_load()is inlined or not, since the x18 value _should_ be invariant from the PoV of the task.
Note that there's also a call to scs_save in cpu_die, because the current task's shadow stack pointer is only stored in x18 and we don't want to lose it.
Quoted 9 lines
We just need to add a
TSK_TI_SCStoasm-offsets.c, and then insert a single LDR at the end:mov sp, x9
msrsp_el0, x1#ifdef CONFIG_SHADOW_CALL_STACK ldr x18, [x1, TSK_TI_SCS]#endif
ret
TSK_TI_SCS is already defined, so yes, we could move this to cpu_switch_to. I would still prefer to have the overflow check that's in scs_thread_switch though.
Sami
Mark Rutland
reviewerRe: [PATCH 18/18] arm64: implement Shadow Call Stack
Quoted 20 lines · Sami Tolvanen
Quoted 4 lines · Mark Rutland
I think
scs_save()would better live in assembly incpu_switch_to(), where we switch the stack and current. It shouldn't matter whetherscs_load()is inlined or not, since the x18 value _should_ be invariant from the PoV of the task.Note that there's also a call to
scs_saveincpu_die, because the current task's shadow stack pointer is only stored in x18 and we don't want to lose it.Quoted 9 lines
We just need to add a
TSK_TI_SCStoasm-offsets.c, and then insert a single LDR at the end:mov sp, x9
msrsp_el0, x1#ifdef CONFIG_SHADOW_CALL_STACK ldr x18, [x1, TSK_TI_SCS]#endif
ret
Quoted 3 lines
TSK_TI_SCSis already defined, so yes, we could move this tocpu_switch_to. I would still prefer to have the overflow check that's inscs_thread_switchthough.
The only bit that I think needs to be in cpu_switch_to() is the install of the next task's shadow addr into x18.
Having a separate scs_check_overflow() sounds fine to me, as that only has to read from the shadow stack. IIUC that's also for the prev task, not next, in the current patches.
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Masami Hiramatsu
maintainerHi,
Quoted 5 lines · On Fri, 18 Oct 2019 09:10:15 -0700 Sami Tolvanen <[email protected]>…
This patch series adds support for Clang's Shadow Call Stack (SCS) mitigation, which uses a separately allocated shadow stack to protect against return address overwrites. More information can be found here:
Looks interesting, and like what function-graph tracing does...
Quoted 7 lines
SCS is currently supported only on arm64, where the compiler requires the x18 register to be reserved for holding the current task's shadow stack pointer. Because of this, the series includes four patches from Ard to remove x18 usage from assembly code and to reserve the register from general allocation.
Quoted 7 lines
With -fsanitize=shadow-call-stack, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning. As a result, SCS is incompatible with features that rely on modifying function return addresses to alter control flow, such as function graph tracing and kretprobes. A copy of the return address is still kept in the kernel stack for compatibility with stack unwinding, for example.
Is it possible that kretprobes and function graph tracing modify the SCS directly instead of changing real stack in that case?
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
v2 cover add support for Clang's Shadow Call Stack
In reply to Sami TolvanenThis patch series adds support for Clang's Shadow Call Stack (SCS) mitigation, which uses a separately allocated shadow stack to protect against return address overwrites. More information can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
SCS provides better protection against traditional buffer overflows than CONFIG_STACKPROTECTOR_*, but it should be noted that SCS security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
SCS is currently supported only on arm64, where the compiler requires the x18 register to be reserved for holding the current task's shadow stack pointer. Because of this, the series includes patches from Ard to remove x18 usage from assembly code.
With -fsanitize=shadow-call-stack, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack, and unconditionally load it again before returning. As a result, SCS is currently incompatible with features that rely on modifying function return addresses to alter control flow, such as function graph tracing and kretprobes, although it may be possible to later change these feature to modify the shadow stack instead. A copy of the return address is still kept in the kernel stack for compatibility with stack unwinding, for example.
SCS has a minimal performance overhead, but allocating shadow stacks increases kernel memory usage. The feature is therefore mostly useful on hardware that lacks support for PAC instructions.
Changes in v2:
- Changed Ard's KVM patch to use x29 instead of x18 for the guest context, which makes restore_callee_saved_regs cleaner
- Updated help text (and commit messages) to point out differences in security properties compared to user space SCS
- Cleaned up config options: removed the ROP protection choice, replaced the CC_IS_CLANG dependency with an arch-specific cc-option test, and moved disabling of incompatible config options to an arch-specific Kconfig
- Added CC_FLAGS_SCS, which are filtered out where needed instead of using DISABLE_SCS
- Added a __has_feature guard around __noscs for older clang versions
- Changed the shadow stack overflow check for vmapped SCS to use SCS_SIZE to avoid surprises when changing configs
- Renamed SCS_GFP to GFP_SCS
- Dropped the patch to reserve x18 unconditionally, it's now only reserved with SCS
- Added a clarification why restoring x18 in the EFI RT wrapper is safe
- Added a missing change to arch/arm64/include/asm/suspend.h, and a comment to arch/arm64/mm/proc.S to remind that struct cpu_suspend_ctx must be kept in sync with the code
- Moved x18 loading/storing during a context switch to cpu_switch_to(), renamed scs_thread_switch() to scs_overflow_check(), and removed the now unused scs_load()
- Added compile-time initialization for init_shadow_call_stack and removed scs_set_init_magic()
Ard Biesheuvel (2):
arm64/lib: copy_page: avoid x18 register in assembler code
arm64: kernel: avoid x18 as an arbitrary temp register
Sami Tolvanen (15):
arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
arm64: kvm: stop treating register x18 as caller save add support for Clang's Shadow Call Stack (SCS)
scs: add accounting
scs: add support for stack usage debugging
kprobes: fix compilation without CONFIG_KRETPROBES
arm64: disable function graph tracing with SCS
arm64: disable kretprobes with SCS
arm64: reserve x18 from general allocation with SCS
arm64: preserve x18 when CPU is suspended
arm64: efi: restore x18 if it was corrupted
arm64: vdso: disable Shadow Call Stack
arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
arm64: disable SCS for hypervisor code
arm64: implement Shadow Call Stack
Makefile | 6 + arch/Kconfig | 33 +++++ arch/arm64/Kconfig | 9 +- arch/arm64/Makefile | 4 + arch/arm64/include/asm/scs.h | 45 ++++++ arch/arm64/include/asm/stacktrace.h | 4 + arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/include/asm/thread_info.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 + arch/arm64/kernel/cpu-reset.S | 4 +- arch/arm64/kernel/efi-rt-wrapper.S | 7 +- arch/arm64/kernel/entry.S | 28 ++++ arch/arm64/kernel/head.S | 9 ++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/probes/kprobes.c | 2 + arch/arm64/kernel/process.c | 2 + arch/arm64/kernel/scs.c | 39 +++++ arch/arm64/kernel/smp.c | 4 + arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kvm/hyp/Makefile | 3 + arch/arm64/kvm/hyp/entry.S | 41 +++-- arch/arm64/lib/copy_page.S | 38 ++--- arch/arm64/mm/proc.S | 72 +++++---- drivers/base/node.c | 6 + fs/proc/meminfo.c | 4 + include/linux/compiler-clang.h | 6 + include/linux/compiler_types.h | 4 + include/linux/mmzone.h | 3 + include/linux/scs.h | 78 ++++++++++ init/init_task.c | 8 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/kprobes.c | 38 ++--- kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 214 +++++++++++++++++++++++++++ mm/page_alloc.c | 6 + mm/vmstat.c | 3 + 39 files changed, 649 insertions(+), 97 deletions(-) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 1/17 arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
idmap_kpti_install_ng_mappings uses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
arch/arm64/mm/proc.S +32 -31
@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */
@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*
@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v2 01/17] arm64: mm: don't use x18 in idmap_kpti_install_ng_mappings
Quoted 6 lines · On Thu, Oct 24, 2019 at 03:51:16PM -0700, [email protected] wrote:
idmap_kpti_install_ng_mappingsuses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Nick Desaulniers <[email protected]>
AFAICT the new register assignment is sound, so FWIW:
I was going to suggest adding menmonics for the remamining raw register names, but after having a go locally I think it's cleaner as-is given the registers are used in different widths for multiple purposes.
Thanks,
Mark.
Quoted 116 lines
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)arch/arm64/mm/proc.S +32
-31@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endifSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 2/17 arm64/lib: copy_page: avoid x18 register in assembler code
In reply to [email protected]From: Ard Biesheuvel <[email protected]>
Register x18 will no longer be used as a caller save register in the future, so stop using it in the copy_page() code.
--- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/arm64/lib/copy_page.S +19 -19
@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1) alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #-240] ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #-224] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #-208] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #-192] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #-176] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #-160] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #-144] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80] - stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #-240] + stnp x6, x7, [x0, #-224] + stnp x8, x9, [x0, #-208] + stnp x10, x11, [x0, #-192] + stnp x12, x13, [x0, #-176] + stnp x14, x15, [x0, #-160] + stnp x16, x17, [x0, #-144] ret ENDPROC(copy_page)
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v2 02/17] arm64/lib: copy_page: avoid x18 register in assembler code
Quoted 27 lines · On Thu, Oct 24, 2019 at 03:51:17PM -0700, [email protected] wrote:
From: Ard Biesheuvel <[email protected]>
Register x18 will no longer be used as a caller save register in the future, so stop using it in the
copy_page()code.Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)arch/arm64/lib/copy_page.S +2
-2@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1)
arch/arm64/lib/copy_page.S alternative_else_nop_endif · line 49
Quoted 10 lines
arch/arm64/lib/copy_page.S +2
-2alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #-240] ldp x4, x5, [x1, #16]
For legibility, could we make the offset and bias explicit in the STNPs so that these line up? e.g.
stnp x4, x5, [x0, #16 - 256] ldp x4, x5, [x1, #16]
... that'd make it much easier to see by eye that this is sound, much as I trust my mental arithmetic. ;)
Quoted 31 lines
arch/arm64/lib/copy_page.S +7
-13- stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #-224] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #-208] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #-192] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #-176] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #-160] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #-144] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80]
arch/arm64/lib/copy_page.S alternative_else_nop_endif · line 75
Quoted 10 lines
arch/arm64/lib/copy_page.S +8
-2- stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #-240] + stnp x6, x7, [x0, #-224] + stnp x8, x9, [x0, #-208] + stnp x10, x11, [x0, #-192] + stnp x12, x13, [x0, #-176] + stnp x14, x15, [x0, #-160] + stnp x16, x17, [x0, #-144]
... likewise here:
stnp xt1, xt2, [x0, #off - 256]
I don't see a nicer way to write this sequence without using an additional register, so with those changes:
Thanks,
Mark.
Quoted 6 lines
arch/arm64/lib/copy_page.S +0
-0ret ENDPROC(copy_page)Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v2 02/17] arm64/lib: copy_page: avoid x18 register in assembler code
Quoted 29 lines · Mark Rutland
Quoted 15 lines
arch/arm64/lib/copy_page.S +2
-2@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1)arch/arm64/lib/copy_page.S alternative_else_nop_endif · line 49
Quoted 10 lines
arch/arm64/lib/copy_page.S +2
-2alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #-240] ldp x4, x5, [x1, #16]For legibility, could we make the offset and bias explicit in the STNPs so that these line up? e.g.
Quoted 5 lines
stnp x4, x5, [x0, #16 - 256] ldp x4, x5, [x1, #16]... that'd make it much easier to see by eye that this is sound, much as I trust my mental arithmetic. ;)
Sure, that makes sense. I'll change this in v3.
Sami
PATCH v2 3/17 arm64: kvm: stop treating register x18 as caller save
In reply to [email protected]In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to __guest_enter(). Instead, preserve its value and restore it upon return.
--- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
arch/arm64/kvm/hyp/entry.S +20 -21
@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28 + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1
@@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret
@@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
@@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 4/17 arm64: kernel: avoid x18 as an arbitrary temp register
In reply to [email protected]From: Ard Biesheuvel <[email protected]>
The code in __cpu_soft_restart() uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/cpu-reset.S +2 -2
@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsection
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v2 04/17] arm64: kernel: avoid x18 as an arbitrary temp register
Minor nit, but could we make the title a bit more specific (and more uniform across the series)? e.g.
arm64: kvm: avoid x18 in __cpu_soft_restart
That makes things a bit nicer when trawling through git logs as the scope of the patch is clearer.
Quoted 8 lines · On Thu, Oct 24, 2019 at 03:51:19PM -0700, [email protected] wrote:
From: Ard Biesheuvel <[email protected]>
The code in
__cpu_soft_restart()uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>
Either way:
Quoted 25 lines
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/arm64/kernel/cpu-reset.S +2
-2@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsectionSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 5/17 add support for Clang's Shadow Call Stack (SCS)
In reply to [email protected]This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
--- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 78 +++++++++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 155 +++++++++++++++++++++++++++++++++ 11 files changed, 303 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Makefile +6 -0
@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
arch/Kconfig +33 -0
@@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool help
include/linux/compiler-clang.h +6 -0
@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack"))) +#else +# define __noscs +#endif
include/linux/compiler_types.h +4 -0
@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif
include/linux/scs.h new file +78 -0
78 lines not on this page · raw message
init/init_task.c +8 -0
@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h>
@@ -184,6 +185,13 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data +__aligned(SCS_SIZE) = { + [(SCS_SIZE /sizeof (long )) -1 ] =SCS_END_MAGIC +}; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry.
kernel/Makefile +1 -0
@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/
kernel/fork.c +9 -0
@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info,
@@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }
@@ -907,6 +912,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)clear_user_return_notifier (tsk);clear_tsk_need_resched (tsk);set_task_stack_end_magic (tsk); +scs_task_init (tsk);#ifdef CONFIG_STACKPROTECTOR tsk->stack_canary =get_random_canary ();
@@ -2022,6 +2028,9 @@ static __latent_entropy struct task_struct *copy_process( args->tls);if (retval) goto bad_fork_cleanup_io; + retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;stackleak_task_init (p);
kernel/sched/core.c +2 -0
@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
kernel/sched/sched.h +1 -0
@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h>
kernel/scs.c new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Mark Rutland
reviewerRe: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 96 lines · On Thu, Oct 24, 2019 at 03:51:20PM -0700, [email protected] wrote:
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 78 +++++++++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 155 +++++++++++++++++++++++++++++++++ 11 files changed, 303 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +6
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)arch/Kconfig +33
-0@@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool helpinclude/linux/compiler-clang.h +0
-0
include/linux/compiler-clang.h line 50
Quoted 9 lines
include/linux/compiler-clang.h +6
-0@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack"))) +#else +# define __noscs +#endif
Huh. I didn't realise it was valid to have a space after the # like this. I see we're very inconsistent about style on that front, so this is fine, I'll just have to get used to it. :)
Quoted 28 lines
include/linux/compiler_types.h +4
-0@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif include/linux/scs.h new file +7
-0@@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Shadow Call Stack support. +* +* Copyright (C) 2018 Google LLC +*/ +
include/linux/scs.h line 17
Quoted 10 lines
include/linux/scs.h new file +10
-0+#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include <linux/gfp.h> +#include <linux/sched.h> +#include <asm/page.h> + +#ifdef CONFIG_SHADOW_CALL_STACK + +#define SCS_SIZE 1024
I think it'd be worth a comment on how this size was chosen. IIRC this empirical?
include/linux/scs.h line 18
Quoted 1 line
include/linux/scs.h new file +1
-0+#define SCS_END_MAGIC 0xaf0194819b1635f6UL
Keyboard smash? ... or is there a prize for whoever figures out the secret? ;)
include/linux/scs.h line 30
Quoted 12 lines
include/linux/scs.h new file +12
-0+ +#define GFP_SCS (GFP_KERNEL | __GFP_ZERO) + +static inline void *task_scs (struct task_struct *tsk) +{ +return task_thread_info (tsk)->shadow_call_stack; +} + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +task_thread_info (tsk)->shadow_call_stack = s; +}
This should probably be named get and set, or have:
#define task_scs(tsk) (task_thread_info(tsk)->shadow_call_stack)
... which can have a trivial implementation as NULL for the !SCS case.
Quoted 4 lines
include/linux/scs.h new file +4
-0+ +extern void scs_init (void ); +extern void scs_task_init (struct task_struct *tsk); +extern void scs_task_reset (struct task_struct *tsk);
include/linux/scs.h line 44
Quoted 10 lines
include/linux/scs.h new file +10
-0+extern int scs_prepare (struct task_struct *tsk,int node); +extern bool scs_corrupted (struct task_struct *tsk); +extern void scs_release (struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void *task_scs (struct task_struct *tsk) +{ +return 0 ; +}
For all the trivial wrappers you can put the implementation on the same line as the prototype. That makes it a bit easier to compare against the prototypes on the other side of the ifdeffery.
e.g. this lot can be:
static inline void *task_scs(struct task_struct *tsk) { return 0; } static inline void task_set_scs(struct task_struct *tsk, void *s) { } static inline void scs_init(void) { }
...
Quoted 46 lines
include/linux/scs.h new file +3
-0+#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ init/init_task.c +8
-0@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h> @@ -184,6 +185,13 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data +__aligned(SCS_SIZE) = { + [(SCS_SIZE /sizeof (long )) -1 ] =SCS_END_MAGIC +}; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry. kernel/Makefile +1
-0@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/fork.c +0
-0
kernel/fork.c line 97
Quoted 4 lines
kernel/fork.c +1
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h>
Nit: alphabetical order, please (this should come before stackleak.h).
Quoted 30 lines
kernel/fork.c +5
-0#include <asm/pgtable.h> #include <asm/pgalloc.h> @@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }@@ -907,6 +912,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)clear_user_return_notifier (tsk);clear_tsk_need_resched (tsk);set_task_stack_end_magic (tsk); +scs_task_init (tsk);#ifdef CONFIG_STACKPROTECTOR tsk->stack_canary =get_random_canary ();
kernel/fork.c static __latent_entropy struct task_struct *copy_process( · line 2033
Quoted 6 lines
kernel/fork.c +3
-0@@ -2022,6 +2028,9 @@ static __latent_entropy struct task_struct *copy_process( args->tls);if (retval) goto bad_fork_cleanup_io; + retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;
Can we please fold scs_prepare() into scs_task_init() and do this in dup_task_struct()? That way we set this up consistently in one place, where we're also allocating the regular stack.
Arguably stackleak_task_init() would better fit there too.
kernel/sched/core.c void init_idle(struct task_struct *idle, int cpu) · line 6016
Quoted 12 lines
kernel/fork.c +0
-0stackleak_task_init (p);kernel/sched/core.c +1
-0@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle);
I'm a bit confused by this -- please see comments below on scs_task_reset().
Quoted 23 lines
kernel/sched/core.c +1
-0+__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();kernel/sched/sched.h +1
-0@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h> kernel/scs.c new file +1
-0@@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0
kernel/scs.c line 11
Quoted 10 lines
kernel/scs.c new file +10
-0+/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/cpuhotplug.h> +#include <linux/mm.h> +#include <linux/slab.h> +#include <linux/scs.h>
Nit: alphabetical order, please.
kernel/scs.c line 18
Quoted 7 lines
kernel/scs.c new file +7
-0+#include <linux/vmalloc.h> +#include <asm/scs.h> + +static inline void *__scs_base (struct task_struct *tsk) +{ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +}
We only ever assign the base to task_scs(tsk), with the current live value being in a register that we don't read. Are we expecting arch code to keep this up-to-date with the register value?
I would have expected that we just leave this as the base (as we do for the regular stack in the task struct), and it's down to arch code to save/restore the current value where necessary.
Am I missing some caveat with that approach?
Quoted 12 lines
kernel/scs.c new file +12
-0+ +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + +/* Keep a cache of shadow stacks */ +#define SCS_CACHE_SIZE 2 +static DEFINE_PER_CPU(void *, scs_cache[SCS_CACHE_SIZE]); + +static void *scs_alloc (int node) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) {
kernel/scs.c line 40
Quoted 10 lines
kernel/scs.c new file +10
-0+void *s; + + s =this_cpu_xchg (scs_cache[i],NULL ); +if (s) { +memset (s,0 ,SCS_SIZE ); +return s; + } + } + +BUILD_BUG_ON (SCS_SIZE >PAGE_SIZE );
It's probably worth a comment on why we rely on SCS_SIZE <= PAGE_SIZE.
Quoted 56 lines
kernel/scs.c new file +56
-0+ +return __vmalloc_node_range (PAGE_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +GFP_SCS ,PAGE_KERNEL ,0 , + node,__builtin_return_address (0 )); +} + +static void scs_free (void *s) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +if (this_cpu_cmpxchg (scs_cache[i],0 , s) !=0 ) +continue ; + +return ; + } + +vfree_atomic (s); +} + +static int scs_cleanup (unsigned int cpu) +{ +int i; +void **cache =per_cpu_ptr (scs_cache, cpu); + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +vfree (cache[i]); + cache[i] =NULL ; + } + +return 0 ; +} + +void __init scs_init (void ) +{ +cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , + scs_cleanup); +} + +#else /* !CONFIG_SHADOW_CALL_STACK_VMAP */ + +static struct kmem_cache *scs_cache; + +static inline void *scs_alloc (int node) +{ +return kmem_cache_alloc_node (scs_cache,GFP_SCS , node); +} + +static inline void scs_free (void *s) +{ +kmem_cache_free (scs_cache, s); +} + +void __init scs_init (void ) +{
kernel/scs.c line 106
Quoted 10 lines
kernel/scs.c new file +10
-0+ scs_cache =kmem_cache_create ("scs_cache" ,SCS_SIZE ,SCS_SIZE , +0 ,NULL ); + WARN_ON(!scs_cache); +} + +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + +static inline unsigned long *scs_magic (struct task_struct *tsk) +{ +return (unsigned long *)(__scs_base (tsk) +SCS_SIZE -sizeof (long ));
Slightly simpler as:
return (unsigned long *)(__scs_base(tsk) + SCS_SIZE) - 1;
Thanks,
Mark.
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 4 lines · Mark Rutland
Quoted 1 line
+#define
SCS_SIZE1024I think it'd be worth a comment on how this size was chosen. IIRC this empirical?
Correct. I'll add a comment.
Quoted 4 lines
Quoted 1 line
+#define
SCS_END_MAGIC0xaf0194819b1635f6ULKeyboard smash? ... or is there a prize for whoever figures out the secret? ;)
It's a random number, so if someone figures out a secret in it, they'll definitely deserve a prize. :)
Quoted 10 lines
include/linux/scs.h line 30
Quoted 4 lines
include/linux/scs.h new file +4
-0+static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +task_thread_info (tsk)->shadow_call_stack = s; +}This should probably be named get and set, or have:
#define
task_scs(tsk)(task_thread_info(tsk)->shadow_call_stack)... which can have a trivial implementation as NULL for the !SCS case.
Sure, sounds good.
Quoted 9 lines
For all the trivial wrappers you can put the implementation on the same line as the prototype. That makes it a bit easier to compare against the prototypes on the other side of the ifdeffery.
e.g. this lot can be:
static inline void *task_scs(struct
task_struct*tsk) { return 0; } static inline voidtask_set_scs(struct task_struct *tsk, void *s){ } static inline voidscs_init(void){ }
Agreed.
Quoted 11 lines
kernel/fork.c line 97
Quoted 9 lines
kernel/fork.c +1
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> Nit: alphabetical order, please (this should come before
stackleak.h).
The includes in kernel/fork.c aren't in alphabetical order, so I just added this to the end here.
Quoted 7 lines
kernel/fork.c static __latent_entropy struct task_struct *copy_process( · line 2033
Quoted 3 lines
kernel/fork.c +3
-0+ retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;Can we please fold
scs_prepare()intoscs_task_init()and do this indup_task_struct()? That way we set this up consistently in one place, where we're also allocating the regular stack.
Yes, that does sound cleaner. I'll change that.
Quoted 14 lines
Quoted 1 line
+
scs_task_reset(idle);I'm a bit confused by this -- please see comments below on
scs_task_reset().kernel/fork.c
Quoted 4 lines
kernel/fork.c +4
-0+static inline void *__scs_base (struct task_struct *tsk) +{ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +}We only ever assign the base to
task_scs(tsk), with the current live value being in a register that we don't read. Are we expecting arch code to keep this up-to-date with the register value?
Quoted 5 lines
I would have expected that we just leave this as the base (as we do for the regular stack in the task struct), and it's down to arch code to save/restore the current value where necessary.
Am I missing some caveat with that approach?
To keep the address of the currently active shadow stack out of memory, the arm64 implementation clears this field when it loads x18 and saves the current value before a context switch. The generic code doesn't expect the arch code to necessarily do so, but does allow it. This requires us to use __scs_base() when accessing the base pointer and to reset it in idle tasks before they're reused, hence scs_task_reset().
Quoted 3 lines
Quoted 1 line
+
BUILD_BUG_ON(SCS_SIZE > PAGE_SIZE);It's probably worth a comment on why we rely on
SCS_SIZE<=PAGE_SIZE.
Ack.
Quoted 7 lines
kernel/fork.c
Quoted 3 lines
kernel/fork.c +3
-0+static inline unsigned long *scs_magic (struct task_struct *tsk) +{ +return (unsigned long *)(__scs_base (tsk) +SCS_SIZE -sizeof (long ));Slightly simpler as:
return (unsigned long *)(__scs_base(tsk) + SCS_SIZE) - 1;
Yes, that's a bit cleaner.
I'll fix these in v3. Thank you for the review!
Sami
Mark Rutland
reviewerRe: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 8 lines · Sami Tolvanen
Quoted 4 lines · Mark Rutland
Quoted 1 line
+#define
SCS_END_MAGIC0xaf0194819b1635f6ULKeyboard smash? ... or is there a prize for whoever figures out the secret? ;)
It's a random number, so if someone figures out a secret in it, they'll definitely deserve a prize. :)
I'll Cc some treasure hunters. :)
Quoted 12 lines
Quoted 11 lines
kernel/fork.c line 97
Quoted 9 lines
kernel/fork.c +1
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> Nit: alphabetical order, please (this should come before
stackleak.h).
Quoted 2 lines
The includes in
kernel/fork.caren't in alphabetical order, so I just added this to the end here.
Fair enough. It looked otherwise in the context, and we generally aim for that as a soft rule.
[...]
Quoted 15 lines
Quoted 9 lines
kernel/fork.c
Quoted 4 lines
kernel/fork.c +4
-0+static inline void *__scs_base (struct task_struct *tsk) +{ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +}We only ever assign the base to
task_scs(tsk), with the current live value being in a register that we don't read. Are we expecting arch code to keep this up-to-date with the register value?Quoted 5 lines
I would have expected that we just leave this as the base (as we do for the regular stack in the task struct), and it's down to arch code to save/restore the current value where necessary.
Am I missing some caveat with that approach?
Quoted 7 lines
To keep the address of the currently active shadow stack out of memory, the arm64 implementation clears this field when it loads x18 and saves the current value before a context switch. The generic code doesn't expect the arch code to necessarily do so, but does allow it. This requires us to use
__scs_base()when accessing the base pointer and to reset it in idle tasks before they're reused, hencescs_task_reset().
Ok. That'd be worth a comment somewhere, since it adds a number of things which would otherwise be unnecessary.
IIUC this assumes an adversary who knows the address of a task's thread_info, and has an arbitrary-read (to extract the SCS base from thead_info) and an arbitrary-write (to modify the SCS area).
Assuming that's the case, I don't think this buys much. If said adversary controls two userspace threads A and B, they only need to wait until A is context-switched out or in userspace, and read A's SCS base using B.
Given that, I'd rather always store the SCS base in the thread_info, and simplify the rest of the code manipulating it.
Thanks,
Mark.
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 16 lines · Mark Rutland
Quoted 7 lines · Sami Tolvanen
To keep the address of the currently active shadow stack out of memory, the arm64 implementation clears this field when it loads x18 and saves the current value before a context switch. The generic code doesn't expect the arch code to necessarily do so, but does allow it. This requires us to use
__scs_base()when accessing the base pointer and to reset it in idle tasks before they're reused, hencescs_task_reset().Ok. That'd be worth a comment somewhere, since it adds a number of things which would otherwise be unnecessary.
IIUC this assumes an adversary who knows the address of a task's
thread_info, and has an arbitrary-read (to extract the SCS base fromthead_info) and an arbitrary-write (to modify the SCS area).
Quoted 7 lines
Assuming that's the case, I don't think this buys much. If said adversary controls two userspace threads A and B, they only need to wait until A is context-switched out or in userspace, and read A's SCS base using B.
Given that, I'd rather always store the SCS base in the
thread_info, and simplify the rest of the code manipulating it.
I'd like to keep this as-is since it provides a temporal protection. Having arbitrary kernel read and write at arbitrary time is a very powerful attack primitive, and is, IMO, not very common. Many attacks tend to be chains of bugs that give attackers narrow visibility in to the kernel at specific moments. I would say this design is more about stopping "current" from dumping thread_info (as there are many more opportunities for current to see its own thread_info compared to arbitrary addresses or another task's thread_info). As such, I think it's a reasonable precaution to take.
Signature · 1 line
Kees Cook
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 25 lines · Kees Cook
Quoted 16 lines · Mark Rutland
Quoted 7 lines · Sami Tolvanen
To keep the address of the currently active shadow stack out of memory, the arm64 implementation clears this field when it loads x18 and saves the current value before a context switch. The generic code doesn't expect the arch code to necessarily do so, but does allow it. This requires us to use
__scs_base()when accessing the base pointer and to reset it in idle tasks before they're reused, hencescs_task_reset().Ok. That'd be worth a comment somewhere, since it adds a number of things which would otherwise be unnecessary.
IIUC this assumes an adversary who knows the address of a task's
thread_info, and has an arbitrary-read (to extract the SCS base fromthead_info) and an arbitrary-write (to modify the SCS area).Quoted 7 lines
Assuming that's the case, I don't think this buys much. If said adversary controls two userspace threads A and B, they only need to wait until A is context-switched out or in userspace, and read A's SCS base using B.
Given that, I'd rather always store the SCS base in the
thread_info, and simplify the rest of the code manipulating it.
Quoted 9 lines
I'd like to keep this as-is since it provides a temporal protection. Having arbitrary kernel read and write at arbitrary time is a very powerful attack primitive, and is, IMO, not very common. Many attacks tend to be chains of bugs that give attackers narrow visibility in to the kernel at specific moments. I would say this design is more about stopping "current" from dumping
thread_info(as there are many more opportunities for current to see its ownthread_infocompared to arbitrary addresses or another task'sthread_info). As such, I think it's a reasonable precaution to take.
I'm not sure if always storing the base address in thread_info would simplify the code that much. We could remove __scs_base() and scs_task_reset(), which are both trivial, and drop a few instructions in the arch-specific code that clear the field. I do agree that a comment or two would help understand what's going on here though.
Sami
Nick Desaulniers
reviewerRe: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
In reply to [email protected]Quoted 369 lines · On Thu, Oct 24, 2019 at 3:51 PM <[email protected]> wrote:
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 78 +++++++++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 155 +++++++++++++++++++++++++++++++++ 11 files changed, 303 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +6
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)arch/Kconfig +33
-0@@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool helpinclude/linux/compiler-clang.h +6
-0@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack"))) +#else +# define __noscs +#endif include/linux/compiler_types.h +4
-0@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif include/linux/scs.h new file +78
-0@@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Shadow Call Stack support. +* +* Copyright (C) 2018 Google LLC +*/ + +#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include <linux/gfp.h> +#include <linux/sched.h> +#include <asm/page.h> + +#ifdef CONFIG_SHADOW_CALL_STACK + +#define SCS_SIZE 1024 +#define SCS_END_MAGIC 0xaf0194819b1635f6UL + +#define GFP_SCS (GFP_KERNEL | __GFP_ZERO) + +static inline void *task_scs (struct task_struct *tsk) +{ +return task_thread_info (tsk)->shadow_call_stack; +} + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +task_thread_info (tsk)->shadow_call_stack = s; +} + +extern void scs_init (void ); +extern void scs_task_init (struct task_struct *tsk); +extern void scs_task_reset (struct task_struct *tsk); +extern int scs_prepare (struct task_struct *tsk,int node); +extern bool scs_corrupted (struct task_struct *tsk); +extern void scs_release (struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void *task_scs (struct task_struct *tsk) +{ +return 0 ; +} + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +} + +static inline void scs_init (void ) +{ +} + +static inline void scs_task_init (struct task_struct *tsk) +{ +} + +static inline void scs_task_reset (struct task_struct *tsk) +{ +} + +static inline int scs_prepare (struct task_struct *tsk,int node) +{ +return 0 ; +} + +static inline bool scs_corrupted (struct task_struct *tsk) +{ +return false; +} + +static inline void scs_release (struct task_struct *tsk) +{ +} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ init/init_task.c +8
-0@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h> @@ -184,6 +185,13 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data +__aligned(SCS_SIZE) = { + [(SCS_SIZE /sizeof (long )) -1 ] =SCS_END_MAGIC +}; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry. kernel/Makefile +1
-0@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/fork.c +9
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h> @@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }@@ -907,6 +912,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)clear_user_return_notifier (tsk);clear_tsk_need_resched (tsk);set_task_stack_end_magic (tsk); +scs_task_init (tsk);#ifdef CONFIG_STACKPROTECTOR tsk->stack_canary =get_random_canary ();@@ -2022,6 +2028,9 @@ static __latent_entropy struct task_struct *copy_process( args->tls);if (retval) goto bad_fork_cleanup_io; + retval =scs_prepare (p, node); +if (retval) + goto bad_fork_cleanup_thread;stackleak_task_init (p);kernel/sched/core.c +2
-0@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();kernel/sched/sched.h +1
-0@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h> kernel/scs.c new file +47
-0@@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/cpuhotplug.h> +#include <linux/mm.h> +#include <linux/slab.h> +#include <linux/scs.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +static inline void *__scs_base (struct task_struct *tsk) +{ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +} + +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + +/* Keep a cache of shadow stacks */ +#define SCS_CACHE_SIZE 2 +static DEFINE_PER_CPU(void *, scs_cache[SCS_CACHE_SIZE]); + +static void *scs_alloc (int node) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +void *s; + + s =this_cpu_xchg (scs_cache[i],NULL ); +if (s) { +memset (s,0 ,SCS_SIZE ); +return s; + } + } + +BUILD_BUG_ON (SCS_SIZE >PAGE_SIZE ); + +return __vmalloc_node_range (PAGE_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +GFP_SCS ,PAGE_KERNEL ,0 , + node,__builtin_return_address (0 )); +} +
kernel/scs.c line 57
Quoted 10 lines
kernel/scs.c new file +10
-0+static void scs_free (void *s) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +if (this_cpu_cmpxchg (scs_cache[i],0 , s) !=0 ) +continue ; + +return ; + }
prefer:
for ...:
if foo() == 0:
return
to:
for ...:
if foo() != 0:
continue
return
Quoted 101 lines
kernel/scs.c new file +98
-1+ +vfree_atomic (s); +} + +static int scs_cleanup (unsigned int cpu) +{ +int i; +void **cache =per_cpu_ptr (scs_cache, cpu); + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +vfree (cache[i]); + cache[i] =NULL ; + } + +return 0 ; +} + +void __init scs_init (void ) +{ +cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , + scs_cleanup); +} + +#else /* !CONFIG_SHADOW_CALL_STACK_VMAP */ + +static struct kmem_cache *scs_cache; + +static inline void *scs_alloc (int node) +{ +return kmem_cache_alloc_node (scs_cache,GFP_SCS , node); +} + +static inline void scs_free (void *s) +{ +kmem_cache_free (scs_cache, s); +} + +void __init scs_init (void ) +{ + scs_cache =kmem_cache_create ("scs_cache" ,SCS_SIZE ,SCS_SIZE , +0 ,NULL ); +WARN_ON (!scs_cache); +} + +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + +static inline unsigned long *scs_magic (struct task_struct *tsk) +{ +return (unsigned long *)(__scs_base (tsk) +SCS_SIZE -sizeof (long )); +} + +static inline void scs_set_magic (struct task_struct *tsk) +{ + *scs_magic (tsk) =SCS_END_MAGIC ; +} + +void scs_task_init (struct task_struct *tsk) +{ +task_set_scs (tsk,NULL ); +} + +void scs_task_reset (struct task_struct *tsk) +{ +task_set_scs (tsk,__scs_base (tsk)); +} + +int scs_prepare (struct task_struct *tsk,int node) +{ +void *s; + + s =scs_alloc (node); +if (!s) +return -ENOMEM ; + +task_set_scs (tsk, s); +scs_set_magic (tsk); + +return 0 ; +} + +bool scs_corrupted (struct task_struct *tsk) +{ +return *scs_magic (tsk) !=SCS_END_MAGIC ; +} + +void scs_release (struct task_struct *tsk) +{ +void *s; + + s =__scs_base (tsk); +if (!s) +return ; + +WARN_ON (scs_corrupted (tsk)); + +scs_task_init (tsk); +scs_free (s); +} --2.24.0.rc0.303.g954a862665-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 17 lines · Nick Desaulniers
kernel/scs.c line 57
Quoted 10 lines
kernel/scs.c new file +10
-0+static void scs_free (void *s) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +if (this_cpu_cmpxchg (scs_cache[i],0 , s) !=0 ) +continue ; + +return ; + }prefer:
for ...:
iffoo()== 0:
return
Quoted 6 lines
to:
for ...:
iffoo()!= 0:
continue
return
This was essentially copied from free_thread_stack in kernel/fork.c, but I agree, your way is cleaner. I'll change this in the next version. Thanks!
Sami
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
In reply to [email protected]Quoted 3 lines · On Thu, 2019-10-24 at 15:51 -0700, [email protected] wrote:
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
[]
include/linux/compiler-clang.h
Quoted 1 line
include/linux/compiler-clang.h +0
-0
[]
include/linux/compiler-clang.h line 47
Quoted 7 lines
include/linux/compiler-clang.h +3
-0@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack")))
__no_sanitize__
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Hi Joe,
Quoted 4 lines · Joe Perches
include/linux/compiler-clang.h line 47
Quoted 2 lines
include/linux/compiler-clang.h +2
-0+#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack")))
__no_sanitize__
Sorry, I missed your earlier message about this. I'm following Clang's documentation for the attribute:
https://clang.llvm.org/docs/ShadowCallStack.html#attribute-no-sanitize-shadow-call-stack
Although __no_sanitize__ seems to work too. Is there a particular reason to prefer that form over the one in the documentation?
Sami
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Hi Sami,
Quoted 8 lines · On Mon, 28 Oct 2019 at 16:20 Sami Tolvanen <[email protected]> wrote:
Hi Joe,
Quoted 4 lines · Joe Perches
include/linux/compiler-clang.h line 47
Quoted 2 lines
include/linux/compiler-clang.h +2
-0+#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((no_sanitize("shadow-call-stack")))
__no_sanitize__
Quoted 8 lines
Sorry, I missed your earlier message about this. I'm following Clang's documentation for the attribute:
https://clang.llvm.org/docs/ShadowCallStack.html#attribute-no-sanitize-shadow-call-stack
Although
__no_sanitize__seems to work too. Is there a particular reason to prefer that form over the one in the documentation?
We decided to do it like that when I introduced compiler_attributes.h.
Given it is hidden behind a definition, we don't care about which one we use internally; therefore the idea was to avoid clashes as much as possible with other names/definitions/etc.
The syntax is supported in the compilers we care about (for docs on attributes, the best reference is GCC's by the way).
Cheers,
Miguel
Signature · 2 lines
Cheers, Miguel
Re: [PATCH v2 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 5 lines · Miguel Ojeda
We decided to do it like that when I introduced
compiler_attributes.h.Given it is hidden behind a definition, we don't care about which one we use internally; therefore the idea was to avoid clashes as much as possible with other names/definitions/etc.
The syntax is supported in the compilers we care about (for docs on attributes, the best reference is GCC's by the way).
Got it, thank you for explaining. I'll change this to __no_sanitize__ in v3 since Clang seems to be happy with either version.
Sami
PATCH v2 6/17 scs: add accounting
In reply to [email protected]This change adds accounting for the memory allocated for shadow stacks.
--- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 20 ++++++++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 42 insertions(+)
drivers/base/node.c +6 -0
@@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif"Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n"
@@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_ANON_MAPPED )), nid,K (i.sharedram), nid,sum_zone_node_page_state (nid,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK + nid,sum_zone_node_page_state (nid,NR_KERNEL_SCS_BYTES ) /1024 , +#endif nid,K (sum_zone_node_page_state (nid,NR_PAGETABLE )), nid,K (node_page_state (pgdat,NR_UNSTABLE_NFS )), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
fs/proc/meminfo.c +4 -0
@@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)show_val_kb (m,"SUnreclaim: " , sunreclaim);seq_printf (m,"KernelStack: %8lu kB\n" ,global_zone_page_state (NR_KERNEL_STACK_KB )); +#ifdef CONFIG_SHADOW_CALL_STACK +seq_printf (m,"ShadowCallStack:%8lu kB\n" , +global_zone_page_state (NR_KERNEL_SCS_BYTES ) /1024 ); +#endif show_val_kb (m,"PageTables: " ,global_zone_page_state (NR_PAGETABLE ));
include/linux/mmzone.h +3 -0
@@ -200,6 +200,9 @@ enum zone_stat_item {NR_MLOCK ,/* mlock()ed pages found and moved off LRU */ NR_PAGETABLE ,/* used for pagetables */ NR_KERNEL_STACK_KB ,/* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +NR_KERNEL_SCS_BYTES ,/* measured in bytes */ +#endif/* Second 128 byte cacheline */ NR_BOUNCE , #if IS_ENABLED(CONFIG_ZSMALLOC)
kernel/scs.c +20 -0
@@ -7,9 +7,11 @@#include <linux/cpuhotplug.h> #include <linux/mm.h> +#include <linux/mmzone.h> #include <linux/slab.h> #include <linux/scs.h> #include <linux/vmalloc.h> +#include <linux/vmstat.h> #include <asm/scs.h> static inline void *__scs_base (struct task_struct *tsk)
@@ -59,6 +61,11 @@ static void scs_free(void *s)vfree_atomic (s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return vmalloc_to_page (__scs_base (tsk)); +} +static int scs_cleanup (unsigned int cpu) {int i;
@@ -92,6 +99,11 @@ static inline void scs_free(void *s)kmem_cache_free (scs_cache, s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return virt_to_page (__scs_base (tsk)); +} +void __init scs_init (void ) { scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE,
@@ -121,6 +133,12 @@ void scs_task_reset(struct task_struct *tsk)task_set_scs (tsk,__scs_base (tsk)); } +static void scs_account (struct task_struct *tsk,int account) +{ +mod_zone_page_state (page_zone (__scs_page (tsk)),NR_KERNEL_SCS_BYTES , + account *SCS_SIZE ); +} +int scs_prepare (struct task_struct *tsk,int node) {void *s;
@@ -131,6 +149,7 @@ int scs_prepare(struct task_struct *tsk, int node)task_set_scs (tsk, s);scs_set_magic (tsk); +scs_account (tsk,1 );return 0 ; }
@@ -150,6 +169,7 @@ void scs_release(struct task_struct *tsk)WARN_ON (scs_corrupted (tsk)); +scs_account (tsk,-1 );scs_task_init (tsk);scs_free (s); }
mm/page_alloc.c +6 -0
@@ -5361,6 +5361,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif" pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB"
@@ -5382,6 +5385,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)K (zone_managed_pages (zone)),K (zone_page_state (zone,NR_MLOCK )),zone_page_state (zone,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK +zone_page_state (zone,NR_KERNEL_SCS_BYTES ) /1024 , +#endifK (zone_page_state (zone,NR_PAGETABLE )),K (zone_page_state (zone,NR_BOUNCE )), K(free_pcp),
mm/vmstat.c +3 -0
@@ -1118,6 +1118,9 @@ const char * const vmstat_text[] = {"nr_mlock" ,"nr_page_table_pages" ,"nr_kernel_stack" , +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +"nr_shadow_call_stack_bytes" , +#endif"nr_bounce" , #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages",
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 7/17 scs: add support for stack usage debugging
In reply to [email protected]Implements CONFIG_DEBUG_STACK_USAGE for shadow stacks.
--- kernel/scs.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
kernel/scs.c +39 -0
@@ -154,6 +154,44 @@ int scs_prepare(struct task_struct *tsk, int node)return 0 ; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long scs_used (struct task_struct *tsk) +{ +unsigned long *p =__scs_base (tsk); +unsigned long *end =scs_magic (tsk); +uintptr_t s = (uintptr_t )p; + +while (p < end && *p) + p++; + +return (uintptr_t )p - s; +} + +static void scs_check_usage (struct task_struct *tsk) +{ +static DEFINE_SPINLOCK (lock); +static unsigned long highest; +unsigned long used =scs_used (tsk); + +if (used <= highest) +return ; + +spin_lock (&lock); + +if (used > highest) { +pr_info ("%s: highest shadow stack usage %lu bytes\n" , + __func__, used); + highest = used; + } + +spin_unlock (&lock); +} +#else +static inline void scs_check_usage (struct task_struct *tsk) +{ +} +#endif +bool scs_corrupted (struct task_struct *tsk) {return *scs_magic (tsk) !=SCS_END_MAGIC ;
@@ -168,6 +206,7 @@ void scs_release(struct task_struct *tsk)return ;WARN_ON (scs_corrupted (tsk)); +scs_check_usage (tsk);scs_account (tsk,-1 );scs_task_init (tsk);
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 8/17 kprobes: fix compilation without CONFIG_KRETPROBES
In reply to [email protected]kprobe_on_func_entry and arch_kprobe_on_func_entry need to be available even if CONFIG_KRETPROBES is not selected.
--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
kernel/kprobes.c +19 -19
@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe
@@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 9/17 arm64: disable function graph tracing with SCS
In reply to [email protected]With CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified in ftrace_graph_caller and prepare_ftrace_return to redirect control flow to ftrace_return_to_handler. This is incompatible with SCS.
--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/Kconfig +1 -1
@@ -148,7 +148,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if !SHADOW_CALL_STACK select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Mark Rutland
reviewerRe: [PATCH v2 09/17] arm64: disable function graph tracing with SCS
Quoted 4 lines · On Thu, Oct 24, 2019 at 03:51:24PM -0700, [email protected] wrote:
With
CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified inftrace_graph_callerandprepare_ftrace_returnto redirect control flow toftrace_return_to_handler. This is incompatible with SCS.
I'm guessing it's difficult to always figure out the SCS slot for an instrumented callsite unless we pass this explicitly from the ftrace entry code, so we'd probably have to change some common infrastructure for that.
We have a similar issue with pointer authentication, and we're solving that with -fpatchable-function-entry, which allows us to hook the callsite before it does anything with the return address. IIUC we could use the same mechanism here (and avoid introducing a third).
Are there plans to implement -fpatchable-function-entry on the clang side?
Thanks,
Mark.
Quoted 22 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/Kconfig +1
-1@@ -148,7 +148,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if !SHADOW_CALL_STACK select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTINGSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v2 09/17] arm64: disable function graph tracing with SCS
Quoted 5 lines · Mark Rutland
I'm guessing it's difficult to always figure out the SCS slot for an instrumented callsite unless we pass this explicitly from the ftrace entry code, so we'd probably have to change some common infrastructure for that.
Quoted 7 lines
We have a similar issue with pointer authentication, and we're solving that with -fpatchable-function-entry, which allows us to hook the callsite before it does anything with the return address. IIUC we could use the same mechanism here (and avoid introducing a third).
Are there plans to implement -fpatchable-function-entry on the clang side?
I'm not sure if there are plans at the moment, but if this feature is needed for PAC, adding it to clang shouldn't be a problem. Nick, did you have any thoughts on this?
Sami
Nick Desaulniers
reviewerRe: [PATCH v2 09/17] arm64: disable function graph tracing with SCS
Quoted 7 lines · Sami Tolvanen
Quoted 5 lines · Mark Rutland
We have a similar issue with pointer authentication, and we're solving that with -fpatchable-function-entry, which allows us to hook the callsite before it does anything with the return address. IIUC we could use the same mechanism here (and avoid introducing a third).
Quoted 6 lines
Quoted 2 lines
Are there plans to implement -fpatchable-function-entry on the clang side?
I'm not sure if there are plans at the moment, but if this feature is needed for PAC, adding it to clang shouldn't be a problem. Nick, did you have any thoughts on this?
I didn't see anything explicitly in LLVM's issue tracker. I also didn't see -fpatchable-function-entry currently in -next other than under arch/parisc. Are there patches I can look at?
Has ARM's kernel team expressed the need to ARM's LLVM team?
Signature · 2 lines
Thanks, ~Nick Desaulniers
PATCH v2 10/17 arm64: disable kretprobes with SCS
In reply to [email protected]With CONFIG_KRETPROBES, function return addresses are modified to redirect control flow to kretprobe_trampoline. This is incompatible with SCS.
--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/Kconfig +1 -1
@@ -165,7 +165,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if !SHADOW_CALL_STACK select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 11/17 arm64: reserve x18 from general allocation with SCS
In reply to [email protected]Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled.
--- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/Makefile +4 -0
@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 12/17 arm64: preserve x18 when CPU is suspended
In reply to [email protected]Don't lose the current task's shadow stack when the CPU is suspended.
--- arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/mm/proc.S | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-)
arch/arm64/include/asm/suspend.h +1 -1
@@ -2,7 +2,7 @@#ifndef __ASM_SUSPEND_H#define __ASM_SUSPEND_H -#define NR_CTX_REGS 12 +#define NR_CTX_REGS 13#define NR_CALLEE_SAVED_REGS 12/*
arch/arm64/mm/proc.S +9 -0
@@ -49,6 +49,8 @@ * cpu_do_suspend - save CPU registers context * * x0: virtual address of context pointer + * + * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>. */ ENTRY(cpu_do_suspend) mrs x2, tpidr_el0
@@ -73,6 +75,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #96] +#endif ret ENDPROC(cpu_do_suspend)
@@ -89,6 +94,10 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 13/17 arm64: efi: restore x18 if it was corrupted
In reply to [email protected]If we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code. This is safe, because the wrapper is called with preemption disabled and a separate shadow stack is used for interrupt handling.
--- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kernel/efi-rt-wrapper.S +6 -1
@@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper)
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 14/17 arm64: vdso: disable Shadow Call Stack
In reply to [email protected]--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := n
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 15/17 arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
In reply to [email protected]This allows CONFIG_KRETPROBES to be disabled without disabling kprobes entirely.
--- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kernel/probes/kprobes.c +2 -0
@@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe (struct kretprobe_instance *ri,struct pt_regs *regs) {
@@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) {return 0 ; } +#endif int __init arch_init_kprobes (void ) {
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v2 16/17 arm64: disable SCS for hypervisor code
In reply to [email protected]--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/kvm/hyp/Makefile +3 -0
@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +ORIG_CFLAGS := $(KBUILD_CFLAGS) +KBUILD_CFLAGS = $(subst $(CC_FLAGS_SCS),,$(ORIG_CFLAGS))
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Steven Rostedt
maintainerRe: [PATCH v2 16/17] arm64: disable SCS for hypervisor code
On Thu, 24 Oct 2019 15:51:31 -0700
[email protected] wrote:
;-)
Quoted 10 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/kvm/hyp/Makefile +0
-0
arch/arm64/kvm/hyp/Makefile GCOV_PROFILE := n · line 33
Quoted 6 lines
arch/arm64/kvm/hyp/Makefile +3
-0@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +ORIG_CFLAGS := $(KBUILD_CFLAGS) +KBUILD_CFLAGS = $(subst $(CC_FLAGS_SCS),,$(ORIG_CFLAGS))
May want a comment above that that states:
# remove the SCS flags from all objects in this directory
-- Steve
Re: [PATCH v2 16/17] arm64: disable SCS for hypervisor code
In reply to [email protected]Quoted 11 lines · On Fri, Oct 25, 2019 at 7:52 AM <[email protected]> wrote:
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/kvm/hyp/Makefile +0
-0
arch/arm64/kvm/hyp/Makefile GCOV_PROFILE := n · line 33
Quoted 6 lines
arch/arm64/kvm/hyp/Makefile +3
-0@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +ORIG_CFLAGS := $(KBUILD_CFLAGS) +KBUILD_CFLAGS = $(subst $(CC_FLAGS_SCS),,$(ORIG_CFLAGS))
$(subst ... ) is not the correct use here.
It works like sed, s/$(CC_CFLAGS_SCS)//
instead of matching by word.
KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))
is more correct, and simpler.
Signature · 2 lines
Best Regards Masahiro Yamada
Steven Rostedt
maintainerRe: [PATCH v2 16/17] arm64: disable SCS for hypervisor code
Quoted 28 lines · On Fri, 25 Oct 2019 10:29:47 +0900 Masahiro Yamada…
Quoted 11 lines · On Fri, Oct 25, 2019 at 7:52 AM <[email protected]> wrote:
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/kvm/hyp/Makefile +0
-0arch/arm64/kvm/hyp/Makefile GCOV_PROFILE := n · line 33
Quoted 6 lines
arch/arm64/kvm/hyp/Makefile +3
-0@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +ORIG_CFLAGS := $(KBUILD_CFLAGS) +KBUILD_CFLAGS = $(subst $(CC_FLAGS_SCS),,$(ORIG_CFLAGS))$(subst ... ) is not the correct use here.
It works like sed, s/$(CC_CFLAGS_SCS)//
instead of matching by word.
Quoted 3 lines
KBUILD_CFLAGS:= $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))is more correct, and simpler.
I guess that would work too. Not sure why I never used it. I see mips used it for their -pg flags.
-- Steve
Re: [PATCH v2 16/17] arm64: disable SCS for hypervisor code
In reply to Masahiro YamadaQuoted 11 lines · Masahiro Yamada
$(subst ... ) is not the correct use here.
It works like sed, s/$(CC_CFLAGS_SCS)//
instead of matching by word.
KBUILD_CFLAGS:= $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))is more correct, and simpler.
Thanks, I will change this in v3.
Sami
PATCH v2 17/17 arm64: implement Shadow Call Stack
In reply to [email protected]This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
--- arch/arm64/Kconfig | 5 ++++ arch/arm64/include/asm/scs.h | 45 ++++++++++++++++++++++++++++ arch/arm64/include/asm/stacktrace.h | 4 +++ arch/arm64/include/asm/thread_info.h | 3 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 ++ arch/arm64/kernel/entry.S | 28 +++++++++++++++++ arch/arm64/kernel/head.S | 9 ++++++ arch/arm64/kernel/irq.c | 2 ++ arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/scs.c | 39 ++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 4 +++ 12 files changed, 145 insertions(+) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c
arch/arm64/Kconfig +5 -0
@@ -66,6 +66,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING
@@ -948,6 +949,10 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_ENABLE_SPLIT_PMD_PTLOCK def_bool y if PGTABLE_LEVELS > 2 +# Supported by clang >= 7.0 +config CC_HAVE_SHADOW_CALL_STACK + def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help---
arch/arm64/include/asm/scs.h new file +45 -0
45 lines not on this page · raw message
arch/arm64/include/asm/stacktrace.h +4 -0
@@ -68,6 +68,10 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif +static inline bool on_irq_stack (unsigned long sp,struct stack_info *info) {
arch/arm64/include/asm/thread_info.h +3 -0
@@ -41,6 +41,9 @@ struct thread_info {#endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK +void *shadow_call_stack; +#endif };#define thread_saved_pc (tsk) \
arch/arm64/kernel/Makefile +1 -0
@@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/
arch/arm64/kernel/asm-offsets.c +3 -0
@@ -33,6 +33,9 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit));#ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack));#endif DEFINE (TSK_STACK , offsetof(struct task_struct , stack));#ifdef CONFIG_STACKPROTECTOR
arch/arm64/kernel/entry.S +28 -0
@@ -172,6 +172,10 @@ alternative_cb_end apply_ssbd 1, x22, x23 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack + str xzr, [tsk, #TSK_TI_SCS] +#endif .else add x21, sp, #S_FRAME_SIZE get_current_task tsk
@@ -278,6 +282,12 @@ alternative_else_nop_endif ct_user_enter .endif +#ifdef CONFIG_SHADOW_CALL_STACK + .if \el == 0 + str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack + .endif +#endif + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR
@@ -383,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x20, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack.
@@ -400,6 +413,12 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + ldr_this_cpu x18, irq_shadow_call_stack_ptr, x26 +#endif + 9998: .endm
@@ -409,6 +428,10 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + /* x20 is also preserved */ + mov x18, x20 +#endif .endm /* GPRs used by entry code */
@@ -1155,6 +1178,11 @@ ENTRY(cpu_switch_to) ldr lr, [x8] mov sp, x9 msr sp_el0, x1 +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #TSK_TI_SCS] + ldr x18, [x1, #TSK_TI_SCS] + str xzr, [x1, #TSK_TI_SCS] +#endif ret ENDPROC(cpu_switch_to) NOKPROBE(cpu_switch_to)
arch/arm64/kernel/head.S +9 -0
@@ -27,6 +27,7 @@ #include <asm/pgtable-hwdef.h> #include <asm/pgtable.h> #include <asm/page.h> +#include <asm/scs.h> #include <asm/smp.h> #include <asm/sysreg.h> #include <asm/thread_info.h>
@@ -424,6 +425,10 @@ __primary_switched: stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between
@@ -731,6 +736,10 @@ __secondary_switched: ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack + str xzr, [x2, #TSK_TI_SCS] +#endif mov x29, #0 mov x30, #0 b secondary_start_kernel
arch/arm64/kernel/irq.c +2 -0
@@ -21,6 +21,7 @@#include <linux/vmalloc.h> #include <asm/daifflags.h> #include <asm/vmap_stack.h> +#include <asm/scs.h> unsigned long irq_err_count;
@@ -63,6 +64,7 @@ static void init_irq_stacks(void)void __init init_IRQ (void ) {init_irq_stacks (); +scs_init_irq ();irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." );
arch/arm64/kernel/process.c +2 -0
@@ -52,6 +52,7 @@#include <asm/mmu_context.h> #include <asm/processor.h> #include <asm/pointer_auth.h> +#include <asm/scs.h> #include <asm/stacktrace.h> #if defined(CONFIG_STACKPROTECTOR ) && !defined(CONFIG_STACKPROTECTOR_PER_TASK )
@@ -507,6 +508,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,uao_thread_switch (next);ptrauth_thread_switch (next);ssbs_thread_switch (next); +scs_overflow_check (next);/* * Complete any pending TLB or cache maintenance on this CPU in case
arch/arm64/kernel/scs.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/percpu.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); + +#ifndef CONFIG_SHADOW_CALL_STACK_VMAP +DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack) +__aligned(SCS_SIZE) ; +#endif + +void scs_init_irq (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) { +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP +unsigned long *p; + + p =__vmalloc_node_range (SCS_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +SCS_GFP ,PAGE_KERNEL , +0 ,cpu_to_node (cpu), +__builtin_return_address (0 )); + +per_cpu (irq_shadow_call_stack_ptr, cpu) = p; +#else +per_cpu (irq_shadow_call_stack_ptr, cpu) = +per_cpu (irq_shadow_call_stack, cpu); +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + } +}
arch/arm64/kernel/smp.c +4 -0
@@ -44,6 +44,7 @@#include <asm/pgtable.h> #include <asm/pgalloc.h> #include <asm/processor.h> +#include <asm/scs.h> #include <asm/smp_plat.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -357,6 +358,9 @@ void cpu_die(void) {unsigned int cpu =smp_processor_id (); +/* Save the shadow stack pointer before exiting the idle task */ +scs_save (current); +idle_task_exit ();local_daif_mask ();
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
v3 cover add support for Clang's Shadow Call Stack
In reply to Sami TolvanenThis patch series adds support for Clang's Shadow Call Stack (SCS) mitigation, which uses a separately allocated shadow stack to protect against return address overwrites. More information can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
SCS provides better protection against traditional buffer overflows than CONFIG_STACKPROTECTOR_*, but it should be noted that SCS security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
SCS is currently supported only on arm64, where the compiler requires the x18 register to be reserved for holding the current task's shadow stack pointer. Because of this, the series includes patches from Ard to remove x18 usage from assembly code.
With -fsanitize=shadow-call-stack, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack, and unconditionally load it again before returning. As a result, SCS is currently incompatible with features that rely on modifying function return addresses to alter control flow, such as function graph tracing and kretprobes, although it may be possible to later change these features to modify the shadow stack instead. A copy of the return address is still kept in the kernel stack for compatibility with stack unwinding, for example.
SCS has a minimal performance overhead, but allocating shadow stacks increases kernel memory usage. The feature is therefore mostly useful on hardware that lacks support for PAC instructions.
Changes in v3:
- Switched to filter-out for removing SCS flags in Makefiles
- Changed the __noscs attribute to use __no_sanitize__("...") instead of no_sanitize("...")
- Cleaned up inline function definitions and moved task_scs() into a macro
- Cleaned up scs_free() and scs_magic()
- Moved SCS initialization into dup_task_struct() and removed the now unused scs_task_init()
- Added comments to __scs_base() and scs_task_reset() to better document design choices
- Changed copy_page to make the offset and bias explicit
Changes in v2:
- Changed Ard's KVM patch to use x29 instead of x18 for the guest context, which makes restore_callee_saved_regs cleaner
- Updated help text (and commit messages) to point out differences in security properties compared to user space SCS
- Cleaned up config options: removed the ROP protection choice, replaced the CC_IS_CLANG dependency with an arch-specific cc-option test, and moved disabling of incompatible config options to an arch-specific Kconfig
- Added CC_FLAGS_SCS, which are filtered out where needed instead of using DISABLE_SCS
- Added a __has_feature guard around __noscs for older clang versions
Ard Biesheuvel (1):
arm64: kernel: avoid x18 __cpu_soft_restart
Sami Tolvanen (16):
arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings arm64/lib: copy_page: avoid x18 register in assembler code
arm64: kvm: stop treating register x18 as caller save add support for Clang's Shadow Call Stack (SCS)
scs: add accounting
scs: add support for stack usage debugging
kprobes: fix compilation without CONFIG_KRETPROBES
arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
arm64: disable kretprobes with SCS
arm64: disable function graph tracing with SCS
arm64: reserve x18 from general allocation with SCS
arm64: preserve x18 when CPU is suspended
arm64: efi: restore x18 if it was corrupted
arm64: vdso: disable Shadow Call Stack
arm64: disable SCS for hypervisor code
arm64: implement Shadow Call Stack
Makefile | 6 + arch/Kconfig | 33 ++++ arch/arm64/Kconfig | 9 +- arch/arm64/Makefile | 4 + arch/arm64/include/asm/scs.h | 37 +++++ arch/arm64/include/asm/stacktrace.h | 4 + arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/include/asm/thread_info.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 + arch/arm64/kernel/cpu-reset.S | 4 +- arch/arm64/kernel/efi-rt-wrapper.S | 7 +- arch/arm64/kernel/entry.S | 28 ++++ arch/arm64/kernel/head.S | 9 ++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/probes/kprobes.c | 2 + arch/arm64/kernel/process.c | 2 + arch/arm64/kernel/scs.c | 39 +++++ arch/arm64/kernel/smp.c | 4 + arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kvm/hyp/Makefile | 3 + arch/arm64/kvm/hyp/entry.S | 41 +++-- arch/arm64/lib/copy_page.S | 38 ++--- arch/arm64/mm/proc.S | 72 +++++---- drivers/base/node.c | 6 + fs/proc/meminfo.c | 4 + include/linux/compiler-clang.h | 6 + include/linux/compiler_types.h | 4 + include/linux/mmzone.h | 3 + include/linux/scs.h | 54 +++++++ init/init_task.c | 8 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/kprobes.c | 38 ++--- kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 227 +++++++++++++++++++++++++++ mm/page_alloc.c | 6 + mm/vmstat.c | 3 + 39 files changed, 630 insertions(+), 97 deletions(-) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v3 1/17 arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
idmap_kpti_install_ng_mappings uses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
arch/arm64/mm/proc.S +32 -31
@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */
@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*
@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
PATCH v3 2/17 arm64/lib: copy_page: avoid x18 register in assembler code
In reply to [email protected]Register x18 will no longer be used as a caller save register in the future, so stop using it in the copy_page() code.
--- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/arm64/lib/copy_page.S +19 -19
@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1) alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #16 - 256] ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #32 - 256] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #48 - 256] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #64 - 256] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #80 - 256] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #96 - 256] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #112 - 256] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80] - stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #16 - 256] + stnp x6, x7, [x0, #32 - 256] + stnp x8, x9, [x0, #48 - 256] + stnp x10, x11, [x0, #64 - 256] + stnp x12, x13, [x0, #80 - 256] + stnp x14, x15, [x0, #96 - 256] + stnp x16, x17, [x0, #112 - 256] ret ENDPROC(copy_page)
Signature · 7 lines
2.24.0.rc0.303.g954a862665-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v3 3/17 arm64: kvm: stop treating register x18 as caller save
In reply to [email protected]In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to __guest_enter(). Instead, preserve its value and restore it upon return.
--- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
arch/arm64/kvm/hyp/entry.S +20 -21
@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28 + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1
@@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret
@@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
@@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 03/17] arm64: kvm: stop treating register x18 as caller save
Quoted 10 lines · On Thu, Oct 31, 2019 at 09:46:23AM -0700, [email protected] wrote:
In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.Co-developed-by Ard Biesheuvel <[email protected]>[ updated commit message, switched from x18 to x29 for the guest context ]Signed-off-by Sami Tolvanen <[email protected]>
-Kees
Quoted 103 lines
--- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-)arch/arm64/kvm/hyp/entry.S +20
-21@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28 + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1@@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret@@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]@@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 4/17 arm64: kernel: avoid x18 __cpu_soft_restart
In reply to [email protected]From: Ard Biesheuvel <[email protected]>
The code in __cpu_soft_restart() uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/cpu-reset.S +2 -2
@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsection
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 04/17] arm64: kernel: avoid x18 __cpu_soft_restart
Quoted 7 lines · On Thu, Oct 31, 2019 at 09:46:24AM -0700, [email protected] wrote:
From: Ard Biesheuvel <[email protected]>
The code in
__cpu_soft_restart()uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.
This one is easy to validate! ;)
-Kees
Quoted 27 lines
Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Mark Rutland <[email protected]>--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/arm64/kernel/cpu-reset.S +2
-2@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsectionSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 5/17 add support for Clang's Shadow Call Stack (SCS)
In reply to [email protected]This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
--- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 54 +++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 169 +++++++++++++++++++++++++++++++++ 11 files changed, 293 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
Makefile +6 -0
@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
arch/Kconfig +33 -0
@@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool help
include/linux/compiler-clang.h +6 -0
@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((__no_sanitize__("shadow-call-stack"))) +#else +# define __noscs +#endif
include/linux/compiler_types.h +4 -0
@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif
include/linux/scs.h new file +54 -0
54 lines not on this page · raw message
init/init_task.c +8 -0
@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h>
@@ -184,6 +185,13 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data +__aligned(SCS_SIZE) = { + [(SCS_SIZE /sizeof (long )) -1 ] =SCS_END_MAGIC +}; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry.
kernel/Makefile +1 -0
@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/
kernel/fork.c +9 -0
@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info,
@@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }
@@ -893,6 +898,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)if (err) goto free_stack; + err =scs_prepare (tsk, node); +if (err) + goto free_stack; +#ifdef CONFIG_SECCOMP /* * We must handle setting up seccomp filters once we're under
kernel/sched/core.c +2 -0
@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
kernel/sched/sched.h +1 -0
@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h>
kernel/scs.c new file +169 -0
169 lines not on this page · raw message
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 142 lines · On Thu, Oct 31, 2019 at 09:46:25AM -0700, [email protected] wrote:
This change adds generic support for Clang's Shadow Call Stack, which uses a shadow stack to protect return addresses from being overwritten by an attacker. Details are available here:
https://clang.llvm.org/docs/ShadowCallStack.html
Note that security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
Signed-off-by Sami Tolvanen <[email protected]>--- Makefile | 6 ++ arch/Kconfig | 33 +++++++ include/linux/compiler-clang.h | 6 ++ include/linux/compiler_types.h | 4 + include/linux/scs.h | 54 +++++++++++ init/init_task.c | 8 ++ kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 169 +++++++++++++++++++++++++++++++++ 11 files changed, 293 insertions(+) create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.cMakefile +6
-0@@ -846,6 +846,12 @@ ifdef CONFIG_LIVEPATCH KBUILD_CFLAGS += $(call cc-option, -flive-patching=inline-clone) endif +ifdef CONFIG_SHADOW_CALL_STACK +CC_FLAGS_SCS := -fsanitize=shadow-call-stack +KBUILD_CFLAGS += $(CC_FLAGS_SCS) +export CC_FLAGS_SCS +endif + # arch Makefile may override CC so keep this after arch Makefile is included NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)arch/Kconfig +33
-0@@ -521,6 +521,39 @@ config STACKPROTECTOR_STRONG about 20% of all kernel functions, which increases the kernel code size by about 2%. +config ARCH_SUPPORTS_SHADOW_CALL_STACK + bool + help + An architecture should select this if it supports Clang's Shadow + Call Stack, has asm/scs.h, and implements runtime support for shadow + stack switching. + +config SHADOW_CALL_STACK_VMAP + bool + depends on SHADOW_CALL_STACK + help + Use virtually mapped shadow call stacks. Selecting this option + provides better stack exhaustion protection, but increases per-thread + memory consumption as a full page is allocated for each shadow stack. + +config SHADOW_CALL_STACK + bool "Clang Shadow Call Stack" + depends on ARCH_SUPPORTS_SHADOW_CALL_STACK + help + This option enables Clang's Shadow Call Stack, which uses a + shadow stack to protect function return addresses from being + overwritten by an attacker. More information can be found from + Clang's documentation: + + https://clang.llvm.org/docs/ShadowCallStack.html + + Note that security guarantees in the kernel differ from the ones + documented for user space. The kernel must store addresses of shadow + stacks used by other tasks and interrupt handlers in memory, which + means an attacker capable reading and writing arbitrary memory may + be able to locate them and hijack control flow by modifying shadow + stacks that are not currently in use. + config HAVE_ARCH_WITHIN_STACK_FRAMES bool helpinclude/linux/compiler-clang.h +6
-0@@ -42,3 +42,9 @@* compilers, like ICC. */ #define barrier () __asm__ __volatile__("" : : : "memory") + +#if __has_feature (shadow_call_stack) +# define __noscs __attribute__((__no_sanitize__("shadow-call-stack"))) +#else +# define __noscs +#endif include/linux/compiler_types.h +4
-0@@ -202,6 +202,10 @@ struct ftrace_likely_data {# define randomized_struct_fields_end#endif +#ifndef __noscs +# define __noscs +#endif +#ifndef asm_volatile_goto #define asm_volatile_goto(x...) asm goto(x)#endif include/linux/scs.h new file +16
-0@@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#ifndef _LINUX_SCS_H +#define _LINUX_SCS_H + +#include <linux/gfp.h> +#include <linux/sched.h> +#include <asm/page.h> + +#ifdef CONFIG_SHADOW_CALL_STACK +
include/linux/scs.h line 26
Quoted 10 lines
include/linux/scs.h new file +10
-0+/* +* In testing, 1 KiB shadow stack size (i.e. 128 stack frames on a 64-bit +* architecture) provided ~40% safety margin on stack usage while keeping +* memory allocation overhead reasonable. +*/ +#define SCS_SIZE 1024 +#define GFP_SCS (GFP_KERNEL | __GFP_ZERO) + +/* A random number to mark the end of the shadow stack. */ +#define SCS_END_MAGIC 0xaf0194819b1635f6UL
Is 0xaf.... non-canonical for arm64? While "random", it should also likely be an "impossible" value to find on the call stack.
Otherwise, all looks great. :)
-Kees
Quoted 310 lines
include/linux/scs.h new file +28
-0+ +#define task_scs (tsk) (task_thread_info(tsk)->shadow_call_stack) + +static inline void task_set_scs (struct task_struct *tsk,void *s) +{ +task_scs (tsk) = s; +} + +extern void scs_init (void ); +extern void scs_task_reset (struct task_struct *tsk); +extern int scs_prepare (struct task_struct *tsk,int node); +extern bool scs_corrupted (struct task_struct *tsk); +extern void scs_release (struct task_struct *tsk); + +#else /* CONFIG_SHADOW_CALL_STACK */ + +#define task_scs (tsk) NULL + +static inline void task_set_scs (struct task_struct *tsk,void *s) {} +static inline void scs_init (void ) {} +static inline void scs_task_reset (struct task_struct *tsk) {} +static inline int scs_prepare (struct task_struct *tsk,int node) {return 0 ; } +static inline bool scs_corrupted (struct task_struct *tsk) {return false; } +static inline void scs_release (struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* _LINUX_SCS_H */ init/init_task.c +8
-0@@ -11,6 +11,7 @@#include <linux/mm.h> #include <linux/audit.h> #include <linux/numa.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <linux/uaccess.h> @@ -184,6 +185,13 @@ struct task_struct init_task };EXPORT_SYMBOL (init_task); +#ifdef CONFIG_SHADOW_CALL_STACK +unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] __init_task_data +__aligned(SCS_SIZE) = { + [(SCS_SIZE /sizeof (long )) -1 ] =SCS_END_MAGIC +}; +#endif +/* * Initial thread structure. Alignment of this is handled by a special * linker map entry. kernel/Makefile +1
-0@@ -102,6 +102,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/fork.c +9
-0@@ -94,6 +94,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/scs.h> #include <asm/pgtable.h> #include <asm/pgalloc.h> @@ -451,6 +452,8 @@ void put_task_stack(struct task_struct *tsk)void free_task (struct task_struct *tsk) { +scs_release (tsk); +#ifndef CONFIG_THREAD_INFO_IN_TASK /* * The task is finally done with both the stack and thread_info, @@ -834,6 +837,8 @@ void __init fork_init(void) NULL, free_vm_stack_cache);#endif +scs_init (); +lockdep_init_task (&init_task);uprobes_init (); }@@ -893,6 +898,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)if (err) goto free_stack; + err =scs_prepare (tsk, node); +if (err) + goto free_stack; +#ifdef CONFIG_SECCOMP /* * We must handle setting up seccomp filters once we're under kernel/sched/core.c +2
-0@@ -6013,6 +6013,8 @@ void init_idle(struct task_struct *idle, int cpu)raw_spin_lock_irqsave (&idle->pi_lock, flags);raw_spin_lock (&rq->lock); +scs_task_reset (idle); +__sched_fork (0 , idle); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();kernel/sched/sched.h +1
-0@@ -58,6 +58,7 @@#include <linux/profile.h> #include <linux/psi.h> #include <linux/rcupdate_wait.h> +#include <linux/scs.h> #include <linux/security.h> #include <linux/stop_machine.h> #include <linux/suspend.h> kernel/scs.c new file +169
-0@@ -0,0 +1,169 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/cpuhotplug.h> +#include <linux/mm.h> +#include <linux/mmzone.h> +#include <linux/scs.h> +#include <linux/slab.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +static inline void *__scs_base (struct task_struct *tsk) +{ +/* +* We allow architectures to use the shadow_call_stack field in +* struct thread_info to store the current shadow stack pointer +* during context switches. +* +* This allows the implementation to also clear the field when +* the task is active to avoid keeping pointers to the current +* task's shadow stack in memory. This can make it harder for an +* attacker to locate the shadow stack, but also requires us to +* compute the base address when needed. +* +* We assume the stack is aligned to SCS_SIZE. +*/ +return (void *)((uintptr_t )task_scs (tsk) & ~(SCS_SIZE -1 )); +} + +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP + +/* Keep a cache of shadow stacks */ +#define SCS_CACHE_SIZE 2 +static DEFINE_PER_CPU(void *, scs_cache[SCS_CACHE_SIZE]); + +static void *scs_alloc (int node) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +void *s; + + s =this_cpu_xchg (scs_cache[i],NULL ); +if (s) { +memset (s,0 ,SCS_SIZE ); +return s; + } + } + +/* +* We allocate a full page for the shadow stack, which should be +* more than we need. Check the assumption nevertheless. +*/ +BUILD_BUG_ON (SCS_SIZE >PAGE_SIZE ); + +return __vmalloc_node_range (PAGE_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +GFP_SCS ,PAGE_KERNEL ,0 , + node,__builtin_return_address (0 )); +} + +static void scs_free (void *s) +{ +int i; + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) +if (this_cpu_cmpxchg (scs_cache[i],0 , s) ==0 ) +return ; + +vfree_atomic (s); +} + +static int scs_cleanup (unsigned int cpu) +{ +int i; +void **cache =per_cpu_ptr (scs_cache, cpu); + +for (i =0 ; i <SCS_CACHE_SIZE ; i++) { +vfree (cache[i]); + cache[i] =NULL ; + } + +return 0 ; +} + +void __init scs_init (void ) +{ +cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"scs:scs_cache" ,NULL , + scs_cleanup); +} + +#else /* !CONFIG_SHADOW_CALL_STACK_VMAP */ + +static struct kmem_cache *scs_cache; + +static inline void *scs_alloc (int node) +{ +return kmem_cache_alloc_node (scs_cache,GFP_SCS , node); +} + +static inline void scs_free (void *s) +{ +kmem_cache_free (scs_cache, s); +} + +void __init scs_init (void ) +{ + scs_cache =kmem_cache_create ("scs_cache" ,SCS_SIZE ,SCS_SIZE , +0 ,NULL ); +WARN_ON (!scs_cache); +} + +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + +static inline unsigned long *scs_magic (struct task_struct *tsk) +{ +return (unsigned long *)(__scs_base (tsk) +SCS_SIZE ) -1 ; +} + +static inline void scs_set_magic (struct task_struct *tsk) +{ + *scs_magic (tsk) =SCS_END_MAGIC ; +} + +void scs_task_reset (struct task_struct *tsk) +{ +/* +* Reset the shadow stack to the base address in case the task +* is reused. +*/ +task_set_scs (tsk,__scs_base (tsk)); +} + +int scs_prepare (struct task_struct *tsk,int node) +{ +void *s; + + s =scs_alloc (node); +if (!s) +return -ENOMEM ; + +task_set_scs (tsk, s); +scs_set_magic (tsk); + +return 0 ; +} + +bool scs_corrupted (struct task_struct *tsk) +{ +return *scs_magic (tsk) !=SCS_END_MAGIC ; +} + +void scs_release (struct task_struct *tsk) +{ +void *s; + + s =__scs_base (tsk); +if (!s) +return ; + +WARN_ON (scs_corrupted (tsk)); + +task_set_scs (tsk,NULL ); +scs_free (s); +}Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
Re: [PATCH v3 05/17] add support for Clang's Shadow Call Stack (SCS)
Quoted 5 lines · Kees Cook
include/linux/scs.h line 26
Quoted 2 lines
include/linux/scs.h new file +2
-0+/* A random number to mark the end of the shadow stack. */ +#define SCS_END_MAGIC 0xaf0194819b1635f6ULIs
0xaf.... non-canonical for arm64? While "random", it should also likely be an "impossible" value to find on the call stack.
Agreed, and yes, this is non-canonical for arm64 and AFAIK all 64-bit architectures the kernel supports. I'll add a note about it.
Sami
PATCH v3 6/17 scs: add accounting
In reply to [email protected]This change adds accounting for the memory allocated for shadow stacks.
--- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 19 +++++++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 41 insertions(+)
drivers/base/node.c +6 -0
@@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif"Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n"
@@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_ANON_MAPPED )), nid,K (i.sharedram), nid,sum_zone_node_page_state (nid,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK + nid,sum_zone_node_page_state (nid,NR_KERNEL_SCS_BYTES ) /1024 , +#endif nid,K (sum_zone_node_page_state (nid,NR_PAGETABLE )), nid,K (node_page_state (pgdat,NR_UNSTABLE_NFS )), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
fs/proc/meminfo.c +4 -0
@@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)show_val_kb (m,"SUnreclaim: " , sunreclaim);seq_printf (m,"KernelStack: %8lu kB\n" ,global_zone_page_state (NR_KERNEL_STACK_KB )); +#ifdef CONFIG_SHADOW_CALL_STACK +seq_printf (m,"ShadowCallStack:%8lu kB\n" , +global_zone_page_state (NR_KERNEL_SCS_BYTES ) /1024 ); +#endif show_val_kb (m,"PageTables: " ,global_zone_page_state (NR_PAGETABLE ));
include/linux/mmzone.h +3 -0
@@ -200,6 +200,9 @@ enum zone_stat_item {NR_MLOCK ,/* mlock()ed pages found and moved off LRU */ NR_PAGETABLE ,/* used for pagetables */ NR_KERNEL_STACK_KB ,/* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +NR_KERNEL_SCS_BYTES ,/* measured in bytes */ +#endif/* Second 128 byte cacheline */ NR_BOUNCE , #if IS_ENABLED(CONFIG_ZSMALLOC)
kernel/scs.c +19 -0
@@ -11,6 +11,7 @@#include <linux/scs.h> #include <linux/slab.h> #include <linux/vmalloc.h> +#include <linux/vmstat.h> #include <asm/scs.h> static inline void *__scs_base (struct task_struct *tsk)
@@ -74,6 +75,11 @@ static void scs_free(void *s)vfree_atomic (s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return vmalloc_to_page (__scs_base (tsk)); +} +static int scs_cleanup (unsigned int cpu) {int i;
@@ -107,6 +113,11 @@ static inline void scs_free(void *s)kmem_cache_free (scs_cache, s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return virt_to_page (__scs_base (tsk)); +} +void __init scs_init (void ) { scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE,
@@ -135,6 +146,12 @@ void scs_task_reset(struct task_struct *tsk)task_set_scs (tsk,__scs_base (tsk)); } +static void scs_account (struct task_struct *tsk,int account) +{ +mod_zone_page_state (page_zone (__scs_page (tsk)),NR_KERNEL_SCS_BYTES , + account *SCS_SIZE ); +} +int scs_prepare (struct task_struct *tsk,int node) {void *s;
@@ -145,6 +162,7 @@ int scs_prepare(struct task_struct *tsk, int node)task_set_scs (tsk, s);scs_set_magic (tsk); +scs_account (tsk,1 );return 0 ; }
@@ -164,6 +182,7 @@ void scs_release(struct task_struct *tsk)WARN_ON (scs_corrupted (tsk)); +scs_account (tsk,-1 );task_set_scs (tsk,NULL );scs_free (s); }
mm/page_alloc.c +6 -0
@@ -5361,6 +5361,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif" pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB"
@@ -5382,6 +5385,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)K (zone_managed_pages (zone)),K (zone_page_state (zone,NR_MLOCK )),zone_page_state (zone,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK +zone_page_state (zone,NR_KERNEL_SCS_BYTES ) /1024 , +#endifK (zone_page_state (zone,NR_PAGETABLE )),K (zone_page_state (zone,NR_BOUNCE )), K(free_pcp),
mm/vmstat.c +3 -0
@@ -1118,6 +1118,9 @@ const char * const vmstat_text[] = {"nr_mlock" ,"nr_page_table_pages" ,"nr_kernel_stack" , +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +"nr_shadow_call_stack_bytes" , +#endif"nr_bounce" , #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages",
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 06/17] scs: add accounting
Quoted 3 lines · On Thu, Oct 31, 2019 at 09:46:26AM -0700, [email protected] wrote:
This change adds accounting for the memory allocated for shadow stacks.
Signed-off-by Sami Tolvanen <[email protected]>
A nice bit of stats to have.
-Kees
Quoted 168 lines
--- drivers/base/node.c | 6 ++++++ fs/proc/meminfo.c | 4 ++++ include/linux/mmzone.h | 3 +++ kernel/scs.c | 19 +++++++++++++++++++ mm/page_alloc.c | 6 ++++++ mm/vmstat.c | 3 +++ 6 files changed, 41 insertions(+)drivers/base/node.c +6
-0@@ -415,6 +415,9 @@ static ssize_t node_read_meminfo(struct device *dev, "Node %d AnonPages: %8lu kB\n" "Node %d Shmem: %8lu kB\n" "Node %d KernelStack: %8lu kB\n" +#ifdef CONFIG_SHADOW_CALL_STACK + "Node %d ShadowCallStack:%8lu kB\n" +#endif"Node %d PageTables: %8lu kB\n" "Node %d NFS_Unstable: %8lu kB\n" "Node %d Bounce: %8lu kB\n" @@ -438,6 +441,9 @@ static ssize_t node_read_meminfo(struct device *dev, nid,K (node_page_state (pgdat,NR_ANON_MAPPED )), nid,K (i.sharedram), nid,sum_zone_node_page_state (nid,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK + nid,sum_zone_node_page_state (nid,NR_KERNEL_SCS_BYTES ) /1024 , +#endif nid,K (sum_zone_node_page_state (nid,NR_PAGETABLE )), nid,K (node_page_state (pgdat,NR_UNSTABLE_NFS )), nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),fs/proc/meminfo.c +4
-0@@ -103,6 +103,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)show_val_kb (m,"SUnreclaim: " , sunreclaim);seq_printf (m,"KernelStack: %8lu kB\n" ,global_zone_page_state (NR_KERNEL_STACK_KB )); +#ifdef CONFIG_SHADOW_CALL_STACK +seq_printf (m,"ShadowCallStack:%8lu kB\n" , +global_zone_page_state (NR_KERNEL_SCS_BYTES ) /1024 ); +#endif show_val_kb (m,"PageTables: " ,global_zone_page_state (NR_PAGETABLE ));include/linux/mmzone.h +3
-0@@ -200,6 +200,9 @@ enum zone_stat_item {NR_MLOCK ,/* mlock()ed pages found and moved off LRU */ NR_PAGETABLE ,/* used for pagetables */ NR_KERNEL_STACK_KB ,/* measured in KiB */ +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +NR_KERNEL_SCS_BYTES ,/* measured in bytes */ +#endif/* Second 128 byte cacheline */ NR_BOUNCE , #if IS_ENABLED(CONFIG_ZSMALLOC)kernel/scs.c +19
-0@@ -11,6 +11,7 @@#include <linux/scs.h> #include <linux/slab.h> #include <linux/vmalloc.h> +#include <linux/vmstat.h> #include <asm/scs.h> static inline void *__scs_base (struct task_struct *tsk)@@ -74,6 +75,11 @@ static void scs_free(void *s)vfree_atomic (s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return vmalloc_to_page (__scs_base (tsk)); +} +static int scs_cleanup (unsigned int cpu) {int i;@@ -107,6 +113,11 @@ static inline void scs_free(void *s)kmem_cache_free (scs_cache, s); } +static struct page *__scs_page (struct task_struct *tsk) +{ +return virt_to_page (__scs_base (tsk)); +} +void __init scs_init (void ) { scs_cache = kmem_cache_create("scs_cache", SCS_SIZE, SCS_SIZE,@@ -135,6 +146,12 @@ void scs_task_reset(struct task_struct *tsk)task_set_scs (tsk,__scs_base (tsk)); } +static void scs_account (struct task_struct *tsk,int account) +{ +mod_zone_page_state (page_zone (__scs_page (tsk)),NR_KERNEL_SCS_BYTES , + account *SCS_SIZE ); +} +int scs_prepare (struct task_struct *tsk,int node) {void *s;@@ -145,6 +162,7 @@ int scs_prepare(struct task_struct *tsk, int node)task_set_scs (tsk, s);scs_set_magic (tsk); +scs_account (tsk,1 );return 0 ; }@@ -164,6 +182,7 @@ void scs_release(struct task_struct *tsk)WARN_ON (scs_corrupted (tsk)); +scs_account (tsk,-1 );task_set_scs (tsk,NULL );scs_free (s); }mm/page_alloc.c +6
-0@@ -5361,6 +5361,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask) " managed:%lukB" " mlocked:%lukB" " kernel_stack:%lukB" +#ifdef CONFIG_SHADOW_CALL_STACK + " shadow_call_stack:%lukB" +#endif" pagetables:%lukB" " bounce:%lukB" " free_pcp:%lukB" @@ -5382,6 +5385,9 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)K (zone_managed_pages (zone)),K (zone_page_state (zone,NR_MLOCK )),zone_page_state (zone,NR_KERNEL_STACK_KB ), +#ifdef CONFIG_SHADOW_CALL_STACK +zone_page_state (zone,NR_KERNEL_SCS_BYTES ) /1024 , +#endifK (zone_page_state (zone,NR_PAGETABLE )),K (zone_page_state (zone,NR_BOUNCE )), K(free_pcp),mm/vmstat.c +3
-0@@ -1118,6 +1118,9 @@ const char * const vmstat_text[] = {"nr_mlock" ,"nr_page_table_pages" ,"nr_kernel_stack" , +#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) +"nr_shadow_call_stack_bytes" , +#endif"nr_bounce" , #if IS_ENABLED(CONFIG_ZSMALLOC) "nr_zspages",Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 7/17 scs: add support for stack usage debugging
In reply to [email protected]Implements CONFIG_DEBUG_STACK_USAGE for shadow stacks.
--- kernel/scs.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
kernel/scs.c +39 -0
@@ -167,6 +167,44 @@ int scs_prepare(struct task_struct *tsk, int node)return 0 ; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long scs_used (struct task_struct *tsk) +{ +unsigned long *p =__scs_base (tsk); +unsigned long *end =scs_magic (tsk); +uintptr_t s = (uintptr_t )p; + +while (p < end && *p) + p++; + +return (uintptr_t )p - s; +} + +static void scs_check_usage (struct task_struct *tsk) +{ +static DEFINE_SPINLOCK (lock); +static unsigned long highest; +unsigned long used =scs_used (tsk); + +if (used <= highest) +return ; + +spin_lock (&lock); + +if (used > highest) { +pr_info ("%s: highest shadow stack usage %lu bytes\n" , + __func__, used); + highest = used; + } + +spin_unlock (&lock); +} +#else +static inline void scs_check_usage (struct task_struct *tsk) +{ +} +#endif +bool scs_corrupted (struct task_struct *tsk) {return *scs_magic (tsk) !=SCS_END_MAGIC ;
@@ -181,6 +219,7 @@ void scs_release(struct task_struct *tsk)return ;WARN_ON (scs_corrupted (tsk)); +scs_check_usage (tsk);scs_account (tsk,-1 );task_set_scs (tsk,NULL );
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 07/17] scs: add support for stack usage debugging
Quoted 1 line · On Thu, Oct 31, 2019 at 09:46:27AM -0700, [email protected] wrote:
Implements
CONFIG_DEBUG_STACK_USAGEfor shadow stacks.
Did I miss it, or is there no Kconfig section for this? I just realized I can't find it. I was going to say "this commit log should explain why/when this option is used", but then figured it might be explained in the Kconfig ... but I couldn't find it. ;)
-Kees
Quoted 66 lines
Signed-off-by Sami Tolvanen <[email protected]>--- kernel/scs.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)kernel/scs.c +39
-0@@ -167,6 +167,44 @@ int scs_prepare(struct task_struct *tsk, int node)return 0 ; } +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long scs_used (struct task_struct *tsk) +{ +unsigned long *p =__scs_base (tsk); +unsigned long *end =scs_magic (tsk); +uintptr_t s = (uintptr_t )p; + +while (p < end && *p) + p++; + +return (uintptr_t )p - s; +} + +static void scs_check_usage (struct task_struct *tsk) +{ +static DEFINE_SPINLOCK (lock); +static unsigned long highest; +unsigned long used =scs_used (tsk); + +if (used <= highest) +return ; + +spin_lock (&lock); + +if (used > highest) { +pr_info ("%s: highest shadow stack usage %lu bytes\n" , + __func__, used); + highest = used; + } + +spin_unlock (&lock); +} +#else +static inline void scs_check_usage (struct task_struct *tsk) +{ +} +#endif +bool scs_corrupted (struct task_struct *tsk) {return *scs_magic (tsk) !=SCS_END_MAGIC ;@@ -181,6 +219,7 @@ void scs_release(struct task_struct *tsk)return ;WARN_ON (scs_corrupted (tsk)); +scs_check_usage (tsk);scs_account (tsk,-1 );task_set_scs (tsk,NULL );Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
Re: [PATCH v3 07/17] scs: add support for stack usage debugging
Quoted 8 lines · Kees Cook
Quoted 1 line · On Thu, Oct 31, 2019 at 09:46:27AM -0700, [email protected] wrote:
Implements
CONFIG_DEBUG_STACK_USAGEfor shadow stacks.Did I miss it, or is there no
Kconfigsection for this? I just realized I can't find it. I was going to say "this commit log should explain why/when this option is used", but then figured it might be explained in theKconfig... but I couldn't find it. ;)
It's in lib/Kconfig.debug. But yes, I will add a commit message in v4.
Sami
Re: [PATCH v3 07/17] scs: add support for stack usage debugging
Quoted 11 lines · Sami Tolvanen
Quoted 8 lines · Kees Cook
Quoted 1 line · On Thu, Oct 31, 2019 at 09:46:27AM -0700, [email protected] wrote:
Implements
CONFIG_DEBUG_STACK_USAGEfor shadow stacks.Did I miss it, or is there no
Kconfigsection for this? I just realized I can't find it. I was going to say "this commit log should explain why/when this option is used", but then figured it might be explained in theKconfig... but I couldn't find it. ;)It's in
lib/Kconfig.debug. But yes, I will add a commit message in v4.
Oh duh -- it's an existing option. Cool; I'm all good. :)
Signature · 1 line
Kees Cook
PATCH v3 8/17 kprobes: fix compilation without CONFIG_KRETPROBES
In reply to [email protected]kprobe_on_func_entry and arch_kprobe_on_func_entry need to be available even if CONFIG_KRETPROBES is not selected.
--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
kernel/kprobes.c +19 -19
@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe
@@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 08/17] kprobes: fix compilation without CONFIG_KRETPROBES
Quoted 4 lines · On Thu, Oct 31, 2019 at 09:46:28AM -0700, [email protected] wrote:
kprobe_on_func_entryandarch_kprobe_on_func_entryneed to be available even ifCONFIG_KRETPROBESis not selected.Signed-off-by Sami Tolvanen <[email protected]>
FWIW:
-Kees
Quoted 64 lines
Acked-by Masami Hiramatsu <[email protected]>--- kernel/kprobes.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)kernel/kprobes.c +19
-19@@ -1829,6 +1829,25 @@ unsigned long __weak arch_deref_entry_point(void *entry)return (unsigned long )entry; } +bool __weak arch_kprobe_on_func_entry (unsigned long offset) +{ +return !offset; +} + +bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) +{ +kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); + +if (IS_ERR (kp_addr)) +return false; + +if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || + !arch_kprobe_on_func_entry (offset)) +return false; + +return true; +} +#ifdef CONFIG_KRETPROBES /* * This kprobe pre_handler is registered with every kretprobe. When probe @@ -1885,25 +1904,6 @@ static int pre_handler_kretprobe(struct kprobe *p, struct pt_regs *regs) }NOKPROBE_SYMBOL (pre_handler_kretprobe); -bool __weak arch_kprobe_on_func_entry (unsigned long offset) -{ -return !offset; -} - -bool kprobe_on_func_entry (kprobe_opcode_t *addr,const char *sym,unsigned long offset) -{ -kprobe_opcode_t *kp_addr =_kprobe_addr (addr, sym, offset); - -if (IS_ERR (kp_addr)) -return false; - -if (!kallsyms_lookup_size_offset ((unsigned long )kp_addr,NULL , &offset) || - !arch_kprobe_on_func_entry (offset)) -return false; - -return true; -} -int register_kretprobe (struct kretprobe *rp) {int ret =0 ;Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 9/17 arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
In reply to [email protected]This allows CONFIG_KRETPROBES to be disabled without disabling kprobes entirely.
--- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kernel/probes/kprobes.c +2 -0
@@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe (struct kretprobe_instance *ri,struct pt_regs *regs) {
@@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) {return 0 ; } +#endif int __init arch_init_kprobes (void ) {
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 09/17] arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
Quoted 4 lines · On Thu, Oct 31, 2019 at 09:46:29AM -0700, [email protected] wrote:
This allows
CONFIG_KRETPROBESto be disabled without disabling kprobes entirely.Signed-off-by Sami Tolvanen <[email protected]>
Is this worth folding this into the prior kprobes patch? Regardless:
-Kees
Quoted 27 lines
--- arch/arm64/kernel/probes/kprobes.c | 2 ++ 1 file changed, 2 insertions(+)arch/arm64/kernel/probes/kprobes.c +2
-0@@ -551,6 +551,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)return (void *)orig_ret_address; } +#ifdef CONFIG_KRETPROBES void __kprobes arch_prepare_kretprobe (struct kretprobe_instance *ri,struct pt_regs *regs) {@@ -564,6 +565,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p) {return 0 ; } +#endif int __init arch_init_kprobes (void ) {Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 10/17 arm64: disable kretprobes with SCS
In reply to [email protected]With CONFIG_KRETPROBES, function return addresses are modified to redirect control flow to kretprobe_trampoline. This is incompatible with SCS.
--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/Kconfig +1 -1
@@ -165,7 +165,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if !SHADOW_CALL_STACK select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAIN
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 10/17] arm64: disable kretprobes with SCS
Quoted 5 lines · On Thu, Oct 31, 2019 at 09:46:30AM -0700, [email protected] wrote:
With
CONFIG_KRETPROBES, function return addresses are modified to redirect control flow tokretprobe_trampoline. This is incompatible with SCS.Signed-off-by Sami Tolvanen <[email protected]>
-Kees
Quoted 20 lines
--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/Kconfig +1
-1@@ -165,7 +165,7 @@ config ARM64 select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS select HAVE_KPROBES - select HAVE_KRETPROBES + select HAVE_KRETPROBES if !SHADOW_CALL_STACK select HAVE_GENERIC_VDSO select IOMMU_DMA if IOMMU_SUPPORT select IRQ_DOMAINSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 11/17 arm64: disable function graph tracing with SCS
In reply to [email protected]With CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified in ftrace_graph_caller and prepare_ftrace_return to redirect control flow to ftrace_return_to_handler. This is incompatible with SCS.
--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/Kconfig +1 -1
@@ -148,7 +148,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if !SHADOW_CALL_STACK select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTING
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 11/17] arm64: disable function graph tracing with SCS
Quoted 4 lines · On Thu, Oct 31, 2019 at 09:46:31AM -0700, [email protected] wrote:
With
CONFIG_FUNCTION_GRAPH_TRACER, function return addresses are modified inftrace_graph_callerandprepare_ftrace_returnto redirect control flow toftrace_return_to_handler. This is incompatible with SCS.
IIRC, the argument was to disable these on a per-arch basis instead of doing it as a "depends on !SHADOW_CALL_STACK" in the top-level function graph tracer Kconfig? (I'm just thinking ahead to doing this again for other architectures, though, I guess, there is much more work than just that for, say, x86.)
Regardless:
-Kees
Quoted 22 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/Kconfig +1
-1@@ -148,7 +148,7 @@ config ARM64 select HAVE_FTRACE_MCOUNT_RECORD select HAVE_FUNCTION_TRACER select HAVE_FUNCTION_ERROR_INJECTION - select HAVE_FUNCTION_GRAPH_TRACER + select HAVE_FUNCTION_GRAPH_TRACER if !SHADOW_CALL_STACK select HAVE_GCC_PLUGINS select HAVE_HW_BREAKPOINT if PERF_EVENTS select HAVE_IRQ_TIME_ACCOUNTINGSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
Re: [PATCH v3 11/17] arm64: disable function graph tracing with SCS
Quoted 3 lines · Kees Cook
IIRC, the argument was to disable these on a per-arch basis instead of doing it as a "depends on !SHADOW_CALL_STACK" in the top-level function graph tracer
Kconfig?
Yes, that's correct.
Quoted 3 lines
(I'm just thinking ahead to doing this again for
other architectures, though, I guess, there is much more work than just that for, say, x86.)
We can always change this later if needed, and possibly figure out how to make function graph tracing and kretprobes work with SCS.
Sami
PATCH v3 12/17 arm64: reserve x18 from general allocation with SCS
In reply to [email protected]Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled.
--- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/Makefile +4 -0
@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 12/17] arm64: reserve x18 from general allocation with SCS
Quoted 5 lines · On Thu, Oct 31, 2019 at 9:47 AM <[email protected]> wrote:
Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled.
Inline/out-of-line assembly will also need to be careful not to accidentally overwrite the current task's shadow stack pointer.
Without automated checking for that, we'll have to rely on a "gentlemen's promise" for code reviewers, though we'd likely hit such a regression with Android+mainline.
Quoted 24 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/Makefile +4
-1@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__ --2.24.0.rc0.303.g954a862665-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v3 12/17] arm64: reserve x18 from general allocation with SCS
In reply to [email protected]Quoted 6 lines · On Thu, Oct 31, 2019 at 09:46:32AM -0700, [email protected] wrote:
Reserve the x18 register from general allocation when SCS is enabled, because the compiler uses the register to store the current task's shadow stack pointer. Note that all external kernel modules must also be compiled with -ffixed-x18 if the kernel has SCS enabled.
Signed-off-by Sami Tolvanen <[email protected]>
-Kees
Quoted 22 lines
--- arch/arm64/Makefile | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/Makefile +4
-0@@ -72,6 +72,10 @@ stack_protector_prepare: prepare0 include/generated/asm-offsets.h)) endif +ifeq ($(CONFIG_SHADOW_CALL_STACK), y) +KBUILD_CFLAGS += -ffixed-x18 +endif + ifeq ($(CONFIG_CPU_BIG_ENDIAN), y) KBUILD_CPPFLAGS += -mbig-endian CHECKFLAGS += -D__AARCH64EB__Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 13/17 arm64: preserve x18 when CPU is suspended
In reply to [email protected]Don't lose the current task's shadow stack when the CPU is suspended.
--- arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/mm/proc.S | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-)
arch/arm64/include/asm/suspend.h +1 -1
@@ -2,7 +2,7 @@#ifndef __ASM_SUSPEND_H#define __ASM_SUSPEND_H -#define NR_CTX_REGS 12 +#define NR_CTX_REGS 13#define NR_CALLEE_SAVED_REGS 12/*
arch/arm64/mm/proc.S +9 -0
@@ -49,6 +49,8 @@ * cpu_do_suspend - save CPU registers context * * x0: virtual address of context pointer + * + * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>. */ ENTRY(cpu_do_suspend) mrs x2, tpidr_el0
@@ -73,6 +75,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #96] +#endif ret ENDPROC(cpu_do_suspend)
@@ -89,6 +94,10 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96] +#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Nick Desaulniers
reviewerRe: [PATCH v3 13/17] arm64: preserve x18 when CPU is suspended
Quoted 46 lines · On Thu, Oct 31, 2019 at 9:47 AM <[email protected]> wrote:
Don't lose the current task's shadow stack when the CPU is suspended.
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/mm/proc.S | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-)arch/arm64/include/asm/suspend.h +1
-1@@ -2,7 +2,7 @@#ifndef __ASM_SUSPEND_H#define __ASM_SUSPEND_H -#define NR_CTX_REGS 12 +#define NR_CTX_REGS 13#define NR_CALLEE_SAVED_REGS 12/* arch/arm64/mm/proc.S +5
-0@@ -49,6 +49,8 @@ * cpu_do_suspend - save CPU registers context * * x0: virtual address of context pointer + * + * This must be kept in sync with struct cpu_suspend_ctx in <asm/suspend.h>. */ ENTRY(cpu_do_suspend) mrs x2, tpidr_el0@@ -73,6 +75,9 @@ alternative_endif stp x8, x9, [x0, #48] stp x10, x11, [x0, #64] stp x12, x13, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #96] +#endif ret ENDPROC(cpu_do_suspend)
arch/arm64/mm/proc.S ENTRY(cpu_do_resume) · line 99
Quoted 6 lines
arch/arm64/mm/proc.S +3
-0@@ -89,6 +94,10 @@ ENTRY(cpu_do_resume) ldp x9, x10, [x0, #48] ldp x11, x12, [x0, #64] ldp x13, x14, [x0, #80] +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96]
How come we zero out x0+#96, but not for other offsets? Is this str necessary?
Quoted 7 lines
arch/arm64/mm/proc.S +1
-1+#endif msr tpidr_el0, x2 msr tpidrro_el0, x3 msr contextidr_el1, x4 --2.24.0.rc0.303.g954a862665-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v3 13/17] arm64: preserve x18 when CPU is suspended
Quoted 5 lines · Nick Desaulniers
arch/arm64/mm/proc.S ENTRY(cpu_do_resume) · line 99
Quoted 3 lines
arch/arm64/mm/proc.S +3
-0+#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96]How come we zero out x0+#96, but not for other offsets? Is this str necessary?
It clears the shadow stack pointer from the sleep state buffer, which is not strictly speaking necessary, but leaves one fewer place to find it.
Sami
Re: [PATCH v3 13/17] arm64: preserve x18 when CPU is suspended
Quoted 7 lines · Sami Tolvanen
Quoted 4 lines · Nick Desaulniers
Quoted 3 lines
arch/arm64/mm/proc.S +3
-0+#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96]
Quoted 5 lines
Quoted 1 line
How come we zero out x0+#96, but not for other offsets? Is this str necessary?
It clears the shadow stack pointer from the sleep state buffer, which is not strictly speaking necessary, but leaves one fewer place to find it.
That sounds like a good idea. Consider adding comments or to the commit message so that the str doesn't get removed accidentally in the future.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v3 13/17] arm64: preserve x18 when CPU is suspended
Quoted 11 lines · Nick Desaulniers
Quoted 9 lines · Sami Tolvanen
Quoted 5 lines · Nick Desaulniers
arch/arm64/mm/proc.S ENTRY(cpu_do_resume) · line 99
Quoted 3 lines
arch/arm64/mm/proc.S +3
-0+#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96]How come we zero out x0+#96, but not for other offsets? Is this str necessary?
Quoted 7 lines
Quoted 3 lines
It clears the shadow stack pointer from the sleep state buffer, which is not strictly speaking necessary, but leaves one fewer place to find it.
That sounds like a good idea. Consider adding comments or to the commit message so that the str doesn't get removed accidentally in the future.
Sure, I'll add a comment in the next version.
Sami
Kees Cook
Reviewed-byRe: [PATCH v3 13/17] arm64: preserve x18 when CPU is suspended
In reply to Nick DesaulniersQuoted 10 lines · Nick Desaulniers
Quoted 9 lines · Sami Tolvanen
Quoted 5 lines · Nick Desaulniers
arch/arm64/mm/proc.S ENTRY(cpu_do_resume) · line 99
Quoted 3 lines
arch/arm64/mm/proc.S +3
-0+#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x0, #96] + str xzr, [x0, #96]How come we zero out x0+#96, but not for other offsets? Is this str necessary?
Quoted 8 lines
Quoted 3 lines
It clears the shadow stack pointer from the sleep state buffer, which is not strictly speaking necessary, but leaves one fewer place to find it.
That sounds like a good idea. Consider adding comments or to the commit message so that the str doesn't get removed accidentally in the future.
Reviewed-by Nick Desaulniers <[email protected]>
Yeah, with the comment added:
-Kees
Signature · 1 line
Kees Cook
PATCH v3 14/17 arm64: efi: restore x18 if it was corrupted
In reply to [email protected]If we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code. This is safe, because the wrapper is called with preemption disabled and a separate shadow stack is used for interrupt handling.
--- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/arm64/kernel/efi-rt-wrapper.S +6 -1
@@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper)
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 14/17] arm64: efi: restore x18 if it was corrupted
Quoted 6 lines · On Thu, Oct 31, 2019 at 09:46:34AM -0700, [email protected] wrote:
If we detect a corrupted x18 and SCS is enabled, restore the register before jumping back to instrumented code. This is safe, because the wrapper is called with preemption disabled and a separate shadow stack is used for interrupt handling.
Signed-off-by Sami Tolvanen <[email protected]>
-Kees
Quoted 23 lines
--- arch/arm64/kernel/efi-rt-wrapper.S | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)arch/arm64/kernel/efi-rt-wrapper.S +6
-1@@ -34,5 +34,10 @@ ENTRY(__efi_rt_asm_wrapper) ldp x29, x30, [sp], #32 b.ne 0f ret -0: b efi_handle_corrupted_x18 // tail call +0: +#ifdef CONFIG_SHADOW_CALL_STACK + /* Restore x18 before returning to instrumented code. */ + mov x18, x2 +#endif + b efi_handle_corrupted_x18 // tail call ENDPROC(__efi_rt_asm_wrapper)Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 15/17 arm64: vdso: disable Shadow Call Stack
In reply to [email protected]--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := n
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 15/17] arm64: vdso: disable Shadow Call Stack
On Thu, Oct 31, 2019 at 9:47 AM <[email protected]> wrote:
I've gotten slapped down before for -ENOCOMMITMSG; maybe include more info if there's a v4? Maintainers can take the safe position of always saying "no," so it is useful to always provide an answer to the implicit question, "why should I take this patch?"
Quoted 11 lines
Signed-off-by Sami Tolvanen <[email protected]>--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/kernel/vdso/Makefile +0
-0
arch/arm64/kernel/vdso/Makefile ccflags-y += -DDISABLE_BRANCH_PROFILING · line 28
Quoted 5 lines
arch/arm64/kernel/vdso/Makefile +1
-1@@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS)
Looks like vgettimeofday is the only remaining source written in C, so we shouldn't need to strip it from other assembly source files.
Quoted 6 lines
arch/arm64/kernel/vdso/Makefile +0
-1KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := n --2.24.0.rc0.303.g954a862665-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Re: [PATCH v3 15/17] arm64: vdso: disable Shadow Call Stack
In reply to [email protected]Quoted 1 line · On Thu, Oct 31, 2019 at 09:46:35AM -0700, [email protected] wrote:
Signed-off-by Sami Tolvanen <[email protected]>
I think Nick already mentioned the missing commit log. With that:
-Kees
Quoted 20 lines
--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/arm64/kernel/vdso/Makefile +1
-1@@ -25,7 +25,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING VDSO_LDFLAGS := -Bsymbolic -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) KBUILD_CFLAGS += $(DISABLE_LTO) KASAN_SANITIZE := n UBSAN_SANITIZE := nSignature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 6 lines
Kees Cook _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v3 16/17 arm64: disable SCS for hypervisor code
In reply to [email protected]Filter out CC_FLAGS_SCS for code that runs at a different exception level.
--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/kvm/hyp/Makefile +3 -0
@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +# remove the SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 16/17] arm64: disable SCS for hypervisor code
Quoted 5 lines · On Thu, Oct 31, 2019 at 09:46:36AM -0700, [email protected] wrote:
Filter out
CC_FLAGS_SCSfor code that runs at a different exception level.Suggested-by Steven Rostedt (VMware) <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>
-Kees
Quoted 18 lines
--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/kvm/hyp/Makefile +3
-0@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +# remove the SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
Re: [PATCH v3 16/17] arm64: disable SCS for hypervisor code
In reply to [email protected]Quoted 5 lines · On Thu, Oct 31, 2019 at 09:46:36AM -0700, [email protected] wrote:
Filter out
CC_FLAGS_SCSfor code that runs at a different exception level.Suggested-by Steven Rostedt (VMware) <[email protected]>Signed-off-by Sami Tolvanen <[email protected]>
(Does it make any sense to merge all of these "disable under SCS" patches? I guess not, since they're each different areas...)
-Kees
Quoted 18 lines
--- arch/arm64/kvm/hyp/Makefile | 3 +++ 1 file changed, 3 insertions(+)arch/arm64/kvm/hyp/Makefile +3
-0@@ -28,3 +28,6 @@ GCOV_PROFILE := n KASAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n + +# remove the SCS flags from all objects in this directory +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS))Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
PATCH v3 17/17 arm64: implement Shadow Call Stack
In reply to [email protected]This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
--- arch/arm64/Kconfig | 5 ++++ arch/arm64/include/asm/scs.h | 37 ++++++++++++++++++++++++++ arch/arm64/include/asm/stacktrace.h | 4 +++ arch/arm64/include/asm/thread_info.h | 3 +++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry.S | 28 ++++++++++++++++++++ arch/arm64/kernel/head.S | 9 +++++++ arch/arm64/kernel/irq.c | 2 ++ arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/scs.c | 39 ++++++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 4 +++ 12 files changed, 137 insertions(+) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c
arch/arm64/Kconfig +5 -0
@@ -66,6 +66,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING
@@ -948,6 +949,10 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_ENABLE_SPLIT_PMD_PTLOCK def_bool y if PGTABLE_LEVELS > 2 +# Supported by clang >= 7.0 +config CC_HAVE_SHADOW_CALL_STACK + def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help---
arch/arm64/include/asm/scs.h new file +37 -0
@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifndef __ASSEMBLY__ + +#include <linux/scs.h> + +#ifdef CONFIG_SHADOW_CALL_STACK + +extern void scs_init_irq (void ); + +static __always_inline void scs_save(struct task_struct *tsk) +{ +void *s; + + asmvolatile ("mov %0, x18" :"=r" (s)); +task_set_scs (tsk, s); +} + +static inline void scs_overflow_check (struct task_struct *tsk) +{ +if (unlikely (scs_corrupted (tsk))) +panic ("corrupted shadow stack detected inside scheduler\n" ); +} + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void scs_init_irq (void ) {} +static inline void scs_save (struct task_struct *tsk) {} +static inline void scs_overflow_check (struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* __ASSEMBLY __ */ + +#endif /* _ASM_SCS_H */
arch/arm64/include/asm/stacktrace.h +4 -0
@@ -68,6 +68,10 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif +static inline bool on_irq_stack (unsigned long sp,struct stack_info *info) {
arch/arm64/include/asm/thread_info.h +3 -0
@@ -41,6 +41,9 @@ struct thread_info {#endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK +void *shadow_call_stack; +#endif };#define thread_saved_pc (tsk) \
arch/arm64/kernel/Makefile +1 -0
@@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/
arch/arm64/kernel/asm-offsets.c +3 -0
@@ -33,6 +33,9 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit));#ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack));#endif DEFINE (TSK_STACK , offsetof(struct task_struct , stack));#ifdef CONFIG_STACKPROTECTOR
arch/arm64/kernel/entry.S +28 -0
@@ -172,6 +172,10 @@ alternative_cb_end apply_ssbd 1, x22, x23 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack + str xzr, [tsk, #TSK_TI_SCS] +#endif .else add x21, sp, #S_FRAME_SIZE get_current_task tsk
@@ -278,6 +282,12 @@ alternative_else_nop_endif ct_user_enter .endif +#ifdef CONFIG_SHADOW_CALL_STACK + .if \el == 0 + str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack + .endif +#endif + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR
@@ -383,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x20, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack.
@@ -400,6 +413,12 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + ldr_this_cpu x18, irq_shadow_call_stack_ptr, x26 +#endif + 9998: .endm
@@ -409,6 +428,10 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + /* x20 is also preserved */ + mov x18, x20 +#endif .endm /* GPRs used by entry code */
@@ -1155,6 +1178,11 @@ ENTRY(cpu_switch_to) ldr lr, [x8] mov sp, x9 msr sp_el0, x1 +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #TSK_TI_SCS] + ldr x18, [x1, #TSK_TI_SCS] + str xzr, [x1, #TSK_TI_SCS] +#endif ret ENDPROC(cpu_switch_to) NOKPROBE(cpu_switch_to)
arch/arm64/kernel/head.S +9 -0
@@ -27,6 +27,7 @@ #include <asm/pgtable-hwdef.h> #include <asm/pgtable.h> #include <asm/page.h> +#include <asm/scs.h> #include <asm/smp.h> #include <asm/sysreg.h> #include <asm/thread_info.h>
@@ -424,6 +425,10 @@ __primary_switched: stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between
@@ -731,6 +736,10 @@ __secondary_switched: ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack + str xzr, [x2, #TSK_TI_SCS] +#endif mov x29, #0 mov x30, #0 b secondary_start_kernel
arch/arm64/kernel/irq.c +2 -0
@@ -21,6 +21,7 @@#include <linux/vmalloc.h> #include <asm/daifflags.h> #include <asm/vmap_stack.h> +#include <asm/scs.h> unsigned long irq_err_count;
@@ -63,6 +64,7 @@ static void init_irq_stacks(void)void __init init_IRQ (void ) {init_irq_stacks (); +scs_init_irq ();irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." );
arch/arm64/kernel/process.c +2 -0
@@ -52,6 +52,7 @@#include <asm/mmu_context.h> #include <asm/processor.h> #include <asm/pointer_auth.h> +#include <asm/scs.h> #include <asm/stacktrace.h> #if defined(CONFIG_STACKPROTECTOR ) && !defined(CONFIG_STACKPROTECTOR_PER_TASK )
@@ -507,6 +508,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,uao_thread_switch (next);ptrauth_thread_switch (next);ssbs_thread_switch (next); +scs_overflow_check (next);/* * Complete any pending TLB or cache maintenance on this CPU in case
arch/arm64/kernel/scs.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/percpu.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); + +#ifndef CONFIG_SHADOW_CALL_STACK_VMAP +DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack) +__aligned(SCS_SIZE) ; +#endif + +void scs_init_irq (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) { +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP +unsigned long *p; + + p =__vmalloc_node_range (SCS_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +SCS_GFP ,PAGE_KERNEL , +0 ,cpu_to_node (cpu), +__builtin_return_address (0 )); + +per_cpu (irq_shadow_call_stack_ptr, cpu) = p; +#else +per_cpu (irq_shadow_call_stack_ptr, cpu) = +per_cpu (irq_shadow_call_stack, cpu); +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + } +}
arch/arm64/kernel/smp.c +4 -0
@@ -44,6 +44,7 @@#include <asm/pgtable.h> #include <asm/pgalloc.h> #include <asm/processor.h> +#include <asm/scs.h> #include <asm/smp_plat.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -357,6 +358,9 @@ void cpu_die(void) {unsigned int cpu =smp_processor_id (); +/* Save the shadow stack pointer before exiting the idle task */ +scs_save (current); +idle_task_exit ();local_daif_mask ();
Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Re: [PATCH v3 17/17] arm64: implement Shadow Call Stack
Quoted 4 lines · On Thu, Oct 31, 2019 at 09:46:37AM -0700, [email protected] wrote:
This change implements shadow stack switching, initial SCS set-up, and interrupt shadow stacks for arm64.
Signed-off-by Sami Tolvanen <[email protected]>
The only suggestion would be calling attention the clearing step (both in comments below and in the commit log), just to record the earlier discussion about it.
-Kees
Quoted 201 lines
--- arch/arm64/Kconfig | 5 ++++ arch/arm64/include/asm/scs.h | 37 ++++++++++++++++++++++++++ arch/arm64/include/asm/stacktrace.h | 4 +++ arch/arm64/include/asm/thread_info.h | 3 +++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry.S | 28 ++++++++++++++++++++ arch/arm64/kernel/head.S | 9 +++++++ arch/arm64/kernel/irq.c | 2 ++ arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/scs.c | 39 ++++++++++++++++++++++++++++ arch/arm64/kernel/smp.c | 4 +++ 12 files changed, 137 insertions(+) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.carch/arm64/Kconfig +5
-0@@ -66,6 +66,7 @@ config ARM64 select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_SUPPORTS_MEMORY_FAILURE + select ARCH_SUPPORTS_SHADOW_CALL_STACK if CC_HAVE_SHADOW_CALL_STACK select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_INT128 if GCC_VERSION >= 50000 || CC_IS_CLANG select ARCH_SUPPORTS_NUMA_BALANCING@@ -948,6 +949,10 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_ENABLE_SPLIT_PMD_PTLOCK def_bool y if PGTABLE_LEVELS > 2 +# Supported by clang >= 7.0 +config CC_HAVE_SHADOW_CALL_STACK + def_bool $(cc-option, -fsanitize=shadow-call-stack -ffixed-x18) + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help---arch/arm64/include/asm/scs.h new file +37
-0@@ -0,0 +1,37 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifndef __ASSEMBLY__ + +#include <linux/scs.h> + +#ifdef CONFIG_SHADOW_CALL_STACK + +extern void scs_init_irq (void ); + +static __always_inline void scs_save(struct task_struct *tsk) +{ +void *s; + + asmvolatile ("mov %0, x18" :"=r" (s)); +task_set_scs (tsk, s); +} + +static inline void scs_overflow_check (struct task_struct *tsk) +{ +if (unlikely (scs_corrupted (tsk))) +panic ("corrupted shadow stack detected inside scheduler\n" ); +} + +#else /* CONFIG_SHADOW_CALL_STACK */ + +static inline void scs_init_irq (void ) {} +static inline void scs_save (struct task_struct *tsk) {} +static inline void scs_overflow_check (struct task_struct *tsk) {} + +#endif /* CONFIG_SHADOW_CALL_STACK */ + +#endif /* __ASSEMBLY __ */ + +#endif /* _ASM_SCS_H */ arch/arm64/include/asm/stacktrace.h +4
-0@@ -68,6 +68,10 @@ extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk); DECLARE_PER_CPU(unsigned long *, irq_stack_ptr); +#ifdef CONFIG_SHADOW_CALL_STACK +DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); +#endif +static inline bool on_irq_stack (unsigned long sp,struct stack_info *info) {arch/arm64/include/asm/thread_info.h +3
-0@@ -41,6 +41,9 @@ struct thread_info {#endif } preempt; }; +#ifdef CONFIG_SHADOW_CALL_STACK +void *shadow_call_stack; +#endif };#define thread_saved_pc (tsk) \arch/arm64/kernel/Makefile +1
-0@@ -63,6 +63,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_SSBD) += ssbd.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o +obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-y += vdso/ probes/ obj-$(CONFIG_COMPAT_VDSO) += vdso32/arch/arm64/kernel/asm-offsets.c +3
-0@@ -33,6 +33,9 @@ int main(void) DEFINE(TSK_TI_ADDR_LIMIT, offsetof(struct task_struct, thread_info.addr_limit));#ifdef CONFIG_ARM64_SW_TTBR0_PAN DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0)); +#endif +#ifdef CONFIG_SHADOW_CALL_STACK + DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack));#endif DEFINE (TSK_STACK , offsetof(struct task_struct , stack));#ifdef CONFIG_STACKPROTECTOR arch/arm64/kernel/entry.S +23
-0@@ -172,6 +172,10 @@ alternative_cb_end apply_ssbd 1, x22, x23 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack + str xzr, [tsk, #TSK_TI_SCS] +#endif .else add x21, sp, #S_FRAME_SIZE get_current_task tsk@@ -278,6 +282,12 @@ alternative_else_nop_endif ct_user_enter .endif +#ifdef CONFIG_SHADOW_CALL_STACK + .if \el == 0 + str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack + .endif +#endif + #ifdef CONFIG_ARM64_SW_TTBR0_PAN /* * Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR@@ -383,6 +393,9 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 .macro irq_stack_entry mov x19, sp // preserve the original sp +#ifdef CONFIG_SHADOW_CALL_STACK + mov x20, x18 // preserve the original shadow stack +#endif /* * Compare sp with the base of the task stack.@@ -400,6 +413,12 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 /* switch to the irq stack */ mov sp, x26 + +#ifdef CONFIG_SHADOW_CALL_STACK + /* also switch to the irq shadow stack */ + ldr_this_cpu x18, irq_shadow_call_stack_ptr, x26 +#endif + 9998: .endm@@ -409,6 +428,10 @@ alternative_insn eret, nop, ARM64_UNMAP_KERNEL_AT_EL0 */ .macro irq_stack_exit mov sp, x19 +#ifdef CONFIG_SHADOW_CALL_STACK + /* x20 is also preserved */ + mov x18, x20 +#endif .endm /* GPRs used by entry code */
arch/arm64/kernel/entry.S ENTRY(cpu_switch_to) · line 1184
Quoted 7 lines
arch/arm64/kernel/entry.S +4
-0@@ -1155,6 +1178,11 @@ ENTRY(cpu_switch_to) ldr lr, [x8] mov sp, x9 msr sp_el0, x1 +#ifdef CONFIG_SHADOW_CALL_STACK + str x18, [x0, #TSK_TI_SCS] + ldr x18, [x1, #TSK_TI_SCS] + str xzr, [x1, #TSK_TI_SCS]
For example here.
Quoted 28 lines
arch/arm64/kernel/entry.S +1
-0+#endif ret ENDPROC(cpu_switch_to) NOKPROBE(cpu_switch_to)arch/arm64/kernel/head.S +5
-0@@ -27,6 +27,7 @@ #include <asm/pgtable-hwdef.h> #include <asm/pgtable.h> #include <asm/page.h> +#include <asm/scs.h> #include <asm/smp.h> #include <asm/sysreg.h> #include <asm/thread_info.h>@@ -424,6 +425,10 @@ __primary_switched: stp xzr, x30, [sp, #-16]! mov x29, sp +#ifdef CONFIG_SHADOW_CALL_STACK + adr_l x18, init_shadow_call_stack // Set shadow call stack +#endif + str_l x21, __fdt_pointer, x5 // Save FDT pointer ldr_l x4, kimage_vaddr // Save the offset between
arch/arm64/kernel/head.S __secondary_switched: · line 741
Quoted 6 lines
arch/arm64/kernel/head.S +3
-0@@ -731,6 +736,10 @@ __secondary_switched: ldr x2, [x0, #CPU_BOOT_TASK] cbz x2, __secondary_too_slow msr sp_el0, x2 +#ifdef CONFIG_SHADOW_CALL_STACK + ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack + str xzr, [x2, #TSK_TI_SCS]
And here. Maybe something like "// Limit visibility of saved SCS"
-Kees
Quoted 114 lines
arch/arm64/kernel/head.S +1
-0+#endif mov x29, #0 mov x30, #0 b secondary_start_kernelarch/arm64/kernel/irq.c +2
-0@@ -21,6 +21,7 @@#include <linux/vmalloc.h> #include <asm/daifflags.h> #include <asm/vmap_stack.h> +#include <asm/scs.h> unsigned long irq_err_count;@@ -63,6 +64,7 @@ static void init_irq_stacks(void)void __init init_IRQ (void ) {init_irq_stacks (); +scs_init_irq ();irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." );arch/arm64/kernel/process.c +2
-0@@ -52,6 +52,7 @@#include <asm/mmu_context.h> #include <asm/processor.h> #include <asm/pointer_auth.h> +#include <asm/scs.h> #include <asm/stacktrace.h> #if defined(CONFIG_STACKPROTECTOR ) && !defined(CONFIG_STACKPROTECTOR_PER_TASK )@@ -507,6 +508,7 @@ __notrace_funcgraph struct task_struct *__switch_to(struct task_struct *prev,uao_thread_switch (next);ptrauth_thread_switch (next);ssbs_thread_switch (next); +scs_overflow_check (next);/* * Complete any pending TLB or cache maintenance on this CPU in case arch/arm64/kernel/scs.c new file +39
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Shadow Call Stack support. +* +* Copyright (C) 2019 Google LLC +*/ + +#include <linux/percpu.h> +#include <linux/vmalloc.h> +#include <asm/scs.h> + +DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr); + +#ifndef CONFIG_SHADOW_CALL_STACK_VMAP +DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack) +__aligned(SCS_SIZE) ; +#endif + +void scs_init_irq (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) { +#ifdef CONFIG_SHADOW_CALL_STACK_VMAP +unsigned long *p; + + p =__vmalloc_node_range (SCS_SIZE ,SCS_SIZE , +VMALLOC_START ,VMALLOC_END , +SCS_GFP ,PAGE_KERNEL , +0 ,cpu_to_node (cpu), +__builtin_return_address (0 )); + +per_cpu (irq_shadow_call_stack_ptr, cpu) = p; +#else +per_cpu (irq_shadow_call_stack_ptr, cpu) = +per_cpu (irq_shadow_call_stack, cpu); +#endif /* CONFIG_SHADOW_CALL_STACK_VMAP */ + } +}arch/arm64/kernel/smp.c +4
-0@@ -44,6 +44,7 @@#include <asm/pgtable.h> #include <asm/pgalloc.h> #include <asm/processor.h> +#include <asm/scs.h> #include <asm/smp_plat.h> #include <asm/sections.h> #include <asm/tlbflush.h> @@ -357,6 +358,9 @@ void cpu_die(void) {unsigned int cpu =smp_processor_id (); +/* Save the shadow stack pointer before exiting the idle task */ +scs_save (current); +idle_task_exit ();local_daif_mask ();Signature · 1 line
2.24.0.rc0.303.g954a862665-goog
Signature · 1 line
Kees Cook
Re: [PATCH v3 17/17] arm64: implement Shadow Call Stack
Quoted 3 lines · Kees Cook
The only suggestion would be calling attention the clearing step (both in comments below and in the commit log), just to record the earlier discussion about it.
Sure, makes sense. I'll add comments about this in v4.
Sami
v4 cover add support for Clang's Shadow Call Stack
In reply to Sami TolvanenThis patch series adds support for Clang's Shadow Call Stack (SCS) mitigation, which uses a separately allocated shadow stack to protect against return address overwrites. More information can be found here:
https://clang.llvm.org/docs/ShadowCallStack.html
SCS provides better protection against traditional buffer overflows than CONFIG_STACKPROTECTOR_*, but it should be noted that SCS security guarantees in the kernel differ from the ones documented for user space. The kernel must store addresses of shadow stacks used by other tasks and interrupt handlers in memory, which means an attacker capable reading and writing arbitrary memory may be able to locate them and hijack control flow by modifying shadow stacks that are not currently in use.
SCS is currently supported only on arm64, where the compiler requires the x18 register to be reserved for holding the current task's shadow stack pointer. Because of this, the series includes patches from Ard to remove x18 usage from assembly code.
With -fsanitize=shadow-call-stack, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack, and unconditionally load it again before returning. As a result, SCS is currently incompatible with features that rely on modifying function return addresses to alter control flow, such as function graph tracing and kretprobes, although it may be possible to later change these features to modify the shadow stack instead. A copy of the return address is still kept in the kernel stack for compatibility with stack unwinding, for example.
SCS has a minimal performance overhead, but allocating shadow stacks increases kernel memory usage. The feature is therefore mostly useful on hardware that lacks support for PAC instructions.
Changes in v4:
- Fixed authorship for Ard's patches
- Added missing commit messages
- Commented code that clears SCS from thread_info
- Added a comment about SCS_END_MAGIC being non-canonical
Changes in v3:
- Switched to filter-out for removing SCS flags in Makefiles
- Changed the __noscs attribute to use __no_sanitize__("...") instead of no_sanitize("...")
- Cleaned up inline function definitions and moved task_scs() into a macro
- Cleaned up scs_free() and scs_magic()
- Moved SCS initialization into dup_task_struct() and removed the now unused scs_task_init()
- Added comments to __scs_base() and scs_task_reset() to better document design choices
- Changed copy_page to make the offset and bias explicit
Changes in v2:
- Changed Ard's KVM patch to use x29 instead of x18 for the guest context, which makes restore_callee_saved_regs cleaner
- Updated help text (and commit messages) to point out differences in security properties compared to user space SCS
- Cleaned up config options: removed the ROP protection choice, replaced the CC_IS_CLANG dependency with an arch-specific cc-option test, and moved disabling of incompatible config options to an arch-specific Kconfig
- Added CC_FLAGS_SCS, which are filtered out where needed instead of using DISABLE_SCS
- Added a __has_feature guard around __noscs for older clang versions
Ard Biesheuvel (3):
arm64/lib: copy_page: avoid x18 register in assembler code
arm64: kvm: stop treating register x18 as caller save
arm64: kernel: avoid x18 __cpu_soft_restart
Sami Tolvanen (14):
arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings add support for Clang's Shadow Call Stack (SCS)
scs: add accounting
scs: add support for stack usage debugging
kprobes: fix compilation without CONFIG_KRETPROBES
arm64: kprobes: fix kprobes without CONFIG_KRETPROBES
arm64: disable kretprobes with SCS
arm64: disable function graph tracing with SCS
arm64: reserve x18 from general allocation with SCS
arm64: preserve x18 when CPU is suspended
arm64: efi: restore x18 if it was corrupted
arm64: vdso: disable Shadow Call Stack
arm64: disable SCS for hypervisor code
arm64: implement Shadow Call Stack
Makefile | 6 + arch/Kconfig | 33 ++++ arch/arm64/Kconfig | 9 +- arch/arm64/Makefile | 4 + arch/arm64/include/asm/scs.h | 37 +++++ arch/arm64/include/asm/stacktrace.h | 4 + arch/arm64/include/asm/suspend.h | 2 +- arch/arm64/include/asm/thread_info.h | 3 + arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/asm-offsets.c | 3 + arch/arm64/kernel/cpu-reset.S | 4 +- arch/arm64/kernel/efi-rt-wrapper.S | 7 +- arch/arm64/kernel/entry.S | 28 ++++ arch/arm64/kernel/head.S | 9 ++ arch/arm64/kernel/irq.c | 2 + arch/arm64/kernel/probes/kprobes.c | 2 + arch/arm64/kernel/process.c | 2 + arch/arm64/kernel/scs.c | 39 +++++ arch/arm64/kernel/smp.c | 4 + arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kvm/hyp/Makefile | 3 + arch/arm64/kvm/hyp/entry.S | 41 +++-- arch/arm64/lib/copy_page.S | 38 ++--- arch/arm64/mm/proc.S | 73 +++++---- drivers/base/node.c | 6 + fs/proc/meminfo.c | 4 + include/linux/compiler-clang.h | 6 + include/linux/compiler_types.h | 4 + include/linux/mmzone.h | 3 + include/linux/scs.h | 57 +++++++ init/init_task.c | 8 + kernel/Makefile | 1 + kernel/fork.c | 9 ++ kernel/kprobes.c | 38 ++--- kernel/sched/core.c | 2 + kernel/sched/sched.h | 1 + kernel/scs.c | 227 +++++++++++++++++++++++++++ mm/page_alloc.c | 6 + mm/vmstat.c | 3 + 39 files changed, 634 insertions(+), 97 deletions(-) create mode 100644 arch/arm64/include/asm/scs.h create mode 100644 arch/arm64/kernel/scs.c create mode 100644 include/linux/scs.h create mode 100644 kernel/scs.c
base-commit: 0dbe6cb8f7e05bc9611602ef45980a6c57b245a3
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v4 1/17 arm64: mm: avoid x18 in idmap_kpti_install_ng_mappings
idmap_kpti_install_ng_mappings uses x18 as a temporary register, which will result in a conflict when x18 is reserved. Use x16 and x17 instead where needed.
--- arch/arm64/mm/proc.S | 63 ++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 31 deletions(-)
arch/arm64/mm/proc.S +32 -31
@@ -250,15 +250,15 @@ ENTRY(idmap_kpti_install_ng_mappings) /* We're the boot CPU. Wait for the others to catch up */ sevl 1: wfe - ldaxr w18, [flag_ptr] - eor w18, w18, num_cpus - cbnz w18, 1b + ldaxr w17, [flag_ptr] + eor w17, w17, num_cpus + cbnz w17, 1b /* We need to walk swapper, so turn off the MMU. */ pre_disable_mmu_workaround - mrs x18, sctlr_el1 - bic x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + bic x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /* Everybody is enjoying the idmap, so we can rewrite swapper. */
@@ -281,9 +281,9 @@ skip_pgd: isb /* We're done: fire up the MMU again */ - mrs x18, sctlr_el1 - orr x18, x18, #SCTLR_ELx_M - msr sctlr_el1, x18 + mrs x17, sctlr_el1 + orr x17, x17, #SCTLR_ELx_M + msr sctlr_el1, x17 isb /*
@@ -353,46 +353,47 @@ skip_pte: b.ne do_pte b next_pmd + .unreq cpu + .unreq num_cpus + .unreq swapper_pa + .unreq cur_pgdp + .unreq end_pgdp + .unreq pgd + .unreq cur_pudp + .unreq end_pudp + .unreq pud + .unreq cur_pmdp + .unreq end_pmdp + .unreq pmd + .unreq cur_ptep + .unreq end_ptep + .unreq pte + /* Secondary CPUs end up here */ __idmap_kpti_secondary: /* Uninstall swapper before surgery begins */ - __idmap_cpu_set_reserved_ttbr1 x18, x17 + __idmap_cpu_set_reserved_ttbr1 x16, x17 /* Increment the flag to let the boot CPU we're ready */ -1: ldxr w18, [flag_ptr] - add w18, w18, #1 - stxr w17, w18, [flag_ptr] +1: ldxr w16, [flag_ptr] + add w16, w16, #1 + stxr w17, w16, [flag_ptr] cbnz w17, 1b /* Wait for the boot CPU to finish messing around with swapper */ sevl 1: wfe - ldxr w18, [flag_ptr] - cbnz w18, 1b + ldxr w16, [flag_ptr] + cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x18 + offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret - .unreq cpu - .unreq num_cpus - .unreq swapper_pa .unreq swapper_ttb .unreq flag_ptr - .unreq cur_pgdp - .unreq end_pgdp - .unreq pgd - .unreq cur_pudp - .unreq end_pudp - .unreq pud - .unreq cur_pmdp - .unreq end_pmdp - .unreq pmd - .unreq cur_ptep - .unreq end_ptep - .unreq pte ENDPROC(idmap_kpti_install_ng_mappings) .popsection #endif
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v4 2/17 arm64/lib: copy_page: avoid x18 register in assembler code
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
Register x18 will no longer be used as a caller save register in the future, so stop using it in the copy_page() code.
--- arch/arm64/lib/copy_page.S | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-)
arch/arm64/lib/copy_page.S +19 -19
@@ -34,45 +34,45 @@ alternative_else_nop_endif ldp x14, x15, [x1, #96] ldp x16, x17, [x1, #112] - mov x18, #(PAGE_SIZE - 128) + add x0, x0, #256 add x1, x1, #128 1: - subs x18, x18, #128 + tst x0, #(PAGE_SIZE - 1) alternative_if ARM64_HAS_NO_HW_PREFETCH prfm pldl1strm, [x1, #384] alternative_else_nop_endif - stnp x2, x3, [x0] + stnp x2, x3, [x0, #-256] ldp x2, x3, [x1] - stnp x4, x5, [x0, #16] + stnp x4, x5, [x0, #16 - 256] ldp x4, x5, [x1, #16] - stnp x6, x7, [x0, #32] + stnp x6, x7, [x0, #32 - 256] ldp x6, x7, [x1, #32] - stnp x8, x9, [x0, #48] + stnp x8, x9, [x0, #48 - 256] ldp x8, x9, [x1, #48] - stnp x10, x11, [x0, #64] + stnp x10, x11, [x0, #64 - 256] ldp x10, x11, [x1, #64] - stnp x12, x13, [x0, #80] + stnp x12, x13, [x0, #80 - 256] ldp x12, x13, [x1, #80] - stnp x14, x15, [x0, #96] + stnp x14, x15, [x0, #96 - 256] ldp x14, x15, [x1, #96] - stnp x16, x17, [x0, #112] + stnp x16, x17, [x0, #112 - 256] ldp x16, x17, [x1, #112] add x0, x0, #128 add x1, x1, #128 - b.gt 1b + b.ne 1b - stnp x2, x3, [x0] - stnp x4, x5, [x0, #16] - stnp x6, x7, [x0, #32] - stnp x8, x9, [x0, #48] - stnp x10, x11, [x0, #64] - stnp x12, x13, [x0, #80] - stnp x14, x15, [x0, #96] - stnp x16, x17, [x0, #112] + stnp x2, x3, [x0, #-256] + stnp x4, x5, [x0, #16 - 256] + stnp x6, x7, [x0, #32 - 256] + stnp x8, x9, [x0, #48 - 256] + stnp x10, x11, [x0, #64 - 256] + stnp x12, x13, [x0, #80 - 256] + stnp x14, x15, [x0, #96 - 256] + stnp x16, x17, [x0, #112 - 256] ret ENDPROC(copy_page)
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
PATCH v4 3/17 arm64: kvm: stop treating register x18 as caller save
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to __guest_enter(). Instead, preserve its value and restore it upon return.
--- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-)
arch/arm64/kvm/hyp/entry.S +20 -21
@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax" .macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28 + ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1
@@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret
@@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
@@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Re: [PATCH v4 03/17] arm64: kvm: stop treating register x18 as caller save
Hi Sami,
Quoted 8 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.
Quoted 6 lines
[Sami: updated commit message, switched from x18 to x29 for the guest context]
Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Kees Cook <[email protected]>
If you intend for this to be merged via the arm64 tree, please add my
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v4 03/17] arm64: kvm: stop treating register x18 as caller save
Quoted 18 lines · Marc Zyngier
Hi Sami,
Quoted 8 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.Quoted 6 lines
[Sami: updated commit message, switched from x18 to x29 for the guest context]
Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Kees Cook <[email protected]>
Quoted 3 lines
If you intend for this to be merged via the arm64 tree, please add my
Reviewed-by Marc Zyngier <[email protected]>
Erm... Muscle memory strikes again. Please ignore the above and use the following instead:
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v4 03/17] arm64: kvm: stop treating register x18 as caller save
In reply to Sami TolvanenQuoted 22 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
In preparation of reserving x18, stop treating it as caller save in the KVM guest entry/exit code. Currently, the code assumes there is no need to preserve it for the host, given that it would have been assumed clobbered anyway by the function call to
__guest_enter(). Instead, preserve its value and restore it upon return.Signed-off-by Ard Biesheuvel <[email protected]>[Sami: updated commit message, switched from x18 to x29 for the guest context]Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Kees Cook <[email protected]>--- arch/arm64/kvm/hyp/entry.S | 41 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 21 deletions(-)arch/arm64/kvm/hyp/entry.S +0
-0
arch/arm64/kvm/hyp/entry.S line 24
Quoted 2 lines
arch/arm64/kvm/hyp/entry.S +0
-0@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax"
Could we please add a note here, e.g.
/*
* We treat x18 as callee-saved as the host may use it as a platform
* register (e.g. for shadow call stack).
*/
... as that will avoid anyone trying to optimize this away in future after reading the AAPCS.
arch/arm64/kvm/hyp/entry.S line 36
Quoted 10 lines
arch/arm64/kvm/hyp/entry.S +2
-0.macro save_callee_saved_regs ctxt + str x18, [\ctxt, #CPU_XREG_OFFSET(18)] stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -32,6 +33,8 @@ .endm .macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28
Probably worth s/assume/require/ here.
Otherwise, this looks godo to me:
Mark.
Quoted 82 lines
arch/arm64/kvm/hyp/entry.S +18
-21+ ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)] ldp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)] ldp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)] ldp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]@@ -48,7 +51,7 @@ ENTRY(__guest_enter) // x0: vcpu // x1: host context // x2-x17: clobbered by macros - // x18: guest context + // x29: guest context // Store the host regs save_callee_saved_regs x1@@ -67,31 +70,28 @@ alternative_else_nop_endif ret 1: - add x18, x0, #VCPU_CONTEXT + add x29, x0, #VCPU_CONTEXT // Macro ptrauth_switch_to_guest format: // ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3) // The below macro to restore guest keys is not implemented in C code // as it may cause Pointer Authentication key signing mismatch errors // when this feature is enabled for kernel code. - ptrauth_switch_to_guest x18, x0, x1, x2 + ptrauth_switch_to_guest x29, x0, x1, x2 // Restore guest regs x0-x17 - ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] - ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] - ldp x4, x5, [x18, #CPU_XREG_OFFSET(4)] - ldp x6, x7, [x18, #CPU_XREG_OFFSET(6)] - ldp x8, x9, [x18, #CPU_XREG_OFFSET(8)] - ldp x10, x11, [x18, #CPU_XREG_OFFSET(10)] - ldp x12, x13, [x18, #CPU_XREG_OFFSET(12)] - ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)] - ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)] - - // Restore guest regs x19-x29, lr - restore_callee_saved_regs x18 - - // Restore guest reg x18 - ldr x18, [x18, #CPU_XREG_OFFSET(18)] + ldp x0, x1, [x29, #CPU_XREG_OFFSET(0)] + ldp x2, x3, [x29, #CPU_XREG_OFFSET(2)] + ldp x4, x5, [x29, #CPU_XREG_OFFSET(4)] + ldp x6, x7, [x29, #CPU_XREG_OFFSET(6)] + ldp x8, x9, [x29, #CPU_XREG_OFFSET(8)] + ldp x10, x11, [x29, #CPU_XREG_OFFSET(10)] + ldp x12, x13, [x29, #CPU_XREG_OFFSET(12)] + ldp x14, x15, [x29, #CPU_XREG_OFFSET(14)] + ldp x16, x17, [x29, #CPU_XREG_OFFSET(16)] + + // Restore guest regs x18-x29, lr + restore_callee_saved_regs x29 // Do not touch any register after this! eret@@ -114,7 +114,7 @@ ENTRY(__guest_exit) // Retrieve the guest regs x0-x1 from the stack ldp x2, x3, [sp], #16 // x0, x1 - // Store the guest regs x0-x1 and x4-x18 + // Store the guest regs x0-x1 and x4-x17 stp x2, x3, [x1, #CPU_XREG_OFFSET(0)] stp x4, x5, [x1, #CPU_XREG_OFFSET(4)] stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]@@ -123,9 +123,8 @@ ENTRY(__guest_exit) stp x12, x13, [x1, #CPU_XREG_OFFSET(12)] stp x14, x15, [x1, #CPU_XREG_OFFSET(14)] stp x16, x17, [x1, #CPU_XREG_OFFSET(16)] - str x18, [x1, #CPU_XREG_OFFSET(18)] - // Store the guest regs x19-x29, lr + // Store the guest regs x18-x29, lr save_callee_saved_regs x1 get_host_ctxt x2, x3Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Re: [PATCH v4 03/17] arm64: kvm: stop treating register x18 as caller save
Quoted 8 lines · Mark Rutland
arch/arm64/kvm/hyp/entry.S line 24
Quoted 5 lines
arch/arm64/kvm/hyp/entry.S +0
-0@@ -23,6 +23,7 @@ .pushsection .hyp.text, "ax"Could we please add a note here, e.g.
Quoted 7 lines
/*
* We treat x18 as callee-saved as the host may use it as a platform
* register (e.g. for shadow call stack).
*/... as that will avoid anyone trying to optimize this away in future after reading the AAPCS.
Sure, that's a good idea.
Quoted 4 lines
arch/arm64/kvm/hyp/entry.S line 36
Quoted 2 lines
arch/arm64/kvm/hyp/entry.S +1
-0.macro restore_callee_saved_regs ctxt + // We assume \ctxt is not x18-x28Probably worth s/assume/require/ here.
Agreed, I'll change this in v5.
Sami
PATCH v4 4/17 arm64: kernel: avoid x18 __cpu_soft_restart
In reply to Sami TolvanenFrom: Ard Biesheuvel <[email protected]>
The code in __cpu_soft_restart() uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/cpu-reset.S +2 -2
@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsection
Signature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Mark Rutland
reviewerRe: [PATCH v4 04/17] arm64: kernel: avoid x18 __cpu_soft_restart
Quoted 11 lines · Sami Tolvanen
From: Ard Biesheuvel <[email protected]>
The code in
__cpu_soft_restart()uses x18 as an arbitrary temp register, which will shortly be disallowed. So use x8 instead.Signed-off-by Ard Biesheuvel <[email protected]>[Sami: updated commit message]Signed-off-by Sami Tolvanen <[email protected]>Reviewed-by Mark Rutland <[email protected]>Reviewed-by Kees Cook <[email protected]>
Trivial nit, but the commit title is missing "in" between x18 and __cpu_soft_restart.
Mark.
Quoted 25 lines
--- arch/arm64/kernel/cpu-reset.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/arm64/kernel/cpu-reset.S +2
-2@@ -42,11 +42,11 @@ ENTRY(__cpu_soft_restart) mov x0, #HVC_SOFT_RESTART hvc #0 // no return -1: mov x18, x1 // entry +1: mov x8, x1 // entry mov x0, x2 // arg0 mov x1, x3 // arg1 mov x2, x4 // arg2 - br x18 + br x8 ENDPROC(__cpu_soft_restart) .popsectionSignature · 1 line
2.24.0.rc1.363.gb1bccd3e3d-goog
Re: [PATCH v4 04/17] arm64: kernel: avoid x18 __cpu_soft_restart
Quoted 2 lines · Mark Rutland
Trivial nit, but the commit title is missing "in" between x18 and
__cpu_soft_restart.
Oops, thanks for pointing that out. I'll fix this in v5.
Sami
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel