Skip to contentSkip to conversation

riscv kernel control flow integrity

RFC PATCH New Deepak Gupta 3 participants 21 messages Apr 9, 2024 – May 13, 2024 RISC-V

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 riscv kernel control flow integrity

Basic overview
---------------
This is a RFC patch series for enabling kernel control flow integrity on riscv architecture. This patch series enables kernel control flow integrity using proposed riscv cpu extensions zicfilp and zicfiss [1].

zicfilp enforces that all indirect calls and jumps must land on a landing pad instruction (lpad). Additionally lpad has 20bit encoded value as part of instruction and cpu will check this 20bit value with t2/x7 register , if they mismatch then cpu will raise an exception software check exception (a new exception with cause=18). In this patch series, a constant label value of 0x1 is used. As series will mature, it will switch to a 20 bit truncated hash over function signature. Label based on function signature allows stricter control flow and fewer call/jmp locations from a callsite.

zicfiss protects the return path from functions where return relies on obtaining return address from stack which is corruptible. zicfiss provides a shadow stack which can be used by software to place return addresses on shadow stack and while returning from function it can be used to compare against return address from regular stack. If they dont match, cpu will raise software check exception. zicfiss based shadow stack are protected against tampering using special page table encodings (please refer to [1])

To obtain more details about zicfiss and zicfilp ISA extension, please refer to [1]. There is an ongoing patchsets for enabling this feature for user mode software here [2]

Enabling on kernel
===================
This patch series introduces new riscv config CONFIG_RISCV_KERNEL_CFI. If this config is selected, it turns on
- forward control flow for kernel using zicfilp
- selects CONFIG_SHADOW_CALL_STACK /w CONFIG_DYNAMIC_SCS to enable backward control flow.

forward control flow for kernel
================================
This patch series simply compiles kernel with march=_zicfilp compiler option. Currently toolchain uses constant label scheme of label = 0x1. This patch series manually fixes some of the assembly callsites and sequences to make sure they are not breaking rules setup by zicfilp.

backward control flow for kernel
=================================
There is an existing support for riscv kernel for shadow call stack [3], which is a software based shadow stack and uses clang /w instrumentation to push/pop return address in prolog and epilog of functions. Although software based shadow stack lacks memory protections and thus suffers from same issue of return address susceptible to hijacking. shadow call stack uses CONFIG_SHADOW_CALL_STACK /w option of CONFIG_DYNAMIC_SCS so that hardware vendors hook into the flow to provide stronger guarantees. This patch uses CONFIG_SHADOW_CALL_STACK flow along with CONFIG_DYNAMIC_SCS to enable return control flow integrity on riscv kernel.

[1] - https://github.com/riscv/riscv-cfi
[2] - https://lore.kernel.org/all/[email protected]/
[3] - https://lore.kernel.org/all/[email protected]/

#1 lore raw

RFC PATCH 1/12 riscv: zicfiss / zicfilp extension csr and bit definitions

zicfiss and zicfilp extension gets enabled via b3 and b2 in *envcfg CSR. menvcfg controls enabling for S/HS mode. henvcfg control enabling for VS while senvcfg controls enabling for U/VU mode.

zicfilp extension extends *status CSR to hold expected landing pad bit. A trap or interrupt can occur between an indirect jmp/call and target instr. expected landing pad bit from CPU is recorded into xstatus CSR so that when supervisor performs xret, expected landing pad state of CPU can be restored.

zicfiss adds one new CSR
- CSR_SSP: CSR_SSP contains current shadow stack pointer.

---
 arch/riscv/include/asm/csr.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
arch/riscv/include/asm/csr.h +16 -0
index 2468c55933cd..9f2b2722b67c 100644
@@ -18,6 +18,15 @@
 #define SR_MPP		_AC(0x00001800, UL) /* Previously Machine */
 #define SR_SUM		_AC(0x00040000, UL) /* Supervisor User Memory Access */
 
+/* zicfilp landing pad status bit */
+#define SR_SPELP	_AC(0x00800000, UL)
+#define SR_MPELP	_AC(0x020000000000, UL)
+#ifdef CONFIG_RISCV_M_MODE
+#define SR_ELP		SR_MPELP
+#else
+#define SR_ELP		SR_SPELP
+#endif
+
 #define SR_FS		_AC(0x00006000, UL) /* Floating-point Status */
 #define SR_FS_OFF	_AC(0x00000000, UL)
 #define SR_FS_INITIAL	_AC(0x00002000, UL)
@@ -196,6 +205,8 @@
 #define ENVCFG_PBMTE			(_AC(1, ULL) << 62)
 #define ENVCFG_CBZE			(_AC(1, UL) << 7)
 #define ENVCFG_CBCFE			(_AC(1, UL) << 6)
+#define ENVCFG_LPE			(_AC(1, UL) << 2)
+#define ENVCFG_SSE			(_AC(1, UL) << 3)
 #define ENVCFG_CBIE_SHIFT		4
 #define ENVCFG_CBIE			(_AC(0x3, UL) << ENVCFG_CBIE_SHIFT)
 #define ENVCFG_CBIE_ILL			_AC(0x0, UL)
@@ -214,6 +225,11 @@
 #define SMSTATEEN0_HSENVCFG		(_ULL(1) << SMSTATEEN0_HSENVCFG_SHIFT)
 #define SMSTATEEN0_SSTATEEN0_SHIFT	63
 #define SMSTATEEN0_SSTATEEN0		(_ULL(1) << SMSTATEEN0_SSTATEEN0_SHIFT)
+/*
+ * zicfiss user mode csr
+ * CSR_SSP holds current shadow stack pointer.
+ */
+#define CSR_SSP                 0x011
 
 /* symbolic CSR names: */
 #define CSR_CYCLE		0xc00
Signature · 1 line
2.43.2
#2 lore raw

RFC PATCH 2/12 riscv: add landing pad for asm routines.

In reply to Deepak Gupta

SYM_* macros are used to define assembly routines. In this patch series, re-define those macros in risc-v arch specific include file to include a landing pad instruction at the beginning. This is done only when the compiler flag for landing pad is enabled (i.e. __riscv_zicfilp).

---
 arch/riscv/include/asm/linkage.h | 42 ++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
arch/riscv/include/asm/linkage.h +42 -0
index 9e88ba23cd2b..bb43ae7dadeb 100644
@@ -6,7 +6,49 @@
 #ifndef _ASM_RISCV_LINKAGE_H
 #define _ASM_RISCV_LINKAGE_H
 
+#ifdef __ASSEMBLY__
+#include <asm/assembler.h>
+#endif
+
 #define __ALIGN		.balign 4
 #define __ALIGN_STR	".balign 4"
 
+#ifdef __riscv_zicfilp
+/*
+ * A landing pad instruction is needed at start of asm routines
+ * re-define macros for asm routines to have a landing pad at
+ * the beginning of function. Currently use label value of 0x1.
+ * Eventually, label should be calculated as a hash over function
+ * signature.
+ */
+#define SYM_FUNC_START(name)				\
+	SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)	\
+	lpad 0x1;
+
+#define SYM_FUNC_START_NOALIGN(name)			\
+	SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE)	\
+	lpad 0x1;
+
+#define SYM_FUNC_START_LOCAL(name)			\
+	SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN)	\
+	lpad 0x1;
+
+#define SYM_FUNC_START_LOCAL_NOALIGN(name)		\
+	SYM_START(name, SYM_L_LOCAL, SYM_A_NONE)	\
+	lpad 0x1;
+
+#define SYM_FUNC_START_WEAK(name)			\
+	SYM_START(name, SYM_L_WEAK, SYM_A_ALIGN)	\
+	lpad 0x1;
+
+#define SYM_FUNC_START_WEAK_NOALIGN(name)		\
+	SYM_START(name, SYM_L_WEAK, SYM_A_NONE)		\
+	lpad 0x1;
+
+#define SYM_TYPED_FUNC_START(name)				\
+	SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)	\
+	lpad 0x1;
+
+#endif
+
 #endif /* _ASM_RISCV_LINKAGE_H */
Signature · 1 line
2.43.2
#6 lore raw

RFC PATCH 3/12 riscv: after saving expected landing pad (elp), clear elp state

In reply to Deepak Gupta

On trap entry, save expected landing pad state and subsequently clear it in sstatus so that if there are traps later on in kernel and sret happens back to same mode, cpu will start faulting.

---
 arch/riscv/kernel/entry.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/kernel/entry.S +1 -1
index 68a24cf9481a..be07355b9eff 100644
@@ -56,7 +56,7 @@ SYM_CODE_START(handle_exception)
 	 * Disable the FPU/Vector to detect illegal usage of floating point
 	 * or vector in kernel space.
 	 */
-	li t0, SR_SUM | SR_FS_VS
+	li t0, SR_SUM | SR_FS_VS | SR_ELP
 
 	REG_L s0, TASK_TI_USER_SP(tp)
 	csrrc s1, CSR_STATUS, t0
Signature · 1 line
2.43.2
#7 lore raw

RFC PATCH 4/12 riscv: update asm call sites with label setup

In reply to Deepak Gupta

All call sites written in asm which will be converted to indirect call form, they need to setup label register (t2/x7) with correct label.

Currently kernel is enabled with consant label of 0x1 for all functions. Thus label is setup with 0x1 at call site. Eventually when hash over function signature based label is adopted, such callsites in asm needs to b updated as well. We need better scheme for that (some macro)

---
 arch/riscv/kernel/entry.S   | 2 ++
 arch/riscv/kernel/head.S    | 1 +
 arch/riscv/lib/clear_page.S | 1 +
 3 files changed, 4 insertions(+)
arch/riscv/kernel/entry.S +2 -0
index be07355b9eff..a35050a3e0ea 100644
@@ -219,6 +219,7 @@ SYM_CODE_START_LOCAL(handle_kernel_stack_overflow)
 	REG_S s4, PT_CAUSE(sp)
 	REG_S s5, PT_TP(sp)
 	move a0, sp
+	lui t2,0x1
 	tail handle_bad_stack
 SYM_CODE_END(handle_kernel_stack_overflow)
 ASM_NOKPROBE(handle_kernel_stack_overflow)
@@ -258,6 +259,7 @@ SYM_FUNC_START(call_on_irq_stack)
 	load_per_cpu t0, irq_stack_ptr, t1
 	li	t1, IRQ_STACK_SIZE
 	add	sp, t0, t1
+	lui t2, 0x1
 	jalr	a1
 
 	/* Switch back to the thread shadow call stack */
arch/riscv/kernel/head.S +1 -0
index 4236a69c35cb..6c311517c3b5 100644
@@ -165,6 +165,7 @@ secondary_start_sbi:
 #endif
 	call .Lsetup_trap_vector
 	scs_load_current
+	lui t2, 0x1
 	tail smp_callin
 #endif /* CONFIG_SMP */
 
arch/riscv/lib/clear_page.S +1 -0
index 20ff03f5b0f2..16e63ea91baa 100644
@@ -69,6 +69,7 @@ SYM_FUNC_START(clear_page)
 	ret
 .Lno_zicboz:
 	li	a1, 0
+	lui t2, 0x1
 	tail	__memset
 SYM_FUNC_END(clear_page)
 EXPORT_SYMBOL(clear_page)
Signature · 1 line
2.43.2
#8 lore raw

RFC PATCH 5/12 riscv: fix certain indirect jumps for kernel cfi

In reply to Deepak Gupta

Handwritten __memset asm routine performs certain static jumps within function and uses a5 to do that. This would require a landing pad instruction at the target. Since its static jump and no memory load is involved, use t2 instead which is exempt from requiring a landing pad.

---
 arch/riscv/lib/memset.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/riscv/lib/memset.S +3 -3
index 35f358e70bdb..e129ebf66986 100644
@@ -56,12 +56,12 @@ SYM_FUNC_START(__memset)
 
 	/* Jump into loop body */
 	/* Assumes 32-bit instruction lengths */
-	la a5, 3f
+	la t2, 3f
 #ifdef CONFIG_64BIT
 	srli a4, a4, 1
 #endif
-	add a5, a5, a4
-	jr a5
+	add t2, t2, a4
+	jr t2
 3:
 	REG_S a1,        0(t0)
 	REG_S a1,    SZREG(t0)
Signature · 1 line
2.43.2
#9 lore raw

RFC PATCH 6/12 scs: place init shadow stack in .shadowstack section

In reply to Deepak Gupta

If compiled for riscv and dynamic scs is turned on, place shadow stack in .shadowstack section. Arch specific linker script can place logic to protect this section against regular stores.

---
 include/linux/init_task.h |  5 +++++
 init/init_task.c          | 12 ++++++++++--
 2 files changed, 15 insertions(+), 2 deletions(-)
include/linux/init_task.h +5 -0
index bccb3f1f6262..f025022e1164 100644
@@ -40,4 +40,9 @@ extern struct cred init_cred;
 /* Attach to the thread_info data structure for proper alignment */
 #define __init_thread_info __section(".data..init_thread_info")
 
+#if defined(CONFIG_RISCV) && defined(CONFIG_DYNAMIC_SCS)
+/* init shadow stack page */
+#define __init_shadow_stack __section(".shadowstack..init")
+#endif
+
 #endif
init/init_task.c +10 -2
index 4daee6d761c8..3c0a01455978 100644
@@ -52,10 +52,18 @@ static struct sighand_struct init_sighand = {
 };
 
 #ifdef CONFIG_SHADOW_CALL_STACK
-unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] = {
+unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)]
+#if defined(CONFIG_RISCV) && defined(CONFIG_DYNAMIC_SCS)
+	/* RISC-V dynamic SCS implements shadow stack and must go in special section */
+	__init_shadow_stack = {
+	[0] = SCS_END_MAGIC
+};
+#else
+	= {
 	[(SCS_SIZE / sizeof(long)) - 1] = SCS_END_MAGIC
 };
-#endif
+#endif /* CONFIG_RISCV && CONFIG_DYNAMIC_SCS */
+#endif /* CONFIG_SHADOW_CALL_STACK */
 
 /*
  * Set up the first task table, touch at your own risk!. Base=0,
Signature · 1 line
2.43.2
#10 lore raw

RFC PATCH 7/12 riscv/mm: prepare shadow stack for init task for kernel cfi

In reply to Deepak Gupta

Under CONFIG_SHADOW_CALL_STACK, shadow call stack goes into data section. Although with CONFIG_DYNAMIC_SCS on riscv, hardware assisted shadow stack are used. Hardware assisted shadow stack on riscv uses PTE.R=0, PTE.W=1 & PTE.X=0 encodings. Without CONFIG_DYNAMIC_SCS, shadow stack for init is placed in data section and thus regular read/write encodings are applied to it. Although with with CONFIG_DYNAMIC_SCS, they need to go into different section. This change places it into .shadowstack section. As part of this change early boot code (setup_vm), applies appropriate PTE encodings to shadow call stack for init placed in .shadowstack section.

---
 arch/riscv/include/asm/pgtable.h     |  4 ++++
 arch/riscv/include/asm/sections.h    | 22 +++++++++++++++++++++
 arch/riscv/include/asm/thread_info.h | 10 ++++++++--
 arch/riscv/kernel/vmlinux.lds.S      | 12 ++++++++++++
 arch/riscv/mm/init.c                 | 29 +++++++++++++++++++++-------
 5 files changed, 68 insertions(+), 9 deletions(-)
arch/riscv/include/asm/pgtable.h +4 -0
index 9f8ea0e33eb1..3409b250390d 100644
@@ -197,6 +197,10 @@ extern struct pt_alloc_ops pt_ops __initdata;
 #define PAGE_KERNEL_READ_EXEC	__pgprot((_PAGE_KERNEL & ~_PAGE_WRITE) \
 					 | _PAGE_EXEC)
 
+#ifdef CONFIG_DYNAMIC_SCS
+#define PAGE_KERNEL_SHADOWSTACK __pgprot(_PAGE_KERNEL & ~(_PAGE_READ | _PAGE_EXEC))
+#endif
+
 #define PAGE_TABLE		__pgprot(_PAGE_TABLE)
 
 #define _PAGE_IOREMAP	((_PAGE_KERNEL & ~_PAGE_MTMASK) | _PAGE_IO)
arch/riscv/include/asm/sections.h +22 -0
index a393d5035c54..4c4154d0021e 100644
@@ -14,6 +14,10 @@ extern char __init_data_begin[], __init_data_end[];
 extern char __init_text_begin[], __init_text_end[];
 extern char __alt_start[], __alt_end[];
 extern char __exittext_begin[], __exittext_end[];
+#ifdef CONFIG_DYNAMIC_SCS
+extern char __init_shstk_start[], __init_shstk_end[];
+#endif
+extern char __end_srodata[];
 
 static inline bool is_va_kernel_text(uintptr_t va)
 {
@@ -31,4 +35,22 @@ static inline bool is_va_kernel_lm_alias_text(uintptr_t va)
 	return va >= start && va < end;
 }
 
+#ifdef CONFIG_DYNAMIC_SCS
+static inline bool is_va_init_shadow_stack_early(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_start));
+	uintptr_t end = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_end));
+
+	return va >= start && va < end;
+}
+
+static inline bool is_va_init_shadow_stack(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(__init_shstk_start);
+	uintptr_t end = (uintptr_t)(__init_shstk_end);
+
+	return va >= start && va < end;
+}
+#endif
+
 #endif /* __ASM_SECTIONS_H */
arch/riscv/include/asm/thread_info.h +8 -2
index 5d473343634b..7ae28d627f84 100644
@@ -63,12 +63,18 @@ struct thread_info {
 };
 
 #ifdef CONFIG_SHADOW_CALL_STACK
+#ifdef CONFIG_DYNAMIC_SCS
 #define INIT_SCS							\
-	.scs_base	= init_shadow_call_stack,			\
+	.scs_base	= init_shadow_call_stack,	\
+	.scs_sp		= &init_shadow_call_stack[SCS_SIZE / sizeof(long)],
+#else
+#define INIT_SCS							\
+	.scs_base	= init_shadow_call_stack,	\
 	.scs_sp		= init_shadow_call_stack,
+#endif /* CONFIG_DYNAMIC_SCS */
 #else
 #define INIT_SCS
-#endif
+#endif /* CONFIG_SHADOW_CALL_STACK */
 
 /*
  * macros/functions for gaining access to the thread information structure
arch/riscv/kernel/vmlinux.lds.S +12 -0
index 002ca58dd998..cccc51f845ab 100644
@@ -126,6 +126,18 @@ SECTIONS
 		*(.srodata*)
 	}
 
+	. = ALIGN(SECTION_ALIGN);
+	__end_srodata = .;
+
+#ifdef CONFIG_DYNAMIC_SCS
+	.shadowstack : AT(ADDR(.shadowstack) - LOAD_OFFSET){
+		__init_shstk_start = .;
+		KEEP(*(.shadowstack..init))
+		. = __init_shstk_start + PAGE_SIZE;
+		__init_shstk_end = .;
+	}
+#endif
+
 	. = ALIGN(SECTION_ALIGN);
 	_data = .;
 
arch/riscv/mm/init.c +22 -7
index fe8e159394d8..5b6f0cfa5719 100644
@@ -713,14 +713,22 @@ static __init pgprot_t pgprot_from_va(uintptr_t va)
 	if (IS_ENABLED(CONFIG_64BIT) && is_va_kernel_lm_alias_text(va))
 		return PAGE_KERNEL_READ;
 
+#ifdef CONFIG_DYNAMIC_SCS
+	/* If init task's shadow stack va, return write only page protections */
+	if (IS_ENABLED(CONFIG_64BIT) && is_va_init_shadow_stack(va)) {
+		pr_info("Shadow stack protections are being applied to for init\n");
+		return PAGE_KERNEL_SHADOWSTACK;
+	}
+#endif
+
 	return PAGE_KERNEL;
 }
 
 void mark_rodata_ro(void)
 {
-	set_kernel_memory(__start_rodata, _data, set_memory_ro);
+	set_kernel_memory(__start_rodata, __end_srodata, set_memory_ro);
 	if (IS_ENABLED(CONFIG_64BIT))
-		set_kernel_memory(lm_alias(__start_rodata), lm_alias(_data),
+		set_kernel_memory(lm_alias(__start_rodata), lm_alias(__end_srodata),
 				  set_memory_ro);
 }
 #else
@@ -913,14 +921,21 @@ static void __init create_kernel_page_table(pgd_t *pgdir,
 static void __init create_kernel_page_table(pgd_t *pgdir, bool early)
 {
 	uintptr_t va, end_va;
+	pgprot_t prot;
 
 	end_va = kernel_map.virt_addr + kernel_map.size;
-	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE)
+	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) {
+		prot = PAGE_KERNEL_EXEC;
+#ifdef CONFIG_DYNAMIC_SCS
+		if (early && is_va_init_shadow_stack_early(va))
+			prot = PAGE_KERNEL_SHADOWSTACK;
+#endif
 		create_pgd_mapping(pgdir, va,
-				   kernel_map.phys_addr + (va - kernel_map.virt_addr),
-				   PMD_SIZE,
-				   early ?
-					PAGE_KERNEL_EXEC : pgprot_from_va(va));
+					kernel_map.phys_addr + (va - kernel_map.virt_addr),
+					PMD_SIZE,
+					early ?
+					prot : pgprot_from_va(va));
+	}
 }
 #endif
 
Signature · 1 line
2.43.2
#13 lore raw

RFC PATCH 8/12 riscv: dynamic (zicfiss) shadow call stack support

In reply to Deepak Gupta

Adding support for dynamic shadow call stack on riscv. zicfiss ISA extn. enables protection for shadow stack against stray writes. This patch enables scs_* macros to use zicfiss shadow stack pointer (CSR_SSP) instead of relying on gp.

Since zicfiss based shadow stack needs to have correct encoding set in PTE init shadow stack can't be established too early. It has to be setup after setup_vm is called. Thus scs_load_init_stack is noped out if CONFIG_DYNAMIC_SCS is selected.

Adds arch_scs_store that can be used in generic scs magic store routine.

---
 arch/riscv/include/asm/asm.h |  2 +-
 arch/riscv/include/asm/scs.h | 47 +++++++++++++++++++++++++++++-------
 arch/riscv/kernel/entry.S    | 14 +++++------
 arch/riscv/kernel/head.S     |  4 +--
 4 files changed, 48 insertions(+), 19 deletions(-)
arch/riscv/include/asm/asm.h +1 -1
index 776354895b81..0304978ea4e4 100644
@@ -109,7 +109,7 @@
 	REG_L \dst, 0(\dst)
 .endm
 
-#ifdef CONFIG_SHADOW_CALL_STACK
+#if defined(CONFIG_SHADOW_CALL_STACK) && !defined(CONFIG_DYNAMIC_SCS)
 /* gp is used as the shadow call stack pointer instead */
 .macro load_global_pointer
 .endm
arch/riscv/include/asm/scs.h +38 -9

84 lines not on this page · raw message

arch/riscv/kernel/entry.S +7 -7
index a35050a3e0ea..0262b46ab064 100644
@@ -81,7 +81,7 @@ SYM_CODE_START(handle_exception)
 	load_global_pointer
 
 	/* Load the kernel shadow call stack pointer if coming from userspace */
-	scs_load_current_if_task_changed s5
+	scs_load_current_if_task_changed s5 t0
 
 #ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
 	move a0, sp
@@ -135,7 +135,7 @@ SYM_CODE_START_NOALIGN(ret_from_exception)
 	REG_S s0, TASK_TI_KERNEL_SP(tp)
 
 	/* Save the kernel shadow call stack pointer */
-	scs_save_current
+	scs_save_current t0
 
 	/*
 	 * Save TP into the scratch register , so we can find the kernel data
@@ -252,8 +252,8 @@ SYM_FUNC_START(call_on_irq_stack)
 	addi	s0, sp, STACKFRAME_SIZE_ON_STACK
 
 	/* Switch to the per-CPU shadow call stack */
-	scs_save_current
-	scs_load_irq_stack t0
+	scs_save_current t0
+	scs_load_irq_stack t0 t1
 
 	/* Switch to the per-CPU IRQ stack and call the handler */
 	load_per_cpu t0, irq_stack_ptr, t1
@@ -263,7 +263,7 @@ SYM_FUNC_START(call_on_irq_stack)
 	jalr	a1
 
 	/* Switch back to the thread shadow call stack */
-	scs_load_current
+	scs_load_current t0
 
 	/* Switch back to the thread stack and restore ra and s0 */
 	addi	sp, s0, -STACKFRAME_SIZE_ON_STACK
@@ -305,7 +305,7 @@ SYM_FUNC_START(__switch_to)
 	REG_S s10, TASK_THREAD_S10_RA(a3)
 	REG_S s11, TASK_THREAD_S11_RA(a3)
 	/* Save the kernel shadow call stack pointer */
-	scs_save_current
+	scs_save_current t0
 	/* Restore context from next->thread */
 	REG_L ra,  TASK_THREAD_RA_RA(a4)
 	REG_L sp,  TASK_THREAD_SP_RA(a4)
@@ -324,7 +324,7 @@ SYM_FUNC_START(__switch_to)
 	/* The offset of thread_info in task_struct is zero. */
 	move tp, a1
 	/* Switch to the next shadow call stack */
-	scs_load_current
+	scs_load_current t0
 	ret
 SYM_FUNC_END(__switch_to)
 
arch/riscv/kernel/head.S +2 -2
index 6c311517c3b5..bc248c137c90 100644
@@ -164,7 +164,7 @@ secondary_start_sbi:
 	call relocate_enable_mmu
 #endif
 	call .Lsetup_trap_vector
-	scs_load_current
+	scs_load_current t0
 	lui t2, 0x1
 	tail smp_callin
 #endif /* CONFIG_SMP */
@@ -313,7 +313,7 @@ SYM_CODE_START(_start_kernel)
 	la tp, init_task
 	la sp, init_thread_union + THREAD_SIZE
 	addi sp, sp, -PT_SIZE_ON_STACK
-	scs_load_current
+	scs_load_current t0
 
 #ifdef CONFIG_KASAN
 	call kasan_early_init
Signature · 1 line
2.43.2
#17 lore raw

RFC PATCH 9/12 scs: kernel shadow stack with hardware assistance

In reply to Deepak Gupta

If shadow stack have memory protections from underlying cpu, use those protections. RISCV uses PAGE_KERNEL_SHADOWSTACK to vmalloc such shadow stack pages. Shadow stack pages on RISCV grows downwards like regular stack. Clang based software shadow call stack grows low to high address. Thus this patch addresses some of those needs due to opposite direction of shadow stack. Furthermore, RISCV hw shadow stack can't be memset because memset uses normal stores. Lastly to store magic word at base of shadow stack, arch specific shadow stack store has to be performed.

---
 include/linux/scs.h | 48 +++++++++++++++++++++++++++++++++------------
 kernel/scs.c        | 28 ++++++++++++++++++++++----
 2 files changed, 59 insertions(+), 17 deletions(-)
include/linux/scs.h +35 -13
index 4ab5bdc898cf..3a31433532d1 100644
@@ -12,6 +12,7 @@
 #include <linux/poison.h>
 #include <linux/sched.h>
 #include <linux/sizes.h>
+#include <asm/scs.h>
 
 #ifdef CONFIG_SHADOW_CALL_STACK
 
@@ -31,6 +32,29 @@ void scs_init(void);
 int scs_prepare(struct task_struct *tsk, int node);
 void scs_release(struct task_struct *tsk);
 
+#ifdef CONFIG_DYNAMIC_SCS
+/* dynamic_scs_enabled set to true if RISCV dynamic SCS */
+#ifdef CONFIG_RISCV
+DECLARE_STATIC_KEY_TRUE(dynamic_scs_enabled);
+#else
+DECLARE_STATIC_KEY_FALSE(dynamic_scs_enabled);
+#endif
+#endif
+
+static inline bool scs_is_dynamic(void)
+{
+	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
+		return false;
+	return static_branch_likely(&dynamic_scs_enabled);
+}
+
+static inline bool scs_is_enabled(void)
+{
+	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
+		return true;
+	return scs_is_dynamic();
+}
+
 static inline void scs_task_reset(struct task_struct *tsk)
 {
 	/*
@@ -42,6 +66,9 @@ static inline void scs_task_reset(struct task_struct *tsk)
 
 static inline unsigned long *__scs_magic(void *s)
 {
+	if (scs_is_dynamic())
+		return (unsigned long *)(s);
+
 	return (unsigned long *)(s + SCS_SIZE) - 1;
 }
 
@@ -50,23 +77,18 @@ static inline bool task_scs_end_corrupted(struct task_struct *tsk)
 	unsigned long *magic = __scs_magic(task_scs(tsk));
 	unsigned long sz = task_scs_sp(tsk) - task_scs(tsk);
 
-	return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC;
-}
-
-DECLARE_STATIC_KEY_FALSE(dynamic_scs_enabled);
+	if (scs_is_dynamic())
+		sz = (task_scs(tsk) + SCS_SIZE) - task_scs_sp(tsk);
 
-static inline bool scs_is_dynamic(void)
-{
-	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
-		return false;
-	return static_branch_likely(&dynamic_scs_enabled);
+	return sz >= SCS_SIZE - 1 || READ_ONCE_NOCHECK(*magic) != SCS_END_MAGIC;
 }
 
-static inline bool scs_is_enabled(void)
+static inline void __scs_store_magic(unsigned long *s, unsigned long magic_val)
 {
-	if (!IS_ENABLED(CONFIG_DYNAMIC_SCS))
-		return true;
-	return scs_is_dynamic();
+	if (scs_is_dynamic())
+		arch_scs_store(s, magic_val);
+	else
+		*__scs_magic(s) = SCS_END_MAGIC;
 }
 
 #else /* CONFIG_SHADOW_CALL_STACK */
kernel/scs.c +24 -4
index d7809affe740..e447483fa9f4 100644
@@ -13,8 +13,13 @@
 #include <linux/vmstat.h>
 
 #ifdef CONFIG_DYNAMIC_SCS
+/* dynamic_scs_enabled set to true if RISCV dynamic SCS */
+#ifdef CONFIG_RISCV
+DEFINE_STATIC_KEY_TRUE(dynamic_scs_enabled);
+#else
 DEFINE_STATIC_KEY_FALSE(dynamic_scs_enabled);
 #endif
+#endif
 
 static void __scs_account(void *s, int account)
 {
@@ -32,19 +37,29 @@ static void *__scs_alloc(int node)
 {
 	int i;
 	void *s;
+	pgprot_t prot = PAGE_KERNEL;
+
+	if (scs_is_dynamic())
+		prot = PAGE_KERNEL_SHADOWSTACK;
 
 	for (i = 0; i < NR_CACHED_SCS; i++) {
 		s = this_cpu_xchg(scs_cache[i], NULL);
 		if (s) {
 			s = kasan_unpoison_vmalloc(s, SCS_SIZE,
 						   KASAN_VMALLOC_PROT_NORMAL);
-			memset(s, 0, SCS_SIZE);
+/*
+ * If either of them undefined, its safe to memset. Else memset is not
+ * possible. memset constitutes stores and stores to shadow stack memory
+ * are disallowed and will fault.
+ */
+			if (!scs_is_dynamic())
+				memset(s, 0, SCS_SIZE);
 			goto out;
 		}
 	}
 
 	s = __vmalloc_node_range(SCS_SIZE, 1, VMALLOC_START, VMALLOC_END,
-				    GFP_SCS, PAGE_KERNEL, 0, node,
+				    GFP_SCS, prot, 0, node,
 				    __builtin_return_address(0));
 
 out:
@@ -59,7 +74,7 @@ void *scs_alloc(int node)
 	if (!s)
 		return NULL;
 
-	*__scs_magic(s) = SCS_END_MAGIC;
+	__scs_store_magic(__scs_magic(s), SCS_END_MAGIC);
 
 	/*
 	 * Poison the allocation to catch unintentional accesses to
@@ -122,7 +137,12 @@ int scs_prepare(struct task_struct *tsk, int node)
 	if (!s)
 		return -ENOMEM;
 
-	task_scs(tsk) = task_scs_sp(tsk) = s;
+	task_scs(tsk) = s;
+	if (scs_is_dynamic())
+		task_scs_sp(tsk) = s + SCS_SIZE;
+	else
+		task_scs_sp(tsk) = s;
+
 	return 0;
 }
 
Signature · 1 line
2.43.2
#18 lore raw

RFC PATCH 10/12 riscv/traps: Introduce software check exception

In reply to Deepak Gupta

zicfiss / zicfilp introduces a new exception to priv isa software check exception with cause code = 18. This patch implements software check exception.

If sw check exception was triggered while in usermode, unknown trap is triggered for usermode. If sw check exception was triggered for kernel mode, kernel dies.

---
 arch/riscv/include/asm/asm-prototypes.h |  1 +
 arch/riscv/kernel/entry.S               |  3 +++
 arch/riscv/kernel/traps.c               | 20 ++++++++++++++++++++
 3 files changed, 24 insertions(+)
arch/riscv/include/asm/asm-prototypes.h +1 -0
index cd627ec289f1..5a27cefd7805 100644
@@ -51,6 +51,7 @@ DECLARE_DO_ERROR_INFO(do_trap_ecall_u);
 DECLARE_DO_ERROR_INFO(do_trap_ecall_s);
 DECLARE_DO_ERROR_INFO(do_trap_ecall_m);
 DECLARE_DO_ERROR_INFO(do_trap_break);
+DECLARE_DO_ERROR_INFO(do_trap_software_check);
 
 asmlinkage void handle_bad_stack(struct pt_regs *regs);
 asmlinkage void do_page_fault(struct pt_regs *regs);
arch/riscv/kernel/entry.S +3 -0
index 0262b46ab064..89aeae803702 100644
@@ -353,6 +353,9 @@ SYM_DATA_START_LOCAL(excp_vect_table)
 	RISCV_PTR do_page_fault   /* load page fault */
 	RISCV_PTR do_trap_unknown
 	RISCV_PTR do_page_fault   /* store page fault */
+	RISCV_PTR do_trap_unknown /* cause=16 */
+	RISCV_PTR do_trap_unknown /* cause=17 */
+	RISCV_PTR do_trap_software_check /* cause=18 is sw check exception */
 SYM_DATA_END_LABEL(excp_vect_table, SYM_L_LOCAL, excp_vect_table_end)
 
 #ifndef CONFIG_MMU
arch/riscv/kernel/traps.c +20 -0
index 05a16b1f0aee..b464355f62b2 100644
@@ -354,6 +354,26 @@ void do_trap_ecall_u(struct pt_regs *regs)
 
 }
 
+/*
+ * software check exception is defined with risc-v cfi spec. Software check
+ * exception is raised when:-
+ * a) An indirect branch doesn't land on 4 byte aligned PC or `lpad`
+ *    instruction or `label` value programmed in `lpad` instr doesn't
+ *    match with value setup in `x7`. reported code in `xtval` is 2.
+ * b) `sspopchk` instruction finds a mismatch between top of shadow stack (ssp)
+ *    and x1/x5. reported code in `xtval` is 3.
+ */
+asmlinkage __visible __trap_section void do_trap_software_check(struct pt_regs *regs)
+{
+	if (user_mode(regs)) {
+		/* deliver unknown trap to usermode */
+		do_trap_unknown(regs);
+	} else {
+		/* sw check exception coming from kernel is a bug in kernel, die */
+		die(regs, "Kernel BUG");
+	}
+}
+
 #ifdef CONFIG_MMU
 asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
 {
Signature · 1 line
2.43.2
#19 lore raw

RFC PATCH 11/12 riscv: Kconfig & Makefile for riscv kernel control flow integrity

In reply to Deepak Gupta

Defines CONFIG_RISCV_KERNEL_CFI and selects SHADOW_CALL_STACK and DYNAMIC_SCS both so that zicfiss can be wired up.

Makefile checks if CONFIG_RISCV_KERNEL_CFI is enabled, then light up zicfiss and zicfilp compiler flags.

---
 arch/riscv/Kconfig  | 36 +++++++++++++++++++++++++++++++++++-
 arch/riscv/Makefile |  6 ++++++
 2 files changed, 41 insertions(+), 1 deletion(-)
arch/riscv/Kconfig +35 -1
index be09c8836d56..5276598bb773 100644
@@ -193,7 +193,7 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE
 	depends on $(cc-option,-fpatchable-function-entry=8)
 
 config HAVE_SHADOW_CALL_STACK
-	def_bool $(cc-option,-fsanitize=shadow-call-stack)
+	def_bool $(cc-option,-fsanitize=shadow-call-stack) || $(cc-option,-mabi=lp64 -march=rv64ima_zicfilp_zicfiss)
 	# https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769
 	depends on $(ld-option,--no-relax-gp)
 
@@ -211,6 +211,30 @@ config ARCH_HAS_BROKEN_DWARF5
 	# https://github.com/llvm/llvm-project/commit/7ffabb61a5569444b5ac9322e22e5471cc5e4a77
 	depends on LD_IS_LLD && LLD_VERSION < 180000
 
+config RISCV_KERNEL_CFI
+	def_bool n
+	bool "hw assisted riscv kernel control flow integrity (kcfi)"
+	depends on 64BIT && $(cc-option,-mabi=lp64 -march=rv64ima_zicfilp_zicfiss)
+	select ARCH_SUPPORTS_SHADOW_CALL_STACK
+	select SHADOW_CALL_STACK
+	select DYNAMIC_SCS
+	help
+	  Provides CPU assisted control flow integrity to for riscv kernel.
+	  Control flow integrity is provided by implementing shadow stack for
+	  backward edge and indirect branch tracking for forward edge. Shadow
+	  stack protection is a hardware feature that detects function return
+	  address corruption. This helps mitigate ROP attacks. RISCV_KERNEL_CFI
+	  selects CONFIG_SHADOW_CALL_STACK which uses software based shadow
+	  stack but is unprotected against stray writes. Selecting RISCV_KERNEL_CFI
+	  will select CONFIG_DYNAMIC_SCS and will enable hardware assisted shadow
+	  stack protection against stray writes.
+	  Indirect branch tracking enforces that all indirect branches must land
+	  on a landing pad instruction else CPU will fault. This enables forward
+	  control flow (call/jmp) protection in kernel and restricts all indirect
+	  call or jump in kernel to a landing pad instruction which mostly likely
+	  will be start of the function.
+	  default n
+
 config ARCH_MMAP_RND_BITS_MIN
 	default 18 if 64BIT
 	default 8
@@ -639,6 +663,16 @@ config RISCV_ISA_ZICBOZ
 
 	   If you don't know what to do here, say Y.
 
+config TOOLCHAIN_HAS_ZICFILP
+	bool
+	default y
+	depends on 64BIT && $(cc-option,-mabi=lp64 -march=rv64ima_zicfilp)
+
+config TOOLCHAIN_HAS_ZICFISS
+	bool
+	default y
+	depends on 64BIT && $(cc-option,-mabi=lp64 -march=rv64ima_zicfiss)
+
 config TOOLCHAIN_HAS_ZIHINTPAUSE
 	bool
 	default y
arch/riscv/Makefile +6 -0
index 5b3115a19852..ae156e37e886 100644
@@ -58,8 +58,10 @@ else ifeq ($(CONFIG_LTO_CLANG),y)
 endif
 
 ifeq ($(CONFIG_SHADOW_CALL_STACK),y)
+ifndef CONFIG_DYNAMIC_SCS
 	KBUILD_LDFLAGS += --no-relax-gp
 endif
+endif
 
 # ISA string setting
 riscv-march-$(CONFIG_ARCH_RV32I)	:= rv32ima
@@ -78,6 +80,10 @@ endif
 # Check if the toolchain supports Zihintpause extension
 riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZIHINTPAUSE) := $(riscv-march-y)_zihintpause
 
+ifeq ($(CONFIG_RISCV_KERNEL_CFI),y)
+riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZICFILP) := $(riscv-march-y)_zicfilp
+riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZICFISS) := $(riscv-march-y)_zicfiss
+endif
 # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
 # matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
 KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')
Signature · 1 line
2.43.2
#20 lore raw

RFC PATCH 12/12 riscv: enable kernel shadow stack and landing pad enforcement

In reply to Deepak Gupta

This patch enables kernel shadow stack and landing pad enforcement by invoking a SBI call. As of now it just issues a SBI_EXT_BASE and a hacked up opensbi implementation sets the LPE/SSE bits in menvcfg

Eventually, we should have fwft [1] interface using which kernel should be able to set this enforcement properly

[1] - https://lists.riscv.org/g/tech-prs/message/833

---
 arch/riscv/kernel/head.S | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/riscv/kernel/head.S +14 -0
index bc248c137c90..1e5bc7b2ee75 100644
@@ -164,6 +164,13 @@ secondary_start_sbi:
 	call relocate_enable_mmu
 #endif
 	call .Lsetup_trap_vector
+	/*
+	 * Temp hack to get menvcfg.SSE=1 and menvcfg.LPE=1 by invoking
+	 * SBI_EXT_BASE
+	 */
+	li a6, 0
+	li a7, 0x10
+	ecall
 	scs_load_current t0
 	lui t2, 0x1
 	tail smp_callin
@@ -313,6 +320,13 @@ SYM_CODE_START(_start_kernel)
 	la tp, init_task
 	la sp, init_thread_union + THREAD_SIZE
 	addi sp, sp, -PT_SIZE_ON_STACK
+	/*
+	 * Temp hack to get menvcfg.SSE=1 and menvcfg.LPE=1 by invoking
+	 * SBI_EXT_BASE
+	 */
+	li a6, 0
+	li a7, 0x10
+	ecall
 	scs_load_current t0
 
 #ifdef CONFIG_KASAN
Signature · 1 line
2.43.2
#14 lore raw

Re: [RFC PATCH 08/12] riscv: dynamic (zicfiss) shadow call stack support

In reply to Deepak Gupta

Hi Deepak,

Thanks for the patches!

Quoted 5 lines · Deepak Gupta

Adding support for dynamic shadow call stack on riscv. zicfiss ISA extn. enables protection for shadow stack against stray writes. This patch enables scs_* macros to use zicfiss shadow stack pointer (CSR_SSP) instead of relying on gp.

CONFIG_DYNAMIC_SCS implies that runtime patching is used to select between software SCS and an alternative hardware implementation (in arm64's case, PAC instead of hardware shadow stacks). I understand this series is still an RFC, but I didn't see runtime patching support. Are you planning on implementing this later?

If there's no plan to actually patch between Zicfiss and SCS at runtime, CONFIG_DYNAMIC_SCS doesn't seem like the appropriate choice and we might need a separate config option that still allows you to reuse most of the software SCS code.

Sami

#3 lore raw

Re: [RFC PATCH 02/12] riscv: add landing pad for asm routines.

In reply to Deepak Gupta
Quoted 24 lines · Deepak Gupta

SYM_* macros are used to define assembly routines. In this patch series, re-define those macros in risc-v arch specific include file to include a landing pad instruction at the beginning. This is done only when the compiler flag for landing pad is enabled (i.e. __riscv_zicfilp).

---
 arch/riscv/include/asm/linkage.h | 42 ++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
arch/riscv/include/asm/linkage.h +4 -0
index 9e88ba23cd2b..bb43ae7dadeb 100644
@@ -6,7 +6,49 @@
 #ifndef _ASM_RISCV_LINKAGE_H
 #define _ASM_RISCV_LINKAGE_H
 
+#ifdef __ASSEMBLY__
+#include <asm/assembler.h>
+#endif
+
 #define __ALIGN                .balign 4

arch/riscv/include/asm/linkage.h line 23

Quoted 10 lines
arch/riscv/include/asm/linkage.h +8 -0
 #define __ALIGN_STR    ".balign 4"
 
+#ifdef __riscv_zicfilp
+/*
+ * A landing pad instruction is needed at start of asm routines
+ * re-define macros for asm routines to have a landing pad at
+ * the beginning of function. Currently use label value of 0x1.
+ * Eventually, label should be calculated as a hash over function
+ * signature.
+ */

I haven't seen the compiler implementation for fine-grained Zicfilp yet, but in the kernel at least, this would ideally reuse as much of the KCFI plumbing as possible. For example, since only C code has type information, we left the type hash computation for the compiler, which allows assembly functions to just reference the appropriate __kcfi_typeid_* symbol.

Sami

#15 lore raw

Re: [RFC PATCH 08/12] riscv: dynamic (zicfiss) shadow call stack support

In reply to Sami Tolvanen
Quoted 11 lines · Sami Tolvanen

Hi Deepak,

Thanks for the patches!

Quoted 5 lines · Deepak Gupta

Adding support for dynamic shadow call stack on riscv. zicfiss ISA extn. enables protection for shadow stack against stray writes. This patch enables scs_* macros to use zicfiss shadow stack pointer (CSR_SSP) instead of relying on gp.

Quoted 5 lines

CONFIG_DYNAMIC_SCS implies that runtime patching is used to select between software SCS and an alternative hardware implementation (in arm64's case, PAC instead of hardware shadow stacks). I understand this series is still an RFC, but I didn't see runtime patching support. Are you planning on implementing this later?

Since I didn't see any example on selecting PAC when CONFIG_DYNAMIC_SCS is selected. So I had that confusion but wasn't sure. I thought of doing it but I don't know how to binary rewrite all the functions. It might be too much. So I went ahead with using CONFIG_DYNAMIC_SCS in this RFC series.

Question:
If arm64 were to use PAC with CONFIG_DYNAMIC_SCS, how would it fixup the code sequences already setup by compiler for shadow stack push and pop in runtime? You expect this to be some offline process using some object editing tool or
a runtime decision?

Quoted 5 lines

If there's no plan to actually patch between Zicfiss and SCS at runtime, CONFIG_DYNAMIC_SCS doesn't seem like the appropriate choice and we might need a separate config option that still allows you to reuse most of the software SCS code.

I wanted to avoid "#ifdef RISCV_SPECIFIC_HW_SHSTK" in arch agnostic scs code. And that's why went with CONFIG_DYNAMIC_SCS which sets dynamic static key once. And then I use is_dynamic everywhere else in arch agnostic scs code.

Quoted 2 lines

Sami

#16 lore raw

Re: [RFC PATCH 08/12] riscv: dynamic (zicfiss) shadow call stack support

Quoted 24 lines · Deepak Gupta
Quoted 11 lines · Sami Tolvanen

Hi Deepak,

Thanks for the patches!

Quoted 5 lines · Deepak Gupta

Adding support for dynamic shadow call stack on riscv. zicfiss ISA extn. enables protection for shadow stack against stray writes. This patch enables scs_* macros to use zicfiss shadow stack pointer (CSR_SSP) instead of relying on gp.

Quoted 5 lines

CONFIG_DYNAMIC_SCS implies that runtime patching is used to select between software SCS and an alternative hardware implementation (in arm64's case, PAC instead of hardware shadow stacks). I understand this series is still an RFC, but I didn't see runtime patching support. Are you planning on implementing this later?

Since I didn't see any example on selecting PAC when CONFIG_DYNAMIC_SCS is selected. So I had that confusion but wasn't sure. I thought of doing it but I don't know how to binary rewrite all the functions. It might be too much. So I went ahead with using CONFIG_DYNAMIC_SCS in this RFC series.

Quoted 5 lines

Question:
If arm64 were to use PAC with CONFIG_DYNAMIC_SCS, how would it fixup the code sequences already setup by compiler for shadow stack push and pop in runtime? You expect this to be some offline process using some object editing tool or a runtime decision?

We use unwind tables for locating instructions to patch, look for UNWIND_PATCH_PAC_INTO_SCS. The actual patching code is in arch/arm64/kernel/pi/patch-scs.c. I suspect this is going to be a bit trickier when patching between two shadow stack options though.

Quoted 8 lines
Quoted 4 lines

If there's no plan to actually patch between Zicfiss and SCS at runtime, CONFIG_DYNAMIC_SCS doesn't seem like the appropriate choice and we might need a separate config option that still allows you to reuse most of the software SCS code.

I wanted to avoid "#ifdef RISCV_SPECIFIC_HW_SHSTK" in arch agnostic scs code. And that's why went with CONFIG_DYNAMIC_SCS which sets dynamic static key once. And then I use is_dynamic everywhere else in arch agnostic scs code.

We could define arch_ functions for any architecture-specific code (with a weak default implementation), and maybe add a config option for specifying which way the shadow stack grows?

Sami

#4 lore raw

Re: [RFC PATCH 02/12] riscv: add landing pad for asm routines.

In reply to Sami Tolvanen
Quoted 27 lines · Sami Tolvanen
Quoted 26 lines · Deepak Gupta

SYM_* macros are used to define assembly routines. In this patch series, re-define those macros in risc-v arch specific include file to include a landing pad instruction at the beginning. This is done only when the compiler flag for landing pad is enabled (i.e. __riscv_zicfilp).

---
 arch/riscv/include/asm/linkage.h | 42 ++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
arch/riscv/include/asm/linkage.h +4 -0
index 9e88ba23cd2b..bb43ae7dadeb 100644
@@ -6,7 +6,49 @@
 #ifndef _ASM_RISCV_LINKAGE_H
 #define _ASM_RISCV_LINKAGE_H
 
+#ifdef __ASSEMBLY__
+#include <asm/assembler.h>
+#endif
+
 #define __ALIGN                .balign 4
 #define __ALIGN_STR    ".balign 4"
 
Quoted 15 lines

arch/riscv/include/asm/linkage.h line 23

Quoted 8 lines
arch/riscv/include/asm/linkage.h +8 -0
+#ifdef __riscv_zicfilp
+/*
+ * A landing pad instruction is needed at start of asm routines
+ * re-define macros for asm routines to have a landing pad at
+ * the beginning of function. Currently use label value of 0x1.
+ * Eventually, label should be calculated as a hash over function
+ * signature.
+ */

I haven't seen the compiler implementation for fine-grained Zicfilp yet, but in the kernel at least, this would ideally reuse as much of the KCFI plumbing as possible. For example, since only C code has type information, we left the type hash computation for the compiler, which allows assembly functions to just reference the appropriate __kcfi_typeid_* symbol.

Fine-grained compiler support hasn't made it in yet.

For reference, compiler that I've been using
https://github.com/sifive/riscv-gnu-toolchain/tree/cfi-dev

Honestly speaking, I didn't realize that kcfi plumbing has made it into riscv as well. I realized that just after sending the patches.

In principle, I agree it should converge with software based kcfi scheme as much as possible. However blocker that I see is hash is placed just before function. This breaks for code mapped as execute only scenarios. And ideally would like to have immediates at callsites instead of loads (purely perf reason and not security).

But yes in next version, I'll take a look and try to converge as much as possible.

Quoted 2 lines

Sami

#5 lore raw

Re: [RFC PATCH 02/12] riscv: add landing pad for asm routines.

Quoted 6 lines · Deepak Gupta

In principle, I agree it should converge with software based kcfi scheme as much as possible. However blocker that I see is hash is placed just before function. This breaks for code mapped as execute only scenarios. And ideally would like to have immediates at callsites instead of loads (purely perf reason and not security).

I'm not saying the schemes have to be compatible, just that it would be great to avoid reinventing type annotations etc. For example, when you implement the fine-grained variant, you could simply override SYM_TYPED_ENTRY (defined in include/linux/cfi_types.h) to move __CFI_TYPE inside the function, and then redefine __CFI_TYPE to emit a landing pad with the correct label. This allows SYM_TYPED_* in assembly code to work with both KCFI and fine-grained Zicfilp. For the coarse-grained variant, your current macros are perfectly fine.

Quoted 2 lines

But yes in next version, I'll take a look and try to converge as much as possible.

Great, sounds good!

Sami

reviewer Reviewed-by on v1 7/12
#11 lore raw

Re: [RFC PATCH 07/12] riscv/mm: prepare shadow stack for init task for kernel cfi

In reply to Deepak Gupta
Quoted 25 lines · Deepak Gupta

Under CONFIG_SHADOW_CALL_STACK, shadow call stack goes into data section. Although with CONFIG_DYNAMIC_SCS on riscv, hardware assisted shadow stack are used. Hardware assisted shadow stack on riscv uses PTE.R=0, PTE.W=1 & PTE.X=0 encodings. Without CONFIG_DYNAMIC_SCS, shadow stack for init is placed in data section and thus regular read/write encodings are applied to it. Although with with CONFIG_DYNAMIC_SCS, they need to go into different section. This change places it into .shadowstack section. As part of this change early boot code (setup_vm), applies appropriate PTE encodings to shadow call stack for init placed in .shadowstack section.

---
  arch/riscv/include/asm/pgtable.h     |  4 ++++
  arch/riscv/include/asm/sections.h    | 22 +++++++++++++++++++++
  arch/riscv/include/asm/thread_info.h | 10 ++++++++--
  arch/riscv/kernel/vmlinux.lds.S      | 12 ++++++++++++
  arch/riscv/mm/init.c                 | 29 +++++++++++++++++++++-------
  5 files changed, 68 insertions(+), 9 deletions(-)
arch/riscv/include/asm/pgtable.h +0 -0
index 9f8ea0e33eb1..3409b250390d 100644

arch/riscv/include/asm/pgtable.h extern struct pt_alloc_ops pt_ops __initdata; · line 203

Quoted 7 lines
arch/riscv/include/asm/pgtable.h +4 -0
@@ -197,6 +197,10 @@ extern struct pt_alloc_ops pt_ops __initdata;
  #define PAGE_KERNEL_READ_EXEC	__pgprot((_PAGE_KERNEL & ~_PAGE_WRITE) \
  					 | _PAGE_EXEC)
 
+#ifdef CONFIG_DYNAMIC_SCS
+#define PAGE_KERNEL_SHADOWSTACK __pgprot(_PAGE_KERNEL & ~(_PAGE_READ | _PAGE_EXEC))
+#endif
+

Not sure the ifdefs are necessary here, but I'll let others jump in. We have a lot of them, so we should try not to add.

Quoted 29 lines
arch/riscv/include/asm/pgtable.h +0 -0
  #define PAGE_TABLE		__pgprot(_PAGE_TABLE)
 
  #define _PAGE_IOREMAP	((_PAGE_KERNEL & ~_PAGE_MTMASK) | _PAGE_IO)
arch/riscv/include/asm/sections.h +11 -0
index a393d5035c54..4c4154d0021e 100644
@@ -14,6 +14,10 @@ extern char __init_data_begin[], __init_data_end[];
  extern char __init_text_begin[], __init_text_end[];
  extern char __alt_start[], __alt_end[];
  extern char __exittext_begin[], __exittext_end[];
+#ifdef CONFIG_DYNAMIC_SCS
+extern char __init_shstk_start[], __init_shstk_end[];
+#endif
+extern char __end_srodata[];
 
  static inline bool is_va_kernel_text(uintptr_t va)
  {
@@ -31,4 +35,22 @@ static inline bool is_va_kernel_lm_alias_text(uintptr_t va)
  	return va >= start && va < end;
  }
 
+#ifdef CONFIG_DYNAMIC_SCS
+static inline bool is_va_init_shadow_stack_early(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_start));
+	uintptr_t end = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_end));
+
+	return va >= start && va < end;

arch/riscv/include/asm/sections.h static inline bool is_va_kernel_lm_alias_text(uintptr_t va) · line 54

Quoted 10 lines
arch/riscv/include/asm/sections.h +10 -0
+}
+
+static inline bool is_va_init_shadow_stack(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(__init_shstk_start);
+	uintptr_t end = (uintptr_t)(__init_shstk_end);
+
+	return va >= start && va < end;
+}
+#endif

You could have used an early flag and have only one function but that's up to you.

Quoted 55 lines
arch/riscv/include/asm/sections.h +1 -0
+
  #endif /* __ASM_SECTIONS_H */
arch/riscv/include/asm/thread_info.h +8 -2
index 5d473343634b..7ae28d627f84 100644
@@ -63,12 +63,18 @@ struct thread_info {
  };
 
  #ifdef CONFIG_SHADOW_CALL_STACK
+#ifdef CONFIG_DYNAMIC_SCS
  #define INIT_SCS							\
-	.scs_base	= init_shadow_call_stack,			\
+	.scs_base	= init_shadow_call_stack,	\
+	.scs_sp		= &init_shadow_call_stack[SCS_SIZE / sizeof(long)],
+#else
+#define INIT_SCS							\
+	.scs_base	= init_shadow_call_stack,	\
  	.scs_sp		= init_shadow_call_stack,
+#endif /* CONFIG_DYNAMIC_SCS */
  #else
  #define INIT_SCS
-#endif
+#endif /* CONFIG_SHADOW_CALL_STACK */
 
  /*
   * macros/functions for gaining access to the thread information structure
arch/riscv/kernel/vmlinux.lds.S +12 -0
index 002ca58dd998..cccc51f845ab 100644
@@ -126,6 +126,18 @@ SECTIONS
  		*(.srodata*)
  	}
 
+	. = ALIGN(SECTION_ALIGN);
+	__end_srodata = .;
+
+#ifdef CONFIG_DYNAMIC_SCS
+	.shadowstack : AT(ADDR(.shadowstack) - LOAD_OFFSET){
+		__init_shstk_start = .;
+		KEEP(*(.shadowstack..init))
+		. = __init_shstk_start + PAGE_SIZE;
+		__init_shstk_end = .;
+	}
+#endif
+
  	. = ALIGN(SECTION_ALIGN);
  	_data = .;
 
arch/riscv/mm/init.c +0 -0
index fe8e159394d8..5b6f0cfa5719 100644

arch/riscv/mm/init.c static __init pgprot_t pgprot_from_va(uintptr_t va) · line 722

Quoted 10 lines
arch/riscv/mm/init.c +7 -0
@@ -713,14 +713,22 @@ static __init pgprot_t pgprot_from_va(uintptr_t va)
  	if (IS_ENABLED(CONFIG_64BIT) && is_va_kernel_lm_alias_text(va))
  		return PAGE_KERNEL_READ;
 
+#ifdef CONFIG_DYNAMIC_SCS
+	/* If init task's shadow stack va, return write only page protections */
+	if (IS_ENABLED(CONFIG_64BIT) && is_va_init_shadow_stack(va)) {
+		pr_info("Shadow stack protections are being applied to for init\n");
+		return PAGE_KERNEL_SHADOWSTACK;
+	}
+#endif

To avoid the ifdef here, I would hide it inis_va_init_shadow_stack().

Quoted 18 lines
arch/riscv/mm/init.c +3 -2
+
  	return PAGE_KERNEL;
  }
 
  void mark_rodata_ro(void)
  {
-	set_kernel_memory(__start_rodata, _data, set_memory_ro);
+	set_kernel_memory(__start_rodata, __end_srodata, set_memory_ro);
  	if (IS_ENABLED(CONFIG_64BIT))
-		set_kernel_memory(lm_alias(__start_rodata), lm_alias(_data),
+		set_kernel_memory(lm_alias(__start_rodata), lm_alias(__end_srodata),
  				  set_memory_ro);
  }
  #else
@@ -913,14 +921,21 @@ static void __init create_kernel_page_table(pgd_t *pgdir,
  static void __init create_kernel_page_table(pgd_t *pgdir, bool early)
  {
  	uintptr_t va, end_va;

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 932

Quoted 10 lines
arch/riscv/mm/init.c +7 -1
+	pgprot_t prot;
 
  	end_va = kernel_map.virt_addr + kernel_map.size;
-	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE)
+	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) {
+		prot = PAGE_KERNEL_EXEC;
+#ifdef CONFIG_DYNAMIC_SCS
+		if (early && is_va_init_shadow_stack_early(va))
+			prot = PAGE_KERNEL_SHADOWSTACK;
+#endif

Ditto here to avoid the ifdef, hide it intois_va_init_shadow_stack_early().

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 936

Quoted 8 lines
arch/riscv/mm/init.c +3 -4
  		create_pgd_mapping(pgdir, va,
-				   kernel_map.phys_addr + (va - kernel_map.virt_addr),
-				   PMD_SIZE,
-				   early ?
-					PAGE_KERNEL_EXEC : pgprot_from_va(va));
+					kernel_map.phys_addr + (va - kernel_map.virt_addr),
+					PMD_SIZE,
+					early ?

The 3 lines above are not modified, so no need to indent them.

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 941

Quoted 5 lines
arch/riscv/mm/init.c +2 -0
+					prot : pgprot_from_va(va));
+	}
  }
  #endif
 

Apart from the nits above, you can add:

Thanks,

Alex

#12 lore raw

Re: [RFC PATCH 07/12] riscv/mm: prepare shadow stack for init task for kernel cfi

Thanks Alex.

Quoted 22 lines · Alexandre Ghiti
Quoted 20 lines · Deepak Gupta

Under CONFIG_SHADOW_CALL_STACK, shadow call stack goes into data section. Although with CONFIG_DYNAMIC_SCS on riscv, hardware assisted shadow stack are used. Hardware assisted shadow stack on riscv uses PTE.R=0, PTE.W=1 & PTE.X=0 encodings. Without CONFIG_DYNAMIC_SCS, shadow stack for init is placed in data section and thus regular read/write encodings are applied to it. Although with with CONFIG_DYNAMIC_SCS, they need to go into different section. This change places it into .shadowstack section. As part of this change early boot code (setup_vm), applies appropriate PTE encodings to shadow call stack for init placed in .shadowstack section.

---
 arch/riscv/include/asm/pgtable.h     |  4 ++++
 arch/riscv/include/asm/sections.h    | 22 +++++++++++++++++++++
 arch/riscv/include/asm/thread_info.h | 10 ++++++++--
 arch/riscv/kernel/vmlinux.lds.S      | 12 ++++++++++++
 arch/riscv/mm/init.c                 | 29 +++++++++++++++++++++-------
 5 files changed, 68 insertions(+), 9 deletions(-)
Quoted 14 lines

arch/riscv/include/asm/pgtable.h extern struct pt_alloc_ops pt_ops __initdata; · line 202

Quoted 11 lines
arch/riscv/include/asm/pgtable.h +4 -0
index 9f8ea0e33eb1..3409b250390d 100644
@@ -197,6 +197,10 @@ extern struct pt_alloc_ops pt_ops __initdata;
 #define PAGE_KERNEL_READ_EXEC	__pgprot((_PAGE_KERNEL & ~_PAGE_WRITE) \
 					 | _PAGE_EXEC)
+#ifdef CONFIG_DYNAMIC_SCS
+#define PAGE_KERNEL_SHADOWSTACK __pgprot(_PAGE_KERNEL & ~(_PAGE_READ | _PAGE_EXEC))
+#endif
+

Not sure the ifdefs are necessary here, but I'll let others jump in. We have a lot of them, so we should try not to add.

I have no hard leanings either way. I was trying to make sure compile fails if shadow stack is not enabled. But there are other places where config selection makes sure of this. So may be not needed here.

Quoted 33 lines
Quoted 33 lines
arch/riscv/include/asm/pgtable.h +0 -0
 #define PAGE_TABLE		__pgprot(_PAGE_TABLE)
 #define _PAGE_IOREMAP	((_PAGE_KERNEL & ~_PAGE_MTMASK) | _PAGE_IO)
arch/riscv/include/asm/sections.h +16 -0
index a393d5035c54..4c4154d0021e 100644
@@ -14,6 +14,10 @@ extern char __init_data_begin[], __init_data_end[];
 extern char __init_text_begin[], __init_text_end[];
 extern char __alt_start[], __alt_end[];
 extern char __exittext_begin[], __exittext_end[];
+#ifdef CONFIG_DYNAMIC_SCS
+extern char __init_shstk_start[], __init_shstk_end[];
+#endif
+extern char __end_srodata[];
 static inline bool is_va_kernel_text(uintptr_t va)
 {
@@ -31,4 +35,22 @@ static inline bool is_va_kernel_lm_alias_text(uintptr_t va)
 	return va >= start && va < end;
 }
+#ifdef CONFIG_DYNAMIC_SCS
+static inline bool is_va_init_shadow_stack_early(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_start));
+	uintptr_t end = (uintptr_t)(kernel_mapping_pa_to_va(__init_shstk_end));
+
+	return va >= start && va < end;
+}
+
+static inline bool is_va_init_shadow_stack(uintptr_t va)
+{
+	uintptr_t start = (uintptr_t)(__init_shstk_start);
Quoted 8 lines

arch/riscv/include/asm/sections.h static inline bool is_va_kernel_lm_alias_text(uintptr_t va) · line 54

Quoted 5 lines
arch/riscv/include/asm/sections.h +5 -0
+	uintptr_t end = (uintptr_t)(__init_shstk_end);
+
+	return va >= start && va < end;
+}
+#endif

You could have used an early flag and have only one function but that's up to you.

Make sense, yeah I'll do that.

Quoted 57 lines
Quoted 57 lines
arch/riscv/include/asm/sections.h +1 -0
+
 #endif /* __ASM_SECTIONS_H */
arch/riscv/include/asm/thread_info.h +8 -2
index 5d473343634b..7ae28d627f84 100644
@@ -63,12 +63,18 @@ struct thread_info {
 };
 #ifdef CONFIG_SHADOW_CALL_STACK
+#ifdef CONFIG_DYNAMIC_SCS
 #define INIT_SCS							\
-	.scs_base	= init_shadow_call_stack,			\
+	.scs_base	= init_shadow_call_stack,	\
+	.scs_sp		= &init_shadow_call_stack[SCS_SIZE / sizeof(long)],
+#else
+#define INIT_SCS							\
+	.scs_base	= init_shadow_call_stack,	\
 	.scs_sp		= init_shadow_call_stack,
+#endif /* CONFIG_DYNAMIC_SCS */
 #else
 #define INIT_SCS
-#endif
+#endif /* CONFIG_SHADOW_CALL_STACK */
 /*
  * macros/functions for gaining access to the thread information structure
arch/riscv/kernel/vmlinux.lds.S +12 -0
index 002ca58dd998..cccc51f845ab 100644
@@ -126,6 +126,18 @@ SECTIONS
 		*(.srodata*)
 	}
+	. = ALIGN(SECTION_ALIGN);
+	__end_srodata = .;
+
+#ifdef CONFIG_DYNAMIC_SCS
+	.shadowstack : AT(ADDR(.shadowstack) - LOAD_OFFSET){
+		__init_shstk_start = .;
+		KEEP(*(.shadowstack..init))
+		. = __init_shstk_start + PAGE_SIZE;
+		__init_shstk_end = .;
+	}
+#endif
+
 	. = ALIGN(SECTION_ALIGN);
 	_data = .;
arch/riscv/mm/init.c +2 -0
index fe8e159394d8..5b6f0cfa5719 100644
@@ -713,14 +713,22 @@ static __init pgprot_t pgprot_from_va(uintptr_t va)
 	if (IS_ENABLED(CONFIG_64BIT) && is_va_kernel_lm_alias_text(va))
 		return PAGE_KERNEL_READ;
+#ifdef CONFIG_DYNAMIC_SCS
+	/* If init task's shadow stack va, return write only page protections */
Quoted 8 lines

arch/riscv/mm/init.c static __init pgprot_t pgprot_from_va(uintptr_t va) · line 722

Quoted 5 lines
arch/riscv/mm/init.c +5 -0
+	if (IS_ENABLED(CONFIG_64BIT) && is_va_init_shadow_stack(va)) {
+		pr_info("Shadow stack protections are being applied to for init\n");
+		return PAGE_KERNEL_SHADOWSTACK;
+	}
+#endif

To avoid the ifdef here, I would hide it inis_va_init_shadow_stack().

Make sense too.

Quoted 23 lines
Quoted 23 lines
arch/riscv/mm/init.c +5 -3
+
 	return PAGE_KERNEL;
 }
 void mark_rodata_ro(void)
 {
-	set_kernel_memory(__start_rodata, _data, set_memory_ro);
+	set_kernel_memory(__start_rodata, __end_srodata, set_memory_ro);
 	if (IS_ENABLED(CONFIG_64BIT))
-		set_kernel_memory(lm_alias(__start_rodata), lm_alias(_data),
+		set_kernel_memory(lm_alias(__start_rodata), lm_alias(__end_srodata),
 				  set_memory_ro);
 }
 #else
@@ -913,14 +921,21 @@ static void __init create_kernel_page_table(pgd_t *pgdir,
 static void __init create_kernel_page_table(pgd_t *pgdir, bool early)
 {
 	uintptr_t va, end_va;
+	pgprot_t prot;
 	end_va = kernel_map.virt_addr + kernel_map.size;
-	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE)
+	for (va = kernel_map.virt_addr; va < end_va; va += PMD_SIZE) {
Quoted 8 lines

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 932

Quoted 5 lines
arch/riscv/mm/init.c +5 -0
+		prot = PAGE_KERNEL_EXEC;
+#ifdef CONFIG_DYNAMIC_SCS
+		if (early && is_va_init_shadow_stack_early(va))
+			prot = PAGE_KERNEL_SHADOWSTACK;
+#endif

Ditto here to avoid the ifdef, hide it intois_va_init_shadow_stack_early().

Yes, will do.

Quoted 13 lines

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 936

Quoted 10 lines
arch/riscv/mm/init.c +3 -4
 		create_pgd_mapping(pgdir, va,
-				   kernel_map.phys_addr + (va - kernel_map.virt_addr),
-				   PMD_SIZE,
-				   early ?
-					PAGE_KERNEL_EXEC : pgprot_from_va(va));
+					kernel_map.phys_addr + (va - kernel_map.virt_addr),
+					PMD_SIZE,
+					early ?

The 3 lines above are not modified, so no need to indent them.

noted.

Quoted 16 lines

arch/riscv/mm/init.c static void __init create_kernel_page_table(pgd_t *pgdir, · line 940

Quoted 6 lines
arch/riscv/mm/init.c +2 -0
+					prot : pgprot_from_va(va));
+	}
 }
 #endif

Apart from the nits above, you can add:

Thanks,

Alex