Skip to contentSkip to conversation

Usermode Indirect Branch Tracking

PATCH In review Richard Patel 3 participants 12 messages Jun 5 – Jun 6

Subsystems (5)
x86 Kselftest ABI/API x86 entry code +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v2 cover Usermode Indirect Branch Tracking

Adds basic support for x86 userspace IBT.

IBT is part of Intel CET. It requires indirect call and jump targets to start with an endbr{32,64} instruction, otherwise throwing #CP.

In summary, this patch does 3 things:
- Config wiring ensuring supervisor XSAVE contains IBT state
- Allow userspace to enable IBT via prctl(PR_CFI_*) for an entire thread
- Enable IBT support (ENDBR instructions) in VDSO

Unlike the arm64 BTI API:
- does not support mixed usermode (all or nothing)
- does not touch page table code
- not enabled automatically (no ELF GNU note parsing)
- temporarily disables IBT enforcement when handling signals These can all be cleanly added later.

The main question is whether glibc is happy with this prctl syscall API.

Changes from v1:
- Removed all signal handler changes
- Removed all uapi changes
(v1 saved state via a redundant uc_flags bit, which was a complete
mess, and not compatible with sigframe_ia32)
- Don't allow unknown PR_CFI_* syscall flags
see RISC-V patch: https://lore.kernel.org/lkml/[email protected]/
- Added 32-bit support

The original usermode IBT patches were written by Yu Cheng years ago, but are quite different (no syscall API to enable IBT, no FRED support)
https://lore.kernel.org/all/[email protected]/

There is one notable gap in this patch series, to do with signals:

000a: mov rax, 0x100a
000f: jmp rax
*** signal occurs ***
*** signal handler runs, does sigreturn ***
100a: nop

The above sequence does not crash.

With IBT, it should crash at the nop (because an endr64 is expected there). The IBT state (WAIT_FOR_ENDBR in IA32_U_CET MSR) is not backed up to the signal frame though. So, when userland does a sigreturn, the CPU has forgotten that it was doing an indirect branch before the signal. (This specifically only occurs with signal handlers that sigreturn.)

This is because IA32_U_CET is part of XSAVE 'supervisor' state, so regular XSAVE/XRSTOR can't access it. Doing a manual backup is tricky.

A related problem is that the signal handler routine is not checked for endbr preamble.

Basic IBT is better than no IBT, though.

Richard Patel (5):
x86: add userspace IBT config option
x86: shstk: don't clobber IBT bits in U_CET MSR
x86: expose user IBT via PR_CFI_BRANCH_LANDING_PADS x86/entry/vdso: build with IBT support selftests/x86: test usermode IBT

 arch/x86/Kconfig                            |  18 ++
 arch/x86/entry/vdso/common/Makefile.include |   3 +-
 arch/x86/include/asm/cpufeatures.h          |   1 +
 arch/x86/include/asm/ibt.h                  |  14 ++
 arch/x86/include/asm/processor.h            |   5 +
 arch/x86/kernel/Makefile                    |   1 +
 arch/x86/kernel/cet.c                       |   3 +-
 arch/x86/kernel/cpu/common.c                |  14 +-
 arch/x86/kernel/ibt.c                       |  98 ++++++++
 arch/x86/kernel/process_64.c                |   2 +
 arch/x86/kernel/shstk.c                     |  12 +-
 tools/arch/x86/include/asm/cpufeatures.h    |   1 +
 tools/testing/selftests/x86/Makefile        |   5 +-
 tools/testing/selftests/x86/user_ibt.c      | 247 ++++++++++++++++++++
 14 files changed, 416 insertions(+), 8 deletions(-)
 create mode 100644 arch/x86/kernel/ibt.c
 create mode 100644 tools/testing/selftests/x86/user_ibt.c

--
2.47.3

#1 lore raw

PATCH v2 1/5 x86: add userspace IBT config option

Adds a X86_USER_IBT Kconfig option and "nouseribt" command-line option. Default disabled for now.

These prepare for userspace support for IBT (forward-edge control flow integrity protection).

User IBT works even if kernel IBT is disabled. However, ibt=off also disables user IBT.

Depends on usermode shadow stack, which enables XSAVES for the IA32_U_CET MSR, containing both SHSTK and IBT state.

---
 arch/x86/Kconfig                         | 18 ++++++++++++++++++
 arch/x86/include/asm/cpufeatures.h       |  1 +
 arch/x86/kernel/cet.c                    |  3 ++-
 arch/x86/kernel/cpu/common.c             | 14 ++++++++++++--
 tools/arch/x86/include/asm/cpufeatures.h |  1 +
 5 files changed, 34 insertions(+), 3 deletions(-)
arch/x86/Kconfig +18 -0
index f3f7cb01d69d..90d5d14fd407 100644
@@ -1901,6 +1901,24 @@ config X86_USER_SHADOW_STACK
 
 	  If unsure, say N.
 
+config X86_USER_IBT
+	bool "X86 userspace indirect branch tracking"
+	depends on X86_64
+	depends on X86_USER_SHADOW_STACK
+	select X86_CET
+	help
+	  Support Indirect Branch Tracking protection for userspace
+	  applications. IBT is a hardware-supported coarse-grained
+	  forward-edge Control Flow Integrity protection feature.
+	  It enforces that all indirect calls must land on an ENDBR
+	  instruction. Applications must be enabled to use it, and old
+	  userspace does not get protection "for free". Enables the
+	  PR_CFI_BRANCH_LANDING_PADS prctl CFI option.
+
+	  CPUs supporting IBT were first released in 2021.
+
+	  If unsure, say N.
+
 config INTEL_TDX_HOST
 	bool "Intel Trust Domain Extensions (TDX) host support"
 	depends on CPU_SUP_INTEL
arch/x86/include/asm/cpufeatures.h +1 -0
index 1d506e5d6f46..1825cbf864c0 100644
@@ -516,6 +516,7 @@
 						      * and purposes if CLEAR_CPU_BUF_VM is set).
 						      */
 #define X86_FEATURE_X2AVIC_EXT		(21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
+#define X86_FEATURE_USER_IBT		(21*32+21) /* Indirect Branch Tracking for user mode applications */
 
 /*
  * BUG word(s)
arch/x86/kernel/cet.c +2 -1
index 99444409c026..3ccf47e82da1 100644
@@ -149,7 +149,8 @@ __setup("ibt=", ibt_setup);
 DEFINE_IDTENTRY_ERRORCODE(exc_control_protection)
 {
 	if (user_mode(regs)) {
-		if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
+		if (cpu_feature_enabled(X86_FEATURE_USER_SHSTK) ||
+		    cpu_feature_enabled(X86_FEATURE_USER_IBT))
 			do_user_cp_fault(regs, error_code);
 		else
 			do_unexpected_cp(regs, error_code);
arch/x86/kernel/cpu/common.c +12 -2
index a4268c47f2bc..2839edd92331 100644
@@ -634,7 +634,7 @@ __noendbr void ibt_restore(u64 save)
 
 static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 {
-	bool user_shstk, kernel_ibt;
+	bool user_shstk, kernel_ibt, user_ibt;
 
 	if (!IS_ENABLED(CONFIG_X86_CET))
 		return;
@@ -642,13 +642,19 @@ static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 	kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT);
 	user_shstk = cpu_feature_enabled(X86_FEATURE_SHSTK) &&
 		     IS_ENABLED(CONFIG_X86_USER_SHADOW_STACK);
+	/* User IBT only needs hardware IBT, not kernel-enabled IBT. */
+	user_ibt = cpu_has(c, X86_FEATURE_IBT) &&
+		   IS_ENABLED(CONFIG_X86_USER_IBT);
 
-	if (!kernel_ibt && !user_shstk)
+	if (!kernel_ibt && !user_shstk && !user_ibt)
 		return;
 
 	if (user_shstk)
 		set_cpu_cap(c, X86_FEATURE_USER_SHSTK);
 
+	if (user_ibt)
+		set_cpu_cap(c, X86_FEATURE_USER_IBT);
+
 	if (kernel_ibt)
 		wrmsrq(MSR_IA32_S_CET, CET_ENDBR_EN);
 	else
@@ -666,6 +672,7 @@ static __always_inline void setup_cet(struct cpuinfo_x86 *c)
 __noendbr void cet_disable(void)
 {
 	if (!(cpu_feature_enabled(X86_FEATURE_IBT) ||
+	      cpu_feature_enabled(X86_FEATURE_USER_IBT) ||
 	      cpu_feature_enabled(X86_FEATURE_SHSTK)))
 		return;
 
@@ -1760,6 +1767,9 @@ static void __init cpu_parse_early_param(void)
 	if (cmdline_find_option_bool(boot_command_line, "nousershstk"))
 		setup_clear_cpu_cap(X86_FEATURE_USER_SHSTK);
 
+	if (cmdline_find_option_bool(boot_command_line, "nouseribt"))
+		setup_clear_cpu_cap(X86_FEATURE_USER_IBT);
+
 	/* Minimize the gap between FRED is available and available but disabled. */
 	arglen = cmdline_find_option(boot_command_line, "fred", arg, sizeof(arg));
 	if (arglen == 3 && !strncmp(arg, "off", 3))
tools/arch/x86/include/asm/cpufeatures.h +1 -0
index 86d17b195e79..1cf22d27c7a1 100644
@@ -515,6 +515,7 @@
 						      * and purposes if CLEAR_CPU_BUF_VM is set).
 						      */
 #define X86_FEATURE_X2AVIC_EXT		(21*32+20) /* AMD SVM x2AVIC support for 4k vCPUs */
+#define X86_FEATURE_USER_IBT		(21*32+21) /* Indirect Branch Tracking for user mode applications */
 
 /*
  * BUG word(s)
Signature · 1 line
2.47.3
#2 lore raw

PATCH v2 2/5 x86: shstk: don't clobber IBT bits in U_CET MSR

In reply to Richard Patel

Updates usermode shadow stack code to not set IBT-related bits in the U_CET MSR.

---
 arch/x86/kernel/shstk.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
arch/x86/kernel/shstk.c +10 -2
index 0ca64900192f..ff4106dcfec4 100644
@@ -150,6 +150,7 @@ static int shstk_setup(void)
 {
 	struct thread_shstk *shstk = &current->thread.shstk;
 	unsigned long addr, size;
+	u64 msrval;
 
 	/* Already enabled */
 	if (features_enabled(ARCH_SHSTK_SHSTK))
@@ -166,7 +167,10 @@ static int shstk_setup(void)
 
 	fpregs_lock_and_load();
 	wrmsrq(MSR_IA32_PL3_SSP, addr + size);
-	wrmsrq(MSR_IA32_U_CET, CET_SHSTK_EN);
+	rdmsrq(MSR_IA32_U_CET, msrval);
+	msrval &= ~CET_WRSS_EN;
+	msrval |= CET_SHSTK_EN;
+	wrmsrq(MSR_IA32_U_CET, msrval);
 	fpregs_unlock();
 
 	shstk->base = addr;
@@ -520,6 +524,8 @@ static int wrss_control(bool enable)
 
 static int shstk_disable(void)
 {
+	u64 msrval;
+
 	if (!cpu_feature_enabled(X86_FEATURE_USER_SHSTK))
 		return -EOPNOTSUPP;
 
@@ -528,8 +534,10 @@ static int shstk_disable(void)
 		return 0;
 
 	fpregs_lock_and_load();
+	rdmsrq(MSR_IA32_U_CET, msrval);
 	/* Disable WRSS too when disabling shadow stack */
-	wrmsrq(MSR_IA32_U_CET, 0);
+	msrval &= ~(CET_SHSTK_EN | CET_WRSS_EN);
+	wrmsrq(MSR_IA32_U_CET, msrval);
 	wrmsrq(MSR_IA32_PL3_SSP, 0);
 	fpregs_unlock();
 
Signature · 1 line
2.47.3
#3 lore raw

PATCH v2 3/5 x86: expose user IBT via PR_CFI_BRANCH_LANDING_PADS

In reply to Richard Patel

Allows userspace applications to enable IBT (forward-edge control flow integrity protection) using the portable PR_CFI prctl API.

The name 'branch landing pads' is RISC-V specific, but the mechanism is nearly identical in x86.

This setting enables the following MSR_IA32_U_CET bits:
- CET_ENDBR_EN (enforce endbr as indirect branch target)
- CET_NOTRACK_EN (jump modifier to opt-out of IBT checking)

Kernel-mode IBT (as part of CFI) bans notrack. A future prctl flag could be introduced to ban notrack in usermode too.

---
 arch/x86/include/asm/ibt.h       | 14 +++++
 arch/x86/include/asm/processor.h |  5 ++
 arch/x86/kernel/Makefile         |  1 +
 arch/x86/kernel/ibt.c            | 98 ++++++++++++++++++++++++++++++++
 arch/x86/kernel/process_64.c     |  2 +
 5 files changed, 120 insertions(+)
 create mode 100644 arch/x86/kernel/ibt.c
arch/x86/include/asm/ibt.h +14 -0
index 5e45d6424722..586e5fadf844 100644
@@ -114,4 +114,18 @@ static inline void ibt_restore(u64 save) { }
 
 #define ENDBR_INSN_SIZE		(4*HAS_KERNEL_IBT)
 
+#ifndef __ASSEMBLER__
+
+#include <linux/prctl.h>
+
+#define PR_CFI_SUPPORTED_STATUS_MASK (PR_CFI_ENABLE | PR_CFI_DISABLE | PR_CFI_LOCK)
+
+#ifdef CONFIG_X86_USER_IBT
+void reset_thread_ibt(void);
+#else
+static inline void reset_thread_ibt(void) {}
+#endif /* CONFIG_X86_USER_IBT */
+
+#endif /* __ASSEMBLER__ */
+
 #endif /* _ASM_X86_IBT_H */
arch/x86/include/asm/processor.h +5 -0
index 67dd932305db..7fbf10410973 100644
@@ -504,6 +504,11 @@ struct thread_struct {
 
 	unsigned int		iopl_warn:1;
 
+#ifdef CONFIG_X86_USER_IBT
+	unsigned int		ibt:1;
+	unsigned int		ibt_locked:1;
+#endif
+
 	/*
 	 * Protection Keys Register for Userspace.  Loaded immediately on
 	 * context switch. Store it in thread_struct to avoid a lookup in
arch/x86/kernel/Makefile +1 -0
index 47a32f583930..05c87f014552 100644
@@ -169,6 +169,7 @@ obj-$(CONFIG_CALL_THUNKS)		+= callthunks.o
 obj-$(CONFIG_X86_CET)			+= cet.o
 
 obj-$(CONFIG_X86_USER_SHADOW_STACK)	+= shstk.o
+obj-$(CONFIG_X86_USER_IBT)		+= ibt.o
 
 ###
 # 64 bit specific files
arch/x86/kernel/ibt.c new file +98 -0

98 lines not on this page · raw message

arch/x86/kernel/process_64.c +2 -0
index b85e715ebb30..4b727cc7bccb 100644
@@ -59,6 +59,7 @@
 #include <asm/fsgsbase.h>
 #include <asm/fred.h>
 #include <asm/msr.h>
+#include <asm/ibt.h>
 #ifdef CONFIG_IA32_EMULATION
 /* Not included via unistd.h */
 #include <asm/unistd_32_ia32.h>
@@ -540,6 +541,7 @@ start_thread_common(struct pt_regs *regs, unsigned long new_ip,
 	}
 
 	reset_thread_features();
+	reset_thread_ibt();
 
 	loadsegment(fs, 0);
 	loadsegment(es, _ds);
Signature · 1 line
2.47.3
#4 lore raw

PATCH v2 4/5 x86/entry/vdso: build with IBT support

In reply to Richard Patel

VDSO should expose ENDBR instructions now that usermode IBT is available.

---
 arch/x86/entry/vdso/common/Makefile.include | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/entry/vdso/common/Makefile.include +1 -2
index 687b3d89b40d..a0dc69b7a330 100644
@@ -46,8 +46,7 @@ flags-y += -fasynchronous-unwind-tables
 # Reset cf protections enabled by compiler default
 flags-y += $(call cc-option, -fcf-protection=none)
 flags-$(X86_USER_SHADOW_STACK) += $(call cc-option, -fcf-protection=return)
-# When user space IBT is supported, enable this.
-# flags-$(CONFIG_USER_IBT) += $(call cc-option, -fcf-protection=branch)
+flags-$(CONFIG_X86_USER_IBT) += $(call cc-option, -fcf-protection=branch)
 
 flags-$(CONFIG_MITIGATION_RETPOLINE) += $(RETPOLINE_VDSO_CFLAGS)
 
Signature · 1 line
2.47.3
#5 lore raw

Re: [PATCH v2 4/5] x86/entry/vdso: build with IBT support

* Richard Patel:

Quoted 13 lines

VDSO should expose ENDBR instructions now that usermode IBT is available.

---
 arch/x86/entry/vdso/common/Makefile.include | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/entry/vdso/common/Makefile.include +0 -0
index 687b3d89b40d..a0dc69b7a330 100644

arch/x86/entry/vdso/common/Makefile.include flags-y += -fasynchronous-unwind-tables · line 51

Quoted 8 lines
arch/x86/entry/vdso/common/Makefile.include +1 -2
@@ -46,8 +46,7 @@ flags-y += -fasynchronous-unwind-tables
 # Reset cf protections enabled by compiler default
 flags-y += $(call cc-option, -fcf-protection=none)
 flags-$(X86_USER_SHADOW_STACK) += $(call cc-option, -fcf-protection=return)
-# When user space IBT is supported, enable this.
-# flags-$(CONFIG_USER_IBT) += $(call cc-option, -fcf-protection=branch)
+flags-$(CONFIG_X86_USER_IBT) += $(call cc-option, -fcf-protection=branch)
 
 flags-$(CONFIG_MITIGATION_RETPOLINE) += $(RETPOLINE_VDSO_CFLAGS)

Surprisingly, the option is cumulative in GCC. I didn't expect that. So this in fact works with GCC.

However, it's not cumulative in Clang, and -fcf-protection=branch overrides the earlier -fcf-protection=return.

Thanks,
Florian

#6 lore raw

PATCH v2 5/5 selftests/x86: test usermode IBT

In reply to Richard Patel

Adds a basic selftest exercising a usermode IBT violation.

---
 tools/testing/selftests/x86/Makefile   |   5 +-
 tools/testing/selftests/x86/user_ibt.c | 247 +++++++++++++++++++++++++
 2 files changed, 251 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/x86/user_ibt.c
tools/testing/selftests/x86/Makefile +4 -1
index 434065215d12..59f2ba3ec4ea 100644
@@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh "$(CC)" trivial_program.c -no-pie)
 TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \
 			check_initial_reg_state sigreturn iopl ioperm \
 			test_vsyscall mov_ss_trap sigtrap_loop \
-			syscall_arg_fault fsgsbase_restore sigaltstack
+			syscall_arg_fault fsgsbase_restore sigaltstack user_ibt
 TARGETS_C_BOTHBITS += nx_stack
 TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \
 			test_FCMOV test_FCOMI test_FISTTP \
@@ -138,3 +138,6 @@ $(OUTPUT)/avx_64: CFLAGS += -mno-avx -mno-avx512f
 $(OUTPUT)/amx_64: EXTRA_FILES += xstate.c
 $(OUTPUT)/avx_64: EXTRA_FILES += xstate.c
 $(OUTPUT)/apx_64: EXTRA_FILES += xstate.c
+
+$(OUTPUT)/user_ibt_32: CFLAGS += -fcf-protection=branch
+$(OUTPUT)/user_ibt_64: CFLAGS += -fcf-protection=branch
tools/testing/selftests/x86/user_ibt.c new file +247 -0

248 lines not on this page · raw message

Signature · 1 line
2.47.3
#7 lore raw
In reply to Richard Patel
Quoted 10 lines · Richard Patel

The above sequence does not crash.

With IBT, it should crash at the nop (because an endr64 is expected there). The IBT state (WAIT_FOR_ENDBR in IA32_U_CET MSR) is not backed up to the signal frame though. So, when userland does a sigreturn, the CPU has forgotten that it was doing an indirect branch before the signal. (This specifically only occurs with signal handlers that sigreturn.)

This is because IA32_U_CET is part of XSAVE 'supervisor' state, so regular XSAVE/XRSTOR can't access it. Doing a manual backup is tricky.

WAIT_FOR_ENDBR should be part of the exception frame with FRED, so if you're on FRED hardware, this should be fixed IIRC.

#8 lore raw
In reply to Richard Patel

* Richard Patel:

Quoted 10 lines

Adds basic support for x86 userspace IBT.

IBT is part of Intel CET. It requires indirect call and jump targets to start with an endbr{32,64} instruction, otherwise throwing #CP.

In summary, this patch does 3 things:
- Config wiring ensuring supervisor XSAVE contains IBT state
- Allow userspace to enable IBT via prctl(PR_CFI_*) for an entire thread
- Enable IBT support (ENDBR instructions) in VDSO

Quoted 6 lines

Unlike the arm64 BTI API:
- does not support mixed usermode (all or nothing)
- does not touch page table code
- not enabled automatically (no ELF GNU note parsing)
- temporarily disables IBT enforcement when handling signals These can all be cleanly added later.

Adding the ELF GNU note parsing can be added later, but perhaps not cleanly. I'm still a bit worried we might have to rev the markup because too many binaries are in circulation that claim compatibility, have never been tested, and are actually broken. If the kernel does not look at the ELF bits, things a slightly simpler.

How do you detect that handling a signal is complete and IBT can be re-enabled? Or is it re-enabled before entering the userspace signal handler?

Quoted 1 line

The main question is whether glibc is happy with this prctl syscall API.

As far as I can tell, the prctl works for glibc. Re-use of an arch_prctl constant might have been problematic, but the series is not doing that.

Quoted 10 lines

There is one notable gap in this patch series, to do with signals:

000a: mov rax, 0x100a
000f: jmp rax
*** signal occurs ***
*** signal handler runs, does sigreturn ***
100a: nop

The above sequence does not crash.

Quoted 8 lines

With IBT, it should crash at the nop (because an endr64 is expected there). The IBT state (WAIT_FOR_ENDBR in IA32_U_CET MSR) is not backed up to the signal frame though. So, when userland does a sigreturn, the CPU has forgotten that it was doing an indirect branch before the signal. (This specifically only occurs with signal handlers that sigreturn.)

This is because IA32_U_CET is part of XSAVE 'supervisor' state, so regular XSAVE/XRSTOR can't access it. Doing a manual backup is tricky.

That's a bit annoying. Is this restricted to signal handlers, or does it apply to page faults, too?

Quoted 2 lines

A related problem is that the signal handler routine is not checked for endbr preamble.

That's not necessarily a problem because its address cannot be directly overwritten in userspace. Not all indirect branches need to be checked, only those that have tweakable targets. In fact, fewer ENDBR64 markers are better (although we wouldn't drop the marker from a signal handler specifically, of course).

Thanks,
Florian

#9 lore raw
Quoted 3 lines · Florian Weimer

How do you detect that handling a signal is complete and IBT can be re-enabled? Or is it re-enabled before entering the userspace signal handler?

Hi Florian,

In v1, we backed up the IBT CPU state into the (user-accessible) signal frame from FRED/XSAVE, then restored it:
https://lore.kernel.org/lkml/[email protected]/

In v2, when entering the signal handler, the kernel just context switches to the new user rip, bypassing IBT checks (continues executing if the signal handler does not begin with endbr).

IBT stays enabled in both designs, just the IBT state is preserved in v1, and lost in v2.

The same thing happens when doing a sigreturn in v2 (e.g. via trampoline), again IBT is not enforced. IBT stays enabled when doing a siglongjmp, though.

Some time in the future, ideally:
- signal handler is *required* to start with endbr (this is easy)
- sigreturn as in my asm example enforces endbr after returning from a signal handler to a in-progres indirect branc
- libc (sig)longjmp is made IBT-compatible

Btw, I had self-tests for the v1 design, and {signal handle,rt_sigreturn, siglongjmp} with {success case,violation} works flawlessly with Fedora 44 glibc amd64. With glibc i686 I ran into PLT issues, probably my fault.

It is quite surprised that siglongjmp was working, btw, since the glibc longjmp code uses 'jmp *reg' (without notrack prefix). I guess you do an endbr64 at the setjmp side?

Quoted 5 lines
Quoted 1 line

The main question is whether glibc is happy with this prctl syscall API.

As far as I can tell, the prctl works for glibc. Re-use of an arch_prctl constant might have been problematic, but the series is not doing that.

Nice :-)
The alternative would have been to bolt on stuff to ARCH_SHSTK, or create an entirely new arch_prctl. Open to any API.

Quoted 5 lines

Adding the ELF GNU note parsing can be added later, but perhaps not cleanly. I'm still a bit worried we might have to rev the markup because too many binaries are in circulation that claim compatibility, have never been tested, and are actually broken. If the kernel does not look at the ELF bits, things a slightly simpler.

Phew, I was hoping you'd say that.

If you want, I can sketch out glibc IBT enabling and test it on Debian and Fedora, which IIRC already emit compile with -fcf-protection=branch for all OS packages.

Quoted 16 lines
Quoted 16 lines

There is one notable gap in this patch series, to do with signals:

000a: mov rax, 0x100a
000f: jmp rax
*** signal occurs ***
*** signal handler runs, does sigreturn ***
100a: nop

The above sequence does not crash.

With IBT, it should crash at the nop (because an endr64 is expected there). The IBT state (WAIT_FOR_ENDBR in IA32_U_CET MSR) is not backed up to the signal frame though. So, when userland does a sigreturn, the CPU has forgotten that it was doing an indirect branch before the signal. (This specifically only occurs with signal handlers that sigreturn.)

Quoted 5 lines
Quoted 2 lines

This is because IA32_U_CET is part of XSAVE 'supervisor' state, so regular XSAVE/XRSTOR can't access it. Doing a manual backup is tricky.

That's a bit annoying. Is this restricted to signal handlers, or does it apply to page faults, too?

Only signal handlers, page faults don't reset IBT.

Quoted 8 lines
Quoted 2 lines

A related problem is that the signal handler routine is not checked for endbr preamble.

That's not necessarily a problem because its address cannot be directly overwritten in userspace. Not all indirect branches need to be checked, only those that have tweakable targets. In fact, fewer ENDBR64 markers are better (although we wouldn't drop the marker from a signal handler specifically, of course).

Just one concern I have is that people start relying on signal handlers not requiring endbr64, and then a future kernel version breaking them once we enforce it.

Really appreciate your review,

-Richard

#10 lore raw

* Richard Patel:

Quoted 8 lines · Florian Weimer
Quoted 4 lines

How do you detect that handling a signal is complete and IBT can be re-enabled? Or is it re-enabled before entering the userspace signal handler?

Hi Florian,

Quoted 7 lines

In v1, we backed up the IBT CPU state into the (user-accessible) signal frame from FRED/XSAVE, then restored it:
https://lore.kernel.org/lkml/[email protected]/

In v2, when entering the signal handler, the kernel just context switches to the new user rip, bypassing IBT checks (continues executing if the signal handler does not begin with endbr).

What's the reason for this?

Quoted 5 lines

Some time in the future, ideally:
- signal handler is *required* to start with endbr (this is easy)
- sigreturn as in my asm example enforces endbr after returning from a signal handler to a in-progres indirect branc
- libc (sig)longjmp is made IBT-compatible

I think the compiler already emits ENDBR markers for returns-twice functions, which is why longjmp does not use a no-track jump. Other architectures require such a proliferation of markers because they do not support no-track jumps at all. However, longjmp is arguable a corner case. It's not completely safe, like loading a function address from a RELRO GOT and jumping to it.

Quoted 3 lines

Btw, I had self-tests for the v1 design, and {signal handle,rt_sigreturn, siglongjmp} with {success case,violation} works flawlessly with Fedora 44 glibc amd64. With glibc i686 I ran into PLT issues, probably my fault.

There's no IBT support planned for i686, that's why we dropped all marker instructions in Fedora.

Quoted 3 lines

It is quite surprised that siglongjmp was working, btw, since the glibc longjmp code uses 'jmp *reg' (without notrack prefix). I guess you do an endbr64 at the setjmp side?

Yes, compilers generate landing pads for returns-twice functions. Not ideal, but it's the only way to get setjmp working on targets without NOTRACK.

Quoted 11 lines
Quoted 5 lines

Adding the ELF GNU note parsing can be added later, but perhaps not cleanly. I'm still a bit worried we might have to rev the markup because too many binaries are in circulation that claim compatibility, have never been tested, and are actually broken. If the kernel does not look at the ELF bits, things a slightly simpler.

Phew, I was hoping you'd say that.

If you want, I can sketch out glibc IBT enabling and test it on Debian and Fedora, which IIRC already emit compile with -fcf-protection=branch for all OS packages.

For Fedora, please coordinate with Arjun (Cc:ed), who is going through the motions of enabling SHSTK for real.

Quoted 9 lines
Quoted 5 lines

That's not necessarily a problem because its address cannot be directly overwritten in userspace. Not all indirect branches need to be checked, only those that have tweakable targets. In fact, fewer ENDBR64 markers are better (although we wouldn't drop the marker from a signal handler specifically, of course).

Just one concern I have is that people start relying on signal handlers not requiring endbr64, and then a future kernel version breaking them once we enforce it.

Would software enforcement be a possibility? The kernel could check if the landing pad is there.

Thanks,
Florian

#11 lore raw
Quoted 14 lines · Florian Weimer

* Richard Patel:

Quoted 12 lines · Florian Weimer
Quoted 4 lines

How do you detect that handling a signal is complete and IBT can be re-enabled? Or is it re-enabled before entering the userspace signal handler?

Hi Florian,

In v1, we backed up the IBT CPU state into the (user-accessible) signal frame from FRED/XSAVE, then restored it:
https://lore.kernel.org/lkml/[email protected]/

Quoted 5 lines
Quoted 3 lines

In v2, when entering the signal handler, the kernel just context switches to the new user rip, bypassing IBT checks (continues executing if the signal handler does not begin with endbr).

What's the reason for this?

Hi Florian,

We just don't have a nice way to include IBT state in the signal frame right now. v1 had an uabi change (adding a new bit in ucontext_t uc_flags), which was originally proposed by Intel years ago. My preferred way to add IBT state is to carve out an XSAVE area in fpstate, which works well with all the existing signal frame code.

But I figured it's better to just keep the first pass at user IBT super simple, in the hopes upstream is more inclined to accept that.

BTW, OpenBSD uses the v2 approach (don't preserve IBT state across signal handlers), presumably because it's also hard for them to restore IBT state on sigreturn.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

That's not necessarily a problem because its address cannot be directly overwritten in userspace. Not all indirect branches need to be checked, only those that have tweakable targets. In fact, fewer ENDBR64 markers are better (although we wouldn't drop the marker from a signal handler specifically, of course).

Quoted 6 lines
Quoted 3 lines

Just one concern I have is that people start relying on signal handlers not requiring endbr64, and then a future kernel version breaking them once we enforce it.

Would software enforcement be a possibility? The kernel could check if the landing pad is there.

Enforcement is the easy part. I can trivially add back 'check if signal handler starts with endbr64'. Just the backup/restore of the pre-signal handler state ('do I expect an endbr64 after returning') is the tricky part.

Thank you,
-Richard