x86/ibt: Implement FineIBT
Subsystems (6)
Peter Zijlstra
maintainercover x86/ibt: Implement FineIBT
Hi all,
Updated FineIBT series; I've (hopefully) incorporated all feedback from last time with the notable exception of the Kconfig CFI default -- I'm not sure we want to add to the Kconfig space for this, also what would a distro do with it.
Anyway; please have a look, I'm hoping to merge this soonish so we can make the next cycle.
Peter Zijlstra
maintainerPATCH 1/4 objtool: Add --cfi to generate the .cfi_sites section
Add the location of all __cfi_##name symbols (as generated by kCFI) to a section such that we might re-write things at kernel boot.
Notably; boot time re-hashing and FineIBT are the intended use of this.
--- tools/objtool/builtin-check.c | 1 tools/objtool/check.c | 69 ++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 3 files changed, 71 insertions(+)
tools/objtool/builtin-check.c +1 -0
@@ -79,6 +79,7 @@ const struct option check_options[] = {OPT_BOOLEAN ('s' ,"stackval" , &opts.stackval,"validate frame pointer rules" ),OPT_BOOLEAN ('t' ,"static-call" , &opts.static_call,"annotate static calls" ),OPT_BOOLEAN ('u' ,"uaccess" , &opts.uaccess,"validate uaccess rules for SMAP" ), +OPT_BOOLEAN (0 ,"cfi" , &opts.cfi,"annotate kernel control flow integrity (kCFI) function preambles" ),OPT_CALLBACK_OPTARG (0 ,"dump" ,NULL ,NULL ,"orc" ,"dump metadata" , parse_dump), OPT_GROUP("Options:"),
tools/objtool/check.c +69 -0
81 lines not on this page · raw message
tools/objtool/include/objtool/builtin.h +1 -0
@@ -26,6 +26,7 @@ struct opts {bool stackval;bool static_call;bool uaccess; +bool cfi;/* options: */ bool backtrace;
[tip: x86/core] objtool: Add --cfi to generate the .cfi_sites section
The following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 9a479f766be1dd777e12e3e57b6ee4c3028a40a5 Gitweb: https://git.kernel.org/tip/9a479f766be1dd777e12e3e57b6ee4c3028a40a5 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 27 Oct 2022 11:28:13 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 01 Nov 2022 13:44:10 +01:00
objtool: Add --cfi to generate the .cfi_sites section
Add the location of all __cfi_##name symbols (as generated by kCFI) to a section such that we might re-write things at kernel boot.
Notably; boot time re-hashing and FineIBT are the intended use of this.
--- tools/objtool/builtin-check.c | 1 +- tools/objtool/check.c | 69 ++++++++++++++++++++++++- tools/objtool/include/objtool/builtin.h | 1 +- 3 files changed, 71 insertions(+)
tools/objtool/builtin-check.c +1 -0
@@ -80,6 +80,7 @@ const struct option check_options[] = {OPT_BOOLEAN ('s' ,"stackval" , &opts.stackval,"validate frame pointer rules" ),OPT_BOOLEAN ('t' ,"static-call" , &opts.static_call,"annotate static calls" ),OPT_BOOLEAN ('u' ,"uaccess" , &opts.uaccess,"validate uaccess rules for SMAP" ), +OPT_BOOLEAN (0 ,"cfi" , &opts.cfi,"annotate kernel control flow integrity (kCFI) function preambles" ),OPT_CALLBACK_OPTARG (0 ,"dump" ,NULL ,NULL ,"orc" ,"dump metadata" , parse_dump), OPT_GROUP("Options:"),
tools/objtool/check.c +69 -0
81 lines not on this page · raw message
tools/objtool/include/objtool/builtin.h +1 -0
@@ -27,6 +27,7 @@ struct opts {bool static_call;bool uaccess;int prefix; +bool cfi;/* options: */ bool backtrace;
Peter Zijlstra
maintainerPATCH 2/4 x86/ibt: Implement FineIBT
In reply to Peter ZijlstraImplement an alternative CFI scheme that merges both the fine-grained nature of kCFI but also takes full advantage of the coarse grained hardware CFI as provided by IBT.
To contrast:
kCFI is a pure software CFI scheme and relies on being able to read
text -- specifically the instruction *before* the target symbol, and does the hash validation *before* doing the call (otherwise control flow is compromised already).
FineIBT is a software and hardware hybrid scheme; by ensuring every
branch target starts with a hash validation it is possible to place the hash validation after the branch. This has several advantages:
o the (hash) load is avoided; no memop; no RX requirement.
o IBT WAIT-FOR-ENDBR state is a speculation stop; by placing the hash validation in the immediate instruction after the branch target there is a minimal speculation window and the whole is a viable defence against SpectreBHB.
o Kees feels obliged to mention it is slightly more vulnerable when the attacker can write code.
Obviously this patch relies on kCFI, but additionally it also relies on the padding from the call-depth-tracking patches. It uses this padding to place the hash-validation while the call-sites are re-written to modify the indirect target to be 16 bytes in front of the original target, thus hitting this new preamble.
Notably, there is no hardware that needs call-depth-tracking (Skylake) and supports IBT (Tigerlake and onwards).
--- arch/um/kernel/um_arch.c | 5 arch/x86/Kconfig | 12 + arch/x86/Makefile | 2 arch/x86/include/asm/alternative.h | 2 arch/x86/include/asm/linkage.h | 6 arch/x86/kernel/alternative.c | 253 +++++++++++++++++++++++++++++++++++-- arch/x86/kernel/cpu/common.c | 1 arch/x86/kernel/module.c | 20 ++ arch/x86/kernel/vmlinux.lds.S | 9 + include/linux/bpf.h | 2 scripts/Makefile.lib | 1 11 files changed, 293 insertions(+), 20 deletions(-)
arch/um/kernel/um_arch.c +5 -0
@@ -444,6 +444,11 @@ void apply_returns(s32 *start, s32 *end) { } +void apply_fineibt (s32 *start_retpoline,s32 *end_retpoline, +s32 *start_cfi,s32 *end_cfi) +{ +} +void apply_alternatives (struct alt_instr *start,struct alt_instr *end) { }
arch/x86/Kconfig +11 -1
@@ -2463,13 +2463,23 @@ config FUNCTION_PADDING_BYTES default FUNCTION_PADDING_CFI if CFI_CLANG default FUNCTION_ALIGNMENT +config CALL_PADDING + def_bool n + depends on CC_HAS_ENTRY_PADDING && OBJTOOL + select FUNCTION_ALIGNMENT_16B + +config FINEIBT + def_bool y + depends on X86_KERNEL_IBT && CFI_CLANG && RETPOLINE + select CALL_PADDING + config HAVE_CALL_THUNKS def_bool y depends on CC_HAS_ENTRY_PADDING && RETHUNK && OBJTOOL config CALL_THUNKS def_bool n - select FUNCTION_ALIGNMENT_16B + select CALL_PADDING menuconfig SPECULATION_MITIGATIONS bool "Mitigations for speculative execution vulnerabilities"
arch/x86/Makefile +1 -1
@@ -208,7 +208,7 @@ ifdef CONFIG_SLS KBUILD_CFLAGS += -mharden-sls=all endif -ifdef CONFIG_CALL_THUNKS +ifdef CONFIG_CALL_PADDING PADDING_CFLAGS := -fpatchable-function-entry=$(CONFIG_FUNCTION_PADDING_BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) KBUILD_CFLAGS += $(PADDING_CFLAGS) export PADDING_CFLAGS
arch/x86/include/asm/alternative.h +2 -0
@@ -78,6 +78,8 @@ extern void apply_alternatives(struct alextern void apply_retpolines (s32 *start,s32 *end);extern void apply_returns (s32 *start,s32 *end);extern void apply_ibt_endbr (s32 *start,s32 *end); +extern void apply_fineibt (s32 *start_retpoline,s32 *end_retpoine, +s32 *start_cfi,s32 *end_cfi);struct module ;struct paravirt_patch_site ;
arch/x86/include/asm/linkage.h +3 -3
@@ -15,7 +15,7 @@#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT, 0x90;#define __ALIGN_STR __stringify(__ALIGN) -#if defined(CONFIG_CALL_THUNKS ) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO ) +#if defined(CONFIG_CALL_PADDING ) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO )#define FUNCTION_PADDING .skip CONFIG_FUNCTION_ALIGNMENT, 0x90;#else #define FUNCTION_PADDING
@@ -57,7 +57,7 @@#endif /* __ASSEMBLY__ */ /* -* Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_THUNKS) the +* Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_PADDING) the * CFI symbol layout changes. * * Without CALL_THUNKS:
@@ -81,7 +81,7 @@* In both cases the whole thing is FUNCTION_ALIGNMENT aligned and sized. */ -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define CFI_PRE_PADDING #define CFI_POST_PADDING .skip CONFIG_FUNCTION_PADDING_BYTES, 0x90;#else
arch/x86/kernel/alternative.c +240 -13
289 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +1 -0
@@ -609,6 +609,7 @@ static __always_inline void setup_cet(stif (!ibt_selftest ()) {pr_err ("IBT selftest: Failed!\n" ); +wrmsrl (MSR_IA32_S_CET ,0 );setup_clear_cpu_cap (X86_FEATURE_IBT );return ; }
arch/x86/kernel/module.c +19 -1
@@ -255,7 +255,7 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *s, *text =NULL , *alt =NULL , *locks =NULL , *para =NULL , *orc =NULL , *orc_ip =NULL , *retpolines =NULL , *returns =NULL , *ibt_endbr =NULL , - *calls =NULL ; + *calls =NULL , *cfi =NULL ;char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
@@ -277,6 +277,8 @@ int module_finalize(const Elf_Ehdr *hdr, returns = s;if (!strcmp (".call_sites" , secstrings + s->sh_name)) calls = s; +if (!strcmp (".cfi_sites" , secstrings + s->sh_name)) + cfi = s;if (!strcmp (".ibt_endbr_seal" , secstrings + s->sh_name)) ibt_endbr = s; }
@@ -289,6 +291,22 @@ int module_finalize(const Elf_Ehdr *hdr,void *pseg = (void *)para->sh_addr;apply_paravirt (pseg, pseg + para->sh_size); } +if (retpolines || cfi) { +void *rseg =NULL , *cseg =NULL ; +unsigned int rsize =0 , csize =0 ; + +if (retpolines) { + rseg = (void *)retpolines->sh_addr; + rsize = retpolines->sh_size; + } + +if (cfi) { + cseg = (void *)cfi->sh_addr; + csize = cfi->sh_size; + } + +apply_fineibt (rseg, rseg + rsize, cseg, cseg + csize); + }if (retpolines) {void *rseg = (void *)retpolines->sh_addr;apply_retpolines (rseg, rseg + retpolines->sh_size);
arch/x86/kernel/vmlinux.lds.S +9 -0
@@ -309,6 +309,15 @@ SECTIONS } #endif +#ifdef CONFIG_FINEIBT + . = ALIGN(8); + .cfi_sites : AT(ADDR(.cfi_sites) - LOAD_OFFSET) { + __cfi_sites = .; + *(.cfi_sites) + __cfi_sites_end = .; + } +#endif + /* * struct alt_inst entries. From the header (alternative.h): * "Alternative instructions for different CPU types or capabilities"
include/linux/bpf.h +1 -1
@@ -984,7 +984,7 @@ int arch_prepare_bpf_dispatcher(void *im }#ifdef CONFIG_X86_64 -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5+CONFIG_FUNCTION_PADDING_BYTES,CONFIG_FUNCTION_PADDING_BYTES)))#else #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
scripts/Makefile.lib +1 -0
@@ -256,6 +256,7 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HA objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) += --hacks=noinstr objtool-args-$(CONFIG_CALL_DEPTH_TRACKING) += --hacks=skylake objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt +objtool-args-$(CONFIG_FINEIBT) += --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount objtool-args-$(CONFIG_UNWINDER_ORC) += --orc objtool-args-$(CONFIG_RETPOLINE) += --retpoline
Peter Zijlstra
maintainerscripts/Makefile.lib objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HA · line 262
Quoted 10 lines · Peter Zijlstra
scripts/Makefile.lib +1
-0@@ -256,6 +256,7 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HA objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) += --hacks=noinstr objtool-args-$(CONFIG_CALL_DEPTH_TRACKING) += --hacks=skylake objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt +objtool-args-$(CONFIG_FINEIBT) += --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount objtool-args-$(CONFIG_UNWINDER_ORC) += --orc objtool-args-$(CONFIG_RETPOLINE) += --retpoline
I seem to have lost a refresh; consider that whitespace fail fixed.
Quoted 31 lines · Peter Zijlstra
Implement an alternative CFI scheme that merges both the fine-grained nature of kCFI but also takes full advantage of the coarse grained hardware CFI as provided by IBT.
To contrast:
kCFI is a pure software CFI scheme and relies on being able to read
text -- specifically the instruction *before* the target symbol, and does the hash validation *before* doing the call (otherwise control flow is compromised already).FineIBT is a software and hardware hybrid scheme; by ensuring every
branch target starts with a hash validation it is possible to place the hash validation after the branch. This has several advantages:o the (hash) load is avoided; no memop; no RX requirement.
o IBT WAIT-FOR-ENDBR state is a speculation stop; by placing the hash validation in the immediate instruction after the branch target there is a minimal speculation window and the whole is a viable defence against SpectreBHB.
o Kees feels obliged to mention it is slightly more vulnerable when the attacker can write code.
Obviously this patch relies on kCFI, but additionally it also relies on the padding from the call-depth-tracking patches. It uses this padding to place the hash-validation while the call-sites are re-written to modify the indirect target to be 16 bytes in front of the original target, thus hitting this new preamble.
Quoted 5 lines
Notably, there is no hardware that needs call-depth-tracking (Skylake) and supports IBT (Tigerlake and onwards).
Suggested-by Joao Moreira (Intel) <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Signature · 1 line
Kees Cook
The following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 931ab63664f02b17d2213ef36b83e1e50190a0aa Gitweb: https://git.kernel.org/tip/931ab63664f02b17d2213ef36b83e1e50190a0aa Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 27 Oct 2022 11:28:14 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 01 Nov 2022 13:44:10 +01:00
x86/ibt: Implement FineIBT
Implement an alternative CFI scheme that merges both the fine-grained nature of kCFI but also takes full advantage of the coarse grained hardware CFI as provided by IBT.
To contrast:
kCFI is a pure software CFI scheme and relies on being able to read
text -- specifically the instruction *before* the target symbol, and does the hash validation *before* doing the call (otherwise control flow is compromised already).
FineIBT is a software and hardware hybrid scheme; by ensuring every
branch target starts with a hash validation it is possible to place the hash validation after the branch. This has several advantages:
o the (hash) load is avoided; no memop; no RX requirement.
o IBT WAIT-FOR-ENDBR state is a speculation stop; by placing the hash validation in the immediate instruction after the branch target there is a minimal speculation window and the whole is a viable defence against SpectreBHB.
o Kees feels obliged to mention it is slightly more vulnerable when the attacker can write code.
Obviously this patch relies on kCFI, but additionally it also relies on the padding from the call-depth-tracking patches. It uses this padding to place the hash-validation while the call-sites are re-written to modify the indirect target to be 16 bytes in front of the original target, thus hitting this new preamble.
Notably, there is no hardware that needs call-depth-tracking (Skylake) and supports IBT (Tigerlake and onwards).
--- arch/um/kernel/um_arch.c | 5 +- arch/x86/Kconfig | 14 +- arch/x86/Makefile | 2 +- arch/x86/include/asm/alternative.h | 2 +- arch/x86/include/asm/linkage.h | 6 +- arch/x86/kernel/alternative.c | 253 ++++++++++++++++++++++++++-- arch/x86/kernel/cpu/common.c | 1 +- arch/x86/kernel/module.c | 20 +- arch/x86/kernel/vmlinux.lds.S | 9 +- include/linux/bpf.h | 2 +- scripts/Makefile.lib | 1 +- 11 files changed, 294 insertions(+), 21 deletions(-)
arch/um/kernel/um_arch.c +5 -0
@@ -444,6 +444,11 @@ void apply_returns(s32 *start, s32 *end) { } +void apply_fineibt (s32 *start_retpoline,s32 *end_retpoline, +s32 *start_cfi,s32 *end_cfi) +{ +} +void apply_alternatives (struct alt_instr *start,struct alt_instr *end) { }
arch/x86/Kconfig +12 -2
@@ -2463,17 +2463,27 @@ config FUNCTION_PADDING_BYTES default FUNCTION_PADDING_CFI if CFI_CLANG default FUNCTION_ALIGNMENT +config CALL_PADDING + def_bool n + depends on CC_HAS_ENTRY_PADDING && OBJTOOL + select FUNCTION_ALIGNMENT_16B + +config FINEIBT + def_bool y + depends on X86_KERNEL_IBT && CFI_CLANG && RETPOLINE + select CALL_PADDING + config HAVE_CALL_THUNKS def_bool y depends on CC_HAS_ENTRY_PADDING && RETHUNK && OBJTOOL config CALL_THUNKS def_bool n - select FUNCTION_ALIGNMENT_16B + select CALL_PADDING config PREFIX_SYMBOLS def_bool y - depends on CALL_THUNKS && !CFI_CLANG + depends on CALL_PADDING && !CFI_CLANG menuconfig SPECULATION_MITIGATIONS bool "Mitigations for speculative execution vulnerabilities"
arch/x86/Makefile +1 -1
@@ -208,7 +208,7 @@ ifdef CONFIG_SLS KBUILD_CFLAGS += -mharden-sls=all endif -ifdef CONFIG_CALL_THUNKS +ifdef CONFIG_CALL_PADDING PADDING_CFLAGS := -fpatchable-function-entry=$(CONFIG_FUNCTION_PADDING_BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) KBUILD_CFLAGS += $(PADDING_CFLAGS) export PADDING_CFLAGS
arch/x86/include/asm/alternative.h +2 -0
@@ -78,6 +78,8 @@ extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);extern void apply_retpolines (s32 *start,s32 *end);extern void apply_returns (s32 *start,s32 *end);extern void apply_ibt_endbr (s32 *start,s32 *end); +extern void apply_fineibt (s32 *start_retpoline,s32 *end_retpoine, +s32 *start_cfi,s32 *end_cfi);struct module ;struct paravirt_patch_site ;
arch/x86/include/asm/linkage.h +3 -3
@@ -15,7 +15,7 @@#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT, 0x90;#define __ALIGN_STR __stringify(__ALIGN) -#if defined(CONFIG_CALL_THUNKS ) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO ) +#if defined(CONFIG_CALL_PADDING ) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO )#define FUNCTION_PADDING .skip CONFIG_FUNCTION_ALIGNMENT, 0x90;#else #define FUNCTION_PADDING
@@ -57,7 +57,7 @@#endif /* __ASSEMBLY__ */ /* -* Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_THUNKS) the +* Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_PADDING) the * CFI symbol layout changes. * * Without CALL_THUNKS:
@@ -81,7 +81,7 @@* In both cases the whole thing is FUNCTION_ALIGNMENT aligned and sized. */ -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define CFI_PRE_PADDING #define CFI_POST_PADDING .skip CONFIG_FUNCTION_PADDING_BYTES, 0x90;#else
arch/x86/kernel/alternative.c +240 -13
289 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +1 -0
@@ -609,6 +609,7 @@ static __always_inline void setup_cet(struct cpuinfo_x86 *c)if (!ibt_selftest ()) {pr_err ("IBT selftest: Failed!\n" ); +wrmsrl (MSR_IA32_S_CET ,0 );setup_clear_cpu_cap (X86_FEATURE_IBT );return ; }
arch/x86/kernel/module.c +19 -1
@@ -255,7 +255,7 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *s, *text =NULL , *alt =NULL , *locks =NULL , *para =NULL , *orc =NULL , *orc_ip =NULL , *retpolines =NULL , *returns =NULL , *ibt_endbr =NULL , - *calls =NULL ; + *calls =NULL , *cfi =NULL ;char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
@@ -277,6 +277,8 @@ int module_finalize(const Elf_Ehdr *hdr, returns = s;if (!strcmp (".call_sites" , secstrings + s->sh_name)) calls = s; +if (!strcmp (".cfi_sites" , secstrings + s->sh_name)) + cfi = s;if (!strcmp (".ibt_endbr_seal" , secstrings + s->sh_name)) ibt_endbr = s; }
@@ -289,6 +291,22 @@ int module_finalize(const Elf_Ehdr *hdr,void *pseg = (void *)para->sh_addr;apply_paravirt (pseg, pseg + para->sh_size); } +if (retpolines || cfi) { +void *rseg =NULL , *cseg =NULL ; +unsigned int rsize =0 , csize =0 ; + +if (retpolines) { + rseg = (void *)retpolines->sh_addr; + rsize = retpolines->sh_size; + } + +if (cfi) { + cseg = (void *)cfi->sh_addr; + csize = cfi->sh_size; + } + +apply_fineibt (rseg, rseg + rsize, cseg, cseg + csize); + }if (retpolines) {void *rseg = (void *)retpolines->sh_addr;apply_retpolines (rseg, rseg + retpolines->sh_size);
arch/x86/kernel/vmlinux.lds.S +9 -0
@@ -309,6 +309,15 @@ SECTIONS } #endif +#ifdef CONFIG_FINEIBT + . = ALIGN(8); + .cfi_sites : AT(ADDR(.cfi_sites) - LOAD_OFFSET) { + __cfi_sites = .; + *(.cfi_sites) + __cfi_sites_end = .; + } +#endif + /* * struct alt_inst entries. From the header (alternative.h): * "Alternative instructions for different CPU types or capabilities"
include/linux/bpf.h +1 -1
@@ -984,7 +984,7 @@ int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_func }#ifdef CONFIG_X86_64 -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5+CONFIG_FUNCTION_PADDING_BYTES,CONFIG_FUNCTION_PADDING_BYTES)))#else #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
scripts/Makefile.lib +1 -0
@@ -256,6 +256,7 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HACK) += --hacks=jump_label objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) += --hacks=noinstr objtool-args-$(CONFIG_CALL_DEPTH_TRACKING) += --hacks=skylake objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt +objtool-args-$(CONFIG_FINEIBT) += --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount objtool-args-$(CONFIG_UNWINDER_ORC) += --orc objtool-args-$(CONFIG_RETPOLINE) += --retpoline
Peter Zijlstra
maintainerPATCH 3/4 x86/cfi: Boot time selection of CFI scheme
In reply to Peter ZijlstraAdd the "cfi=" boot parameter to allow people to select a CFI scheme at boot time. Mostly useful for development / debugging.
--- arch/x86/kernel/alternative.c | 103 +++++++++++++++++++++++++++++++++--------- 1 file changed, 83 insertions(+), 20 deletions(-)
arch/x86/kernel/alternative.c +83 -20
@@ -702,6 +702,47 @@ void __init_or_module noinline apply_ibt#endif /* CONFIG_X86_KERNEL_IBT */ #ifdef CONFIG_FINEIBT + +enum cfi_mode { +CFI_DEFAULT , +CFI_OFF , +CFI_KCFI , +CFI_FINEIBT , +}; + +static enum cfi_mode cfi_mode__ro_after_init =CFI_DEFAULT ; + +static __init int cfi_parse_cmdline (char *str) +{ +if (!str) +return -EINVAL ; + +while (str) { +char *next =strchr (str,',' ); +if (next) { + *next =0 ; + next++; + } + +if (!strcmp (str,"auto" )) { + cfi_mode =CFI_DEFAULT ; + }else if (!strcmp (str,"off" )) { + cfi_mode =CFI_OFF ; + }else if (!strcmp (str,"kcfi" )) { + cfi_mode =CFI_KCFI ; + }else if (!strcmp (str,"fineibt" )) { + cfi_mode =CFI_FINEIBT ; + }else { +pr_err ("Ignoring unknown cfi option (%s)." , str); + } + + str = next; + } + +return 0 ; +} +early_param ("cfi" , cfi_parse_cmdline); +/* * kCFI FineIBT *
@@ -868,30 +909,52 @@ static void __apply_fineibt(s32 *start_r "FineIBT preamble wrong size: %ld", fineibt_preamble_size))return ; -if (!HAS_KERNEL_IBT || !cpu_feature_enabled (X86_FEATURE_IBT )) +if (cfi_mode ==CFI_DEFAULT ) { + cfi_mode =CFI_KCFI ; +if (HAS_KERNEL_IBT &&cpu_feature_enabled (X86_FEATURE_IBT )) + cfi_mode =CFI_FINEIBT ; + } + +switch (cfi_mode) { +case CFI_OFF : + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + +if (builtin) +pr_info ("Disabling CFI\n" );return ; -/* -* Rewrite the callers to not use the __cfi_ stubs, such that we might -* rewrite them. This disables all CFI. If this succeeds but any of the -* later stages fails, we're without CFI. -*/ - ret =cfi_disable_callers (start_retpoline, end_retpoline); -if (ret) - goto err; - - ret =cfi_rewrite_preamble (start_cfi, end_cfi); -if (ret) - goto err; - - ret =cfi_rewrite_callers (start_retpoline, end_retpoline); -if (ret) - goto err; +case CFI_KCFI : +if (builtin) +pr_info ("Using kCFI\n" ); +return ; -if (builtin) -pr_info ("Using FineIBT CFI\n" ); +case CFI_FINEIBT : +/* +* Rewrite the callers to not use the __cfi_ stubs, such that we might +* rewrite them. This disables all CFI. If this succeeds but any of the +* later stages fails, we're without CFI. +*/ + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + + ret =cfi_rewrite_preamble (start_cfi, end_cfi); +if (ret) + goto err; + + ret =cfi_rewrite_callers (start_retpoline, end_retpoline); +if (ret) + goto err; -return ; +if (builtin) +pr_info ("Using FineIBT CFI\n" ); +return ; + +default : +break ; + } err:pr_err ("Something went horribly wrong trying to rewrite the CFI implementation.\n" );
Re: [PATCH 3/4] x86/cfi: Boot time selection of CFI scheme
Quoted 5 lines · Peter Zijlstra
Add the "cfi=" boot parameter to allow people to select a CFI scheme at boot time. Mostly useful for development / debugging.
Requested-by Kees Cook <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Signature · 1 line
Kees Cook
[tip: x86/core] x86/cfi: Boot time selection of CFI scheme
In reply to Peter ZijlstraThe following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 082c4c815252ea333b0f3a51e336df60c2314fe2 Gitweb: https://git.kernel.org/tip/082c4c815252ea333b0f3a51e336df60c2314fe2 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 27 Oct 2022 11:28:15 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 01 Nov 2022 13:44:11 +01:00
x86/cfi: Boot time selection of CFI scheme
Add the "cfi=" boot parameter to allow people to select a CFI scheme at boot time. Mostly useful for development / debugging.
--- arch/x86/kernel/alternative.c | 99 +++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 18 deletions(-)
arch/x86/kernel/alternative.c +81 -18
@@ -702,6 +702,47 @@ void __init_or_module noinline apply_ibt_endbr(s32 *start, s32 *end) { }#endif /* CONFIG_X86_KERNEL_IBT */ #ifdef CONFIG_FINEIBT + +enum cfi_mode { +CFI_DEFAULT , +CFI_OFF , +CFI_KCFI , +CFI_FINEIBT , +}; + +static enum cfi_mode cfi_mode__ro_after_init =CFI_DEFAULT ; + +static __init int cfi_parse_cmdline (char *str) +{ +if (!str) +return -EINVAL ; + +while (str) { +char *next =strchr (str,',' ); +if (next) { + *next =0 ; + next++; + } + +if (!strcmp (str,"auto" )) { + cfi_mode =CFI_DEFAULT ; + }else if (!strcmp (str,"off" )) { + cfi_mode =CFI_OFF ; + }else if (!strcmp (str,"kcfi" )) { + cfi_mode =CFI_KCFI ; + }else if (!strcmp (str,"fineibt" )) { + cfi_mode =CFI_FINEIBT ; + }else { +pr_err ("Ignoring unknown cfi option (%s)." , str); + } + + str = next; + } + +return 0 ; +} +early_param ("cfi" , cfi_parse_cmdline); +/* * kCFI FineIBT *
@@ -868,30 +909,52 @@ static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, "FineIBT preamble wrong size: %ld", fineibt_preamble_size))return ; -if (!HAS_KERNEL_IBT || !cpu_feature_enabled (X86_FEATURE_IBT )) +if (cfi_mode ==CFI_DEFAULT ) { + cfi_mode =CFI_KCFI ; +if (HAS_KERNEL_IBT &&cpu_feature_enabled (X86_FEATURE_IBT )) + cfi_mode =CFI_FINEIBT ; + } + +switch (cfi_mode) { +case CFI_OFF : + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + +if (builtin) +pr_info ("Disabling CFI\n" );return ; -/* -* Rewrite the callers to not use the __cfi_ stubs, such that we might -* rewrite them. This disables all CFI. If this succeeds but any of the -* later stages fails, we're without CFI. -*/ - ret =cfi_disable_callers (start_retpoline, end_retpoline); -if (ret) - goto err; +case CFI_KCFI : +if (builtin) +pr_info ("Using kCFI\n" ); +return ; - ret =cfi_rewrite_preamble (start_cfi, end_cfi); -if (ret) - goto err; +case CFI_FINEIBT : +/* +* Rewrite the callers to not use the __cfi_ stubs, such that we might +* rewrite them. This disables all CFI. If this succeeds but any of the +* later stages fails, we're without CFI. +*/ + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + + ret =cfi_rewrite_preamble (start_cfi, end_cfi); +if (ret) + goto err; - ret =cfi_rewrite_callers (start_retpoline, end_retpoline); -if (ret) - goto err; + ret =cfi_rewrite_callers (start_retpoline, end_retpoline); +if (ret) + goto err; -if (builtin) -pr_info ("Using FineIBT CFI\n" ); +if (builtin) +pr_info ("Using FineIBT CFI\n" ); +return ; -return ; +default : +break ; + } err:pr_err ("Something went horribly wrong trying to rewrite the CFI implementation.\n" );
Peter Zijlstra
maintainerPATCH 4/4 x86/cfi: Add boot time hash randomization
In reply to Peter ZijlstraIn order to avoid known hashes (from knowing the boot image), randomize the CFI hashes with a per-boot random seed.
--- arch/x86/kernel/alternative.c | 120 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 108 insertions(+), 12 deletions(-)
arch/x86/kernel/alternative.c +108 -12
@@ -711,6 +711,24 @@ enum cfi_mode { };static enum cfi_mode cfi_mode__ro_after_init =CFI_DEFAULT ; +static bool cfi_rand__ro_after_init = true; +static u32 cfi_seed__ro_after_init ; + +/* +* Re-hash the CFI hash with a boot-time seed while making sure the result is +* not a valid ENDBR instruction. +*/ +static u32 cfi_rehash (u32 hash) +{ + hash ^= cfi_seed; +while (unlikely (is_endbr (hash) ||is_endbr (-hash))) { +bool lsb = hash &1 ; + hash >>=1 ; +if (lsb) + hash ^=0x80200003 ; + } +return hash; +}static __init int cfi_parse_cmdline (char *str) {
@@ -728,10 +746,13 @@ static __init int cfi_parse_cmdline(char cfi_mode =CFI_DEFAULT ; }else if (!strcmp (str,"off" )) { cfi_mode =CFI_OFF ; + cfi_rand = false; }else if (!strcmp (str,"kcfi" )) { cfi_mode =CFI_KCFI ; }else if (!strcmp (str,"fineibt" )) { cfi_mode =CFI_FINEIBT ; + }else if (!strcmp (str,"norand" )) { + cfi_rand = false; }else {pr_err ("Ignoring unknown cfi option (%s)." , str); }
@@ -856,7 +877,50 @@ static int cfi_disable_callers(s32 *starreturn 0 ; } +static int cfi_enable_callers (s32 *start,s32 *end) +{ +/* +* Re-enable kCFI, undo what cfi_disable_callers() did. +*/ +const u8 mov[] = {0x41 ,0xba }; +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + addr -= fineibt_caller_size; + hash =decode_caller_hash (addr); +if (!hash)/* nocfi callers */ +continue ; + +text_poke_early (addr, mov,2 ); + } + +return 0 ; +} +/* .cfi_sites */ +static int cfi_rand_preamble (s32 *start,s32 *end) +{ +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + hash =decode_preamble_hash (addr); +if (WARN (!hash,"no CFI hash found at: %pS %px %*ph\n" , + addr, addr,5 , addr)) +return -EINVAL ; + + hash =cfi_rehash (hash); +text_poke_early (addr +1 , &hash,4 ); + } + +return 0 ; +} +static int cfi_rewrite_preamble (s32 *start,s32 *end) {s32 *s;
@@ -879,6 +943,25 @@ static int cfi_rewrite_preamble(s32 *sta }/* .retpoline_sites */ +static int cfi_rand_callers (s32 *start,s32 *end) +{ +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + addr -= fineibt_caller_size; + hash =decode_caller_hash (addr); +if (hash) { + hash = -cfi_rehash (hash); +text_poke_early (addr +2 , &hash,4 ); + } + } + +return 0 ; +} +static int cfi_rewrite_callers (s32 *start,s32 *end) {s32 *s;
@@ -915,31 +998,44 @@ static void __apply_fineibt(s32 *start_r cfi_mode =CFI_FINEIBT ; } -switch (cfi_mode) { -case CFI_OFF : - ret =cfi_disable_callers (start_retpoline, end_retpoline); +/* +* Rewrite the callers to not use the __cfi_ stubs, such that we might +* rewrite them. This disables all CFI. If this succeeds but any of the +* later stages fails, we're without CFI. +*/ + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + +if (cfi_rand) { +if (builtin) + cfi_seed =get_random_u32 (); + + ret =cfi_rand_preamble (start_cfi, end_cfi);if (ret) goto err; + ret =cfi_rand_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + } + +switch (cfi_mode) { +case CFI_OFF :if (builtin)pr_info ("Disabling CFI\n" );return ;case CFI_KCFI : + ret =cfi_enable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; +if (builtin)pr_info ("Using kCFI\n" );return ;case CFI_FINEIBT : -/* -* Rewrite the callers to not use the __cfi_ stubs, such that we might -* rewrite them. This disables all CFI. If this succeeds but any of the -* later stages fails, we're without CFI. -*/ - ret =cfi_disable_callers (start_retpoline, end_retpoline); -if (ret) - goto err; - ret =cfi_rewrite_preamble (start_cfi, end_cfi);if (ret) goto err;
Re: [PATCH 4/4] x86/cfi: Add boot time hash randomization
Quoted 23 lines · Peter Zijlstra
In order to avoid known hashes (from knowing the boot image), randomize the CFI hashes with a per-boot random seed.
Suggested-by Kees Cook <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/kernel/alternative.c | 120 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 108 insertions(+), 12 deletions(-)arch/x86/kernel/alternative.c +8
-0@@ -711,6 +711,24 @@ enum cfi_mode { };static enum cfi_mode cfi_mode__ro_after_init =CFI_DEFAULT ; +static bool cfi_rand__ro_after_init = true; +static u32 cfi_seed__ro_after_init ; + +/* +* Re-hash the CFI hash with a boot-time seed while making sure the result is +* not a valid ENDBR instruction. +*/ +static u32 cfi_rehash (u32 hash)
arch/x86/kernel/alternative.c enum cfi_mode { · line 731
Quoted 10 lines
arch/x86/kernel/alternative.c +10
-0+{ + hash ^= cfi_seed; +while (unlikely (is_endbr (hash) ||is_endbr (-hash))) { +bool lsb = hash &1 ; + hash >>=1 ; +if (lsb) + hash ^=0x80200003 ; + } +return hash; +}
I guess this risks hash collision with existing hashes, but meeeh. I'm glad to have the randomization. :)
Signature · 1 line
Kees Cook
[tip: x86/core] x86/cfi: Add boot time hash randomization
In reply to Peter ZijlstraThe following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 0c3e806ec0f9771fa1f34c60499097d9260a8bb7 Gitweb: https://git.kernel.org/tip/0c3e806ec0f9771fa1f34c60499097d9260a8bb7 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 27 Oct 2022 11:28:16 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 01 Nov 2022 13:44:11 +01:00
x86/cfi: Add boot time hash randomization
In order to avoid known hashes (from knowing the boot image), randomize the CFI hashes with a per-boot random seed.
--- arch/x86/kernel/alternative.c | 120 +++++++++++++++++++++++++++++---- 1 file changed, 108 insertions(+), 12 deletions(-)
arch/x86/kernel/alternative.c +108 -12
@@ -711,6 +711,24 @@ enum cfi_mode { };static enum cfi_mode cfi_mode__ro_after_init =CFI_DEFAULT ; +static bool cfi_rand__ro_after_init = true; +static u32 cfi_seed__ro_after_init ; + +/* +* Re-hash the CFI hash with a boot-time seed while making sure the result is +* not a valid ENDBR instruction. +*/ +static u32 cfi_rehash (u32 hash) +{ + hash ^= cfi_seed; +while (unlikely (is_endbr (hash) ||is_endbr (-hash))) { +bool lsb = hash &1 ; + hash >>=1 ; +if (lsb) + hash ^=0x80200003 ; + } +return hash; +}static __init int cfi_parse_cmdline (char *str) {
@@ -728,10 +746,13 @@ static __init int cfi_parse_cmdline(char *str) cfi_mode =CFI_DEFAULT ; }else if (!strcmp (str,"off" )) { cfi_mode =CFI_OFF ; + cfi_rand = false; }else if (!strcmp (str,"kcfi" )) { cfi_mode =CFI_KCFI ; }else if (!strcmp (str,"fineibt" )) { cfi_mode =CFI_FINEIBT ; + }else if (!strcmp (str,"norand" )) { + cfi_rand = false; }else {pr_err ("Ignoring unknown cfi option (%s)." , str); }
@@ -856,7 +877,50 @@ static int cfi_disable_callers(s32 *start, s32 *end)return 0 ; } +static int cfi_enable_callers (s32 *start,s32 *end) +{ +/* +* Re-enable kCFI, undo what cfi_disable_callers() did. +*/ +const u8 mov[] = {0x41 ,0xba }; +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + addr -= fineibt_caller_size; + hash =decode_caller_hash (addr); +if (!hash)/* nocfi callers */ +continue ; + +text_poke_early (addr, mov,2 ); + } + +return 0 ; +} +/* .cfi_sites */ +static int cfi_rand_preamble (s32 *start,s32 *end) +{ +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + hash =decode_preamble_hash (addr); +if (WARN (!hash,"no CFI hash found at: %pS %px %*ph\n" , + addr, addr,5 , addr)) +return -EINVAL ; + + hash =cfi_rehash (hash); +text_poke_early (addr +1 , &hash,4 ); + } + +return 0 ; +} +static int cfi_rewrite_preamble (s32 *start,s32 *end) {s32 *s;
@@ -879,6 +943,25 @@ static int cfi_rewrite_preamble(s32 *start, s32 *end) }/* .retpoline_sites */ +static int cfi_rand_callers (s32 *start,s32 *end) +{ +s32 *s; + +for (s = start; s < end; s++) { +void *addr = (void *)s + *s; +u32 hash; + + addr -= fineibt_caller_size; + hash =decode_caller_hash (addr); +if (hash) { + hash = -cfi_rehash (hash); +text_poke_early (addr +2 , &hash,4 ); + } + } + +return 0 ; +} +static int cfi_rewrite_callers (s32 *start,s32 *end) {s32 *s;
@@ -915,31 +998,44 @@ static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, cfi_mode =CFI_FINEIBT ; } -switch (cfi_mode) { -case CFI_OFF : - ret =cfi_disable_callers (start_retpoline, end_retpoline); +/* +* Rewrite the callers to not use the __cfi_ stubs, such that we might +* rewrite them. This disables all CFI. If this succeeds but any of the +* later stages fails, we're without CFI. +*/ + ret =cfi_disable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + +if (cfi_rand) { +if (builtin) + cfi_seed =get_random_u32 (); + + ret =cfi_rand_preamble (start_cfi, end_cfi);if (ret) goto err; + ret =cfi_rand_callers (start_retpoline, end_retpoline); +if (ret) + goto err; + } + +switch (cfi_mode) { +case CFI_OFF :if (builtin)pr_info ("Disabling CFI\n" );return ;case CFI_KCFI : + ret =cfi_enable_callers (start_retpoline, end_retpoline); +if (ret) + goto err; +if (builtin)pr_info ("Using kCFI\n" );return ;case CFI_FINEIBT : -/* -* Rewrite the callers to not use the __cfi_ stubs, such that we might -* rewrite them. This disables all CFI. If this succeeds but any of the -* later stages fails, we're without CFI. -*/ - ret =cfi_disable_callers (start_retpoline, end_retpoline); -if (ret) - goto err; - ret =cfi_rewrite_preamble (start_cfi, end_cfi);if (ret) goto err;
From: Peter Zijlstra
Quoted 10 lines
Sent: 27 October 2022 10:28
Hi all,
Updated FineIBT series; I've (hopefully) incorporated all feedback from last time with the notable exception of the
KconfigCFI default -- I'm not sure we want to add to theKconfigspace for this, also what would a distro do with it.Anyway; please have a look, I'm hoping to merge this soonish so we can make the next cycle.
Is there a test to ensure that modules are actually compiled with the required endbra, function prologue gap (etc). Having the module load fail is somewhat better than a crash.
It is almost certainly quite easy to generate an out of tree module that is missing all of those (even if compiled at the same time as the kernel). (Never mind issues with modules that contain binary blobs.)
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Peter Zijlstra
maintainerQuoted 12 lines · David Laight
From: Peter Zijlstra
Quoted 10 lines
Sent: 27 October 2022 10:28
Hi all,
Updated FineIBT series; I've (hopefully) incorporated all feedback from last time with the notable exception of the
KconfigCFI default -- I'm not sure we want to add to theKconfigspace for this, also what would a distro do with it.Anyway; please have a look, I'm hoping to merge this soonish so we can make the next cycle.
Quoted 7 lines
Is there a test to ensure that modules are actually compiled with the required endbra, function prologue gap (etc). Having the module load fail is somewhat better than a crash.
It is almost certainly quite easy to generate an out of tree module that is missing all of those (even if compiled at the same time as the kernel). (Never mind issues with modules that contain binary blobs.)
There is not; it is always possible to load a 'malformed' module. We have no sanity checking on modules. It is no different from any other binary compatilibity issue; if you build a dud module, you get to keep the pieces.