x86/kprobes: Fixes for CONFIG_RETHUNK
Subsystems (3)
v2 cover x86/kprobes: Fixes for CONFIG_RETHUNK
Hi,
Here is the 2nd version of the patches to fix kprobes and optprobe with CONFIG_RETHUNK and CONFIG_SLS.
Previous version is here;
https://lore.kernel.org/all/166251211081.632004.1842371136165709807.stgit@devnote2/
In this version, I updated the Fixed tag and the decoding function to memorize all branch targets and decode those code blocks besed on Peter's idea. (Thanks!)
With these configs, the kernel functions may includes INT3 for stopping speculative execution in the function code block (body) in addition to the gaps between functions.
Since kprobes on x86 has to ensure the probe address is an instruction bondary, it decodes the instructions in the function until the address. If it finds an INT3 which is not embedded by kprobe, it stops decoding because usually the INT3 is used for debugging as a software breakpoint and such INT3 will replace the first byte of an original instruction. Without recovering it, kprobes can not continue to decode it. Thus the kprobes returns -EILSEQ as below.
# echo "p:probe/vfs_truncate_L19 vfs_truncate+98" >> kprobe_events sh: write error: Invalid or incomplete multibyte or wide character
Actually, those INT3s are just for padding and can be ignored.
To avoid this issue, memorize the branch target address during decoding and if there is INT3, restart decoding from unchecked target address so that this decodes all instructions in the kernel (in both kprobes and optprobe.)
With thses fixes, kprobe and optprobe can probe the kernel again with CONFIG_RETHUNK=y.
# echo "p:probe/vfs_truncate_L19 vfs_truncate+98" >> kprobe_events # echo 1 > events/probe/vfs_truncate_L19/enable # cat /sys/kernel/debug/kprobes/list ffffffff81307b52 k vfs_truncate+0x62 [OPTIMIZED]
Thank you,
---
Masami Hiramatsu (Google) (2):
x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
arch/x86/kernel/kprobes/common.h | 28 +++++ arch/x86/kernel/kprobes/core.c | 227 +++++++++++++++++++++++++++++++++----- arch/x86/kernel/kprobes/opt.c | 96 ++++------------ 3 files changed, 249 insertions(+), 102 deletions(-)
--
Masami Hiramatsu (Google) <[email protected]>
PATCH v2 1/2 x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
From: Masami Hiramatsu (Google) <[email protected]>
Since the CONFIG_RETHUNK and CONFIG_SLS will use INT3 for stopping speculative execution after RET instruction, kprobes always failes to check the probed instruction boundary by decoding the function body if the probed address is after such sequence. (Note that some conditional code blocks will be placed after function return, if compiler decides it is not on the hot path.)
This is because kprobes expects someone (e.g. kgdb) puts the INT3 as a software breakpoint and it will replace the original instruction. But these INT3 are not such purpose, it doesn't need to recover the original instruction.
To avoid this issue, memorize the branch target address during decoding and if there is INT3, restart decoding from unchecked target address.
---
Changes in v2:
- Use GNU AS names for mnemonic in comments
- Decode all code blocks in the function to find the instruction boundary
- Update Fixes tag to specify CONFIG_SLS.
--- arch/x86/kernel/kprobes/common.h | 28 +++++ arch/x86/kernel/kprobes/core.c | 227 +++++++++++++++++++++++++++++++++----- arch/x86/kernel/kprobes/opt.c | 23 ---- 3 files changed, 225 insertions(+), 53 deletions(-)
arch/x86/kernel/kprobes/common.h +28 -0
@@ -92,6 +92,34 @@ extern int __copy_instruction(u8 *dest, u8 *src, u8 *real, struct insn *insn);extern void synthesize_reljump (void *dest,void *from,void *to);extern void synthesize_relcall (void *dest,void *from,void *to); +/* Return the jump target address or 0 */ +static inline unsigned long insn_get_branch_addr (struct insn *insn) +{ +switch (insn->opcode.bytes[0 ]) { +case 0xe0 :/* loopne */ +case 0xe1 :/* loope */ +case 0xe2 :/* loop */ +case 0xe3 :/* Jcxz */ +case 0xe9 :/* JMP.d32 */ +case 0xeb :/* JMP.d8 */ +break ; +case 0x0f : +if ((insn->opcode.bytes[1 ] &0xf0 ) ==0x80 )/* Jcc.d32 */ +break ; +return 0 ; + case 0x70 ... 0x7f: /* Jcc.d8 */ +break ; + +default : +return 0 ; + } +return (unsigned long )insn->next_byte + insn->immediate.value; +} + +int every_insn_in_func (unsigned long faddr, +int (*callback)(struct insn *,void *), +void *data); +#ifdef CONFIG_OPTPROBES extern int setup_detour_execution (struct kprobe *p,struct pt_regs *regs,int reenter);extern unsigned long __recover_optprobed_insn (kprobe_opcode_t *buf,unsigned long addr);
arch/x86/kernel/kprobes/core.c +195 -32
242 lines not on this page · raw message
arch/x86/kernel/kprobes/opt.c +2 -21
@@ -235,28 +235,9 @@ static int __insn_is_indirect_jump(struct insn *insn)/* Check whether insn jumps into specified address range */ static int insn_jump_into_range (struct insn *insn,unsigned long start,int len) { -unsigned long target =0 ; - -switch (insn->opcode.bytes[0 ]) { -case 0xe0 :/* loopne */ -case 0xe1 :/* loope */ -case 0xe2 :/* loop */ -case 0xe3 :/* jcxz */ -case 0xe9 :/* near relative jump */ -case 0xeb :/* short relative jump */ -break ; -case 0x0f : -if ((insn->opcode.bytes[1 ] &0xf0 ) ==0x80 )/* jcc near */ -break ; -return 0 ; -default : -if ((insn->opcode.bytes[0 ] &0xf0 ) ==0x70 )/* jcc short */ -break ; -return 0 ; - } - target = (unsigned long )insn->next_byte + insn->immediate.value; +unsigned long target =insn_get_branch_addr (insn); -return (start <= target && target <= start + len); +return target ? (start <= target && target <= start + len) :0 ; }static int insn_is_indirect_jump (struct insn *insn)
PATCH v2 2/2 x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Since the CONFIG_RETHUNK and CONFIG_SLS will use INT3 for stopping speculative execution after function return, kprobe jump optimization always fails on the functions with such INT3 inside the function body. (It already checks the INT3 padding between functions, but not inside
the function)
To avoid this issue, as same as kprobes, decoding the all code blocks in the function to check the kprobe can be optimized.
---
Changes in v2:
- Reuse the kprobes decoding function.
--- arch/x86/kernel/kprobes/opt.c | 73 +++++++++++++---------------------------- 1 file changed, 24 insertions(+), 49 deletions(-)
arch/x86/kernel/kprobes/opt.c +24 -49
@@ -260,25 +260,36 @@ static int insn_is_indirect_jump(struct insn *insn)return ret; } -static bool is_padding_int3 (unsigned long addr,unsigned long eaddr) +static int optimize_check_cb (struct insn *insn,void *data) { -unsigned char ops; +unsigned long paddr = (unsigned long )data; -for (; addr < eaddr; addr++) { -if (get_kernel_nofault (ops, (void *)addr) <0 || - ops !=INT3_INSN_OPCODE ) -return false; - } +if (search_exception_tables ((unsigned long )insn->kaddr)) +/* +* Since some fixup code will jumps into this function, +* we can't optimize kprobe in this function. +*/ +return 1 ; + +/* Check any instructions don't jump into target */ +if (insn_is_indirect_jump (insn) || +insn_jump_into_range (insn, paddr +INT3_INSN_SIZE , +DISP32_SIZE )) +return 1 ; + +/* Check whether an INT3 is involved. */ +if (insn->opcode.bytes[0 ] ==INT3_INSN_OPCODE && + paddr <= (unsigned long )insn->kaddr && + (unsigned long )insn->kaddr <= paddr +JMP32_INSN_SIZE ) +return 1 ; -return true; +return 0 ; }/* Decode whole function to ensure any instructions don't jump into target */ static int can_optimize (unsigned long paddr) { -unsigned long addr, size =0 , offset =0 ; -struct insn insn; -kprobe_opcode_t buf[MAX_INSN_SIZE ]; +unsigned long size =0 , offset =0 ;/* Lookup symbol including addr */ if (!kallsyms_lookup_size_offset (paddr, &size, &offset))
@@ -296,44 +307,8 @@ static int can_optimize(unsigned long paddr)if (size - offset <JMP32_INSN_SIZE )return 0 ; -/* Decode instructions */ - addr = paddr - offset; -while (addr < paddr - offset + size) {/* Decode until function end */ -unsigned long recovered_insn; -int ret; - -if (search_exception_tables (addr)) -/* -* Since some fixup code will jumps into this function, -* we can't optimize kprobe in this function. -*/ -return 0 ; - recovered_insn =recover_probed_instruction (buf, addr); -if (!recovered_insn) -return 0 ; - - ret =insn_decode_kernel (&insn, (void *)recovered_insn); -if (ret <0 ) -return 0 ; - -/* -* In the case of detecting unknown breakpoint, this could be -* a padding INT3 between functions. Let's check that all the -* rest of the bytes are also INT3. -*/ -if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE ) -return is_padding_int3 (addr, paddr - offset + size) ?1 :0 ; - -/* Recover address */ - insn.kaddr = (void *)addr; - insn.next_byte = (void *)(addr + insn.length); -/* Check any instructions don't jump into target */ -if (insn_is_indirect_jump (&insn) || -insn_jump_into_range (&insn, paddr +INT3_INSN_SIZE , -DISP32_SIZE )) -return 0 ; - addr += insn.length; - } +if (every_insn_in_func (paddr, optimize_check_cb, (void *)paddr)) +return 0 ;return 1 ; }
Re: [PATCH v2 1/2] x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
In reply to Masami Hiramatsu (Google)Quoted 9 lines · Masami Hiramatsu (Google)
From: Masami Hiramatsu (Google) <[email protected]>
Since the
CONFIG_RETHUNKandCONFIG_SLSwill use INT3 for stopping speculative execution after RET instruction, kprobes always failes to check the probed instruction boundary by decoding the function body if the probed address is after such sequence. (Note that some conditional code blocks will be placed after function return, if compiler decides it is not on the hot path.)
Quoted 7 lines
This is because kprobes expects someone (e.g. kgdb) puts the INT3 as a software breakpoint and it will replace the original instruction. But these INT3 are not such purpose, it doesn't need to recover the original instruction.
To avoid this issue, memorize the branch target address during decoding and if there is INT3, restart decoding from unchecked target address.
Hm, is kprobes conflicting with kgdb actually a realistic concern? Seems like a dangerous combination
Either way, this feels overengineered. Sort of like implementing objtool in the kernel.
And it's incomplete: for a switch statement jump table (or C goto jump table like in BPF), you can't detect the potential targets of the indirect branch.
Wouldn't it be much simpler to just encode the knowledge that
if (CONFIG_RETHUNK && !X86_FEATURE_RETHUNK)
// all rets are followed by four INT3s else if (CONFIG_SLS) // all rets are followed by one INT3
?
Signature · 1 line
Josh
Re: [PATCH v2 1/2] x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
Quoted 15 lines · Josh Poimboeuf
Quoted 14 lines · Masami Hiramatsu (Google)
From: Masami Hiramatsu (Google) <[email protected]>
Since the
CONFIG_RETHUNKandCONFIG_SLSwill use INT3 for stopping speculative execution after RET instruction, kprobes always failes to check the probed instruction boundary by decoding the function body if the probed address is after such sequence. (Note that some conditional code blocks will be placed after function return, if compiler decides it is not on the hot path.)This is because kprobes expects someone (e.g. kgdb) puts the INT3 as a software breakpoint and it will replace the original instruction. But these INT3 are not such purpose, it doesn't need to recover the original instruction.
Quoted 5 lines
Quoted 2 lines
To avoid this issue, memorize the branch target address during decoding and if there is INT3, restart decoding from unchecked target address.
Hm, is kprobes conflicting with kgdb actually a realistic concern? Seems like a dangerous combination
Yeah, not in my book. If you use kgdb you'd better be ready to hold pieces anyway, that thing is terrible.
Quoted 6 lines
Wouldn't it be much simpler to just encode the knowledge that
if (
CONFIG_RETHUNK&& !X86_FEATURE_RETHUNK)// all rets are followed by four INT3s else if (CONFIG_SLS) // all rets are followed by one INT3
At the very least that doesn't deal with the INT3 after JMP thing the compilers should do once they catch up. This issue seems to keep getting lost, but is now part of the AMD Branch Type Confusion (it was disclosed in an earlier thing, but I keep forgetting what that was called).
Once that lands the rules are:
0-5 INT3 after RET, !CONFIG_RETHUNK && !CONFIG_SLS: 0
CONFIG_SLS: 1
CONFIG_RETHUNK: 4-5 depending on compiler version
0-1 INT3 after RET: !CONFIG_SLS: 0
CONFIG_SLS: 0-1 depending on compiler version
Now, given we know the compiler version at build time, this could be determined and used in kprobes, but meh.
We also *should* put an INT3 after indirect jumps when patching the retpolines. We don't appear to do so, but that's recommended even for Intel I think.
Let me go do a patch.
PATCH 1/1 x86,retpoline: Be sure to emit INT3 after JMP *%\reg
Quoted 1 line · Peter Zijlstra
Let me go do a patch.
---
Subject: x86,retpoline: Be sure to emit INT3 after JMP *%\reg
Both AMD and Intel recommend using INT3 after an indirect JMP. Make sure to emit one when rewriting the retpoline JMP irrespective of compiler SLS options or even CONFIG_SLS.
--- arch/x86/kernel/alternative.c | 9 +++++++++ arch/x86/net/bpf_jit_comp.c | 3 ++- 2 files changed, 11 insertions(+), 1 deletion(-)
arch/x86/kernel/alternative.c +9 -0
@@ -453,6 +453,15 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)return ret; i += ret; +/* +* The compiler is supposed to EMIT an INT3 after every unconditional +* JMP instruction due to AMD BTC. However, if the compiler is too old +* or SLS isn't enabled, we still need an INT3 after indirect JMPs +* even on Intel. +*/ +if (op ==JMP32_INSN_OPCODE && i < insn->length) + bytes[i++] =INT3_INSN_OPCODE ; +for (; i < insn->length;) bytes[i++] =BYTES_NOP1 ;
arch/x86/net/bpf_jit_comp.c +2 -1
@@ -419,7 +419,8 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +EMIT1 (0xCC );/* int3 */ } *pprog = prog;
Re: [PATCH v2 1/2] x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
Once that lands the rules are:
0-5 INT3 after RET, !CONFIG_RETHUNK && !CONFIG_SLS: 0
CONFIG_SLS: 1
CONFIG_RETHUNK: 4-5 depending on compiler version0-1 INT3 after RET: !CONFIG_SLS: 0
CONFIG_SLS: 0-1 depending on compiler versionNow, given we know the compiler version at build time, this could be determined and used in kprobes, but meh.
Oh also, for giggles, we have a number of sites that have ret;int3 independent of CONFIG_SLS because that was easier than figuring out what all should be done.
Re: [PATCH v2 1/2] x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
In reply to Josh PoimboeufQuoted 15 lines · On Wed, 7 Sep 2022 22:08:55 -0700 Josh Poimboeuf <[email protected]> wrote:
Quoted 14 lines · Masami Hiramatsu (Google)
From: Masami Hiramatsu (Google) <[email protected]>
Since the
CONFIG_RETHUNKandCONFIG_SLSwill use INT3 for stopping speculative execution after RET instruction, kprobes always failes to check the probed instruction boundary by decoding the function body if the probed address is after such sequence. (Note that some conditional code blocks will be placed after function return, if compiler decides it is not on the hot path.)This is because kprobes expects someone (e.g. kgdb) puts the INT3 as a software breakpoint and it will replace the original instruction. But these INT3 are not such purpose, it doesn't need to recover the original instruction.
Quoted 5 lines
Quoted 2 lines
To avoid this issue, memorize the branch target address during decoding and if there is INT3, restart decoding from unchecked target address.
Hm, is kprobes conflicting with kgdb actually a realistic concern? Seems like a dangerous combination
I'm actually not sure, I don't recommend it. But it is safe just having fail-safe.
Quoted 7 lines
Either way, this feels overengineered. Sort of like implementing objtool in the kernel.
And it's incomplete: for a switch statement jump table (or C goto jump table like in BPF), you can't detect the potential targets of the indirect branch.
In that case, it just fails to detect instruction boundary (and anyway optprobe just stops optimization if it finds the indirect jump). So it is still fail safe.
Quoted 7 lines
Wouldn't it be much simpler to just encode the knowledge that
if (
CONFIG_RETHUNK&& !X86_FEATURE_RETHUNK)// all rets are followed by four INT3s else if (CONFIG_SLS) // all rets are followed by one INT3
Maybe we should just ask kgdb if it is using breakpoint on that function, and if so, just reject kprobe on it. Then, all INT3 can be just skipped. That may be more realistic solution.
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
In reply to Peter ZijlstraQuoted 42 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
Let me go do a patch.
---
Subject: x86,retpoline: Be sure to emit INT3 after JMP *%\regBoth AMD and Intel recommend using INT3 after an indirect JMP. Make sure to emit one when rewriting the retpoline JMP irrespective of compiler SLS options or even
CONFIG_SLS.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/kernel/alternative.c | 9 +++++++++ arch/x86/net/bpf_jit_comp.c | 3 ++- 2 files changed, 11 insertions(+), 1 deletion(-)arch/x86/kernel/alternative.c +9
-0@@ -453,6 +453,15 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)return ret; i += ret; +/* +* The compiler is supposed to EMIT an INT3 after every unconditional +* JMP instruction due to AMD BTC. However, if the compiler is too old +* or SLS isn't enabled, we still need an INT3 after indirect JMPs +* even on Intel. +*/ +if (op ==JMP32_INSN_OPCODE && i < insn->length) + bytes[i++] =INT3_INSN_OPCODE ; +for (; i < insn->length;) bytes[i++] =BYTES_NOP1 ;arch/x86/net/bpf_jit_comp.c +0
-0
arch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip) · line 423
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +2
-1@@ -419,7 +419,8 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +EMIT1 (0xCC );/* int3 */
Hmm. Why is this unconditional?
Shouldn't it be guarded with CONFIG_xx or cpu_feature_enabled ? People that don't care about hw speculation vulnerabilities shouldn't pay the price of increased code size.
v3 cover x86/kprobes: Fixes for CONFIG_RETHUNK
In reply to Masami Hiramatsu (Google)Hi Peter and Josh,
So here is 3rd version of the patches to fix kprobes and optprobe with CONFIG_RETHUNK and CONFIG_SLS.
Previous version is here;
https://lore.kernel.org/all/166260087224.759381.4170102827490658262.stgit@devnote2/
In this version, I simplified all code and just checks the INT3 comes from kgdb or not. Other INT3 are treated as one-byte instruction.
With CONFIG_RETHUNK/CONFIG_SLS, the kernel functions may includes INT3 for stopping speculative execution in the function code block (body) in addition to the gaps between functions.
Since kprobes on x86 has to ensure the probe address is an instruction bondary, it decodes the instructions in the function until the address. If it finds an INT3 which is not embedded by kprobe, it stops decoding because usually the INT3 is used for debugging as a software breakpoint and such INT3 will replace the first byte of an original instruction. Without recovering it, kprobes can not continue to decode it. Thus the kprobes returns -EILSEQ as below.
# echo "p:probe/vfs_truncate_L19 vfs_truncate+98" >> kprobe_events sh: write error: Invalid or incomplete multibyte or wide character
Actually, such INT3 can be ignored except the INT3 installed by kgdb.
To avoid this issue, just check whether the INT3 is owned by kgdb or not and if so, it just stopped and return failure.
With thses fixes, kprobe and optprobe can probe the kernel again with CONFIG_RETHUNK=y.
# echo "p:probe/vfs_truncate_L19 vfs_truncate+98" >> kprobe_events # echo 1 > events/probe/vfs_truncate_L19/enable # cat /sys/kernel/debug/kprobes/list ffffffff81307b52 k vfs_truncate+0x62 [OPTIMIZED]
Thank you,
---
Masami Hiramatsu (Google) (2):
x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
arch/x86/kernel/kprobes/core.c | 10 +++++++--- arch/x86/kernel/kprobes/opt.c | 28 ++++++++-------------------- 2 files changed, 15 insertions(+), 23 deletions(-)
--
Masami Hiramatsu (Google) <[email protected]>
PATCH v3 1/2 x86/kprobes: Fix kprobes instruction boudary check with CONFIG_RETHUNK
From: Masami Hiramatsu (Google) <[email protected]>
Since the CONFIG_RETHUNK and CONFIG_SLS will use INT3 for stopping speculative execution after RET instruction, kprobes always failes to check the probed instruction boundary by decoding the function body if the probed address is after such sequence. (Note that some conditional code blocks will be placed after function return, if compiler decides it is not on the hot path.)
This is because kprobes expects kgdb puts the INT3 as a software breakpoint and it will replace the original instruction. But these INT3 are not such purpose, it doesn't need to recover the original instruction.
To avoid this issue, kprobes checks whether the INT3 is owned by kgdb or not, and if so, stop decoding and make it fail. The other INT3 will come from CONFIG_RETHUNK/CONFIG_SLS and those can be treated as a one-byte instruction.
--- arch/x86/kernel/kprobes/core.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes/core.c +7 -3
@@ -37,6 +37,7 @@#include <linux/extable.h> #include <linux/kdebug.h> #include <linux/kallsyms.h> +#include <linux/kgdb.h> #include <linux/ftrace.h> #include <linux/kasan.h> #include <linux/moduleloader.h>
@@ -283,12 +284,15 @@ static int can_probe(unsigned long paddr)if (ret <0 )return 0 ; +#ifdef CONFIG_KGDB /* -* Another debugging subsystem might insert this breakpoint. -* In that case, we can't recover it. +* If there is a dynamically installed kgdb sw breakpoint, +* this function should not be probed. */ -if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE ) +if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE && +kgdb_has_hit_break (addr))return 0 ; +#endif addr += insn.length; }
PATCH v3 2/2 x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
In reply to Masami Hiramatsu (Google)From: Masami Hiramatsu (Google) <[email protected]>
Since the CONFIG_RETHUNK and CONFIG_SLS will use INT3 for stopping speculative execution after function return, kprobe jump optimization always fails on the functions with such INT3 inside the function body. (It already checks the INT3 padding between functions, but not inside
the function)
To avoid this issue, as same as kprobes, check whether the INT3 comes from kgdb or not, and if so, stop decoding and make it fail. The other INT3 will come from CONFIG_RETHUNK/CONFIG_SLS and those can be treated as a one-byte instruction.
--- arch/x86/kernel/kprobes/opt.c | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-)
arch/x86/kernel/kprobes/opt.c +8 -20
@@ -15,6 +15,7 @@#include <linux/extable.h> #include <linux/kdebug.h> #include <linux/kallsyms.h> +#include <linux/kgdb.h> #include <linux/ftrace.h> #include <linux/objtool.h> #include <linux/pgtable.h>
@@ -279,19 +280,6 @@ static int insn_is_indirect_jump(struct insn *insn)return ret; } -static bool is_padding_int3 (unsigned long addr,unsigned long eaddr) -{ -unsigned char ops; - -for (; addr < eaddr; addr++) { -if (get_kernel_nofault (ops, (void *)addr) <0 || - ops !=INT3_INSN_OPCODE ) -return false; - } - -return true; -} -/* Decode whole function to ensure any instructions don't jump into target */ static int can_optimize (unsigned long paddr) {
@@ -334,15 +322,15 @@ static int can_optimize(unsigned long paddr) ret =insn_decode_kernel (&insn, (void *)recovered_insn);if (ret <0 )return 0 ; - +#ifdef CONFIG_KGDB /* -* In the case of detecting unknown breakpoint, this could be -* a padding INT3 between functions. Let's check that all the -* rest of the bytes are also INT3. +* If there is a dynamically installed kgdb sw breakpoint, +* this function should not be probed. */ -if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE ) -return is_padding_int3 (addr, paddr - offset + size) ?1 :0 ; - +if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE && +kgdb_has_hit_break (addr)) +return 0 ; +#endif /* Recover address */ insn.kaddr = (void *)addr; insn.next_byte = (void *)(addr + insn.length);
Quoted 10 lines · Masami Hiramatsu (Google)
Hi Peter and Josh,
So here is 3rd version of the patches to fix kprobes and optprobe with
CONFIG_RETHUNKandCONFIG_SLS.
Previous version is here;https://lore.kernel.org/all/166260087224.759381.4170102827490658262.stgit@devnote2/
In this version, I simplified all code and just checks the INT3 comes from kgdb or not. Other INT3 are treated as one-byte instruction.
Looks good to me.
I was confused by the naming of kgdb_has_hit_break(), because in this case, with the function being called from outside the stopped kgdb context, the breakpoint hasn't actually been hit. But it still seems to do the right thing: it checks for BP_ACTIVE which means the breakpoint has been written.
Signature · 1 line
Josh
Re: [PATCH] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
In reply to Alexei StarovoitovQuoted 8 lines · Alexei Starovoitov
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -419,7 +419,8 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else {
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip) · line 423
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +2
-1-EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +EMIT1 (0xCC );/* int3 */ Hmm. Why is this unconditional?
Shouldn't it be guarded withCONFIG_xxorcpu_feature_enabled? People that don't care about hw speculation vulnerabilities shouldn't pay the price of increased code size.
Sure, like so then?
---
Subject: x86,retpoline: Be sure to emit INT3 after JMP *%\reg
From: Peter Zijlstra <[email protected]>
Date: Thu, 8 Sep 2022 12:04:50 +0200
Both AMD and Intel recommend using INT3 after an indirect JMP. Make sure to emit one when rewriting the retpoline JMP irrespective of compiler SLS options or even CONFIG_SLS.
---
arch/x86/kernel/alternative.c | 9 +++++++++ arch/x86/net/bpf_jit_comp.c | 4 +++- 2 files changed, 12 insertions(+), 1 deletion(-)
arch/x86/kernel/alternative.c +9 -0
@@ -453,6 +453,15 @@ static int patch_retpoline(void *addr, sreturn ret; i += ret; +/* +* The compiler is supposed to EMIT an INT3 after every unconditional +* JMP instruction due to AMD BTC. However, if the compiler is too old +* or SLS isn't enabled, we still need an INT3 after indirect JMPs +* even on Intel. +*/ +if (op ==JMP32_INSN_OPCODE && i < insn->length) + bytes[i++] =INT3_INSN_OPCODE ; +for (; i < insn->length;) bytes[i++] =BYTES_NOP1 ;
arch/x86/net/bpf_jit_comp.c +3 -1
@@ -419,7 +419,9 @@ static void emit_indirect_jump(u8 **pproOPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +if (IS_ENABLED (CONFIG_RETPOLINE ) ||IS_ENABLED (CONFIG_SLS )) +EMIT1 (0xCC );/* int3 */ } *pprog = prog;
Re: [PATCH] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
Quoted 59 lines · Peter Zijlstra
Quoted 8 lines · Alexei Starovoitov
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +0
-0@@ -419,7 +419,8 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else {Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **ppro · line 424
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +2
-1-EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +EMIT1 (0xCC );/* int3 */ Hmm. Why is this unconditional?
Shouldn't it be guarded withCONFIG_xxorcpu_feature_enabled? People that don't care about hw speculation vulnerabilities shouldn't pay the price of increased code size.Sure, like so then?
---
Subject: x86,retpoline: Be sure to emit INT3 after JMP *%\reg
From: Peter Zijlstra <[email protected]>
Date: Thu, 8 Sep 2022 12:04:50 +0200Both AMD and Intel recommend using INT3 after an indirect JMP. Make sure to emit one when rewriting the retpoline JMP irrespective of compiler SLS options or even
CONFIG_SLS.Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
arch/x86/kernel/alternative.c | 9 +++++++++ arch/x86/net/bpf_jit_comp.c | 4 +++- 2 files changed, 12 insertions(+), 1 deletion(-)arch/x86/kernel/alternative.c +9
-0@@ -453,6 +453,15 @@ static int patch_retpoline(void *addr, sreturn ret; i += ret; +/* +* The compiler is supposed to EMIT an INT3 after every unconditional +* JMP instruction due to AMD BTC. However, if the compiler is too old +* or SLS isn't enabled, we still need an INT3 after indirect JMPs +* even on Intel. +*/ +if (op ==JMP32_INSN_OPCODE && i < insn->length) + bytes[i++] =INT3_INSN_OPCODE ; +for (; i < insn->length;) bytes[i++] =BYTES_NOP1 ;
arch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **ppro · line 424
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -419,7 +419,9 @@ static void emit_indirect_jump(u8 **pproOPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +if (IS_ENABLED (CONFIG_RETPOLINE ) ||IS_ENABLED (CONFIG_SLS )) +EMIT1 (0xCC );/* int3 */
Looks better. Ack.
Re: [PATCH] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
In reply to Peter Zijlstraarch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **ppro · line 425
Quoted 10 lines · Peter Zijlstra
+++
b/arch/x86/net/bpf_jit_comp.carch/x86/net/bpf_jit_comp.c +3
-1@@ -419,7 +419,9 @@ static void emit_indirect_jump(u8 **pproOPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +if (IS_ENABLED (CONFIG_RETPOLINE ) ||IS_ENABLED (CONFIG_SLS )) +EMIT1 (0xCC );/* int3 */ }
Hm, if you have retpolines disabled at runtime, why would you want this.
Signature · 1 line
Josh
Re: [PATCH] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
Quoted 13 lines · Josh Poimboeuf
arch/x86/net/bpf_jit_comp.c static void emit_indirect_jump(u8 **ppro · line 425
Quoted 10 lines · Peter Zijlstra
+++
b/arch/x86/net/bpf_jit_comp.carch/x86/net/bpf_jit_comp.c +3
-1@@ -419,7 +419,9 @@ static void emit_indirect_jump(u8 **pproOPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +if (IS_ENABLED (CONFIG_RETPOLINE ) ||IS_ENABLED (CONFIG_SLS )) +EMIT1 (0xCC );/* int3 */ }Hm, if you have retpolines disabled at runtime, why would you want this.
Because I don't think eIBRS guarantees it will not SLS.
[tip: x86/core] x86,retpoline: Be sure to emit INT3 after JMP *%\reg
In reply to Peter ZijlstraThe following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: 8c03af3e090e9d57d90f482d344563dd4bae1e66 Gitweb: https://git.kernel.org/tip/8c03af3e090e9d57d90f482d344563dd4bae1e66 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 08 Sep 2022 12:04:50 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 15 Sep 2022 16:13:53 +02:00
x86,retpoline: Be sure to emit INT3 after JMP *%\reg
Both AMD and Intel recommend using INT3 after an indirect JMP. Make sure to emit one when rewriting the retpoline JMP irrespective of compiler SLS options or even CONFIG_SLS.
--- arch/x86/kernel/alternative.c | 9 +++++++++ arch/x86/net/bpf_jit_comp.c | 4 +++- 2 files changed, 12 insertions(+), 1 deletion(-)
arch/x86/kernel/alternative.c +9 -0
@@ -453,6 +453,15 @@ static int patch_retpoline(void *addr, struct insn *insn, u8 *bytes)return ret; i += ret; +/* +* The compiler is supposed to EMIT an INT3 after every unconditional +* JMP instruction due to AMD BTC. However, if the compiler is too old +* or SLS isn't enabled, we still need an INT3 after indirect JMPs +* even on Intel. +*/ +if (op ==JMP32_INSN_OPCODE && i < insn->length) + bytes[i++] =INT3_INSN_OPCODE ; +for (; i < insn->length;) bytes[i++] =BYTES_NOP1 ;
arch/x86/net/bpf_jit_comp.c +3 -1
@@ -419,7 +419,9 @@ static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)OPTIMIZER_HIDE_VAR (reg);emit_jump (&prog, &__x86_indirect_thunk_array[reg], ip); }else { -EMIT2 (0xFF ,0xE0 + reg); +EMIT2 (0xFF ,0xE0 + reg);/* jmp *%\reg */ +if (IS_ENABLED (CONFIG_RETPOLINE ) ||IS_ENABLED (CONFIG_SLS )) +EMIT1 (0xCC );/* int3 */ } *pprog = prog;
Re: [PATCH v3 2/2] x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
In reply to Masami Hiramatsu (Google)Quoted 69 lines · Masami Hiramatsu (Google)
From: Masami Hiramatsu (Google) <[email protected]>
Since the
CONFIG_RETHUNKandCONFIG_SLSwill use INT3 for stopping speculative execution after function return, kprobe jump optimization always fails on the functions with such INT3 inside the function body. (It already checks the INT3 padding between functions, but not inside the function)To avoid this issue, as same as kprobes, check whether the INT3 comes from kgdb or not, and if so, stop decoding and make it fail. The other INT3 will come from CONFIG_RETHUNK/CONFIG_SLS and those can be treated as a one-byte instruction.
Signed-off-by Masami Hiramatsu (Google) <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Fixes e463a09af2f0 ("x86: Add straight-line-speculation mitigation")---
arch/x86/kernel/kprobes/opt.c| 28 ++++++++--------------------1 file changed, 8 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes/opt.c +4
-17@@ -15,6 +15,7 @@#include <linux/extable.h> #include <linux/kdebug.h> #include <linux/kallsyms.h> +#include <linux/kgdb.h> #include <linux/ftrace.h> #include <linux/objtool.h> #include <linux/pgtable.h> @@ -279,19 +280,6 @@ static int insn_is_indirect_jump(struct insn *insn)return ret; } -static bool is_padding_int3 (unsigned long addr,unsigned long eaddr) -{ -unsigned char ops; - -for (; addr < eaddr; addr++) { -if (get_kernel_nofault (ops, (void *)addr) <0 || - ops !=INT3_INSN_OPCODE ) -return false; - } - -return true; -} -/* Decode whole function to ensure any instructions don't jump into target */ static int can_optimize (unsigned long paddr) {@@ -334,15 +322,15 @@ static int can_optimize(unsigned long paddr) ret =insn_decode_kernel (&insn, (void *)recovered_insn);if (ret <0 )return 0 ; - +#ifdef CONFIG_KGDB /* -* In the case of detecting unknown breakpoint, this could be -* a padding INT3 between functions. Let's check that all the -* rest of the bytes are also INT3. +* If there is a dynamically installed kgdb sw breakpoint, +* this function should not be probed. */
arch/x86/kernel/kprobes/opt.c static int can_optimize(unsigned long paddr) · line 336
Quoted 10 lines
arch/x86/kernel/kprobes/opt.c +4
-3-if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE ) -return is_padding_int3 (addr, paddr - offset + size) ?1 :0 ; - +if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE && +kgdb_has_hit_break (addr)) +return 0 ; +#endif /* Recover address */ insn.kaddr = (void *)addr; insn.next_byte = (void *)(addr + insn.length);
Hi Masami,
I encountered a similar issue with can_probe(). I see that your patches were not upstreamed, at least to 6.1.
So I was wondering whether it they are going to be upstreamed, and whether you want also to update can_probe().
Thanks,
Nadav
Re: [PATCH v3 2/2] x86/kprobes: Fix optprobe optimization check with CONFIG_RETHUNK
Quoted 82 lines · On Thu, 15 Dec 2022 03:31:25 +0000 Nadav Amit <[email protected]> wrote:
Quoted 69 lines · Masami Hiramatsu (Google)
From: Masami Hiramatsu (Google) <[email protected]>
Since the
CONFIG_RETHUNKandCONFIG_SLSwill use INT3 for stopping speculative execution after function return, kprobe jump optimization always fails on the functions with such INT3 inside the function body. (It already checks the INT3 padding between functions, but not inside the function)To avoid this issue, as same as kprobes, check whether the INT3 comes from kgdb or not, and if so, stop decoding and make it fail. The other INT3 will come from CONFIG_RETHUNK/CONFIG_SLS and those can be treated as a one-byte instruction.
Signed-off-by Masami Hiramatsu (Google) <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Fixes e463a09af2f0 ("x86: Add straight-line-speculation mitigation")---
arch/x86/kernel/kprobes/opt.c| 28 ++++++++--------------------1 file changed, 8 insertions(+), 20 deletions(-)arch/x86/kernel/kprobes/opt.c +4
-17@@ -15,6 +15,7 @@#include <linux/extable.h> #include <linux/kdebug.h> #include <linux/kallsyms.h> +#include <linux/kgdb.h> #include <linux/ftrace.h> #include <linux/objtool.h> #include <linux/pgtable.h> @@ -279,19 +280,6 @@ static int insn_is_indirect_jump(struct insn *insn)return ret; } -static bool is_padding_int3 (unsigned long addr,unsigned long eaddr) -{ -unsigned char ops; - -for (; addr < eaddr; addr++) { -if (get_kernel_nofault (ops, (void *)addr) <0 || - ops !=INT3_INSN_OPCODE ) -return false; - } - -return true; -} -/* Decode whole function to ensure any instructions don't jump into target */ static int can_optimize (unsigned long paddr) {@@ -334,15 +322,15 @@ static int can_optimize(unsigned long paddr) ret =insn_decode_kernel (&insn, (void *)recovered_insn);if (ret <0 )return 0 ; - +#ifdef CONFIG_KGDB /* -* In the case of detecting unknown breakpoint, this could be -* a padding INT3 between functions. Let's check that all the -* rest of the bytes are also INT3. +* If there is a dynamically installed kgdb sw breakpoint, +* this function should not be probed. */ arch/x86/kernel/kprobes/opt.c static int can_optimize(unsigned long paddr) · line 336
Quoted 10 lines
arch/x86/kernel/kprobes/opt.c +4
-3-if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE ) -return is_padding_int3 (addr, paddr - offset + size) ?1 :0 ; - +if (insn.opcode.bytes[0 ] ==INT3_INSN_OPCODE && +kgdb_has_hit_break (addr)) +return 0 ; +#endif /* Recover address */ insn.kaddr = (void *)addr; insn.next_byte = (void *)(addr + insn.length);Hi Masami,
Quoted 2 lines
I encountered a similar issue with
can_probe(). I see that your patches were not upstreamed, at least to 6.1.
Oops, I should update and resend the series. This patch must go through x86 (tip) tree since it is x86 specific issue. Moreover, this is a kind of bugfix because kprobes doesn't work correctly with this issue.
Quoted 3 lines
So I was wondering whether it they are going to be upstreamed, and whether you want also to update
can_probe().
Yeah, I want to push it.
Thank you!
Quoted 3 lines
Thanks,
Nadav
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>