kcov, objtool: Make runtime functions noinstr-compatible
PATCH v2 1/2 kcov, objtool: Make runtime functions noinstr-compatible
While we lack a compiler attribute to add to noinstr that would disable KCOV, make the KCOV runtime functions return if the caller is in a noinstr section. We then whitelist __sanitizer_cov_*() functions in objtool. __sanitizer_cov_*() cannot safely become safe noinstr functions as-is, as they may fault due to accesses to vmalloc's memory.
Declare write_comp_data() as __always_inline to ensure it is inlined, and reduce stack usage and remove one extra call from the fast-path.
In future, our compilers may provide an attribute to implement __no_sanitize_coverage, which can then be added to noinstr, and the checks added in this patch can be guarded by an #ifdef checking if the compiler has such an attribute or not.
---
Apply after:
https://lkml.kernel.org/r/[email protected]
v2:
* Rewrite based on Peter's and Andrey's feedback -- v1 worked because we got lucky. Let's not rely on luck, as it will be difficult to ensure the same conditions remain true in future.
v1: https://lkml.kernel.org/r/[email protected]
Note: There are a set of KCOV patches from Andrey in -next:
https://lkml.kernel.org/r/[email protected] -- Git cleanly merges this patch with those patches, and no merge conflict is expected.
--- kernel/kcov.c | 19 +++++++++++++++++-- tools/objtool/check.c | 7 +++++++ 2 files changed, 24 insertions(+), 2 deletions(-)
kernel/kcov.c +17 -2
@@ -24,6 +24,7 @@#include <linux/refcount.h> #include <linux/log2.h> #include <asm/setup.h> +#include <asm/sections.h> #define kcov_debug (fmt, ...) pr_debug("%s: " fmt, __func__, ##__VA_ARGS__)
@@ -172,6 +173,12 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)return ip; } +static __always_inline bool in_noinstr_section(unsigned long ip) +{ +return (unsigned long )__noinstr_text_start <= ip && + ip < (unsigned long )__noinstr_text_end; +} +/* * Entry point from instrumented code. * This is called once per basic-block/edge.
@@ -180,13 +187,18 @@ void notrace __sanitizer_cov_trace_pc(void) {struct task_struct *t;unsigned long *area; -unsigned long ip =canonicalize_ip (_RET_IP_); +unsigned long ip;unsigned long pos; +if (unlikely (in_noinstr_section (_RET_IP_))) +return ; + t = current;if (!check_kcov_mode (KCOV_MODE_TRACE_PC , t))return ; + ip =canonicalize_ip (_RET_IP_); + area = t->kcov_area;/* The first 64-bit word is the number of subsequent PCs. */ pos =READ_ONCE (area[0 ]) +1 ;
@@ -198,12 +210,15 @@ void notrace __sanitizer_cov_trace_pc(void)EXPORT_SYMBOL (__sanitizer_cov_trace_pc);#ifdef CONFIG_KCOV_ENABLE_COMPARISONS -static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) +static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) {struct task_struct *t;u64 *area;u64 count, start_index, end_pos, max_pos; +if (unlikely (in_noinstr_section (ip))) +return ; + t = current;if (!check_kcov_mode (KCOV_MODE_TRACE_CMP , t))return ;
tools/objtool/check.c +7 -0
@@ -2213,6 +2213,13 @@ static inline bool noinstr_call_dest(struct symbol *func)if (!strncmp (func->name,"__ubsan_handle_" ,15 ))return true; +/* +* The __sanitizer_cov_*() calls include a check if the caller is in the +* noinstr section, and simply return if that is the case. +*/ +if (!strncmp (func->name,"__sanitizer_cov_" ,16 )) +return true; +return false; }
Signature · 1 line
2.27.0.rc2.251.g90737beb825-goog
PATCH v2 2/2 kcov: Pass -fno-stack-protector with Clang
For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist with Clang.
--- kernel/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/Makefile +1 -1
@@ -35,7 +35,7 @@ KCOV_INSTRUMENT_stacktrace.o := n KCOV_INSTRUMENT_kcov.o := n KASAN_SANITIZE_kcov.o := n KCSAN_SANITIZE_kcov.o := n -CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) +CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) $(call cc-option, -fno-stack-protector) # cond_syscall is currently not LTO compatible CFLAGS_sys_ni.o = $(DISABLE_LTO)
Signature · 1 line
2.27.0.rc2.251.g90737beb825-goog
Re: [PATCH v2 2/2] kcov: Pass -fno-stack-protector with Clang
Quoted 14 lines · Marco Elver' via Clang Built Linux
For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist with Clang.
Signed-off-by Marco Elver <[email protected]>--- kernel/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/Makefile +0
-0
kernel/Makefile KCOV_INSTRUMENT_stacktrace.o := n · line 38
Quoted 5 lines
kernel/Makefile +1
-1@@ -35,7 +35,7 @@ KCOV_INSTRUMENT_stacktrace.o := n KCOV_INSTRUMENT_kcov.o := n KASAN_SANITIZE_kcov.o := n KCSAN_SANITIZE_kcov.o := n -CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector) +CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) $(call cc-option, -fno-stack-protector)
All supported compiler versions understand -fno-stack-protector. Please unwrap it from cc-option. That's one less compiler invocation at build time.
Quoted 4 lines
kernel/Makefile +0
-1# cond_syscall is currently not LTO compatible CFLAGS_sys_ni.o = $(DISABLE_LTO) --
Signature · 2 lines
Thanks, ~Nick Desaulniers
Peter Zijlstra
maintainerQuoted 7 lines · Marco Elver
While we lack a compiler attribute to add to noinstr that would disable KCOV, make the KCOV runtime functions return if the caller is in a noinstr section. We then whitelist __sanitizer_cov_*() functions in objtool.
__sanitizer_cov_*() cannot safely become safe noinstr functions as-is, as they may fault due to accesses to vmalloc's memory.
I would feel very much better with those actually in noinstr, because without it, there is nothing stopping us from adding a kprobe/hw- breakpoint or other funny to the function.
Even if they almost instra-return, having a kprobe on the function entry or condition check is enough to utterly wreck things.
So something like:
void noinstr __sanitizer_cov_trace_*(...)
{
if (within_noinstr_section(ip))
return;
instrumentation_begin();
write_comp_data(...);
instrumentation_end();
}Would make me feel a whole lot better.
kernel/kcov.c static notrace unsigned long canonicalize_ip(unsigned long ip) · line 180
Quoted 5 lines
kernel/kcov.c +5
-0+static __always_inline bool in_noinstr_section(unsigned long ip) +{ +return (unsigned long )__noinstr_text_start <= ip && + ip < (unsigned long )__noinstr_text_end; +}
.entry.text is also considered noinstr, although I suppose that all being in .S files avoids it having annotations inserted, but perhaps a comment?
[-Cc most]
[+Cc Mark]
On Thu, 4 Jun 2020 at 17:25, Peter Zijlstra <[email protected]> wrote:
[...]
Quoted 10 lines
I would feel very much better with those actually in noinstr, because without it, there is nothing stopping us from adding a kprobe/hw- breakpoint or other funny to the function.
Even if they almost instra-return, having a kprobe on the function entry or condition check is enough to utterly wreck things.
So something like:
void noinstr __sanitizer_cov_trace_*(...)
Quoted 8 lines
{
if (within_noinstr_section(ip))
return;
instrumentation_begin();
write_comp_data(...);
instrumentation_end();
}Apologies for resurrecting this. :-)
It seems I'll need to use this approach soon for upcoming KCSAN instrumentation for memory barriers. I'm able to use the same objtool feature that erases __sanitizer_cov* calls on x86 to erase memory barrier instrumentation, but arm64 will still be a problem because of lack of objtool support.
Mark, on arm64, is the approach above that Peter proposed ~1y ago acceptable in general to make instrumentation noinstr-safe?
Thanks,
-- Marco