Skip to contentSkip to conversation

kcov: Make runtime functions noinstr-compatible

PATCH Superseded Marco Elver 3 participants 8 messages Jun 4, 2020 KCOV

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kcov: Make runtime functions noinstr-compatible

The KCOV runtime is very minimal, only updating a field in 'current', and none of __sanitizer_cov-functions generates reports nor calls any other external functions.

Therefore we can make the KCOV runtime noinstr-compatible by:

1. always-inlining internal functions and marking __sanitizer_cov-functions noinstr. The function write_comp_data() is now guaranteed to be inlined into __sanitize_cov_trace_*cmp() functions, which saves a call in the fast-path and reduces stack pressure due to the first argument being a constant.

2. For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist on Clang.

The major benefit compared to adding another attribute to 'noinstr' to not collect coverage information, is that we retain coverage visibility in noinstr functions. We also currently lack such an attribute in both GCC and Clang.

---
Note: There are a set of KCOV patches from Andrey in -next:
https://lkml.kernel.org/r/[email protected] -- Git cleanly merges this patch with those patches, and no merge conflict is expected.

---
 kernel/Makefile |  2 +-
 kernel/kcov.c   | 26 +++++++++++++-------------
 2 files changed, 14 insertions(+), 14 deletions(-)
kernel/Makefile +1 -1
index 5d935b63f812..8e282c611a72 100644
@@ -35,7 +35,7 @@ KCOV_INSTRUMENT_stacktrace.o := n
 KCOV_INSTRUMENT_kcov.o := n
 KASAN_SANITIZE_kcov.o := n
 KCSAN_SANITIZE_kcov.o := n
-CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack -fno-stack-protector)
+CFLAGS_kcov.o := $(call cc-option, -fno-conserve-stack) $(call cc-option, -fno-stack-protector)
 
 # cond_syscall is currently not LTO compatible
 CFLAGS_sys_ni.o = $(DISABLE_LTO)
kernel/kcov.c +13 -13
index 8accc9722a81..d6e3be2d0570 100644
@@ -142,7 +142,7 @@ static void kcov_remote_area_put(struct kcov_remote_area *area,
 	list_add(&area->list, &kcov_remote_areas);
 }
 
-static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t)
+static __always_inline bool check_kcov_mode(enum kcov_mode needed_mode, struct task_struct *t)
 {
 	unsigned int mode;
 
@@ -164,7 +164,7 @@ static notrace bool check_kcov_mode(enum kcov_mode needed_mode, struct task_stru
 	return mode == needed_mode;
 }
 
-static notrace unsigned long canonicalize_ip(unsigned long ip)
+static __always_inline unsigned long canonicalize_ip(unsigned long ip)
 {
 #ifdef CONFIG_RANDOMIZE_BASE
 	ip -= kaslr_offset();
@@ -176,7 +176,7 @@ static notrace unsigned long canonicalize_ip(unsigned long ip)
  * Entry point from instrumented code.
  * This is called once per basic-block/edge.
  */
-void notrace __sanitizer_cov_trace_pc(void)
+void noinstr __sanitizer_cov_trace_pc(void)
 {
 	struct task_struct *t;
 	unsigned long *area;
@@ -198,7 +198,7 @@ void notrace __sanitizer_cov_trace_pc(void)
 EXPORT_SYMBOL(__sanitizer_cov_trace_pc);
 
 #ifdef CONFIG_KCOV_ENABLE_COMPARISONS
-static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
+static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 {
 	struct task_struct *t;
 	u64 *area;
@@ -231,59 +231,59 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 	}
 }
 
-void notrace __sanitizer_cov_trace_cmp1(u8 arg1, u8 arg2)
+void noinstr __sanitizer_cov_trace_cmp1(u8 arg1, u8 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(0), arg1, arg2, _RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_cmp1);
 
-void notrace __sanitizer_cov_trace_cmp2(u16 arg1, u16 arg2)
+void noinstr __sanitizer_cov_trace_cmp2(u16 arg1, u16 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(1), arg1, arg2, _RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_cmp2);
 
-void notrace __sanitizer_cov_trace_cmp4(u32 arg1, u32 arg2)
+void noinstr __sanitizer_cov_trace_cmp4(u32 arg1, u32 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(2), arg1, arg2, _RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_cmp4);
 
-void notrace __sanitizer_cov_trace_cmp8(u64 arg1, u64 arg2)
+void noinstr __sanitizer_cov_trace_cmp8(u64 arg1, u64 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(3), arg1, arg2, _RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_cmp8);
 
-void notrace __sanitizer_cov_trace_const_cmp1(u8 arg1, u8 arg2)
+void noinstr __sanitizer_cov_trace_const_cmp1(u8 arg1, u8 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(0) | KCOV_CMP_CONST, arg1, arg2,
 			_RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_const_cmp1);
 
-void notrace __sanitizer_cov_trace_const_cmp2(u16 arg1, u16 arg2)
+void noinstr __sanitizer_cov_trace_const_cmp2(u16 arg1, u16 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(1) | KCOV_CMP_CONST, arg1, arg2,
 			_RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_const_cmp2);
 
-void notrace __sanitizer_cov_trace_const_cmp4(u32 arg1, u32 arg2)
+void noinstr __sanitizer_cov_trace_const_cmp4(u32 arg1, u32 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(2) | KCOV_CMP_CONST, arg1, arg2,
 			_RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_const_cmp4);
 
-void notrace __sanitizer_cov_trace_const_cmp8(u64 arg1, u64 arg2)
+void noinstr __sanitizer_cov_trace_const_cmp8(u64 arg1, u64 arg2)
 {
 	write_comp_data(KCOV_CMP_SIZE(3) | KCOV_CMP_CONST, arg1, arg2,
 			_RET_IP_);
 }
 EXPORT_SYMBOL(__sanitizer_cov_trace_const_cmp8);
 
-void notrace __sanitizer_cov_trace_switch(u64 val, u64 *cases)
+void noinstr __sanitizer_cov_trace_switch(u64 val, u64 *cases)
 {
 	u64 i;
 	u64 count = cases[0];
Signature · 1 line
2.27.0.rc2.251.g90737beb825-goog
#1 lore raw
Quoted 3 lines · Marco Elver

The KCOV runtime is very minimal, only updating a field in 'current', and none of __sanitizer_cov-functions generates reports nor calls any other external functions.

Not quite true; it writes to t->kcov_area, and we need to make absolutely sure that doesn't take faults or triggers anything else untowards.

Quoted 23 lines

Therefore we can make the KCOV runtime noinstr-compatible by:

1. always-inlining internal functions and marking __sanitizer_cov-functions noinstr. The function write_comp_data() is now guaranteed to be inlined into __sanitize_cov_trace_*cmp() functions, which saves a call in the fast-path and reduces stack pressure due to the first argument being a constant.

2. For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist on Clang.

The major benefit compared to adding another attribute to 'noinstr' to not collect coverage information, is that we retain coverage visibility in noinstr functions. We also currently lack such an attribute in both GCC and Clang.

kernel/kcov.c +1 -1
-static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
+static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 {
 	struct task_struct *t;
 	u64 *area;

kernel/kcov.c static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) · line 232

Quoted 2 lines
kernel/kcov.c +0 -0
@@ -231,59 +231,59 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 	}
 }

This thing; that appears to be the meat of it, right?

I can't find where t->kcov_area comes from.. is that always kcov_mmap()'s vmalloc_user() ?

That whole kcov_remote stuff confuses me.

KCOV_ENABLE() has kcov_fault_in_area(), which supposedly takes the vmalloc faults for the current task, but who does it for the remote?

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

So no #PF from writing to t->kcov_area then, under the assumption that the vmalloc_user() is the only allocation site.

But then there's hardware watchpoints, if someone goes and sets a data watchpoint in the kcov_area we're screwed. Nothing actively prevents that from happening. Then again, the same is currently true for much of current :/

Also, I think you need __always_inline on kaslr_offset()

And, unrelated to this patch in specific, I suppose I'm going to have to extend objtool to look for data that is used from noinstr, to make sure we exclude it from inspection and stuff, like that kaslr offset crud for example.

Anyway, yes, it appears you're lucky (for having Joerg remove vmalloc faults) and this mostly should work as is.

#2 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Marco Elver

The KCOV runtime is very minimal, only updating a field in 'current', and none of __sanitizer_cov-functions generates reports nor calls any other external functions.

Not quite true; it writes to t->kcov_area, and we need to make absolutely sure that doesn't take faults or triggers anything else untowards.

Ah, right.

Quoted 28 lines
Quoted 23 lines

Therefore we can make the KCOV runtime noinstr-compatible by:

1. always-inlining internal functions and marking __sanitizer_cov-functions noinstr. The function write_comp_data() is now guaranteed to be inlined into __sanitize_cov_trace_*cmp() functions, which saves a call in the fast-path and reduces stack pressure due to the first argument being a constant.

2. For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist on Clang.

The major benefit compared to adding another attribute to 'noinstr' to not collect coverage information, is that we retain coverage visibility in noinstr functions. We also currently lack such an attribute in both GCC and Clang.

kernel/kcov.c +1 -1
-static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
+static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 {
      struct task_struct *t;
      u64 *area;

kernel/kcov.c static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) · line 232

Quoted 2 lines
kernel/kcov.c +0 -0
@@ -231,59 +231,59 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
      }
 }

This thing; that appears to be the meat of it, right?

Quoted 2 lines

I can't find where t->kcov_area comes from.. is that always kcov_mmap()'s vmalloc_user() ?

Yeah, looks like it.

Quoted 19 lines

That whole kcov_remote stuff confuses me.

KCOV_ENABLE() has kcov_fault_in_area(), which supposedly takes the vmalloc faults for the current task, but who does it for the remote?

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

So no #PF from writing to t->kcov_area then, under the assumption that the vmalloc_user() is the only allocation site.

But then there's hardware watchpoints, if someone goes and sets a data watchpoint in the kcov_area we're screwed. Nothing actively prevents that from happening. Then again, the same is currently true for much of current :/

Also, I think you need __always_inline on kaslr_offset()

Quoted 7 lines

And, unrelated to this patch in specific, I suppose I'm going to have to extend objtool to look for data that is used from noinstr, to make sure we exclude it from inspection and stuff, like that kaslr offset crud for example.

Anyway, yes, it appears you're lucky (for having Joerg remove vmalloc faults) and this mostly should work as is.

Hmm, looks like this doesn't generally work then. :-/

An alternative would be to check if '__noinstr_text_start <= _RET_IP_ < __noinstr_text_end' in __sanitizer_cov-functions and return if that's the case. This could be #ifdef'd when we get a compiler that can do __no_sanitize_coverage. At least that way we get working KCOV for now.

Would that work?

Thanks,
-- Marco

#3 lore raw
In reply to Peter Zijlstra
Quoted 17 lines · Peter Zijlstra
Quoted 3 lines · Marco Elver

The KCOV runtime is very minimal, only updating a field in 'current', and none of __sanitizer_cov-functions generates reports nor calls any other external functions.

Not quite true; it writes to t->kcov_area, and we need to make absolutely sure that doesn't take faults or triggers anything else untowards.

Quoted 7 lines

Therefore we can make the KCOV runtime noinstr-compatible by:

1. always-inlining internal functions and marking __sanitizer_cov-functions noinstr. The function write_comp_data() is now guaranteed to be inlined into __sanitize_cov_trace_*cmp() functions, which saves a call in the fast-path and reduces stack pressure due to the first argument being a constant.

Maybe we could do CFLAGS_REMOVE_kcov.o = $(CC_FLAGS_FTRACE) the same way we do it for KASAN? And drop notrace/noinstr from kcov. Would it resolve the issue? I'm not sure which solution is better though.

Quoted 21 lines
Quoted 16 lines

2. For Clang, correctly pass -fno-stack-protector via a separate cc-option, as -fno-conserve-stack does not exist on Clang.

The major benefit compared to adding another attribute to 'noinstr' to not collect coverage information, is that we retain coverage visibility in noinstr functions. We also currently lack such an attribute in both GCC and Clang.

kernel/kcov.c +1 -1
-static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
+static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 {
      struct task_struct *t;
      u64 *area;

kernel/kcov.c static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) · line 232

Quoted 2 lines
kernel/kcov.c +0 -0
@@ -231,59 +231,59 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
      }
 }

This thing; that appears to be the meat of it, right?

Quoted 7 lines

I can't find where t->kcov_area comes from.. is that always kcov_mmap()'s vmalloc_user() ?

That whole kcov_remote stuff confuses me.

KCOV_ENABLE() has kcov_fault_in_area(), which supposedly takes the vmalloc faults for the current task, but who does it for the remote?

Hm, no one. This might be an issue, thanks for noticing!

Quoted 2 lines

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

Could you point me to those patches?

Even though it might work fine now, we might get issues if we backport remote kcov to older kernels.

Quoted 19 lines

So no #PF from writing to t->kcov_area then, under the assumption that the vmalloc_user() is the only allocation site.

But then there's hardware watchpoints, if someone goes and sets a data watchpoint in the kcov_area we're screwed. Nothing actively prevents that from happening. Then again, the same is currently true for much of current :/

Also, I think you need __always_inline on kaslr_offset()

And, unrelated to this patch in specific, I suppose I'm going to have to extend objtool to look for data that is used from noinstr, to make sure we exclude it from inspection and stuff, like that kaslr offset crud for example.

Anyway, yes, it appears you're lucky (for having Joerg remove vmalloc faults) and this mostly should work as is.

#4 lore raw
Quoted 20 lines · Andrey Konovalov
Quoted 17 lines · Peter Zijlstra
Quoted 3 lines · Marco Elver

The KCOV runtime is very minimal, only updating a field in 'current', and none of __sanitizer_cov-functions generates reports nor calls any other external functions.

Not quite true; it writes to t->kcov_area, and we need to make absolutely sure that doesn't take faults or triggers anything else untowards.

Quoted 7 lines

Therefore we can make the KCOV runtime noinstr-compatible by:

1. always-inlining internal functions and marking __sanitizer_cov-functions noinstr. The function write_comp_data() is now guaranteed to be inlined into __sanitize_cov_trace_*cmp() functions, which saves a call in the fast-path and reduces stack pressure due to the first argument being a constant.

Quoted 3 lines

Maybe we could do CFLAGS_REMOVE_kcov.o = $(CC_FLAGS_FTRACE) the same way we do it for KASAN? And drop notrace/noinstr from kcov. Would it resolve the issue? I'm not sure which solution is better though.

Sadly no. 'noinstr' implies 'notrace', but also places the function in the .noinstr.text section for the purpose of objtool checking. But: we should only mark a function 'noinstr' if it (and its callees) satisfies the requirements that Peter outlined (are the requirements documented somewhere?). In particular, we need to worry about vmalloc faults.

[...]

Quoted 21 lines
Quoted 11 lines

kernel/kcov.c static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) · line 232

Quoted 8 lines
kernel/kcov.c +1 -1
-static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
+static __always_inline void write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
 {
      struct task_struct *t;
      u64 *area;
@@ -231,59 +231,59 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip)
      }
 }

This thing; that appears to be the meat of it, right?

Quoted 7 lines

I can't find where t->kcov_area comes from.. is that always kcov_mmap()'s vmalloc_user() ?

That whole kcov_remote stuff confuses me.

KCOV_ENABLE() has kcov_fault_in_area(), which supposedly takes the vmalloc faults for the current task, but who does it for the remote?

Hm, no one. This might be an issue, thanks for noticing!

Quoted 27 lines
Quoted 2 lines

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

Could you point me to those patches?

Even though it might work fine now, we might get issues if we backport remote kcov to older kernels.

Quoted 19 lines

So no #PF from writing to t->kcov_area then, under the assumption that the vmalloc_user() is the only allocation site.

But then there's hardware watchpoints, if someone goes and sets a data watchpoint in the kcov_area we're screwed. Nothing actively prevents that from happening. Then again, the same is currently true for much of current :/

Also, I think you need __always_inline on kaslr_offset()

And, unrelated to this patch in specific, I suppose I'm going to have to extend objtool to look for data that is used from noinstr, to make sure we exclude it from inspection and stuff, like that kaslr offset crud for example.

Anyway, yes, it appears you're lucky (for having Joerg remove vmalloc faults) and this mostly should work as is.

Now I am a bit worried that, even though we're lucky today, with what Andrey said about e.g. kcov_remote faults, it'll be hard to ensure we won't break in future. The exact set of conditions that mean we're lucky today may change and we have no way of checking this.

I'll try to roll a v2 based on the "if (_RET_IP_ in noinstr section) return;" and whitelist in objtool approach. Unless you see something very wrong with that. And I do hope we'll get compiler attributes eventually.

Thanks,
-- Marco

#5 lore raw
Quoted 2 lines · Marco Elver

Sadly no. 'noinstr' implies 'notrace', but also places the function in the .noinstr.text section for the purpose of objtool checking.

Not only the compile time checking, but also for purpose of runtime exclusion for things like kprobes and hw-breakpoints.

#6 lore raw
In reply to Andrey Konovalov
Quoted 4 lines · Andrey Konovalov
Quoted 2 lines

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

Could you point me to those patches?

git log 7f0a002b5a21302d9f4b29ba83c96cd433ff3769...d8626138009ba58ae2c22356966c2edaa1f1c3b5

#7 lore raw
In reply to Andrey Konovalov
Quoted 9 lines · Andrey Konovalov
Quoted 4 lines · Peter Zijlstra

That whole kcov_remote stuff confuses me.

KCOV_ENABLE() has kcov_fault_in_area(), which supposedly takes the vmalloc faults for the current task, but who does it for the remote?

Hm, no one. This might be an issue, thanks for noticing!

Quoted 7 lines
Quoted 2 lines

Now, luckily Joerg went and ripped out the vmalloc faults, let me check where those patches are... w00t, they're upstream in this merge window.

Could you point me to those patches?

Even though it might work fine now, we might get issues if we backport remote kcov to older kernels.

Thinking more about this; you can't actually pre-fault for kernel threads, as kernel threads will run with the mm of whatever regular thread ran before them, and who knows if they have that vmalloc region faulted in.

So Joerg's patches are pretty much the only way to guarantee remotes will not his the vmalloc fault.