kcsan: Support detecting a subset of missing memory barriers
Subsystems (13)
Marco Elver
maintainerv2 cover kcsan: Support detecting a subset of missing memory barriers
Detection of some missing memory barriers has been on the KCSAN feature wishlist for some time: this series adds support for modeling a subset of weak memory as defined by the LKMM, which enables detection of a subset of data races due to missing memory barriers.
KCSAN's approach to detecting missing memory barriers is based on modeling access reordering. Each memory access for which a watchpoint is set up, is also selected for simulated reordering within the scope of its function (at most 1 in-flight access).
We are limited to modeling the effects of "buffering" (delaying the access), since the runtime cannot "prefetch" accesses. Once an access has been selected for reordering, it is checked along every other access until the end of the function scope. If an appropriate memory barrier is encountered, the access will no longer be considered for reordering.
When the result of a memory operation should be ordered by a barrier, KCSAN can then detect data races where the conflict only occurs as a result of a missing barrier due to reordering accesses.
Some more details and an example are captured in the updated <Documentation/dev-tools/kcsan.rst>.
Some light fuzzing with the feature also resulted in a discussion [1] around an issue which appears to be allowed, but unlikely in practice.
[1] https://lkml.kernel.org/r/[email protected]
The first half of the series are core KCSAN changes, documentation updates, and test changes. The second half adds instrumentation to barriers, atomics, bitops, along with enabling barrier instrumentation for some currently uninstrumented subsystems. The last two patches are objtool changes to add the usual entries to the uaccess whitelist, but also instruct objtool to remove memory barrier instrumentation from noinstr code (on x86).
---
v2:
* Rewrite objtool patch after rebase to v5.16-rc1.
* Note the reason in documentation that address or control dependencies do not require special handling.
* Rename kcsan_atomic_release() to kcsan_atomic_builtin_memorder() to avoid confusion.
* Define kcsan_noinstr as noinline if we rely on objtool nop'ing out calls, to avoid things like LTO inlining it.
v1: https://lore.kernel.org/all/[email protected]/
Marco Elver (23):
kcsan: Refactor reading of instrumented memory
kcsan: Remove redundant zero-initialization of globals
kcsan: Avoid checking scoped accesses from nested contexts
kcsan: Add core support for a subset of weak memory modeling
kcsan: Add core memory barrier instrumentation functions kcsan, kbuild: Add option for barrier instrumentation only
kcsan: Call scoped accesses reordered in reports
kcsan: Show location access was reordered to
kcsan: Document modeling of weak memory
kcsan: test: Match reordered or normal accesses
kcsan: test: Add test cases for memory barrier instrumentation
kcsan: Ignore GCC 11+ warnings about TSan runtime support
kcsan: selftest: Add test case to check memory barrier instrumentation locking/barriers, kcsan: Add instrumentation for barriers locking/barriers, kcsan: Support generic instrumentation locking/atomics, kcsan: Add instrumentation for barriers asm-generic/bitops, kcsan: Add instrumentation for barriers x86/barriers, kcsan: Use generic instrumentation for non-smp barriers x86/qspinlock, kcsan: Instrument barrier of pv_queued_spin_unlock() mm, kcsan: Enable barrier instrumentation
sched, kcsan: Enable memory barrier instrumentation objtool, kcsan: Add memory barrier instrumentation to whitelist objtool, kcsan: Remove memory barrier instrumentation from noinstr
Documentation/dev-tools/kcsan.rst | 76 +++- arch/x86/include/asm/barrier.h | 10 +- arch/x86/include/asm/qspinlock.h | 1 + include/asm-generic/barrier.h | 54 ++- .../asm-generic/bitops/instrumented-atomic.h | 3 + .../asm-generic/bitops/instrumented-lock.h | 3 + include/linux/atomic/atomic-instrumented.h | 135 +++++- include/linux/kcsan-checks.h | 51 ++- include/linux/kcsan.h | 11 +- include/linux/sched.h | 3 + include/linux/spinlock.h | 2 +- init/init_task.c | 9 +- kernel/kcsan/Makefile | 2 + kernel/kcsan/core.c | 326 +++++++++++--- kernel/kcsan/kcsan_test.c | 416 ++++++++++++++++-- kernel/kcsan/report.c | 51 ++- kernel/kcsan/selftest.c | 141 ++++++ kernel/sched/Makefile | 7 +- lib/Kconfig.kcsan | 16 + mm/Makefile | 2 + scripts/Makefile.kcsan | 15 +- scripts/Makefile.lib | 5 + scripts/atomic/gen-atomic-instrumented.sh | 41 +- tools/objtool/check.c | 41 +- tools/objtool/include/objtool/elf.h | 2 +- 25 files changed, 1248 insertions(+), 175 deletions(-)
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 1/23 kcsan: Refactor reading of instrumented memory
Factor out the switch statement reading instrumented memory into a helper read_instrumented_memory().
No functional change.
--- kernel/kcsan/core.c | 51 +++++++++++++++------------------------------ 1 file changed, 17 insertions(+), 34 deletions(-)
kernel/kcsan/core.c +17 -34
@@ -325,6 +325,21 @@ static void delay_access(int type)udelay (delay); } +/* +* Reads the instrumented memory for value change detection; value change +* detection is currently done for accesses up to a size of 8 bytes. +*/ +static __always_inline u64 read_instrumented_memory(const volatile void *ptr, size_t size) +{ +switch (size) { +case 1 :return READ_ONCE (*(const u8 *)ptr); +case 2 :return READ_ONCE (*(const u16 *)ptr); +case 4 :return READ_ONCE (*(const u32 *)ptr); +case 8 :return READ_ONCE (*(const u64 *)ptr); +default :return 0 ;/* Ignore; we do not diff the values. */ + } +} +void kcsan_save_irqtrace (struct task_struct *task) {#ifdef CONFIG_TRACE_IRQFLAGS
@@ -482,23 +497,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned* Read the current value, to later check and infer a race if the data * was modified via a non-instrumented access, e.g. from a device. */ - old =0 ; -switch (size) { -case 1 : - old =READ_ONCE (*(const u8 *)ptr); -break ; -case 2 : - old =READ_ONCE (*(const u16 *)ptr); -break ; -case 4 : - old =READ_ONCE (*(const u32 *)ptr); -break ; -case 8 : - old =READ_ONCE (*(const u64 *)ptr); -break ; -default : -break ;/* ignore; we do not diff the values */ - } + old =read_instrumented_memory (ptr, size);/* * Delay this thread, to increase probability of observing a racy
@@ -511,23 +510,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned* racy access. */ access_mask = ctx->access_mask; - new =0 ; -switch (size) { -case 1 : - new =READ_ONCE (*(const u8 *)ptr); -break ; -case 2 : - new =READ_ONCE (*(const u16 *)ptr); -break ; -case 4 : - new =READ_ONCE (*(const u32 *)ptr); -break ; -case 8 : - new =READ_ONCE (*(const u64 *)ptr); -break ; -default : -break ;/* ignore; we do not diff the values */ - } + new =read_instrumented_memory (ptr, size); diff = old ^ new;if (access_mask)
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH v2 01/23] kcsan: Refactor reading of instrumented memory
Quoted 6 lines · Marco Elver
Factor out the switch statement reading instrumented memory into a helper
read_instrumented_memory().No functional change.
Signed-off-by Marco Elver <[email protected]>
Nice cleanup!
Mark.
Quoted 83 lines
--- kernel/kcsan/core.c | 51 +++++++++++++++------------------------------ 1 file changed, 17 insertions(+), 34 deletions(-)kernel/kcsan/core.c +17
-34@@ -325,6 +325,21 @@ static void delay_access(int type)udelay (delay); } +/* +* Reads the instrumented memory for value change detection; value change +* detection is currently done for accesses up to a size of 8 bytes. +*/ +static __always_inline u64 read_instrumented_memory(const volatile void *ptr, size_t size) +{ +switch (size) { +case 1 :return READ_ONCE (*(const u8 *)ptr); +case 2 :return READ_ONCE (*(const u16 *)ptr); +case 4 :return READ_ONCE (*(const u32 *)ptr); +case 8 :return READ_ONCE (*(const u64 *)ptr); +default :return 0 ;/* Ignore; we do not diff the values. */ + } +} +void kcsan_save_irqtrace (struct task_struct *task) {#ifdef CONFIG_TRACE_IRQFLAGS @@ -482,23 +497,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned* Read the current value, to later check and infer a race if the data * was modified via a non-instrumented access, e.g. from a device. */ - old =0 ; -switch (size) { -case 1 : - old =READ_ONCE (*(const u8 *)ptr); -break ; -case 2 : - old =READ_ONCE (*(const u16 *)ptr); -break ; -case 4 : - old =READ_ONCE (*(const u32 *)ptr); -break ; -case 8 : - old =READ_ONCE (*(const u64 *)ptr); -break ; -default : -break ;/* ignore; we do not diff the values */ - } + old =read_instrumented_memory (ptr, size);/* * Delay this thread, to increase probability of observing a racy @@ -511,23 +510,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned* racy access. */ access_mask = ctx->access_mask; - new =0 ; -switch (size) { -case 1 : - new =READ_ONCE (*(const u8 *)ptr); -break ; -case 2 : - new =READ_ONCE (*(const u16 *)ptr); -break ; -case 4 : - new =READ_ONCE (*(const u32 *)ptr); -break ; -case 8 : - new =READ_ONCE (*(const u64 *)ptr); -break ; -default : -break ;/* ignore; we do not diff the values */ - } + new =read_instrumented_memory (ptr, size); diff = old ^ new;if (access_mask)Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 2/23 kcsan: Remove redundant zero-initialization of globals
In reply to Marco ElverThey are implicitly zero-initialized, remove explicit initialization. It keeps the upcoming additions to kcsan_ctx consistent with the rest.
No functional change intended.
--- init/init_task.c | 9 +-------- kernel/kcsan/core.c | 5 ----- 2 files changed, 1 insertion(+), 13 deletions(-)
init/init_task.c +1 -8
@@ -181,14 +181,7 @@ struct task_struct init_task .kasan_depth = 1,#endif #ifdef CONFIG_KCSAN - .kcsan_ctx = { - .disable_count = 0, - .atomic_next = 0, - .atomic_nest_count = 0, - .in_flat_atomic = false, - .access_mask = 0, - .scoped_accesses = {LIST_POISON1, NULL}, - }, + .kcsan_ctx = { .scoped_accesses = {LIST_POISON1, NULL} }, #endif #ifdef CONFIG_TRACE_IRQFLAGS .softirqs_enabled = 1,
kernel/kcsan/core.c +0 -5
@@ -44,11 +44,6 @@ bool kcsan_enabled;/* Per-CPU kcsan_ctx for interrupts */ static DEFINE_PER_CPU(struct kcsan_ctx, kcsan_cpu_ctx) = { - .disable_count =0 , - .atomic_next =0 , - .atomic_nest_count =0 , - .in_flat_atomic = false, - .access_mask =0 , .scoped_accesses = {LIST_POISON1 ,NULL }, };
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH v2 02/23] kcsan: Remove redundant zero-initialization of globals
Quoted 18 lines · Marco Elver
They are implicitly zero-initialized, remove explicit initialization. It keeps the upcoming additions to
kcsan_ctxconsistent with the rest.No functional change intended.
Signed-off-by Marco Elver <[email protected]>--- init/init_task.c | 9 +-------- kernel/kcsan/core.c | 5 ----- 2 files changed, 1 insertion(+), 13 deletions(-)init/init_task.c +0
-0@@ -181,14 +181,7 @@ struct task_struct init_task .kasan_depth = 1,#endif
init/init_task.c struct task_struct init_task · line 184
Quoted 10 lines
init/init_task.c +1
-8#ifdef CONFIG_KCSAN - .kcsan_ctx = { - .disable_count = 0, - .atomic_next = 0, - .atomic_nest_count = 0, - .in_flat_atomic = false, - .access_mask = 0, - .scoped_accesses = {LIST_POISON1, NULL}, - }, + .kcsan_ctx = { .scoped_accesses = {LIST_POISON1, NULL} },
I'd recommend leaving this as:
.kcsan_ctx = {
.scoped_accesses = {LIST_POISON1, NULL},
},... which'd be consistent with the DEFINE_PER_CPU() usage below, and makes it easier to add fields to in future without needing structural changes.
Either way:
Quoted 22 lines
init/init_task.c +0
-0#endif #ifdef CONFIG_TRACE_IRQFLAGS .softirqs_enabled = 1,kernel/kcsan/core.c +0
-5@@ -44,11 +44,6 @@ bool kcsan_enabled;/* Per-CPU kcsan_ctx for interrupts */ static DEFINE_PER_CPU(struct kcsan_ctx, kcsan_cpu_ctx) = { - .disable_count =0 , - .atomic_next =0 , - .atomic_nest_count =0 , - .in_flat_atomic = false, - .access_mask =0 , .scoped_accesses = {LIST_POISON1 ,NULL }, };Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 3/23 kcsan: Avoid checking scoped accesses from nested contexts
In reply to Marco ElverAvoid checking scoped accesses from nested contexts (such as nested interrupts or in scheduler code) which share the same kcsan_ctx.
This is to avoid detecting false positive races of accesses in the same thread with currently scoped accesses: consider setting up a watchpoint for a non-scoped (normal) access that also "conflicts" with a current scoped access. In a nested interrupt (or in the scheduler), which shares the same kcsan_ctx, we cannot check scoped accesses set up in the parent context -- simply ignore them in this case.
With the introduction of kcsan_ctx::disable_scoped, we can also clean up kcsan_check_scoped_accesses()'s recursion guard, and do not need to modify the list's prev pointer.
--- include/linux/kcsan.h | 1 + kernel/kcsan/core.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/kcsan.h +1 -0
@@ -21,6 +21,7 @@*/ struct kcsan_ctx {int disable_count;/* disable counter */ +int disable_scoped;/* disable scoped access counter */ int atomic_next;/* number of following atomic ops */ /*
kernel/kcsan/core.c +15 -3
@@ -204,15 +204,17 @@ check_access(const volatile void *ptr, size_t size, int type, unsigned long ip); static noinline void kcsan_check_scoped_accesses(void) {struct kcsan_ctx *ctx =get_ctx (); -struct list_head *prev_save = ctx->scoped_accesses.prev;struct kcsan_scoped_access *scoped_access; - ctx->scoped_accesses.prev =NULL ;/* Avoid recursion. */ +if (ctx->disable_scoped) +return ; + + ctx->disable_scoped++;list_for_each_entry (scoped_access, &ctx->scoped_accesses, list) {check_access (scoped_access->ptr, scoped_access->size, scoped_access->type, scoped_access->ip); } - ctx->scoped_accesses.prev = prev_save; + ctx->disable_scoped--; }/* Rules for generic atomic accesses. Called from fast-path. */
@@ -465,6 +467,15 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned goto out; } +/* +* Avoid races of scoped accesses from nested interrupts (or scheduler). +* Assume setting up a watchpoint for a non-scoped (normal) access that +* also conflicts with a current scoped access. In a nested interrupt, +* which shares the context, it would check a conflicting scoped access. +* To avoid, disable scoped access checking. +*/ + ctx->disable_scoped++; +/* * Save and restore the IRQ state trace touched by KCSAN, since KCSAN's * runtime is entered for every memory access, and potentially useful
@@ -578,6 +589,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsignedif (!kcsan_interrupt_watcher)local_irq_restore (irq_flags);kcsan_restore_irqtrace (current); + ctx->disable_scoped--; out:user_access_restore (ua_flags); }
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH v2 03/23] kcsan: Avoid checking scoped accesses from nested contexts
Hi Marco,
Quoted 4 lines · Marco Elver
Avoid checking scoped accesses from nested contexts (such as nested interrupts or in scheduler code) which share the same
kcsan_ctx.This is to avoid detecting false positive races of accesses in the same
Could you provide an example for a false positive?
I think we do want to detect the following race:
static int v = SOME_VALUE; // a percpu variable.
static int other_v = ... ;
void foo(..)
{
int tmp;
int other_tmp;
preempt_disable();
{
ASSERT_EXCLUSIVE_ACCESSS_SCOPED(v);
tmp = v;
other_tmp = other_v; // int_handler() may run here
v = tmp + 2;
}
preempt_enabled();
}
void int_handler() // an interrupt handler
{
v++;
}, if I understand correctly, we can detect this currently, but with this patch, we cannot detect this if the interrupt happens while we're doing the check for "other_tmp = other_v;", right? Of course, running tests multiple times may eventually catch this, but I just want to understand what's this patch for, thanks!
Regards,
Boqun
Quoted 80 lines
thread with currently scoped accesses: consider setting up a watchpoint for a non-scoped (normal) access that also "conflicts" with a current scoped access. In a nested interrupt (or in the scheduler), which shares the same
kcsan_ctx, we cannot check scoped accesses set up in the parent context -- simply ignore them in this case.With the introduction of
kcsan_ctx::disable_scoped, we can also clean up kcsan_check_scoped_accesses()'s recursion guard, and do not need to modify the list's prev pointer.Signed-off-by Marco Elver <[email protected]>--- include/linux/kcsan.h | 1 + kernel/kcsan/core.c | 18 +++++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-)include/linux/kcsan.h +1
-0@@ -21,6 +21,7 @@*/ struct kcsan_ctx {int disable_count;/* disable counter */ +int disable_scoped;/* disable scoped access counter */ int atomic_next;/* number of following atomic ops */ /* kernel/kcsan/core.c +15
-3@@ -204,15 +204,17 @@ check_access(const volatile void *ptr, size_t size, int type, unsigned long ip); static noinline void kcsan_check_scoped_accesses(void) {struct kcsan_ctx *ctx =get_ctx (); -struct list_head *prev_save = ctx->scoped_accesses.prev;struct kcsan_scoped_access *scoped_access; - ctx->scoped_accesses.prev =NULL ;/* Avoid recursion. */ +if (ctx->disable_scoped) +return ; + + ctx->disable_scoped++;list_for_each_entry (scoped_access, &ctx->scoped_accesses, list) {check_access (scoped_access->ptr, scoped_access->size, scoped_access->type, scoped_access->ip); } - ctx->scoped_accesses.prev = prev_save; + ctx->disable_scoped--; }/* Rules for generic atomic accesses. Called from fast-path. */ @@ -465,6 +467,15 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned goto out; } +/* +* Avoid races of scoped accesses from nested interrupts (or scheduler). +* Assume setting up a watchpoint for a non-scoped (normal) access that +* also conflicts with a current scoped access. In a nested interrupt, +* which shares the context, it would check a conflicting scoped access. +* To avoid, disable scoped access checking. +*/ + ctx->disable_scoped++; +/* * Save and restore the IRQ state trace touched by KCSAN, since KCSAN's * runtime is entered for every memory access, and potentially useful @@ -578,6 +589,7 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsignedif (!kcsan_interrupt_watcher)local_irq_restore (irq_flags);kcsan_restore_irqtrace (current); + ctx->disable_scoped--; out:user_access_restore (ua_flags); }Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerRe: [PATCH v2 03/23] kcsan: Avoid checking scoped accesses from nested contexts
Quoted 37 lines · Boqun Feng
Hi Marco,
Quoted 4 lines · Marco Elver
Avoid checking scoped accesses from nested contexts (such as nested interrupts or in scheduler code) which share the same
kcsan_ctx.This is to avoid detecting false positive races of accesses in the same
Could you provide an example for a false positive?
I think we do want to detect the following race:
static int v = SOME_VALUE; // a percpu variable. static int other_v = ... ; void foo(..) { int tmp; int other_tmp; preempt_disable(); { ASSERT_EXCLUSIVE_ACCESSS_SCOPED(v); tmp = v; other_tmp = other_v; // int_handler() may run here v = tmp + 2; } preempt_enabled(); } void int_handler() // an interrupt handler { v++; }
Quoted 5 lines
, if I understand correctly, we can detect this currently, but with this patch, we cannot detect this if the interrupt happens while we're doing the check for "
other_tmp=other_v;", right? Of course, running tests multiple times may eventually catch this, but I just want to understand what's this patch for, thanks!
The above will still be detected. Task and interrupt contexts in this case are distinct, i.e. kcsan_ctx differ (see get_ctx()).
But there are rare cases where kcsan_ctx is shared, such as nested interrupts (NMI?), or when entering scheduler code -- which currently has a KCSAN_SANITIZE := n, but I occasionally test it, which is how I found this problem. The problem occurs frequently when enabling KCSAN in kernel/sched and placing a random ASSERT_EXCLUSIVE_ACCESS_SCOPED() in task context, or just enable "weak memory modeling" without this fix. You also need CONFIG_PREEMPT=y + CONFIG_KCSAN_INTERRUPT_WATCHER=y.
The emphasis here really is on _shared kcsan_ctx_, which is not too common. As noted in the commit description, we need to "[...] setting up a watchpoint for a non-scoped (normal) access that also "conflicts" with a current scoped access."
Consider this:
static int v;
int foo(..)
{
ASSERT_EXCLUSIVE_ACCESS_SCOPED(v);
v++; // preempted during watchpoint for 'v++'
}Here we set up a scoped_access to be checked for v. Then on v++, a watchpoint is set up for the normal access. While the watchpoint is set up, the task is preempted and upon entering scheduler code, we're still in_task() and 'current' is still the same, thus get_ctx() returns a kcsan_ctx where the scoped_accesses list is non-empty containing the scoped access for foo()'s ASSERT_EXCLUSIVE.
That means, when instrumenting scheduler code or any other code called by scheduler code or nested interrupts (anything where get_ctx() still returns the same as parent context), it'd now perform checks based on the parent context's scoped access, and because the parent context also has a watchpoint set up on the variable that conflicts with the scoped access we'd report a nonsensical race.
This case is also possible:
static int v;
static int x;
int foo(..)
{
ASSERT_EXCLUSIVE_ACCESS_SCOPED(v);
x++; // preempted during watchpoint for 'v' after checking x++
}Here, all we need is for the scoped access to be checked after x++, end up with a watchpoint for it, then enter scheduler code, which then checked 'v', sees the conflicting watchpoint, and reports a nonsensical race again.
By disallowing scoped access checking for a kcsan_ctx, we simply make sure that in such nested contexts where kcsan_ctx is shared, none of these nonsensical races would be detected nor reported.
Hopefully that clarifies what this is about.
Thanks,
-- Marco
Re: [PATCH v2 03/23] kcsan: Avoid checking scoped accesses from nested contexts
Quoted 38 lines · Marco Elver
Quoted 37 lines · Boqun Feng
Hi Marco,
Quoted 4 lines · Marco Elver
Avoid checking scoped accesses from nested contexts (such as nested interrupts or in scheduler code) which share the same
kcsan_ctx.This is to avoid detecting false positive races of accesses in the same
Could you provide an example for a false positive?
I think we do want to detect the following race:
static int v = SOME_VALUE; // a percpu variable. static int other_v = ... ; void foo(..) { int tmp; int other_tmp; preempt_disable(); { ASSERT_EXCLUSIVE_ACCESSS_SCOPED(v); tmp = v; other_tmp = other_v; // int_handler() may run here v = tmp + 2; } preempt_enabled(); } void int_handler() // an interrupt handler { v++; }
Quoted 9 lines
Quoted 5 lines
, if I understand correctly, we can detect this currently, but with this patch, we cannot detect this if the interrupt happens while we're doing the check for "
other_tmp=other_v;", right? Of course, running tests multiple times may eventually catch this, but I just want to understand what's this patch for, thanks!The above will still be detected. Task and interrupt contexts in this case are distinct, i.e.
kcsan_ctxdiffer (seeget_ctx()).
Ok, I was missing that.
Quoted 8 lines
But there are rare cases where
kcsan_ctxis shared, such as nested interrupts (NMI?), or when entering scheduler code -- which currently has aKCSAN_SANITIZE:= n, but I occasionally test it, which is how I found this problem. The problem occurs frequently when enabling KCSAN in kernel/sched and placing a randomASSERT_EXCLUSIVE_ACCESS_SCOPED()in task context, or just enable "weak memory modeling" without this fix. You also needCONFIG_PREEMPT=y+CONFIG_KCSAN_INTERRUPT_WATCHER=y.
Thanks for the background, it's now more clear that the problem is triggered ;-)
Quoted 21 lines
The emphasis here really is on
_sharedkcsan_ctx_, which is not too common. As noted in the commit description, we need to "[...] setting up a watchpoint for a non-scoped (normal) access that also "conflicts" with a current scoped access."Consider this:
static int v; int foo(..) { ASSERT_EXCLUSIVE_ACCESS_SCOPED(v); v++; // preempted during watchpoint for 'v++' }Here we set up a
scoped_accessto be checked for v. Then on v++, a watchpoint is set up for the normal access. While the watchpoint is set up, the task is preempted and upon entering scheduler code, we're stillin_task()and 'current' is still the same, thusget_ctx()returns akcsan_ctxwhere thescoped_accesseslist is non-empty containing the scoped access for foo()'sASSERT_EXCLUSIVE.
Quoted 7 lines
That means, when instrumenting scheduler code or any other code called by scheduler code or nested interrupts (anything where
get_ctx()still returns the same as parent context), it'd now perform checks based on the parent context's scoped access, and because the parent context also has a watchpoint set up on the variable that conflicts with the scoped access we'd report a nonsensical race.
Agreed.
Quoted 10 lines
This case is also possible:
static int v; static int x; int foo(..) { ASSERT_EXCLUSIVE_ACCESS_SCOPED(v); x++; // preempted during watchpoint for 'v' after checking x++ }
Quoted 5 lines
Here, all we need is for the scoped access to be checked after x++, end up with a watchpoint for it, then enter scheduler code, which then checked 'v', sees the conflicting watchpoint, and reports a nonsensical race again.
Just to be clear, in both examples, the assumption is that 'v' is a variable that scheduler code doesn't access, right? Because if scheduler code does access 'v', then it's a problem that KCSAN should report. Yes, I don't know any variable that scheduler exports, just to make sure here.
Quoted 6 lines
By disallowing scoped access checking for a
kcsan_ctx, we simply make sure that in such nested contexts wherekcsan_ctxis shared, none of these nonsensical races would be detected nor reported.Hopefully that clarifies what this is about.
Make sense to me, thanks.
Regards,
Boqun
Quoted 2 lines
Thanks,
-- Marco
Marco Elver
maintainerRe: [PATCH v2 03/23] kcsan: Avoid checking scoped accesses from nested contexts
On Mon, 29 Nov 2021 at 15:27, Boqun Feng <[email protected]> wrote:
[...]
Quoted 16 lines
Quoted 10 lines
This case is also possible:
static int v; static int x; int foo(..) { ASSERT_EXCLUSIVE_ACCESS_SCOPED(v); x++; // preempted during watchpoint for 'v' after checking x++ }Quoted 5 lines
Here, all we need is for the scoped access to be checked after x++, end up with a watchpoint for it, then enter scheduler code, which then checked 'v', sees the conflicting watchpoint, and reports a nonsensical race again.
Quoted 5 lines
Just to be clear, in both examples, the assumption is that 'v' is a variable that scheduler code doesn't access, right? Because if scheduler code does access 'v', then it's a problem that KCSAN should report. Yes, I don't know any variable that scheduler exports, just to make sure here.
Right. We might miss such cases where an ASSERT_EXCLUSIVE*_SCOPED() could have pointed out a legitimate race with a nested context that share ctx, like in scheduler, where the only time to detect it is if some state change later in the scope makes a concurrent access possible from that point in the scope. I'm willing to bet that there's an extremely small chance we'll ever encounter such a case (famous last words ;-)), i.e. the initial check_access() in kcsan_begin_scoped_access() wouldn't detect it nor would the problem manifest as a regular data race.
Thanks,
-- Marco
Marco Elver
maintainerPATCH v2 4/23 kcsan: Add core support for a subset of weak memory modeling
In reply to Marco ElverAdd support for modeling a subset of weak memory, which will enable detection of a subset of data races due to missing memory barriers.
KCSAN's approach to detecting missing memory barriers is based on modeling access reordering, and enabled if CONFIG_KCSAN_WEAK_MEMORY=y, which depends on CONFIG_KCSAN_STRICT=y. The feature can be enabled or disabled at boot and runtime via the kcsan.weak_memory boot parameter.
Each memory access for which a watchpoint is set up, is also selected for simulated reordering within the scope of its function (at most 1 in-flight access).
We are limited to modeling the effects of "buffering" (delaying the access), since the runtime cannot "prefetch" accesses (therefore no acquire modeling). Once an access has been selected for reordering, it is checked along every other access until the end of the function scope. If an appropriate memory barrier is encountered, the access will no longer be considered for reordering.
When the result of a memory operation should be ordered by a barrier, KCSAN can then detect data races where the conflict only occurs as a result of a missing barrier due to reordering accesses.
---
v2:
* Define kcsan_noinstr as noinline if we rely on objtool nop'ing out calls, to avoid things like LTO inlining it.
--- include/linux/kcsan-checks.h | 10 +- include/linux/kcsan.h | 10 +- include/linux/sched.h | 3 + kernel/kcsan/core.c | 222 ++++++++++++++++++++++++++++++++--- lib/Kconfig.kcsan | 16 +++ scripts/Makefile.kcsan | 9 +- 6 files changed, 251 insertions(+), 19 deletions(-)
include/linux/kcsan-checks.h +9 -1
@@ -99,7 +99,15 @@ void kcsan_set_access_mask(unsigned long mask);/* Scoped access information. */ struct kcsan_scoped_access { -struct list_head list; +union { +struct list_head list;/* scoped_accesses list */ +/* +* Not an entry in scoped_accesses list; stack depth from where +* the access was initialized. +*/ +int stack_depth; + }; +/* Access information. */ const volatile void *ptr;size_t size;
include/linux/kcsan.h +9 -1
@@ -49,8 +49,16 @@ struct kcsan_ctx {*/ unsigned long access_mask; -/* List of scoped accesses. */ +/* List of scoped accesses; likely to be empty. */ struct list_head scoped_accesses; + +#ifdef CONFIG_KCSAN_WEAK_MEMORY +/* +* Scoped access for modeling access reordering to detect missing memory +* barriers; only keep 1 to keep fast-path complexity manageable. +*/ +struct kcsan_scoped_access reorder_access; +#endif };/**
include/linux/sched.h +3 -0
@@ -1339,6 +1339,9 @@ struct task_struct {#ifdef CONFIG_TRACE_IRQFLAGS struct irqtrace_events kcsan_save_irqtrace;#endif +#ifdef CONFIG_KCSAN_WEAK_MEMORY +int kcsan_stack_depth; +#endif #endif #if IS_ENABLED (CONFIG_KUNIT )
kernel/kcsan/core.c +207 -15
348 lines not on this page · raw message
lib/Kconfig.kcsan +16 -0
@@ -191,6 +191,22 @@ config KCSAN_STRICT closely aligns with the rules defined by the Linux-kernel memory consistency model (LKMM). +config KCSAN_WEAK_MEMORY + bool "Enable weak memory modeling to detect missing memory barriers" + default y + depends on KCSAN_STRICT + help + Enable support for modeling a subset of weak memory, which allows + detecting a subset of data races due to missing memory barriers. + + Depends on KCSAN_STRICT, because the options strenghtening certain + plain accesses by default (depending on !KCSAN_STRICT) reduce the + ability to detect any data races invoving reordered accesses, in + particular reordered writes. + + Weak memory modeling relies on additional instrumentation and may + affect performance. + config KCSAN_REPORT_VALUE_CHANGE_ONLY bool "Only report races where watcher observed a data value change" default y
scripts/Makefile.kcsan +7 -2
@@ -9,7 +9,12 @@ endif # Keep most options here optional, to allow enabling more compilers if absence # of some options does not break KCSAN nor causes false positive reports. -export CFLAGS_KCSAN := -fsanitize=thread \ - $(call cc-option,$(call cc-param,tsan-instrument-func-entry-exit=0) -fno-optimize-sibling-calls) \ +kcsan-cflags := -fsanitize=thread -fno-optimize-sibling-calls \ $(call cc-option,$(call cc-param,tsan-compound-read-before-write=1),$(call cc-option,$(call cc-param,tsan-instrument-read-before-write=1))) \ $(call cc-param,tsan-distinguish-volatile=1) + +ifndef CONFIG_KCSAN_WEAK_MEMORY +kcsan-cflags += $(call cc-option,$(call cc-param,tsan-instrument-func-entry-exit=0)) +endif + +export CFLAGS_KCSAN := $(kcsan-cflags)
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 5/23 kcsan: Add core memory barrier instrumentation functions
In reply to Marco ElverAdd the core memory barrier instrumentation functions. These invalidate the current in-flight reordered access based on the rules for the respective barrier types and in-flight access type.
---
v2:
* Rename kcsan_atomic_release() to kcsan_atomic_builtin_memorder() to avoid confusion.
--- include/linux/kcsan-checks.h | 41 ++++++++++++++++++++++++++++++++++-- kernel/kcsan/core.c | 36 +++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 2 deletions(-)
include/linux/kcsan-checks.h +39 -2
@@ -36,6 +36,26 @@*/ void __kcsan_check_access (const volatile void *ptr,size_t size,int type); +/** +* __kcsan_mb - full memory barrier instrumentation +*/ +void __kcsan_mb (void ); + +/** +* __kcsan_wmb - write memory barrier instrumentation +*/ +void __kcsan_wmb (void ); + +/** +* __kcsan_rmb - read memory barrier instrumentation +*/ +void __kcsan_rmb (void ); + +/** +* __kcsan_release - release barrier instrumentation +*/ +void __kcsan_release (void ); +/** * kcsan_disable_current - disable KCSAN for the current context *
@@ -159,6 +179,10 @@ void kcsan_end_scoped_access(struct kcsan_scoped_access *sa);static inline void __kcsan_check_access (const volatile void *ptr,size_t size,int type) { } +static inline void __kcsan_mb (void ) { } +static inline void __kcsan_wmb (void ) { } +static inline void __kcsan_rmb (void ) { } +static inline void __kcsan_release (void ) { }static inline void kcsan_disable_current (void ) { }static inline void kcsan_enable_current (void ) { }static inline void kcsan_enable_current_nowarn (void ) { }
@@ -191,12 +215,25 @@ static inline void kcsan_end_scoped_access(struct kcsan_scoped_access *sa) { }*/ #define __kcsan_disable_current kcsan_disable_current#define __kcsan_enable_current kcsan_enable_current_nowarn -#else +#else /* __SANITIZE_THREAD__ */ static inline void kcsan_check_access (const volatile void *ptr,size_t size,int type) { }static inline void __kcsan_enable_current (void ) { }static inline void __kcsan_disable_current (void ) { } -#endif +#endif /* __SANITIZE_THREAD__ */ + +#if defined(CONFIG_KCSAN_WEAK_MEMORY ) && \ + (defined(__SANITIZE_THREAD__) || defined(__KCSAN_INSTRUMENT_BARRIERS__)) +#define kcsan_mb __kcsan_mb +#define kcsan_wmb __kcsan_wmb +#define kcsan_rmb __kcsan_rmb +#define kcsan_release __kcsan_release +#else /* CONFIG_KCSAN_WEAK_MEMORY && (__SANITIZE_THREAD__ || __KCSAN_INSTRUMENT_BARRIERS__) */ +static inline void kcsan_mb (void ) { } +static inline void kcsan_wmb (void ) { } +static inline void kcsan_rmb (void ) { } +static inline void kcsan_release (void ) { } +#endif /* CONFIG_KCSAN_WEAK_MEMORY && (__SANITIZE_THREAD__ || __KCSAN_INSTRUMENT_BARRIERS__) */ /** * __kcsan_check_read - check regular read access for races
kernel/kcsan/core.c +36 -0
@@ -955,6 +955,28 @@ void __kcsan_check_access(const volatile void *ptr, size_t size, int type) }EXPORT_SYMBOL (__kcsan_check_access); +#define DEFINE_MEMORY_BARRIER (name, order_before_cond) \ + kcsan_noinstr void __kcsan_##name(void) \ + { \ + struct kcsan_scoped_access *sa; \ + if (within_noinstr(_RET_IP_)) \ + return; \ + instrumentation_begin(); \ + sa = get_reorder_access(get_ctx()); \ + if (!sa) \ + goto out; \ + if (order_before_cond) \ + sa->size = 0; \ + out: \ + instrumentation_end(); \ + } \ + EXPORT_SYMBOL(__kcsan_##name) + +DEFINE_MEMORY_BARRIER (mb, true); +DEFINE_MEMORY_BARRIER (wmb, sa->type & (KCSAN_ACCESS_WRITE |KCSAN_ACCESS_COMPOUND )); +DEFINE_MEMORY_BARRIER (rmb, !(sa->type &KCSAN_ACCESS_WRITE ) || (sa->type &KCSAN_ACCESS_COMPOUND )); +DEFINE_MEMORY_BARRIER (release, true); +/* * KCSAN uses the same instrumentation that is emitted by supported compilers * for ThreadSanitizer (TSAN).
@@ -1143,10 +1165,19 @@ EXPORT_SYMBOL(__tsan_init);* functions, whose job is to also execute the operation itself. */ +static __always_inline void kcsan_atomic_builtin_memorder(int memorder) +{ +if (memorder == __ATOMIC_RELEASE || + memorder == __ATOMIC_SEQ_CST || + memorder == __ATOMIC_ACQ_REL) +__kcsan_release (); +} +#define DEFINE_TSAN_ATOMIC_LOAD_STORE (bits) \ u##bits __tsan_atomic##bits##_load(const u##bits *ptr, int memorder); \ u##bits __tsan_atomic##bits##_load(const u##bits *ptr, int memorder) \ { \ + kcsan_atomic_builtin_memorder(memorder); \ if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \ check_access(ptr, bits / BITS_PER_BYTE, KCSAN_ACCESS_ATOMIC, _RET_IP_); \ } \
@@ -1156,6 +1187,7 @@ EXPORT_SYMBOL(__tsan_init); void __tsan_atomic##bits##_store(u##bits *ptr, u##bits v, int memorder); \ void __tsan_atomic##bits##_store(u##bits *ptr, u##bits v, int memorder) \ { \ + kcsan_atomic_builtin_memorder(memorder); \ if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \ check_access(ptr, bits / BITS_PER_BYTE, \ KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ATOMIC, _RET_IP_); \
@@ -1168,6 +1200,7 @@ EXPORT_SYMBOL(__tsan_init); u##bits __tsan_atomic##bits##_##op(u##bits *ptr, u##bits v, int memorder); \ u##bits __tsan_atomic##bits##_##op(u##bits *ptr, u##bits v, int memorder) \ { \ + kcsan_atomic_builtin_memorder(memorder); \ if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \ check_access(ptr, bits / BITS_PER_BYTE, \ KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
@@ -1200,6 +1233,7 @@ EXPORT_SYMBOL(__tsan_init); int __tsan_atomic##bits##_compare_exchange_##strength(u##bits *ptr, u##bits *exp, \ u##bits val, int mo, int fail_mo) \ { \ + kcsan_atomic_builtin_memorder(mo); \ if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \ check_access(ptr, bits / BITS_PER_BYTE, \ KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
@@ -1215,6 +1249,7 @@ EXPORT_SYMBOL(__tsan_init); u##bits __tsan_atomic##bits##_compare_exchange_val(u##bits *ptr, u##bits exp, u##bits val, \ int mo, int fail_mo) \ { \ + kcsan_atomic_builtin_memorder(mo); \ if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \ check_access(ptr, bits / BITS_PER_BYTE, \ KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
@@ -1246,6 +1281,7 @@ DEFINE_TSAN_ATOMIC_OPS(64);void __tsan_atomic_thread_fence (int memorder);void __tsan_atomic_thread_fence (int memorder) { +kcsan_atomic_builtin_memorder (memorder);__atomic_thread_fence (memorder); }EXPORT_SYMBOL (__tsan_atomic_thread_fence);
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 6/23 kcsan, kbuild: Add option for barrier instrumentation only
In reply to Marco ElverSource files that disable KCSAN via KCSAN_SANITIZE := n, remove all instrumentation, including explicit barrier instrumentation. With instrumentation for memory barriers, in few places it is required to enable just the explicit instrumentation for memory barriers to avoid false positives.
Providing the Makefile variable KCSAN_INSTRUMENT_BARRIERS_obj.o or KCSAN_INSTRUMENT_BARRIERS (for all files) set to 'y' only enables the explicit barrier instrumentation.
--- scripts/Makefile.lib | 5 +++++ 1 file changed, 5 insertions(+)
scripts/Makefile.lib +5 -0
@@ -182,6 +182,11 @@ ifeq ($(CONFIG_KCSAN),y) _c_flags += $(if $(patsubst n%,, \ $(KCSAN_SANITIZE_$(basetarget).o)$(KCSAN_SANITIZE)y), \ $(CFLAGS_KCSAN)) +# Some uninstrumented files provide implied barriers required to avoid false +# positives: set KCSAN_INSTRUMENT_BARRIERS for barrier instrumentation only. +_c_flags += $(if $(patsubst n%,, \ + $(KCSAN_INSTRUMENT_BARRIERS_$(basetarget).o)$(KCSAN_INSTRUMENT_BARRIERS)n), \ + -D__KCSAN_INSTRUMENT_BARRIERS__) endif # $(srctree)/$(src) for including checkin headers from generated source files
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 7/23 kcsan: Call scoped accesses reordered in reports
In reply to Marco ElverThe scoping of an access simply denotes the scope in which it may be reordered. However, in reports, it'll be less confusing to say the access is "reordered". This is more accurate when the race occurred.
--- kernel/kcsan/kcsan_test.c | 4 ++-- kernel/kcsan/report.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-)
kernel/kcsan/kcsan_test.c +2 -2
@@ -213,9 +213,9 @@ static bool report_matches(const struct expect_report *r)const bool is_atomic = (ty &KCSAN_ACCESS_ATOMIC );const bool is_scoped = (ty &KCSAN_ACCESS_SCOPED );const char *const access_type_aux = - (is_atomic && is_scoped) ?" (marked, scoped)" + (is_atomic && is_scoped) ?" (marked, reordered)" : (is_atomic ?" (marked)" - : (is_scoped ?" (scoped)" :"" )); + : (is_scoped ?" (reordered)" :"" ));if (i ==1 ) {/* Access 2 */
kernel/kcsan/report.c +8 -8
@@ -215,9 +215,9 @@ static const char *get_access_type(int type)if (type &KCSAN_ACCESS_ASSERT ) {if (type &KCSAN_ACCESS_SCOPED ) {if (type &KCSAN_ACCESS_WRITE ) -return "assert no accesses (scoped)" ; +return "assert no accesses (reordered)" ;else -return "assert no writes (scoped)" ; +return "assert no writes (reordered)" ; }else {if (type &KCSAN_ACCESS_WRITE )return "assert no accesses" ;
@@ -240,17 +240,17 @@ static const char *get_access_type(int type)case KCSAN_ACCESS_COMPOUND |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ATOMIC :return "read-write (marked)" ;case KCSAN_ACCESS_SCOPED : -return "read (scoped)" ; +return "read (reordered)" ;case KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_ATOMIC : -return "read (marked, scoped)" ; +return "read (marked, reordered)" ;case KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_WRITE : -return "write (scoped)" ; +return "write (reordered)" ;case KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ATOMIC : -return "write (marked, scoped)" ; +return "write (marked, reordered)" ;case KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_COMPOUND |KCSAN_ACCESS_WRITE : -return "read-write (scoped)" ; +return "read-write (reordered)" ;case KCSAN_ACCESS_SCOPED |KCSAN_ACCESS_COMPOUND |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ATOMIC : -return "read-write (marked, scoped)" ; +return "read-write (marked, reordered)" ;default :BUG (); }
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 8/23 kcsan: Show location access was reordered to
In reply to Marco ElverAlso show the location the access was reordered to. An example report:
| ================================================================== | BUG: KCSAN: data-race in test_kernel_wrong_memorder / test_kernel_wrong_memorder | | read-write to 0xffffffffc01e61a8 of 8 bytes by task 2311 on cpu 5: | test_kernel_wrong_memorder+0x57/0x90 | access_thread+0x99/0xe0 | kthread+0x2ba/0x2f0 | ret_from_fork+0x22/0x30 | | read-write (reordered) to 0xffffffffc01e61a8 of 8 bytes by task 2310 on cpu 7: | test_kernel_wrong_memorder+0x57/0x90 | access_thread+0x99/0xe0 | kthread+0x2ba/0x2f0 | ret_from_fork+0x22/0x30 | | | +-> reordered to: test_kernel_wrong_memorder+0x80/0x90 | | Reported by Kernel Concurrency Sanitizer on: | CPU: 7 PID: 2310 Comm: access_thread Not tainted 5.14.0-rc1+ #18 | Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 | ==================================================================
--- kernel/kcsan/report.c | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)
kernel/kcsan/report.c +23 -12
@@ -308,10 +308,12 @@ static int get_stack_skipnr(const unsigned long stack_entries[], int num_entries/* * Skips to the first entry that matches the function of @ip, and then replaces -* that entry with @ip, returning the entries to skip. +* that entry with @ip, returning the entries to skip with @replaced containing +* the replaced entry. */ static int -replace_stack_entry (unsigned long stack_entries[],int num_entries,unsigned long ip) +replace_stack_entry (unsigned long stack_entries[],int num_entries,unsigned long ip, +unsigned long *replaced) {unsigned long symbolsize, offset;unsigned long target_func;
@@ -330,6 +332,7 @@ replace_stack_entry(unsigned long stack_entries[], int num_entries, unsigned lon func -= offset;if (func == target_func) { + *replaced = stack_entries[skip]; stack_entries[skip] = ip;return skip; }
@@ -342,9 +345,10 @@ replace_stack_entry(unsigned long stack_entries[], int num_entries, unsigned lon }static int -sanitize_stack_entries (unsigned long stack_entries[],int num_entries,unsigned long ip) +sanitize_stack_entries (unsigned long stack_entries[],int num_entries,unsigned long ip, +unsigned long *replaced) { -return ip ?replace_stack_entry (stack_entries, num_entries, ip) : +return ip ?replace_stack_entry (stack_entries, num_entries, ip, replaced) :get_stack_skipnr (stack_entries, num_entries); }
@@ -360,6 +364,14 @@ static int sym_strcmp(void *addr1, void *addr2)return strncmp (buf1, buf2,sizeof (buf1)); } +static void +print_stack_trace (unsigned long stack_entries[],int num_entries,unsigned long reordered_to) +{ +stack_trace_print (stack_entries, num_entries,0 ); +if (reordered_to) +pr_err (" |\n +-> reordered to: %pS\n" , (void *)reordered_to); +} +static void print_verbose_info (struct task_struct *task) {if (!task)
@@ -378,10 +390,12 @@ static void print_report(enum kcsan_value_change value_change,struct other_info *other_info, u64 old, u64 new, u64 mask) { + unsigned long reordered_to = 0;unsigned long stack_entries[NUM_STACK_ENTRIES ] = {0 };int num_stack_entries =stack_trace_save (stack_entries,NUM_STACK_ENTRIES ,1 ); -int skipnr =sanitize_stack_entries (stack_entries, num_stack_entries, ai->ip); +int skipnr =sanitize_stack_entries (stack_entries, num_stack_entries, ai->ip, &reordered_to);unsigned long this_frame = stack_entries[skipnr]; +unsigned long other_reordered_to =0 ;unsigned long other_frame =0 ;int other_skipnr =0 ;/* silence uninit warnings */
@@ -394,7 +408,7 @@ static void print_report(enum kcsan_value_change value_change,if (other_info) { other_skipnr =sanitize_stack_entries (other_info->stack_entries, other_info->num_stack_entries, - other_info->ai.ip); + other_info->ai.ip, &other_reordered_to); other_frame = other_info->stack_entries[other_skipnr];/* @value_change is only known for the other thread */
@@ -434,10 +448,9 @@ static void print_report(enum kcsan_value_change value_change, other_info->ai.cpu_id);/* Print the other thread's stack trace. */ -stack_trace_print (other_info->stack_entries + other_skipnr, +print_stack_trace (other_info->stack_entries + other_skipnr, other_info->num_stack_entries - other_skipnr, -0 ); - + other_reordered_to);if (IS_ENABLED (CONFIG_KCSAN_VERBOSE ))print_verbose_info (other_info->task);
@@ -451,9 +464,7 @@ static void print_report(enum kcsan_value_change value_change, get_thread_desc(ai->task_pid), ai->cpu_id); }/* Print stack trace of this thread. */ -stack_trace_print (stack_entries + skipnr, num_stack_entries - skipnr, -0 ); - +print_stack_trace (stack_entries + skipnr, num_stack_entries - skipnr, reordered_to);if (IS_ENABLED (CONFIG_KCSAN_VERBOSE ))print_verbose_info (current);
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 9/23 kcsan: Document modeling of weak memory
In reply to Marco ElverDocument how KCSAN models a subset of weak memory and the subset of missing memory barriers it can detect as a result.
---
v2:
* Note the reason that address or control dependencies do not require special handling.
--- Documentation/dev-tools/kcsan.rst | 76 +++++++++++++++++++++++++------ 1 file changed, 63 insertions(+), 13 deletions(-)
Documentation/dev-tools/kcsan.rst +63 -13
@@ -204,17 +204,17 @@ Ultimately this allows to determine the possible executions of concurrent code, and if that code is free from data races. KCSAN is aware of *marked atomic operations* (``READ_ONCE``, ``WRITE_ONCE``, -``atomic_*``, etc.), but is oblivious of any ordering guarantees and simply -assumes that memory barriers are placed correctly. In other words, KCSAN -assumes that as long as a plain access is not observed to race with another -conflicting access, memory operations are correctly ordered. - -This means that KCSAN will not report *potential* data races due to missing -memory ordering. Developers should therefore carefully consider the required -memory ordering requirements that remain unchecked. If, however, missing -memory ordering (that is observable with a particular compiler and -architecture) leads to an observable data race (e.g. entering a critical -section erroneously), KCSAN would report the resulting data race. +``atomic_*``, etc.), and a subset of ordering guarantees implied by memory +barriers. With ``CONFIG_KCSAN_WEAK_MEMORY=y``, KCSAN models load or store +buffering, and can detect missing ``smp_mb()``, ``smp_wmb()``, ``smp_rmb()``, +``smp_store_release()``, and all ``atomic_*`` operations with equivalent +implied barriers. + +Note, KCSAN will not report all data races due to missing memory ordering, +specifically where a memory barrier would be required to prohibit subsequent +memory operation from reordering before the barrier. Developers should +therefore carefully consider the required memory ordering requirements that +remain unchecked. Race Detection Beyond Data Races --------------------------------
@@ -268,6 +268,56 @@ marked operations, if all accesses to a variable that is accessed concurrently are properly marked, KCSAN will never trigger a watchpoint and therefore never report the accesses. +Modeling Weak Memory +~~~~~~~~~~~~~~~~~~~~ + +KCSAN's approach to detecting data races due to missing memory barriers is +based on modeling access reordering (with ``CONFIG_KCSAN_WEAK_MEMORY=y``). +Each plain memory access for which a watchpoint is set up, is also selected for +simulated reordering within the scope of its function (at most 1 in-flight +access). + +Once an access has been selected for reordering, it is checked along every +other access until the end of the function scope. If an appropriate memory +barrier is encountered, the access will no longer be considered for simulated +reordering. + +When the result of a memory operation should be ordered by a barrier, KCSAN can +then detect data races where the conflict only occurs as a result of a missing +barrier. Consider the example:: + + int x, flag; + void T1(void) + { + x = 1; // data race! + WRITE_ONCE(flag, 1); // correct: smp_store_release(&flag, 1) + } + void T2(void) + { + while (!READ_ONCE(flag)); // correct: smp_load_acquire(&flag) + ... = x; // data race! + } + +When weak memory modeling is enabled, KCSAN can consider ``x`` in ``T1`` for +simulated reordering. After the write of ``flag``, ``x`` is again checked for +concurrent accesses: because ``T2`` is able to proceed after the write of +``flag``, a data race is detected. With the correct barriers in place, ``x`` +would not be considered for reordering after the proper release of ``flag``, +and no data race would be detected. + +Deliberate trade-offs in complexity but also practical limitations mean only a +subset of data races due to missing memory barriers can be detected. With +currently available compiler support, the implementation is limited to modeling +the effects of "buffering" (delaying accesses), since the runtime cannot +"prefetch" accesses. Also recall that watchpoints are only set up for plain +accesses, and the only access type for which KCSAN simulates reordering. This +means reordering of marked accesses is not modeled. + +A consequence of the above is that acquire operations do not require barrier +instrumentation (no prefetching). Furthermore, marked accesses introducing +address or control dependencies do not require special handling (the marked +access cannot be reordered, later dependent accesses cannot be prefetched). + Key Properties ~~~~~~~~~~~~~~
@@ -290,8 +340,8 @@ Key Properties 4. **Detects Racy Writes from Devices:** Due to checking data values upon setting up watchpoints, racy writes from devices can also be detected. -5. **Memory Ordering:** KCSAN is *not* explicitly aware of the LKMM's ordering - rules; this may result in missed data races (false negatives). +5. **Memory Ordering:** KCSAN is aware of only a subset of LKMM ordering rules; + this may result in missed data races (false negatives). 6. **Analysis Accuracy:** For observed executions, due to using a sampling strategy, the analysis is *unsound* (false negatives possible), but aims to
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 10/23 kcsan: test: Match reordered or normal accesses
In reply to Marco ElverDue to reordering accesses with weak memory modeling, any access can now appear as "(reordered)".
Match any permutation of accesses if CONFIG_KCSAN_WEAK_MEMORY=y, so that we effectively match an access if it is denoted "(reordered)" or not.
--- kernel/kcsan/kcsan_test.c | 92 +++++++++++++++++++++++++++------------ 1 file changed, 63 insertions(+), 29 deletions(-)
kernel/kcsan/kcsan_test.c +63 -29
@@ -151,7 +151,7 @@ struct expect_report {/* Check observed report matches information in @r. */ __no_kcsan -static bool report_matches(const struct expect_report *r) +static bool __report_matches(const struct expect_report *r) {const bool is_assert = (r->access[0 ].type | r->access[1 ].type) &KCSAN_ACCESS_ASSERT ;bool ret = false;
@@ -253,6 +253,40 @@ static bool report_matches(const struct expect_report *r)return ret; } +static __always_inline const struct expect_report * +__report_set_scoped (struct expect_report *r,int accesses) +{ +BUILD_BUG_ON (accesses >3 ); + +if (accesses &1 ) + r->access[0 ].type |=KCSAN_ACCESS_SCOPED ; +else + r->access[0 ].type &= ~KCSAN_ACCESS_SCOPED ; + +if (accesses &2 ) + r->access[1 ].type |=KCSAN_ACCESS_SCOPED ; +else + r->access[1 ].type &= ~KCSAN_ACCESS_SCOPED ; + +return r; +} + +__no_kcsan +static bool report_matches_any_reordered(struct expect_report *r) +{ +return __report_matches (__report_set_scoped (r,0 )) || +__report_matches (__report_set_scoped (r,1 )) || +__report_matches (__report_set_scoped (r,2 )) || +__report_matches (__report_set_scoped (r,3 )); +} + +#ifdef CONFIG_KCSAN_WEAK_MEMORY +/* Due to reordering accesses, any access may appear as "(reordered)". */ +#define report_matches report_matches_any_reordered +#else +#define report_matches __report_matches +#endif +/* ===== Test kernels ===== */ static long test_sink;
@@ -438,13 +472,13 @@ static noinline void test_kernel_xor_1bit(void)__no_kcsan static void test_basic(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_write, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_read, &test_var,sizeof (test_var),0 }, }, }; -static const struct expect_report never = { +struct expect_report never = { .access = { { test_kernel_read, &test_var,sizeof (test_var),0 }, { test_kernel_read, &test_var,sizeof (test_var),0 },
@@ -469,14 +503,14 @@ static void test_basic(struct kunit *test)__no_kcsan static void test_concurrent_races(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = {/* NULL will match any address. */ { test_kernel_rmw_array,NULL ,0 ,__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) }, { test_kernel_rmw_array,NULL ,0 ,__KCSAN_ACCESS_RW (0 ) }, }, }; -static const struct expect_report never = { +struct expect_report never = { .access = { { test_kernel_rmw_array,NULL ,0 ,0 }, { test_kernel_rmw_array,NULL ,0 ,0 },
@@ -498,13 +532,13 @@ static void test_concurrent_races(struct kunit *test)__no_kcsan static void test_novalue_change(struct kunit *test) { -const struct expect_report expect_rw = { +struct expect_report expect_rw = { .access = { { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_read, &test_var,sizeof (test_var),0 }, }, }; -const struct expect_report expect_ww = { +struct expect_report expect_ww = { .access = { { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE },
@@ -530,13 +564,13 @@ static void test_novalue_change(struct kunit *test)__no_kcsan static void test_novalue_change_exception(struct kunit *test) { -const struct expect_report expect_rw = { +struct expect_report expect_rw = { .access = { { test_kernel_write_nochange_rcu, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_read, &test_var,sizeof (test_var),0 }, }, }; -const struct expect_report expect_ww = { +struct expect_report expect_ww = { .access = { { test_kernel_write_nochange_rcu, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_write_nochange_rcu, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE },
@@ -556,7 +590,7 @@ static void test_novalue_change_exception(struct kunit *test)__no_kcsan static void test_unknown_origin(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_read, &test_var,sizeof (test_var),0 }, {NULL },
@@ -578,7 +612,7 @@ static void test_unknown_origin(struct kunit *test)__no_kcsan static void test_write_write_assume_atomic(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_write, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, { test_kernel_write, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE },
@@ -604,7 +638,7 @@ static void test_write_write_assume_atomic(struct kunit *test)__no_kcsan static void test_write_write_struct(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE }, { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE },
@@ -626,7 +660,7 @@ static void test_write_write_struct(struct kunit *test)__no_kcsan static void test_write_write_struct_part(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE }, { test_kernel_write_struct_part, &test_struct.val[3 ],sizeof (test_struct.val[3 ]),KCSAN_ACCESS_WRITE },
@@ -658,7 +692,7 @@ static void test_read_atomic_write_atomic(struct kunit *test)__no_kcsan static void test_read_plain_atomic_write(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_read, &test_var,sizeof (test_var),0 }, { test_kernel_write_atomic, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE |KCSAN_ACCESS_ATOMIC },
@@ -679,7 +713,7 @@ static void test_read_plain_atomic_write(struct kunit *test)__no_kcsan static void test_read_plain_atomic_rmw(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_read, &test_var,sizeof (test_var),0 }, { test_kernel_atomic_rmw, &test_var,sizeof (test_var),
@@ -701,13 +735,13 @@ static void test_read_plain_atomic_rmw(struct kunit *test)__no_kcsan static void test_zero_size_access(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE }, { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE }, }, }; -const struct expect_report never = { +struct expect_report never = { .access = { { test_kernel_write_struct, &test_struct,sizeof (test_struct),KCSAN_ACCESS_WRITE }, { test_kernel_read_struct_zero_size, &test_struct.val[3 ],0 ,0 },
@@ -741,7 +775,7 @@ static void test_data_race(struct kunit *test)__no_kcsan static void test_assert_exclusive_writer(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_assert_writer, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT }, { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE },
@@ -759,7 +793,7 @@ static void test_assert_exclusive_writer(struct kunit *test)__no_kcsan static void test_assert_exclusive_access(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_assert_access, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE }, { test_kernel_read, &test_var,sizeof (test_var),0 },
@@ -777,19 +811,19 @@ static void test_assert_exclusive_access(struct kunit *test)__no_kcsan static void test_assert_exclusive_access_writer(struct kunit *test) { -const struct expect_report expect_access_writer = { +struct expect_report expect_access_writer = { .access = { { test_kernel_assert_access, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE }, { test_kernel_assert_writer, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT }, }, }; -const struct expect_report expect_access_access = { +struct expect_report expect_access_access = { .access = { { test_kernel_assert_access, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE }, { test_kernel_assert_access, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE }, }, }; -const struct expect_report never = { +struct expect_report never = { .access = { { test_kernel_assert_writer, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT }, { test_kernel_assert_writer, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT },
@@ -813,7 +847,7 @@ static void test_assert_exclusive_access_writer(struct kunit *test)__no_kcsan static void test_assert_exclusive_bits_change(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_assert_bits_change, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT }, { test_kernel_change_bits, &test_var,sizeof (test_var),
@@ -844,13 +878,13 @@ static void test_assert_exclusive_bits_nochange(struct kunit *test)__no_kcsan static void test_assert_exclusive_writer_scoped(struct kunit *test) { -const struct expect_report expect_start = { +struct expect_report expect_start = { .access = { { test_kernel_assert_writer_scoped, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_SCOPED }, { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE }, }, }; -const struct expect_report expect_inscope = { +struct expect_report expect_inscope = { .access = { { test_enter_scope, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_SCOPED }, { test_kernel_write_nochange, &test_var,sizeof (test_var),KCSAN_ACCESS_WRITE },
@@ -871,16 +905,16 @@ static void test_assert_exclusive_writer_scoped(struct kunit *test)__no_kcsan static void test_assert_exclusive_access_scoped(struct kunit *test) { -const struct expect_report expect_start1 = { +struct expect_report expect_start1 = { .access = { { test_kernel_assert_access_scoped, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_SCOPED }, { test_kernel_read, &test_var,sizeof (test_var),0 }, }, }; -const struct expect_report expect_start2 = { +struct expect_report expect_start2 = { .access = { expect_start1.access[0 ], expect_start1.access[0 ] }, }; -const struct expect_report expect_inscope = { +struct expect_report expect_inscope = { .access = { { test_enter_scope, &test_var,sizeof (test_var),KCSAN_ACCESS_ASSERT |KCSAN_ACCESS_WRITE |KCSAN_ACCESS_SCOPED }, { test_kernel_read, &test_var,sizeof (test_var),0 },
@@ -985,7 +1019,7 @@ static void test_atomic_builtins(struct kunit *test)__no_kcsan static void test_1bit_value_change(struct kunit *test) { -const struct expect_report expect = { +struct expect_report expect = { .access = { { test_kernel_read, &test_var,sizeof (test_var),0 }, { test_kernel_xor_1bit, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) },
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 11/23 kcsan: test: Add test cases for memory barrier instrumentation
In reply to Marco ElverAdds test cases to check that memory barriers are instrumented correctly, and detection of missing memory barriers is working as intended if CONFIG_KCSAN_STRICT=y.
--- kernel/kcsan/kcsan_test.c | 320 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 320 insertions(+)
kernel/kcsan/kcsan_test.c +320 -0
@@ -16,9 +16,12 @@#define pr_fmt (fmt) "kcsan_test: " fmt#include <kunit/test.h> +#include <linux/atomic.h> +#include <linux/bitops.h> #include <linux/jiffies.h> #include <linux/kcsan-checks.h> #include <linux/kernel.h> +#include <linux/mutex.h> #include <linux/sched.h> #include <linux/seqlock.h> #include <linux/spinlock.h>
@@ -305,6 +308,16 @@ static DEFINE_SEQLOCK(test_seqlock);__no_kcsan static noinline void sink_value(long v) { WRITE_ONCE(test_sink, v); } +/* +* Generates a delay and some accesses that enter the runtime but do not produce +* data races. +*/ +static noinline void test_delay(int iter) +{ +while (iter--) +sink_value (READ_ONCE (test_sink)); +} + static noinline void test_kernel_read(void) { sink_value(test_var); } static noinline void test_kernel_write(void)
@@ -466,8 +479,220 @@ static noinline void test_kernel_xor_1bit(void)kcsan_nestable_atomic_end (); } +#define TEST_KERNEL_LOCKED (name, acquire, release) \ + static noinline void test_kernel_##name(void) \ + { \ + long *flag = &test_struct.val[0]; \ + long v = 0; \ + if (!(acquire)) \ + return; \ + while (v++ < 100) { \ + test_var++; \ + barrier(); \ + } \ + release; \ + test_delay(10); \ + } + +TEST_KERNEL_LOCKED (with_memorder, +cmpxchg_acquire (flag,0 ,1 ) ==0 , +smp_store_release (flag,0 )); +TEST_KERNEL_LOCKED (wrong_memorder, +cmpxchg_relaxed (flag,0 ,1 ) ==0 , +WRITE_ONCE (*flag,0 )); +TEST_KERNEL_LOCKED (atomic_builtin_with_memorder, +__atomic_compare_exchange_n (flag, &v,1 ,0 , __ATOMIC_ACQUIRE, __ATOMIC_RELAXED), +__atomic_store_n (flag,0 , __ATOMIC_RELEASE)); +TEST_KERNEL_LOCKED (atomic_builtin_wrong_memorder, +__atomic_compare_exchange_n (flag, &v,1 ,0 , __ATOMIC_RELAXED, __ATOMIC_RELAXED), +__atomic_store_n (flag,0 , __ATOMIC_RELAXED)); +/* ===== Test cases ===== */ +/* +* Tests that various barriers have the expected effect on internal state. Not +* exhaustive on atomic_t operations. Unlike the selftest, also checks for +* too-strict barrier instrumentation; these can be tolerated, because it does +* not cause false positives, but at least we should be aware of such cases. +*/ +__no_kcsan +static void test_barrier_nothreads(struct kunit *test) +{ +#ifdef CONFIG_KCSAN_WEAK_MEMORY +struct kcsan_scoped_access *reorder_access = ¤t->kcsan_ctx.reorder_access; +#else +struct kcsan_scoped_access *reorder_access =NULL ; +#endif +arch_spinlock_t arch_spinlock = __ARCH_SPIN_LOCK_UNLOCKED; +DEFINE_SPINLOCK (spinlock); +DEFINE_MUTEX (mutex); +atomic_t dummy; + +KCSAN_TEST_REQUIRES (test, reorder_access !=NULL ); +KCSAN_TEST_REQUIRES (test,IS_ENABLED (CONFIG_SMP )); + +#define __KCSAN_EXPECT_BARRIER (access_type, barrier, order_before, name) \ + do { \ + reorder_access->type = (access_type) | KCSAN_ACCESS_SCOPED; \ + reorder_access->size = sizeof(test_var); \ + barrier; \ + KUNIT_EXPECT_EQ_MSG(test, reorder_access->size, \ + order_before ? 0 : sizeof(test_var), \ + "improperly instrumented type=(" #access_type "): " name); \ + } while (0) +#define KCSAN_EXPECT_READ_BARRIER (b, o) __KCSAN_EXPECT_BARRIER(0, b, o, #b) +#define KCSAN_EXPECT_WRITE_BARRIER (b, o) __KCSAN_EXPECT_BARRIER(KCSAN_ACCESS_WRITE, b, o, #b) +#define KCSAN_EXPECT_RW_BARRIER (b, o) __KCSAN_EXPECT_BARRIER(KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE, b, o, #b) + +/* Force creating a valid entry in reorder_access first. */ + test_var =0 ; +while (test_var++ <1000000 && reorder_access->size !=sizeof (test_var)) +__kcsan_check_read (&test_var,sizeof (test_var)); +KUNIT_ASSERT_EQ (test, reorder_access->size,sizeof (test_var)); + +kcsan_nestable_atomic_begin ();/* No watchpoints in called functions. */ + +KCSAN_EXPECT_READ_BARRIER (mb (), true); +KCSAN_EXPECT_READ_BARRIER (wmb (), false); +KCSAN_EXPECT_READ_BARRIER (rmb (), true); +KCSAN_EXPECT_READ_BARRIER (smp_mb (), true); +KCSAN_EXPECT_READ_BARRIER (smp_wmb (), false); +KCSAN_EXPECT_READ_BARRIER (smp_rmb (), true); +KCSAN_EXPECT_READ_BARRIER (dma_wmb (), false); +KCSAN_EXPECT_READ_BARRIER (dma_rmb (), true); +KCSAN_EXPECT_READ_BARRIER (smp_mb__before_atomic (), true); +KCSAN_EXPECT_READ_BARRIER (smp_mb__after_atomic (), true); +KCSAN_EXPECT_READ_BARRIER (smp_mb__after_spinlock (), true); +KCSAN_EXPECT_READ_BARRIER (smp_store_mb (test_var,0 ), true); +KCSAN_EXPECT_READ_BARRIER (smp_load_acquire (&test_var), false); +KCSAN_EXPECT_READ_BARRIER (smp_store_release (&test_var,0 ), true); +KCSAN_EXPECT_READ_BARRIER (xchg (&test_var,0 ), true); +KCSAN_EXPECT_READ_BARRIER (xchg_release (&test_var,0 ), true); +KCSAN_EXPECT_READ_BARRIER (xchg_relaxed (&test_var,0 ), false); +KCSAN_EXPECT_READ_BARRIER (cmpxchg (&test_var,0 ,0 ), true); +KCSAN_EXPECT_READ_BARRIER (cmpxchg_release (&test_var,0 ,0 ), true); +KCSAN_EXPECT_READ_BARRIER (cmpxchg_relaxed (&test_var,0 ,0 ), false); +KCSAN_EXPECT_READ_BARRIER (atomic_read (&dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_read_acquire (&dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_set (&dummy,0 ), false); +KCSAN_EXPECT_READ_BARRIER (atomic_set_release (&dummy,0 ), true); +KCSAN_EXPECT_READ_BARRIER (atomic_add (1 , &dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_add_return (1 , &dummy), true); +KCSAN_EXPECT_READ_BARRIER (atomic_add_return_acquire (1 , &dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_add_return_release (1 , &dummy), true); +KCSAN_EXPECT_READ_BARRIER (atomic_add_return_relaxed (1 , &dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_fetch_add (1 , &dummy), true); +KCSAN_EXPECT_READ_BARRIER (atomic_fetch_add_acquire (1 , &dummy), false); +KCSAN_EXPECT_READ_BARRIER (atomic_fetch_add_release (1 , &dummy), true); +KCSAN_EXPECT_READ_BARRIER (atomic_fetch_add_relaxed (1 , &dummy), false); +KCSAN_EXPECT_READ_BARRIER (test_and_set_bit (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (test_and_clear_bit (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (test_and_change_bit (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (__clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (clear_bit_unlock_is_negative_byte (0 , &test_var), true); +KCSAN_EXPECT_READ_BARRIER (arch_spin_lock (&arch_spinlock), false); +KCSAN_EXPECT_READ_BARRIER (arch_spin_unlock (&arch_spinlock), true); +KCSAN_EXPECT_READ_BARRIER (spin_lock (&spinlock), false); +KCSAN_EXPECT_READ_BARRIER (spin_unlock (&spinlock), true); +KCSAN_EXPECT_READ_BARRIER (mutex_lock (&mutex), false); +KCSAN_EXPECT_READ_BARRIER (mutex_unlock (&mutex), true); + +KCSAN_EXPECT_WRITE_BARRIER (mb (), true); +KCSAN_EXPECT_WRITE_BARRIER (wmb (), true); +KCSAN_EXPECT_WRITE_BARRIER (rmb (), false); +KCSAN_EXPECT_WRITE_BARRIER (smp_mb (), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_wmb (), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_rmb (), false); +KCSAN_EXPECT_WRITE_BARRIER (dma_wmb (), true); +KCSAN_EXPECT_WRITE_BARRIER (dma_rmb (), false); +KCSAN_EXPECT_WRITE_BARRIER (smp_mb__before_atomic (), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_mb__after_atomic (), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_mb__after_spinlock (), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_store_mb (test_var,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (smp_load_acquire (&test_var), false); +KCSAN_EXPECT_WRITE_BARRIER (smp_store_release (&test_var,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (xchg (&test_var,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (xchg_release (&test_var,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (xchg_relaxed (&test_var,0 ), false); +KCSAN_EXPECT_WRITE_BARRIER (cmpxchg (&test_var,0 ,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (cmpxchg_release (&test_var,0 ,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (cmpxchg_relaxed (&test_var,0 ,0 ), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_read (&dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_read_acquire (&dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_set (&dummy,0 ), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_set_release (&dummy,0 ), true); +KCSAN_EXPECT_WRITE_BARRIER (atomic_add (1 , &dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_add_return (1 , &dummy), true); +KCSAN_EXPECT_WRITE_BARRIER (atomic_add_return_acquire (1 , &dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_add_return_release (1 , &dummy), true); +KCSAN_EXPECT_WRITE_BARRIER (atomic_add_return_relaxed (1 , &dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_fetch_add (1 , &dummy), true); +KCSAN_EXPECT_WRITE_BARRIER (atomic_fetch_add_acquire (1 , &dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (atomic_fetch_add_release (1 , &dummy), true); +KCSAN_EXPECT_WRITE_BARRIER (atomic_fetch_add_relaxed (1 , &dummy), false); +KCSAN_EXPECT_WRITE_BARRIER (test_and_set_bit (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (test_and_clear_bit (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (test_and_change_bit (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (__clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (clear_bit_unlock_is_negative_byte (0 , &test_var), true); +KCSAN_EXPECT_WRITE_BARRIER (arch_spin_lock (&arch_spinlock), false); +KCSAN_EXPECT_WRITE_BARRIER (arch_spin_unlock (&arch_spinlock), true); +KCSAN_EXPECT_WRITE_BARRIER (spin_lock (&spinlock), false); +KCSAN_EXPECT_WRITE_BARRIER (spin_unlock (&spinlock), true); +KCSAN_EXPECT_WRITE_BARRIER (mutex_lock (&mutex), false); +KCSAN_EXPECT_WRITE_BARRIER (mutex_unlock (&mutex), true); + +KCSAN_EXPECT_RW_BARRIER (mb (), true); +KCSAN_EXPECT_RW_BARRIER (wmb (), true); +KCSAN_EXPECT_RW_BARRIER (rmb (), true); +KCSAN_EXPECT_RW_BARRIER (smp_mb (), true); +KCSAN_EXPECT_RW_BARRIER (smp_wmb (), true); +KCSAN_EXPECT_RW_BARRIER (smp_rmb (), true); +KCSAN_EXPECT_RW_BARRIER (dma_wmb (), true); +KCSAN_EXPECT_RW_BARRIER (dma_rmb (), true); +KCSAN_EXPECT_RW_BARRIER (smp_mb__before_atomic (), true); +KCSAN_EXPECT_RW_BARRIER (smp_mb__after_atomic (), true); +KCSAN_EXPECT_RW_BARRIER (smp_mb__after_spinlock (), true); +KCSAN_EXPECT_RW_BARRIER (smp_store_mb (test_var,0 ), true); +KCSAN_EXPECT_RW_BARRIER (smp_load_acquire (&test_var), false); +KCSAN_EXPECT_RW_BARRIER (smp_store_release (&test_var,0 ), true); +KCSAN_EXPECT_RW_BARRIER (xchg (&test_var,0 ), true); +KCSAN_EXPECT_RW_BARRIER (xchg_release (&test_var,0 ), true); +KCSAN_EXPECT_RW_BARRIER (xchg_relaxed (&test_var,0 ), false); +KCSAN_EXPECT_RW_BARRIER (cmpxchg (&test_var,0 ,0 ), true); +KCSAN_EXPECT_RW_BARRIER (cmpxchg_release (&test_var,0 ,0 ), true); +KCSAN_EXPECT_RW_BARRIER (cmpxchg_relaxed (&test_var,0 ,0 ), false); +KCSAN_EXPECT_RW_BARRIER (atomic_read (&dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_read_acquire (&dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_set (&dummy,0 ), false); +KCSAN_EXPECT_RW_BARRIER (atomic_set_release (&dummy,0 ), true); +KCSAN_EXPECT_RW_BARRIER (atomic_add (1 , &dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_add_return (1 , &dummy), true); +KCSAN_EXPECT_RW_BARRIER (atomic_add_return_acquire (1 , &dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_add_return_release (1 , &dummy), true); +KCSAN_EXPECT_RW_BARRIER (atomic_add_return_relaxed (1 , &dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_fetch_add (1 , &dummy), true); +KCSAN_EXPECT_RW_BARRIER (atomic_fetch_add_acquire (1 , &dummy), false); +KCSAN_EXPECT_RW_BARRIER (atomic_fetch_add_release (1 , &dummy), true); +KCSAN_EXPECT_RW_BARRIER (atomic_fetch_add_relaxed (1 , &dummy), false); +KCSAN_EXPECT_RW_BARRIER (test_and_set_bit (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (test_and_clear_bit (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (test_and_change_bit (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (__clear_bit_unlock (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (clear_bit_unlock_is_negative_byte (0 , &test_var), true); +KCSAN_EXPECT_RW_BARRIER (arch_spin_lock (&arch_spinlock), false); +KCSAN_EXPECT_RW_BARRIER (arch_spin_unlock (&arch_spinlock), true); +KCSAN_EXPECT_RW_BARRIER (spin_lock (&spinlock), false); +KCSAN_EXPECT_RW_BARRIER (spin_unlock (&spinlock), true); +KCSAN_EXPECT_RW_BARRIER (mutex_lock (&mutex), false); +KCSAN_EXPECT_RW_BARRIER (mutex_unlock (&mutex), true); + +kcsan_nestable_atomic_end (); +} +/* Simple test with normal data race. */ __no_kcsan static void test_basic(struct kunit *test)
@@ -1039,6 +1264,90 @@ static void test_1bit_value_change(struct kunit *test)KUNIT_EXPECT_TRUE (test, match); } +__no_kcsan +static void test_correct_barrier(struct kunit *test) +{ +struct expect_report expect = { + .access = { + { test_kernel_with_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) }, + { test_kernel_with_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (0 ) }, + }, + }; +bool match_expect = false; + + test_struct.val[0 ] =0 ;/* init unlocked */ +begin_test_checks (test_kernel_with_memorder, test_kernel_with_memorder); +do { + match_expect =report_matches_any_reordered (&expect); + }while (!end_test_checks (match_expect)); +KUNIT_EXPECT_FALSE (test, match_expect); +} + +__no_kcsan +static void test_missing_barrier(struct kunit *test) +{ +struct expect_report expect = { + .access = { + { test_kernel_wrong_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) }, + { test_kernel_wrong_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (0 ) }, + }, + }; +bool match_expect = false; + + test_struct.val[0 ] =0 ;/* init unlocked */ +begin_test_checks (test_kernel_wrong_memorder, test_kernel_wrong_memorder); +do { + match_expect =report_matches_any_reordered (&expect); + }while (!end_test_checks (match_expect)); +if (IS_ENABLED (CONFIG_KCSAN_WEAK_MEMORY )) +KUNIT_EXPECT_TRUE (test, match_expect); +else +KUNIT_EXPECT_FALSE (test, match_expect); +} + +__no_kcsan +static void test_atomic_builtins_correct_barrier(struct kunit *test) +{ +struct expect_report expect = { + .access = { + { test_kernel_atomic_builtin_with_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) }, + { test_kernel_atomic_builtin_with_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (0 ) }, + }, + }; +bool match_expect = false; + + test_struct.val[0 ] =0 ;/* init unlocked */ +begin_test_checks (test_kernel_atomic_builtin_with_memorder, + test_kernel_atomic_builtin_with_memorder); +do { + match_expect =report_matches_any_reordered (&expect); + }while (!end_test_checks (match_expect)); +KUNIT_EXPECT_FALSE (test, match_expect); +} + +__no_kcsan +static void test_atomic_builtins_missing_barrier(struct kunit *test) +{ +struct expect_report expect = { + .access = { + { test_kernel_atomic_builtin_wrong_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (KCSAN_ACCESS_WRITE ) }, + { test_kernel_atomic_builtin_wrong_memorder, &test_var,sizeof (test_var),__KCSAN_ACCESS_RW (0 ) }, + }, + }; +bool match_expect = false; + + test_struct.val[0 ] =0 ;/* init unlocked */ +begin_test_checks (test_kernel_atomic_builtin_wrong_memorder, + test_kernel_atomic_builtin_wrong_memorder); +do { + match_expect =report_matches_any_reordered (&expect); + }while (!end_test_checks (match_expect)); +if (IS_ENABLED (CONFIG_KCSAN_WEAK_MEMORY )) +KUNIT_EXPECT_TRUE (test, match_expect); +else +KUNIT_EXPECT_FALSE (test, match_expect); +} +/* * Generate thread counts for all test cases. Values generated are in interval * [2, 5] followed by exponentially increasing thread counts from 8 to 32.
@@ -1088,6 +1397,7 @@ static const void *nthreads_gen_params(const void *prev, char *desc)#define KCSAN_KUNIT_CASE (test_name) KUNIT_CASE_PARAM(test_name, nthreads_gen_params)static struct kunit_case kcsan_test_cases[] = { +KUNIT_CASE (test_barrier_nothreads),KCSAN_KUNIT_CASE (test_basic),KCSAN_KUNIT_CASE (test_concurrent_races),KCSAN_KUNIT_CASE (test_novalue_change),
@@ -1112,6 +1422,10 @@ static struct kunit_case kcsan_test_cases[] = {KCSAN_KUNIT_CASE (test_seqlock_noreport),KCSAN_KUNIT_CASE (test_atomic_builtins),KCSAN_KUNIT_CASE (test_1bit_value_change), +KCSAN_KUNIT_CASE (test_correct_barrier), +KCSAN_KUNIT_CASE (test_missing_barrier), +KCSAN_KUNIT_CASE (test_atomic_builtins_correct_barrier), + KCSAN_KUNIT_CASE(test_atomic_builtins_missing_barrier), {}, };
@@ -1176,6 +1490,9 @@ static int test_init(struct kunit *test) observed.nlines =0 ;spin_unlock_irqrestore (&observed.lock, flags); +if (strstr (test->name,"nothreads" )) +return 0 ; +if (!torture_init_begin ((char *)test->name,1 ))return -EBUSY ;
@@ -1218,6 +1535,9 @@ static void test_exit(struct kunit *test)struct task_struct **stop_thread;int i; +if (strstr (test->name,"nothreads" )) +return ; +if (torture_cleanup_begin ())return ;
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 12/23 kcsan: Ignore GCC 11+ warnings about TSan runtime support
In reply to Marco ElverGCC 11 has introduced a new warning option, -Wtsan [1], to warn about unsupported operations in the TSan runtime. But KCSAN != TSan runtime, so none of the warnings apply.
[1] https://gcc.gnu.org/onlinedocs/gcc-11.1.0/gcc/Warning-Options.html
Ignore the warnings.
Currently the warning only fires in the test for __atomic_thread_fence():
kernel/kcsan/kcsan_test.c: In function ‘test_atomic_builtins’:kernel/kcsan/kcsan_test.c:1234:17: warning: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Wtsan]
1234 | __atomic_thread_fence(__ATOMIC_SEQ_CST);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~which exists to ensure the KCSAN runtime keeps supporting the builtin instrumentation.
--- scripts/Makefile.kcsan | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/Makefile.kcsan +6 -0
@@ -13,6 +13,12 @@ kcsan-cflags := -fsanitize=thread -fno-optimize-sibling-calls \ $(call cc-option,$(call cc-param,tsan-compound-read-before-write=1),$(call cc-option,$(call cc-param,tsan-instrument-read-before-write=1))) \ $(call cc-param,tsan-distinguish-volatile=1) +ifdef CONFIG_CC_IS_GCC +# GCC started warning about operations unsupported by the TSan runtime. But +# KCSAN != TSan, so just ignore these warnings. +kcsan-cflags += -Wno-tsan +endif + ifndef CONFIG_KCSAN_WEAK_MEMORY kcsan-cflags += $(call cc-option,$(call cc-param,tsan-instrument-func-entry-exit=0)) endif
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 13/23 kcsan: selftest: Add test case to check memory barrier instrumentation
In reply to Marco ElverMemory barrier instrumentation is crucial to avoid false positives. To avoid surprises, run a simple test case in the boot-time selftest to ensure memory barriers are still instrumented correctly.
--- kernel/kcsan/Makefile | 2 + kernel/kcsan/selftest.c | 141 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+)
kernel/kcsan/Makefile +2 -0
@@ -11,6 +11,8 @@ CFLAGS_core.o := $(call cc-option,-fno-conserve-stack) \ -fno-stack-protector -DDISABLE_BRANCH_PROFILING obj-y := core.o debugfs.o report.o + +KCSAN_INSTRUMENT_BARRIERS_selftest.o := y obj-$(CONFIG_KCSAN_SELFTEST) += selftest.o CFLAGS_kcsan_test.o := $(CFLAGS_KCSAN) -g -fno-omit-frame-pointer
kernel/kcsan/selftest.c +141 -0
163 lines not on this page · raw message
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 14/23 locking/barriers, kcsan: Add instrumentation for barriers
In reply to Marco ElverAdds the required KCSAN instrumentation for barriers if CONFIG_SMP. KCSAN supports modeling the effects of:
smp_mb() smp_rmb() smp_wmb() smp_store_release()
--- include/asm-generic/barrier.h | 29 +++++++++++++++-------------- include/linux/spinlock.h | 2 +- 2 files changed, 16 insertions(+), 15 deletions(-)
include/asm-generic/barrier.h +15 -14
@@ -14,6 +14,7 @@#ifndef __ASSEMBLY__#include <linux/compiler.h> +#include <linux/kcsan-checks.h> #include <asm/rwonce.h> #ifndef nop
@@ -62,15 +63,15 @@#ifdef CONFIG_SMP #ifndef smp_mb -#define smp_mb () __smp_mb() +#define smp_mb () do { kcsan_mb(); __smp_mb(); } while (0)#endif #ifndef smp_rmb -#define smp_rmb () __smp_rmb() +#define smp_rmb () do { kcsan_rmb(); __smp_rmb(); } while (0)#endif #ifndef smp_wmb -#define smp_wmb () __smp_wmb() +#define smp_wmb () do { kcsan_wmb(); __smp_wmb(); } while (0)#endif #else /* !CONFIG_SMP */
@@ -123,19 +124,19 @@ do { \#ifdef CONFIG_SMP #ifndef smp_store_mb -#define smp_store_mb (var, value) __smp_store_mb(var, value) +#define smp_store_mb (var, value) do { kcsan_mb(); __smp_store_mb(var, value); } while (0)#endif #ifndef smp_mb__before_atomic -#define smp_mb__before_atomic () __smp_mb__before_atomic() +#define smp_mb__before_atomic () do { kcsan_mb(); __smp_mb__before_atomic(); } while (0)#endif #ifndef smp_mb__after_atomic -#define smp_mb__after_atomic () __smp_mb__after_atomic() +#define smp_mb__after_atomic () do { kcsan_mb(); __smp_mb__after_atomic(); } while (0)#endif #ifndef smp_store_release -#define smp_store_release (p, v) __smp_store_release(p, v) +#define smp_store_release (p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0)#endif #ifndef smp_load_acquire
@@ -178,13 +179,13 @@ do { \#endif /* CONFIG_SMP */ /* Barriers for virtual machine guests when talking to an SMP host */ -#define virt_mb () __smp_mb() -#define virt_rmb () __smp_rmb() -#define virt_wmb () __smp_wmb() -#define virt_store_mb (var, value) __smp_store_mb(var, value) -#define virt_mb__before_atomic () __smp_mb__before_atomic() -#define virt_mb__after_atomic () __smp_mb__after_atomic() -#define virt_store_release (p, v) __smp_store_release(p, v) +#define virt_mb () do { kcsan_mb(); __smp_mb(); } while (0) +#define virt_rmb () do { kcsan_rmb(); __smp_rmb(); } while (0) +#define virt_wmb () do { kcsan_wmb(); __smp_wmb(); } while (0) +#define virt_store_mb (var, value) do { kcsan_mb(); __smp_store_mb(var, value); } while (0) +#define virt_mb__before_atomic () do { kcsan_mb(); __smp_mb__before_atomic(); } while (0) +#define virt_mb__after_atomic () do { kcsan_mb(); __smp_mb__after_atomic(); } while (0) +#define virt_store_release (p, v) do { kcsan_release(); __smp_store_release(p, v); } while (0)#define virt_load_acquire (p) __smp_load_acquire(p)/**
include/linux/spinlock.h +1 -1
@@ -171,7 +171,7 @@ do { \* Architectures that can implement ACQUIRE better need to take care. */ #ifndef smp_mb__after_spinlock -#define smp_mb__after_spinlock () do { } while (0) +#define smp_mb__after_spinlock () kcsan_mb()#endif #ifdef CONFIG_DEBUG_SPINLOCK
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 15/23 locking/barriers, kcsan: Support generic instrumentation
In reply to Marco ElverThus far only smp_*() barriers had been defined by asm-generic/barrier.h based on __smp_*() barriers, because the !SMP case is usually generic.
With the introduction of instrumentation, it also makes sense to have asm-generic/barrier.h assist in the definition of instrumented versions of mb(), rmb(), wmb(), dma_rmb(), and dma_wmb().
Because there is no requirement to distinguish the !SMP case, the definition can be simpler: we can avoid also providing fallbacks for the __ prefixed cases, and only check if defined(__<barrier>), to finally define the KCSAN-instrumented versions.
This also allows for the compiler to complain if an architecture accidentally defines both the normal and __ prefixed variant.
--- include/asm-generic/barrier.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
include/asm-generic/barrier.h +25 -0
@@ -21,6 +21,31 @@#define nop () asm volatile ("nop")#endif +/* +* Architectures that want generic instrumentation can define __ prefixed +* variants of all barriers. +*/ + +#ifdef __mb +#define mb () do { kcsan_mb(); __mb(); } while (0) +#endif + +#ifdef __rmb +#define rmb () do { kcsan_rmb(); __rmb(); } while (0) +#endif + +#ifdef __wmb +#define wmb () do { kcsan_wmb(); __wmb(); } while (0) +#endif + +#ifdef __dma_rmb +#define dma_rmb () do { kcsan_rmb(); __dma_rmb(); } while (0) +#endif + +#ifdef __dma_wmb +#define dma_wmb () do { kcsan_wmb(); __dma_wmb(); } while (0) +#endif +/* * Force strict CPU ordering. And yes, this is required on UP too when we're * talking to devices.
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 16/23 locking/atomics, kcsan: Add instrumentation for barriers
In reply to Marco ElverAdds the required KCSAN instrumentation for barriers of atomics.
--- include/linux/atomic/atomic-instrumented.h | 135 ++++++++++++++++++++- scripts/atomic/gen-atomic-instrumented.sh | 41 +++++-- 2 files changed, 166 insertions(+), 10 deletions(-)
include/linux/atomic/atomic-instrumented.h +134 -1
@@ -45,6 +45,7 @@ atomic_set(atomic_t *v, int i) static __always_inline voidatomic_set_release (atomic_t *v,int i) { +kcsan_release ();instrument_atomic_write (v,sizeof (*v));arch_atomic_set_release (v, i); }
@@ -59,6 +60,7 @@ atomic_add(int i, atomic_t *v) static __always_inline intatomic_add_return (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_add_return (i, v); }
@@ -73,6 +75,7 @@ atomic_add_return_acquire(int i, atomic_t *v) static __always_inline intatomic_add_return_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_add_return_release (i, v); }
@@ -87,6 +90,7 @@ atomic_add_return_relaxed(int i, atomic_t *v) static __always_inline intatomic_fetch_add (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_add (i, v); }
@@ -101,6 +105,7 @@ atomic_fetch_add_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_add_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_add_release (i, v); }
@@ -122,6 +127,7 @@ atomic_sub(int i, atomic_t *v) static __always_inline intatomic_sub_return (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_sub_return (i, v); }
@@ -136,6 +142,7 @@ atomic_sub_return_acquire(int i, atomic_t *v) static __always_inline intatomic_sub_return_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_sub_return_release (i, v); }
@@ -150,6 +157,7 @@ atomic_sub_return_relaxed(int i, atomic_t *v) static __always_inline intatomic_fetch_sub (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_sub (i, v); }
@@ -164,6 +172,7 @@ atomic_fetch_sub_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_sub_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_sub_release (i, v); }
@@ -185,6 +194,7 @@ atomic_inc(atomic_t *v) static __always_inline intatomic_inc_return (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_inc_return (v); }
@@ -199,6 +209,7 @@ atomic_inc_return_acquire(atomic_t *v) static __always_inline intatomic_inc_return_release (atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_inc_return_release (v); }
@@ -213,6 +224,7 @@ atomic_inc_return_relaxed(atomic_t *v) static __always_inline intatomic_fetch_inc (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_inc (v); }
@@ -227,6 +239,7 @@ atomic_fetch_inc_acquire(atomic_t *v) static __always_inline intatomic_fetch_inc_release (atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_inc_release (v); }
@@ -248,6 +261,7 @@ atomic_dec(atomic_t *v) static __always_inline intatomic_dec_return (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_dec_return (v); }
@@ -262,6 +276,7 @@ atomic_dec_return_acquire(atomic_t *v) static __always_inline intatomic_dec_return_release (atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_dec_return_release (v); }
@@ -276,6 +291,7 @@ atomic_dec_return_relaxed(atomic_t *v) static __always_inline intatomic_fetch_dec (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_dec (v); }
@@ -290,6 +306,7 @@ atomic_fetch_dec_acquire(atomic_t *v) static __always_inline intatomic_fetch_dec_release (atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_dec_release (v); }
@@ -311,6 +328,7 @@ atomic_and(int i, atomic_t *v) static __always_inline intatomic_fetch_and (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_and (i, v); }
@@ -325,6 +343,7 @@ atomic_fetch_and_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_and_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_and_release (i, v); }
@@ -346,6 +365,7 @@ atomic_andnot(int i, atomic_t *v) static __always_inline intatomic_fetch_andnot (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_andnot (i, v); }
@@ -360,6 +380,7 @@ atomic_fetch_andnot_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_andnot_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_andnot_release (i, v); }
@@ -381,6 +402,7 @@ atomic_or(int i, atomic_t *v) static __always_inline intatomic_fetch_or (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_or (i, v); }
@@ -395,6 +417,7 @@ atomic_fetch_or_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_or_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_or_release (i, v); }
@@ -416,6 +439,7 @@ atomic_xor(int i, atomic_t *v) static __always_inline intatomic_fetch_xor (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_xor (i, v); }
@@ -430,6 +454,7 @@ atomic_fetch_xor_acquire(int i, atomic_t *v) static __always_inline intatomic_fetch_xor_release (int i,atomic_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_xor_release (i, v); }
@@ -444,6 +469,7 @@ atomic_fetch_xor_relaxed(int i, atomic_t *v) static __always_inline intatomic_xchg (atomic_t *v,int i) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_xchg (v, i); }
@@ -458,6 +484,7 @@ atomic_xchg_acquire(atomic_t *v, int i) static __always_inline intatomic_xchg_release (atomic_t *v,int i) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_xchg_release (v, i); }
@@ -472,6 +499,7 @@ atomic_xchg_relaxed(atomic_t *v, int i) static __always_inline intatomic_cmpxchg (atomic_t *v,int old,int new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_cmpxchg (v, old, new); }
@@ -486,6 +514,7 @@ atomic_cmpxchg_acquire(atomic_t *v, int old, int new) static __always_inline intatomic_cmpxchg_release (atomic_t *v,int old,int new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_cmpxchg_release (v, old, new); }
@@ -500,6 +529,7 @@ atomic_cmpxchg_relaxed(atomic_t *v, int old, int new) static __always_inline boolatomic_try_cmpxchg (atomic_t *v,int *old,int new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic_try_cmpxchg (v, old, new);
@@ -516,6 +546,7 @@ atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new) static __always_inline boolatomic_try_cmpxchg_release (atomic_t *v,int *old,int new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic_try_cmpxchg_release (v, old, new);
@@ -532,6 +563,7 @@ atomic_try_cmpxchg_relaxed(atomic_t *v, int *old, int new) static __always_inline boolatomic_sub_and_test (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_sub_and_test (i, v); }
@@ -539,6 +571,7 @@ atomic_sub_and_test(int i, atomic_t *v) static __always_inline boolatomic_dec_and_test (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_dec_and_test (v); }
@@ -546,6 +579,7 @@ atomic_dec_and_test(atomic_t *v) static __always_inline boolatomic_inc_and_test (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_inc_and_test (v); }
@@ -553,6 +587,7 @@ atomic_inc_and_test(atomic_t *v) static __always_inline boolatomic_add_negative (int i,atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_add_negative (i, v); }
@@ -560,6 +595,7 @@ atomic_add_negative(int i, atomic_t *v) static __always_inline intatomic_fetch_add_unless (atomic_t *v,int a,int u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_fetch_add_unless (v, a, u); }
@@ -567,6 +603,7 @@ atomic_fetch_add_unless(atomic_t *v, int a, int u) static __always_inline boolatomic_add_unless (atomic_t *v,int a,int u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_add_unless (v, a, u); }
@@ -574,6 +611,7 @@ atomic_add_unless(atomic_t *v, int a, int u) static __always_inline boolatomic_inc_not_zero (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_inc_not_zero (v); }
@@ -581,6 +619,7 @@ atomic_inc_not_zero(atomic_t *v) static __always_inline boolatomic_inc_unless_negative (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_inc_unless_negative (v); }
@@ -588,6 +627,7 @@ atomic_inc_unless_negative(atomic_t *v) static __always_inline boolatomic_dec_unless_positive (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_dec_unless_positive (v); }
@@ -595,6 +635,7 @@ atomic_dec_unless_positive(atomic_t *v) static __always_inline intatomic_dec_if_positive (atomic_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_dec_if_positive (v); }
@@ -623,6 +664,7 @@ atomic64_set(atomic64_t *v, s64 i) static __always_inline voidatomic64_set_release (atomic64_t *v,s64 i) { +kcsan_release ();instrument_atomic_write (v,sizeof (*v));arch_atomic64_set_release (v, i); }
@@ -637,6 +679,7 @@ atomic64_add(s64 i, atomic64_t *v) static __always_inline s64atomic64_add_return (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_add_return (i, v); }
@@ -651,6 +694,7 @@ atomic64_add_return_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_add_return_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_add_return_release (i, v); }
@@ -665,6 +709,7 @@ atomic64_add_return_relaxed(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_add (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_add (i, v); }
@@ -679,6 +724,7 @@ atomic64_fetch_add_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_add_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_add_release (i, v); }
@@ -700,6 +746,7 @@ atomic64_sub(s64 i, atomic64_t *v) static __always_inline s64atomic64_sub_return (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_sub_return (i, v); }
@@ -714,6 +761,7 @@ atomic64_sub_return_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_sub_return_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_sub_return_release (i, v); }
@@ -728,6 +776,7 @@ atomic64_sub_return_relaxed(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_sub (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_sub (i, v); }
@@ -742,6 +791,7 @@ atomic64_fetch_sub_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_sub_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_sub_release (i, v); }
@@ -763,6 +813,7 @@ atomic64_inc(atomic64_t *v) static __always_inline s64atomic64_inc_return (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_inc_return (v); }
@@ -777,6 +828,7 @@ atomic64_inc_return_acquire(atomic64_t *v) static __always_inline s64atomic64_inc_return_release (atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_inc_return_release (v); }
@@ -791,6 +843,7 @@ atomic64_inc_return_relaxed(atomic64_t *v) static __always_inline s64atomic64_fetch_inc (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_inc (v); }
@@ -805,6 +858,7 @@ atomic64_fetch_inc_acquire(atomic64_t *v) static __always_inline s64atomic64_fetch_inc_release (atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_inc_release (v); }
@@ -826,6 +880,7 @@ atomic64_dec(atomic64_t *v) static __always_inline s64atomic64_dec_return (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_dec_return (v); }
@@ -840,6 +895,7 @@ atomic64_dec_return_acquire(atomic64_t *v) static __always_inline s64atomic64_dec_return_release (atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_dec_return_release (v); }
@@ -854,6 +910,7 @@ atomic64_dec_return_relaxed(atomic64_t *v) static __always_inline s64atomic64_fetch_dec (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_dec (v); }
@@ -868,6 +925,7 @@ atomic64_fetch_dec_acquire(atomic64_t *v) static __always_inline s64atomic64_fetch_dec_release (atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_dec_release (v); }
@@ -889,6 +947,7 @@ atomic64_and(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_and (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_and (i, v); }
@@ -903,6 +962,7 @@ atomic64_fetch_and_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_and_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_and_release (i, v); }
@@ -924,6 +984,7 @@ atomic64_andnot(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_andnot (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_andnot (i, v); }
@@ -938,6 +999,7 @@ atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_andnot_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_andnot_release (i, v); }
@@ -959,6 +1021,7 @@ atomic64_or(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_or (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_or (i, v); }
@@ -973,6 +1036,7 @@ atomic64_fetch_or_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_or_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_or_release (i, v); }
@@ -994,6 +1058,7 @@ atomic64_xor(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_xor (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_xor (i, v); }
@@ -1008,6 +1073,7 @@ atomic64_fetch_xor_acquire(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_xor_release (s64 i,atomic64_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_xor_release (i, v); }
@@ -1022,6 +1088,7 @@ atomic64_fetch_xor_relaxed(s64 i, atomic64_t *v) static __always_inline s64atomic64_xchg (atomic64_t *v,s64 i) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_xchg (v, i); }
@@ -1036,6 +1103,7 @@ atomic64_xchg_acquire(atomic64_t *v, s64 i) static __always_inline s64atomic64_xchg_release (atomic64_t *v,s64 i) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_xchg_release (v, i); }
@@ -1050,6 +1118,7 @@ atomic64_xchg_relaxed(atomic64_t *v, s64 i) static __always_inline s64atomic64_cmpxchg (atomic64_t *v,s64 old,s64 new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_cmpxchg (v, old, new); }
@@ -1064,6 +1133,7 @@ atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new) static __always_inline s64atomic64_cmpxchg_release (atomic64_t *v,s64 old,s64 new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_cmpxchg_release (v, old, new); }
@@ -1078,6 +1148,7 @@ atomic64_cmpxchg_relaxed(atomic64_t *v, s64 old, s64 new) static __always_inline boolatomic64_try_cmpxchg (atomic64_t *v,s64 *old,s64 new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic64_try_cmpxchg (v, old, new);
@@ -1094,6 +1165,7 @@ atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new) static __always_inline boolatomic64_try_cmpxchg_release (atomic64_t *v,s64 *old,s64 new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic64_try_cmpxchg_release (v, old, new);
@@ -1110,6 +1182,7 @@ atomic64_try_cmpxchg_relaxed(atomic64_t *v, s64 *old, s64 new) static __always_inline boolatomic64_sub_and_test (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_sub_and_test (i, v); }
@@ -1117,6 +1190,7 @@ atomic64_sub_and_test(s64 i, atomic64_t *v) static __always_inline boolatomic64_dec_and_test (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_dec_and_test (v); }
@@ -1124,6 +1198,7 @@ atomic64_dec_and_test(atomic64_t *v) static __always_inline boolatomic64_inc_and_test (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_inc_and_test (v); }
@@ -1131,6 +1206,7 @@ atomic64_inc_and_test(atomic64_t *v) static __always_inline boolatomic64_add_negative (s64 i,atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_add_negative (i, v); }
@@ -1138,6 +1214,7 @@ atomic64_add_negative(s64 i, atomic64_t *v) static __always_inline s64atomic64_fetch_add_unless (atomic64_t *v,s64 a,s64 u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_fetch_add_unless (v, a, u); }
@@ -1145,6 +1222,7 @@ atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u) static __always_inline boolatomic64_add_unless (atomic64_t *v,s64 a,s64 u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_add_unless (v, a, u); }
@@ -1152,6 +1230,7 @@ atomic64_add_unless(atomic64_t *v, s64 a, s64 u) static __always_inline boolatomic64_inc_not_zero (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_inc_not_zero (v); }
@@ -1159,6 +1238,7 @@ atomic64_inc_not_zero(atomic64_t *v) static __always_inline boolatomic64_inc_unless_negative (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_inc_unless_negative (v); }
@@ -1166,6 +1246,7 @@ atomic64_inc_unless_negative(atomic64_t *v) static __always_inline boolatomic64_dec_unless_positive (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_dec_unless_positive (v); }
@@ -1173,6 +1254,7 @@ atomic64_dec_unless_positive(atomic64_t *v) static __always_inline s64atomic64_dec_if_positive (atomic64_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic64_dec_if_positive (v); }
@@ -1201,6 +1283,7 @@ atomic_long_set(atomic_long_t *v, long i) static __always_inline voidatomic_long_set_release (atomic_long_t *v,long i) { +kcsan_release ();instrument_atomic_write (v,sizeof (*v));arch_atomic_long_set_release (v, i); }
@@ -1215,6 +1298,7 @@ atomic_long_add(long i, atomic_long_t *v)static __always_inline long atomic_long_add_return (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_add_return (i, v); }
@@ -1229,6 +1313,7 @@ atomic_long_add_return_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_add_return_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_add_return_release (i, v); }
@@ -1243,6 +1328,7 @@ atomic_long_add_return_relaxed(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_add (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_add (i, v); }
@@ -1257,6 +1343,7 @@ atomic_long_fetch_add_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_add_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_add_release (i, v); }
@@ -1278,6 +1365,7 @@ atomic_long_sub(long i, atomic_long_t *v)static __always_inline long atomic_long_sub_return (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_sub_return (i, v); }
@@ -1292,6 +1380,7 @@ atomic_long_sub_return_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_sub_return_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_sub_return_release (i, v); }
@@ -1306,6 +1395,7 @@ atomic_long_sub_return_relaxed(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_sub (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_sub (i, v); }
@@ -1320,6 +1410,7 @@ atomic_long_fetch_sub_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_sub_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_sub_release (i, v); }
@@ -1341,6 +1432,7 @@ atomic_long_inc(atomic_long_t *v)static __always_inline long atomic_long_inc_return (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_inc_return (v); }
@@ -1355,6 +1447,7 @@ atomic_long_inc_return_acquire(atomic_long_t *v)static __always_inline long atomic_long_inc_return_release (atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_inc_return_release (v); }
@@ -1369,6 +1462,7 @@ atomic_long_inc_return_relaxed(atomic_long_t *v)static __always_inline long atomic_long_fetch_inc (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_inc (v); }
@@ -1383,6 +1477,7 @@ atomic_long_fetch_inc_acquire(atomic_long_t *v)static __always_inline long atomic_long_fetch_inc_release (atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_inc_release (v); }
@@ -1404,6 +1499,7 @@ atomic_long_dec(atomic_long_t *v)static __always_inline long atomic_long_dec_return (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_dec_return (v); }
@@ -1418,6 +1514,7 @@ atomic_long_dec_return_acquire(atomic_long_t *v)static __always_inline long atomic_long_dec_return_release (atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_dec_return_release (v); }
@@ -1432,6 +1529,7 @@ atomic_long_dec_return_relaxed(atomic_long_t *v)static __always_inline long atomic_long_fetch_dec (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_dec (v); }
@@ -1446,6 +1544,7 @@ atomic_long_fetch_dec_acquire(atomic_long_t *v)static __always_inline long atomic_long_fetch_dec_release (atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_dec_release (v); }
@@ -1467,6 +1566,7 @@ atomic_long_and(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_and (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_and (i, v); }
@@ -1481,6 +1581,7 @@ atomic_long_fetch_and_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_and_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_and_release (i, v); }
@@ -1502,6 +1603,7 @@ atomic_long_andnot(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_andnot (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_andnot (i, v); }
@@ -1516,6 +1618,7 @@ atomic_long_fetch_andnot_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_andnot_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_andnot_release (i, v); }
@@ -1537,6 +1640,7 @@ atomic_long_or(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_or (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_or (i, v); }
@@ -1551,6 +1655,7 @@ atomic_long_fetch_or_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_or_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_or_release (i, v); }
@@ -1572,6 +1677,7 @@ atomic_long_xor(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_xor (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_xor (i, v); }
@@ -1586,6 +1692,7 @@ atomic_long_fetch_xor_acquire(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_xor_release (long i,atomic_long_t *v) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_xor_release (i, v); }
@@ -1600,6 +1707,7 @@ atomic_long_fetch_xor_relaxed(long i, atomic_long_t *v)static __always_inline long atomic_long_xchg (atomic_long_t *v,long i) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_xchg (v, i); }
@@ -1614,6 +1722,7 @@ atomic_long_xchg_acquire(atomic_long_t *v, long i)static __always_inline long atomic_long_xchg_release (atomic_long_t *v,long i) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_xchg_release (v, i); }
@@ -1628,6 +1737,7 @@ atomic_long_xchg_relaxed(atomic_long_t *v, long i)static __always_inline long atomic_long_cmpxchg (atomic_long_t *v,long old,long new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_cmpxchg (v, old, new); }
@@ -1642,6 +1752,7 @@ atomic_long_cmpxchg_acquire(atomic_long_t *v, long old, long new)static __always_inline long atomic_long_cmpxchg_release (atomic_long_t *v,long old,long new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_cmpxchg_release (v, old, new); }
@@ -1656,6 +1767,7 @@ atomic_long_cmpxchg_relaxed(atomic_long_t *v, long old, long new) static __always_inline boolatomic_long_try_cmpxchg (atomic_long_t *v,long *old,long new) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic_long_try_cmpxchg (v, old, new);
@@ -1672,6 +1784,7 @@ atomic_long_try_cmpxchg_acquire(atomic_long_t *v, long *old, long new) static __always_inline boolatomic_long_try_cmpxchg_release (atomic_long_t *v,long *old,long new) { +kcsan_release ();instrument_atomic_read_write (v,sizeof (*v));instrument_atomic_read_write (old,sizeof (*old));return arch_atomic_long_try_cmpxchg_release (v, old, new);
@@ -1688,6 +1801,7 @@ atomic_long_try_cmpxchg_relaxed(atomic_long_t *v, long *old, long new) static __always_inline boolatomic_long_sub_and_test (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_sub_and_test (i, v); }
@@ -1695,6 +1809,7 @@ atomic_long_sub_and_test(long i, atomic_long_t *v) static __always_inline boolatomic_long_dec_and_test (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_dec_and_test (v); }
@@ -1702,6 +1817,7 @@ atomic_long_dec_and_test(atomic_long_t *v) static __always_inline boolatomic_long_inc_and_test (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_inc_and_test (v); }
@@ -1709,6 +1825,7 @@ atomic_long_inc_and_test(atomic_long_t *v) static __always_inline boolatomic_long_add_negative (long i,atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_add_negative (i, v); }
@@ -1716,6 +1833,7 @@ atomic_long_add_negative(long i, atomic_long_t *v)static __always_inline long atomic_long_fetch_add_unless (atomic_long_t *v,long a,long u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_fetch_add_unless (v, a, u); }
@@ -1723,6 +1841,7 @@ atomic_long_fetch_add_unless(atomic_long_t *v, long a, long u) static __always_inline boolatomic_long_add_unless (atomic_long_t *v,long a,long u) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_add_unless (v, a, u); }
@@ -1730,6 +1849,7 @@ atomic_long_add_unless(atomic_long_t *v, long a, long u) static __always_inline boolatomic_long_inc_not_zero (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_inc_not_zero (v); }
@@ -1737,6 +1857,7 @@ atomic_long_inc_not_zero(atomic_long_t *v) static __always_inline boolatomic_long_inc_unless_negative (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_inc_unless_negative (v); }
@@ -1744,6 +1865,7 @@ atomic_long_inc_unless_negative(atomic_long_t *v) static __always_inline boolatomic_long_dec_unless_positive (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_dec_unless_positive (v); }
@@ -1751,6 +1873,7 @@ atomic_long_dec_unless_positive(atomic_long_t *v)static __always_inline long atomic_long_dec_if_positive (atomic_long_t *v) { +kcsan_mb ();instrument_atomic_read_write (v,sizeof (*v));return arch_atomic_long_dec_if_positive (v); }
@@ -1758,6 +1881,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define xchg (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_xchg(__ai_ptr, __VA_ARGS__); \ })
@@ -1772,6 +1896,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define xchg_release (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_release(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_xchg_release(__ai_ptr, __VA_ARGS__); \ })
@@ -1786,6 +1911,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define cmpxchg (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_cmpxchg(__ai_ptr, __VA_ARGS__); \ })
@@ -1800,6 +1926,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define cmpxchg_release (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_release(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_cmpxchg_release(__ai_ptr, __VA_ARGS__); \ })
@@ -1814,6 +1941,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define cmpxchg64 (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_cmpxchg64(__ai_ptr, __VA_ARGS__); \ })
@@ -1828,6 +1956,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define cmpxchg64_release (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_release(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_cmpxchg64_release(__ai_ptr, __VA_ARGS__); \ })
@@ -1843,6 +1972,7 @@ atomic_long_dec_if_positive(atomic_long_t *v) ({ \ typeof(ptr) __ai_ptr = (ptr); \ typeof(oldp) __ai_oldp = (oldp); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ arch_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
@@ -1861,6 +1991,7 @@ atomic_long_dec_if_positive(atomic_long_t *v) ({ \ typeof(ptr) __ai_ptr = (ptr); \ typeof(oldp) __ai_oldp = (oldp); \ + kcsan_release(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ arch_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
@@ -1892,6 +2023,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define sync_cmpxchg (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \ })
@@ -1899,6 +2031,7 @@ atomic_long_dec_if_positive(atomic_long_t *v)#define cmpxchg_double (ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \ })
@@ -1912,4 +2045,4 @@ atomic_long_dec_if_positive(atomic_long_t *v) })#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 2a9553f0a9d5619f19151092df5cabbbf16ce835 +// 87c974b93032afd42143613434d1a7788fa598f9
scripts/atomic/gen-atomic-instrumented.sh +32 -9
95 lines not on this page · raw message
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 17/23 asm-generic/bitops, kcsan: Add instrumentation for barriers
In reply to Marco ElverAdds the required KCSAN instrumentation for barriers of atomic bitops.
--- include/asm-generic/bitops/instrumented-atomic.h | 3 +++ include/asm-generic/bitops/instrumented-lock.h | 3 +++ 2 files changed, 6 insertions(+)
include/asm-generic/bitops/instrumented-atomic.h +3 -0
@@ -67,6 +67,7 @@ static inline void change_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_and_set_bit (long nr,volatile unsigned long *addr) { +kcsan_mb ();instrument_atomic_read_write (addr +BIT_WORD (nr),sizeof (long ));return arch_test_and_set_bit (nr, addr); }
@@ -80,6 +81,7 @@ static inline bool test_and_set_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_and_clear_bit (long nr,volatile unsigned long *addr) { +kcsan_mb ();instrument_atomic_read_write (addr +BIT_WORD (nr),sizeof (long ));return arch_test_and_clear_bit (nr, addr); }
@@ -93,6 +95,7 @@ static inline bool test_and_clear_bit(long nr, volatile unsigned long *addr)*/ static inline bool test_and_change_bit (long nr,volatile unsigned long *addr) { +kcsan_mb ();instrument_atomic_read_write (addr +BIT_WORD (nr),sizeof (long ));return arch_test_and_change_bit (nr, addr); }
include/asm-generic/bitops/instrumented-lock.h +3 -0
@@ -22,6 +22,7 @@*/ static inline void clear_bit_unlock (long nr,volatile unsigned long *addr) { +kcsan_release ();instrument_atomic_write (addr +BIT_WORD (nr),sizeof (long ));arch_clear_bit_unlock (nr, addr); }
@@ -37,6 +38,7 @@ static inline void clear_bit_unlock(long nr, volatile unsigned long *addr)*/ static inline void __clear_bit_unlock (long nr,volatile unsigned long *addr) { +kcsan_release ();instrument_write (addr +BIT_WORD (nr),sizeof (long ));arch___clear_bit_unlock (nr, addr); }
@@ -71,6 +73,7 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr)static inline bool clear_bit_unlock_is_negative_byte (long nr,volatile unsigned long *addr) { +kcsan_release ();instrument_atomic_write (addr +BIT_WORD (nr),sizeof (long ));return arch_clear_bit_unlock_is_negative_byte (nr, addr); }
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 18/23 x86/barriers, kcsan: Use generic instrumentation for non-smp barriers
In reply to Marco ElverPrefix all barriers with __, now that asm-generic/barriers.h supports defining the final instrumented version of these barriers. The change is limited to barriers used by x86-64.
--- arch/x86/include/asm/barrier.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/barrier.h +5 -5
@@ -19,9 +19,9 @@#define wmb () asm volatile(ALTERNATIVE("lock; addl $0,-4(%%esp)", "sfence", \ X86_FEATURE_XMM2) ::: "memory", "cc")#else -#define mb () asm volatile("mfence":::"memory") -#define rmb () asm volatile("lfence":::"memory") -#define wmb () asm volatile("sfence" ::: "memory") +#define __mb () asm volatile("mfence":::"memory") +#define __rmb () asm volatile("lfence":::"memory") +#define __wmb () asm volatile("sfence" ::: "memory")#endif /**
@@ -51,8 +51,8 @@ static inline unsigned long array_index_mask_nospec(unsigned long index,/* Prevent speculative execution past this barrier. */ #define barrier_nospec () alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC) -#define dma_rmb () barrier() -#define dma_wmb () barrier() +#define __dma_rmb () barrier() +#define __dma_wmb () barrier()#define __smp_mb () asm volatile("lock; addl $0,-4(%%" _ASM_SP ")" ::: "memory", "cc")
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 19/23 x86/qspinlock, kcsan: Instrument barrier of pv_queued_spin_unlock()
In reply to Marco ElverIf CONFIG_PARAVIRT_SPINLOCKS=y, queued_spin_unlock() is implemented using pv_queued_spin_unlock() which is entirely inline asm based. As such, we do not receive any KCSAN barrier instrumentation via regular atomic operations.
Add the missing KCSAN barrier instrumentation for the CONFIG_PARAVIRT_SPINLOCKS case.
--- arch/x86/include/asm/qspinlock.h | 1 + 1 file changed, 1 insertion(+)
arch/x86/include/asm/qspinlock.h +1 -0
@@ -53,6 +53,7 @@ static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)static inline void queued_spin_unlock (struct qspinlock *lock) { +kcsan_release ();pv_queued_spin_unlock (lock); }
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 20/23 mm, kcsan: Enable barrier instrumentation
In reply to Marco ElverSome memory management calls imply memory barriers that are required to avoid false positives. For example, without the correct instrumentation, we could observe data races of the following variant:
T0 | T1
------------------------+------------------------
|
*a = 42; ---+ |
kfree(a); | |
| | b = kmalloc(..); // b == a
<reordered> <-+ | *b = 42; // not a data race!
|Therefore, instrument memory barriers in all allocator code currently not being instrumented in a default build.
--- mm/Makefile | 2 ++ 1 file changed, 2 insertions(+)
mm/Makefile +2 -0
@@ -15,6 +15,8 @@ KCSAN_SANITIZE_slab_common.o := n KCSAN_SANITIZE_slab.o := n KCSAN_SANITIZE_slub.o := n KCSAN_SANITIZE_page_alloc.o := n +# But enable explicit instrumentation for memory barriers. +KCSAN_INSTRUMENT_BARRIERS := y # These files are disabled because they produce non-interesting and/or # flaky coverage that is not a function of syscall inputs. E.g. slab is out of
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 21/23 sched, kcsan: Enable memory barrier instrumentation
In reply to Marco ElverThere's no fundamental reason to disable KCSAN for scheduler code, except for excessive noise and performance concerns (instrumenting scheduler code is usually a good way to stress test KCSAN itself).
However, several core sched functions imply memory barriers that are invisible to KCSAN without instrumentation, but are required to avoid false positives. Therefore, unconditionally enable instrumentation of memory barriers in scheduler code. Also update the comment to reflect this and be a bit more brief.
--- kernel/sched/Makefile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/sched/Makefile +3 -4
@@ -11,11 +11,10 @@ ccflags-y += $(call cc-disable-warning, unused-but-set-variable) # that is not a function of syscall inputs. E.g. involuntary context switches. KCOV_INSTRUMENT := n -# There are numerous data races here, however, most of them are due to plain accesses. -# This would make it even harder for syzbot to find reproducers, because these -# bugs trigger without specific input. Disable by default, but should re-enable -# eventually. +# Disable KCSAN to avoid excessive noise and performance degradation. To avoid +# false positives ensure barriers implied by sched functions are instrumented. KCSAN_SANITIZE := n +KCSAN_INSTRUMENT_BARRIERS := y ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 22/23 objtool, kcsan: Add memory barrier instrumentation to whitelist
In reply to Marco ElverAdds KCSAN's memory barrier instrumentation to objtool's uaccess whitelist.
--- tools/objtool/check.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/objtool/check.c +4 -0
@@ -849,6 +849,10 @@ static const char *uaccess_safe_builtin[] = {"__asan_report_store16_noabort" ,/* KCSAN */ "__kcsan_check_access" , +"__kcsan_mb" , +"__kcsan_wmb" , +"__kcsan_rmb" , +"__kcsan_release" ,"kcsan_found_watchpoint" ,"kcsan_setup_watchpoint" , "kcsan_check_scoped_accesses",
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Marco Elver
maintainerPATCH v2 23/23 objtool, kcsan: Remove memory barrier instrumentation from noinstr
In reply to Marco ElverTeach objtool to turn instrumentation required for memory barrier modeling into nops in noinstr text.
The __tsan_func_entry/exit calls are still emitted by compilers even with the __no_sanitize_thread attribute. The memory barrier instrumentation will be inserted explicitly (without compiler help), and thus needs to also explicitly be removed.
---
v2:
* Rewrite after rebase to v5.16-rc1.
--- tools/objtool/check.c | 37 ++++++++++++++++++++++------- tools/objtool/include/objtool/elf.h | 2 +- 2 files changed, 30 insertions(+), 9 deletions(-)
tools/objtool/check.c +29 -8
@@ -1071,12 +1071,7 @@ static void annotate_call_site(struct objtool_file *file,return ; } -/* -* Many compilers cannot disable KCOV with a function attribute -* so they need a little help, NOP out any KCOV calls from noinstr -* text. -*/ -if (insn->sec->noinstr && sym->kcov) { +if (insn->sec->noinstr && sym->removable_instr) {if (reloc) { reloc->type =R_NONE ;elf_write_reloc (file->elf, reloc);
@@ -1991,6 +1986,32 @@ static int read_intra_function_calls(struct objtool_file *file)return 0 ; } +static bool is_removable_instr (const char *name) +{ +/* +* Many compilers cannot disable KCOV with a function attribute so they +* need a little help, NOP out any KCOV calls from noinstr text. +*/ +if (!strncmp (name,"__sanitizer_cov_" ,16 )) +return true; + +/* +* Compilers currently do not remove __tsan_func_entry/exit with the +* __no_sanitize_thread attribute, remove them. +* +* Memory barrier instrumentation is not emitted by the compiler, but +* inserted explicitly, so we need to also remove them. +*/ +if (!strncmp (name,"__tsan_func_" ,12 ) || + !strcmp (name,"__kcsan_mb" ) || + !strcmp (name,"__kcsan_wmb" ) || + !strcmp (name,"__kcsan_rmb" ) || + !strcmp (name,"__kcsan_release" )) +return true; + +return false; +} +static int classify_symbols (struct objtool_file *file) {struct section *sec;
@@ -2011,8 +2032,8 @@ static int classify_symbols(struct objtool_file *file)if (!strcmp (func->name,"__fentry__" )) func->fentry = true; -if (!strncmp (func->name,"__sanitizer_cov_" ,16 )) - func->kcov = true; +if (is_removable_instr (func->name)) + func->removable_instr = true; } }
tools/objtool/include/objtool/elf.h +1 -1
@@ -58,7 +58,7 @@ struct symbol { u8 static_call_tramp : 1; u8 retpoline_thunk : 1; u8 fentry : 1; - u8 kcov : 1; + u8 removable_instr : 1;struct list_head pv_target; };
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Re: [PATCH v2 23/23] objtool, kcsan: Remove memory barrier instrumentation from noinstr
Quoted 4 lines · Marco Elver
tools/objtool/check.c +0
-0@@ -1071,12 +1071,7 @@ static void annotate_call_site(struct objtool_file *file,return ; }
tools/objtool/check.c static void annotate_call_site(struct objtool_file *file, · line 1077
Quoted 10 lines
tools/objtool/check.c +1
-6-/* -* Many compilers cannot disable KCOV with a function attribute -* so they need a little help, NOP out any KCOV calls from noinstr -* text. -*/ -if (insn->sec->noinstr && sym->kcov) { +if (insn->sec->noinstr && sym->removable_instr) {if (reloc) { reloc->type =R_NONE ;elf_write_reloc (file->elf, reloc);
I'd love to have a clearer name than 'removable_instr', though I'm having trouble coming up with something.
'profiling_func'?
Profiling isn't really accurate but maybe it gets the point across. I'm definitely open to other suggestions.
Also, the above code isn't very self-evident so there still needs to be a comment there, like:
/*
* Many compilers cannot disable KCOV or sanitizer calls with a
* function attribute so they need a little help, NOP out any
* such calls from noinstr text.
*/
Quoted 4 lines
tools/objtool/check.c +0
-0@@ -1991,6 +1986,32 @@ static int read_intra_function_calls(struct objtool_file *file)return 0 ; }
tools/objtool/check.c static int read_intra_function_calls(struct objtool_file *file) · line 1998
Quoted 10 lines
tools/objtool/check.c +8
-0+static bool is_removable_instr (const char *name) +{ +/* +* Many compilers cannot disable KCOV with a function attribute so they +* need a little help, NOP out any KCOV calls from noinstr text. +*/ +if (!strncmp (name,"__sanitizer_cov_" ,16 )) +return true;
A comment is good here, but the NOP-ing bit seems out of place.
Signature · 1 line
Josh
Marco Elver
maintainerRe: [PATCH v2 23/23] objtool, kcsan: Remove memory barrier instrumentation from noinstr
Quoted 5 lines · Josh Poimboeuf
Quoted 4 lines
+ if (insn->sec->noinstr && sym->removable_instr) { if (reloc) { reloc->type = R_NONE; elf_write_reloc(file->elf, reloc);
Quoted 7 lines
I'd love to have a clearer name than '
removable_instr', though I'm having trouble coming up with something.'
profiling_func'?Profiling isn't really accurate but maybe it gets the point across. I'm definitely open to other suggestions.
Well, this bit is not true for all "profiling functions" either. It's only true for instrumentation functions that appear in 'noinstr' and that the compiler can't remove on its own, but are valid to remove by objtool in noinstr code, hence 'removable_instr'.
I'm really quite indifferent what we call it, so I'll leave you to pick whatever sounds best:
-- profiling_func
-- nop_profiling_func
-- optional_profiling_func
-- noinstr_remove
-- removable_profiling_func
-- noinstr_nop_func
-- noinstr_nop
-- nop_in_noinstr
-- invalid_in_noinstr
?
Quoted 9 lines
Also, the above code isn't very self-evident so there still needs to be a comment there, like:
/*
* Many compilers cannot disable KCOV or sanitizer calls with a
* function attribute so they need a little help, NOP out any
* such calls from noinstr text.
*/
I'll add it.
Quoted 9 lines
tools/objtool/check.c static int read_intra_function_calls(struct objtool_file *file) · line 1996
Quoted 7 lines
tools/objtool/check.c +7
-0+{ +/* +* Many compilers cannot disable KCOV with a function attribute so they +* need a little help, NOP out any KCOV calls from noinstr text. +*/ +if (!strncmp (name,"__sanitizer_cov_" ,16 )) +return true;A comment is good here, but the NOP-ing bit seems out of place.
I'll fix that.
Marco Elver
maintainerRe: [PATCH v2 23/23] objtool, kcsan: Remove memory barrier instrumentation from noinstr
In reply to Josh PoimboeufQuoted 1 line · Josh Poimboeuf
On Thu, Nov 18, 2021 at 09:10:27AM +0100, Marco Elver wrote:
[...]
Quoted 1 line
Quoted 1 line
(quoted diff) +1
-0+ if (insn->sec->noinstr && sym->removable_instr) {
[...]
Quoted 2 lines
I'd love to have a clearer name than '
removable_instr', though I'm having trouble coming up with something.
[...]
I now have the below as v3 of this patch. The naming isn't entirely obvious, but coming up with a short name for this is tricky, but hopefully the comments make it clear. We can of course still pick another name.
Does that look reasonable?
Note, I'd like this series to sit in -next for a while (probably from some time next week after sending v3 if there are no further complaints). By default everything will be picked up by the -rcu tree, and we're targeting Linux 5.18.
If you feel there might be objtool conflicts coming, this patch could be taken through another tree as there are no hard dependencies, as long as this patch reaches mainline before or with the rest.
Thanks,
-- Marco
------ >8 ------
From: Marco Elver <[email protected]>
Date: Mon, 9 Aug 2021 12:11:14 +0200
Subject: [PATCH] objtool, kcsan: Remove memory barrier instrumentation from
noinstr
Teach objtool to turn instrumentation required for memory barrier modeling into nops in noinstr text.
The __tsan_func_entry/exit calls are still emitted by compilers even with the __no_sanitize_thread attribute. The memory barrier instrumentation will be inserted explicitly (without compiler help), and thus needs to also explicitly be removed.
---
v3:
* s/removable_instr/profiling_func/ (suggested by Josh Poimboeuf)
* Fix and add more comments.
v2:
* Rewrite after rebase to v5.16-rc1.
--- tools/objtool/check.c | 41 ++++++++++++++++++++++++----- tools/objtool/include/objtool/elf.h | 2 +- 2 files changed, 36 insertions(+), 7 deletions(-)
tools/objtool/check.c +35 -6
@@ -1072,11 +1072,11 @@ static void annotate_call_site(struct objtool_file *file, }/* -* Many compilers cannot disable KCOV with a function attribute -* so they need a little help, NOP out any KCOV calls from noinstr -* text. +* Many compilers cannot disable KCOV or sanitizer calls with a function +* attribute so they need a little help, NOP out any such calls from +* noinstr text. */ -if (insn->sec->noinstr && sym->kcov) { +if (insn->sec->noinstr && sym->profiling_func) {if (reloc) { reloc->type =R_NONE ;elf_write_reloc (file->elf, reloc);
@@ -1991,6 +1991,35 @@ static int read_intra_function_calls(struct objtool_file *file)return 0 ; } +/* +* Return true if name matches an instrumentation function, where calls to that +* function from noinstr code can safely be removed, but compilers won't do so. +*/ +static bool is_profiling_func (const char *name) +{ +/* +* Many compilers cannot disable KCOV with a function attribute. +*/ +if (!strncmp (name,"__sanitizer_cov_" ,16 )) +return true; + +/* +* Compilers currently do not remove __tsan_func_entry/exit with the +* __no_sanitize_thread attribute, remove them. +* +* Memory barrier instrumentation is not emitted by the compiler, but +* inserted explicitly, so we need to also remove them. +*/ +if (!strncmp (name,"__tsan_func_" ,12 ) || + !strcmp (name,"__kcsan_mb" ) || + !strcmp (name,"__kcsan_wmb" ) || + !strcmp (name,"__kcsan_rmb" ) || + !strcmp (name,"__kcsan_release" )) +return true; + +return false; +} +static int classify_symbols (struct objtool_file *file) {struct section *sec;
@@ -2011,8 +2040,8 @@ static int classify_symbols(struct objtool_file *file)if (!strcmp (func->name,"__fentry__" )) func->fentry = true; -if (!strncmp (func->name,"__sanitizer_cov_" ,16 )) - func->kcov = true; +if (is_profiling_func (func->name)) + func->profiling_func = true; } }
tools/objtool/include/objtool/elf.h +1 -1
@@ -58,7 +58,7 @@ struct symbol { u8 static_call_tramp : 1; u8 retpoline_thunk : 1; u8 fentry : 1; - u8 kcov : 1; + u8 profiling_func : 1;struct list_head pv_target; };
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Josh Poimboeuf
Acked-byRe: [PATCH v2 23/23] objtool, kcsan: Remove memory barrier instrumentation from noinstr
Quoted 21 lines · Marco Elver
Quoted 1 line · Josh Poimboeuf
On Thu, Nov 18, 2021 at 09:10:27AM +0100, Marco Elver wrote:
[...]
Quoted 1 line
Quoted 1 line
(quoted diff) +1
-0+ if (insn->sec->noinstr && sym->removable_instr) {[...]
Quoted 2 lines
I'd love to have a clearer name than '
removable_instr', though I'm having trouble coming up with something.[...]
I now have the below as v3 of this patch. The naming isn't entirely obvious, but coming up with a short name for this is tricky, but hopefully the comments make it clear. We can of course still pick another name.
Does that look reasonable?
Note, I'd like this series to sit in -next for a while (probably from some time next week after sending v3 if there are no further complaints). By default everything will be picked up by the -rcu tree, and we're targeting Linux 5.18.
Quoted 6 lines
If you feel there might be objtool conflicts coming, this patch could be taken through another tree as there are no hard dependencies, as long as this patch reaches mainline before or with the rest.
Thanks,
-- Marco
Looks good to me. I don't know of any upcoming conflicts, feel free to carry it with your series for now.
Signature · 1 line
Josh