x86: mm: disable KMSAN instrumentation for physaddr.c
Subsystems (3)
PATCH 1/3 x86: mm: disable KMSAN instrumentation for physaddr.c
Enabling CONFIG_DEBUG_VIRTUAL=y together with KMSAN led to infinite recursion, because kmsan_get_metadata() ended up calling instrumented __pfn_valid() from arch/x86/mm/physaddr.c.
Prevent it by disabling instrumentation of the whole file.
--- arch/x86/mm/Makefile | 1 + 1 file changed, 1 insertion(+)
arch/x86/mm/Makefile +1 -0
@@ -17,6 +17,7 @@ KCSAN_SANITIZE := n # Avoid recursion by not calling KMSAN hooks for CEA code. KMSAN_SANITIZE_cpu_entry_area.o := n KMSAN_SANITIZE_mem_encrypt_identity.o := n +KMSAN_SANITIZE_physaddr.o := n ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_mem_encrypt.o = -pg
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
PATCH 2/3 lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
At least on x86 KMSAN is seriously slown down by lockdep, as every pfn_valid() call (which is done on every instrumented memory access in the kernel) performs several lockdep checks, all of which, in turn, perform additional memory accesses and call KMSAN instrumentation.
Right now lockdep overflows the stack under KMSAN, but even if we use reentrancy counters to avoid the recursion on the KMSAN side, the slowdown from lockdep remains big enough for the kernel to become unusable.
--- lib/Kconfig.debug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/Kconfig.debug +1 -1
@@ -1339,7 +1339,7 @@ menu "Lock Debugging (spinlocks, mutexes, etc...)" config LOCK_DEBUGGING_SUPPORT bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default y config PROVE_LOCKING
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
Quoted 22 lines · Alexander Potapenko
At least on x86 KMSAN is seriously slown down by lockdep, as every
pfn_valid()call (which is done on every instrumented memory access in the kernel) performs several lockdep checks, all of which, in turn, perform additional memory accesses and call KMSAN instrumentation.Right now lockdep overflows the stack under KMSAN, but even if we use reentrancy counters to avoid the recursion on the KMSAN side, the slowdown from lockdep remains big enough for the kernel to become unusable.
Reported-by Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>--- lib/Kconfig.debug | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/Kconfig.debug +0
-0
lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1342
Quoted 5 lines
lib/Kconfig.debug +1
-1@@ -1339,7 +1339,7 @@ menu "Lock Debugging (spinlocks, mutexes, etc...)" config LOCK_DEBUGGING_SUPPORT bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
Nit: no need to pile everything in one line. Add "depends on !KMSAN" on a separate line.
Otherwise, looks sane.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
In reply to Alexander Potapenkolib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORTlib/Kconfig.debug +1
-1bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default y
This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.
I'd *MUCH* rather hide KMSAN:
config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT
+ depends on !LOCKDEP
Because, frankly, lockdep is way more important than KMSAN.
But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
Quoted 21 lines · Dave Hansen
lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORTlib/Kconfig.debug +1
-1bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default yThis kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.
I'd *MUCH* rather hide KMSAN:
config KMSAN
bool "KMSAN: detector of uninitialized values use" depends onHAVE_ARCH_KMSAN&&HAVE_KMSAN_COMPILERdepends onDEBUG_KERNEL&& !KASAN && !KCSAN depends on !PREEMPT_RT(quoted diff) +1
-0+ depends on !LOCKDEP
Quoted 5 lines
Because, frankly, lockdep is way more important than KMSAN.
But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in
pfn_valid()than no lockdep at all.
The only locks used in pfn_valid() are rcu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replace rcu_read_lock_sched() with preempt_disable() and rcu_read_unlock_sched() with preempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a special rcu_read_lock_sched() or call lockdep_recursion_inc() in instrumented pfn_valid() to disable lockdep temporarily.
[Cc Paul]
Regards,
Boqun
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen
lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORTlib/Kconfig.debug +1
-1bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default yThis kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.
I'd *MUCH* rather hide KMSAN:
config KMSAN
bool "KMSAN: detector of uninitialized values use" depends onHAVE_ARCH_KMSAN&&HAVE_KMSAN_COMPILERdepends onDEBUG_KERNEL&& !KASAN && !KCSAN depends on !PREEMPT_RT(quoted diff) +1
-0+ depends on !LOCKDEPQuoted 5 lines
Because, frankly, lockdep is way more important than KMSAN.
But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in
pfn_valid()than no lockdep at all.The only locks used in
pfn_valid()arercu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replacercu_read_lock_sched()withpreempt_disable()andrcu_read_unlock_sched()withpreempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a specialrcu_read_lock_sched()or calllockdep_recursion_inc()in instrumentedpfn_valid()to disable lockdep temporarily.
Quoted 1 line
[Cc Paul]
Don't tell me what? ;-)
An alternative is to use rcu_read_lock_sched_notrace() and rcu_read_unlock_sched_notrace(). If you really want to use preempt_disable() and preempt_enable() instead, you will likely want the _notrace() variants.
Thanx, Paul
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen
lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORTlib/Kconfig.debug +1
-1bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default yThis kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.
I'd *MUCH* rather hide KMSAN:
config KMSAN
bool "KMSAN: detector of uninitialized values use" depends onHAVE_ARCH_KMSAN&&HAVE_KMSAN_COMPILERdepends onDEBUG_KERNEL&& !KASAN && !KCSAN depends on !PREEMPT_RT(quoted diff) +1
-0+ depends on !LOCKDEPQuoted 5 lines
Because, frankly, lockdep is way more important than KMSAN.
But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in
pfn_valid()than no lockdep at all.The only locks used in
pfn_valid()arercu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replacercu_read_lock_sched()withpreempt_disable()andrcu_read_unlock_sched()withpreempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a specialrcu_read_lock_sched()or calllockdep_recursion_inc()in instrumentedpfn_valid()to disable lockdep temporarily.
Quoted 4 lines
Quoted 1 line
[Cc Paul]
Don't tell me what? ;-)
Turn out that telling you is a good idea ;-)
Quoted 2 lines
An alternative is to use
rcu_read_lock_sched_notrace()andrcu_read_unlock_sched_notrace(). If you really want to use
Yes, I think this is better than what I proposed.
Regards,
Boqun
Quoted 4 lines
preempt_disable()andpreempt_enable()instead, you will likely want the_notrace()variants.Thanx, Paul
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
Quoted 44 lines · Boqun Feng
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen
lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORT
boollib/Kconfig.debug +1
-1- depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default yThis kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.
I'd *MUCH* rather hide KMSAN:
config KMSAN
bool "KMSAN: detector of uninitialized values use" depends onHAVE_ARCH_KMSAN&&HAVE_KMSAN_COMPILERdepends onDEBUG_KERNEL&& !KASAN && !KCSAN depends on !PREEMPT_RT
+ depends on !LOCKDEPQuoted 5 lines
Because, frankly, lockdep is way more important than KMSAN.
But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in
pfn_valid()than no lockdep at all.The only locks used in
pfn_valid()arercu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replacercu_read_lock_sched()withpreempt_disable()andrcu_read_unlock_sched()withpreempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a specialrcu_read_lock_sched()or calllockdep_recursion_inc()in instrumentedpfn_valid()to disable lockdep temporarily.Quoted 4 lines
Quoted 1 line
[Cc Paul]
Don't tell me what? ;-)
Quoted 6 lines
Turn out that telling you is a good idea ;-)
Quoted 2 lines
An alternative is to use
rcu_read_lock_sched_notrace()andrcu_read_unlock_sched_notrace(). If you really want to useYes, I think this is better than what I proposed.
Thanks for your comments!
Yes, that's what I was actually looking into after Dave's answer on the other thread
(https://groups.google.com/g/kasan-dev/c/ZBiGzZL36-I/m/WtNuKqP9EQAJ)
I'll still need to rework the code calling virt_to_page() to avoid deadlocks from there though.
Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN
In reply to Alexander Potapenkolib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343
Quoted 5 lines · Alexander Potapenko
config
LOCK_DEBUGGING_SUPPORTlib/Kconfig.debug +1
-1bool - depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT + depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN default y
This kinda stinks. It ends up doubling the amount of work that ks
PATCH 3/3 x86/traps: fix an objtool warning in handle_bug()
In reply to Alexander PotapenkoBecause handle_bug() is a noinstr function, call to kmsan_unpoison_entry_regs() should be happening within the instrumentation_begin()/instrumentation_end() region.
Fortunately, the same noinstr annotation lets us dereference @regs in handle_bug() without unpoisoning them, so we don't have to move the is_valid_bugaddr(regs->ip) check below instrumentation_begin().
--- arch/x86/kernel/traps.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/kernel/traps.c +9 -6
@@ -217,12 +217,6 @@ static noinstr bool handle_bug(struct pt_regs *regs) {bool handled = false; -/* -* Normally @regs are unpoisoned by irqentry_enter(), but handle_bug() -* is a rare case that uses @regs without passing them to -* irqentry_enter(). -*/ -kmsan_unpoison_entry_regs (regs);if (!is_valid_bugaddr (regs->ip))return handled;
@@ -230,6 +224,15 @@ static noinstr bool handle_bug(struct pt_regs *regs)* All lies, just get the WARN/BUG out. */ instrumentation_begin (); +/* +* Normally @regs are unpoisoned by irqentry_enter(), but handle_bug() +* is a rare case that uses @regs without passing them to +* irqentry_enter(). +* Unpoisoning of @regs should be done before the first access to it, +* but because this is a noinstr function it is fine to postpone +* unpoisoning until the call of instrumentation_begin(). +*/ +kmsan_unpoison_entry_regs (regs);/* * Since we're emulating a CALL with exceptions, restore the interrupt * state to what it was at the exception site.
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
Re: [PATCH 3/3] x86/traps: fix an objtool warning in handle_bug()
Quoted 7 lines · Alexander Potapenko
Because
handle_bug()is a noinstr function, call tokmsan_unpoison_entry_regs()should be happening within the instrumentation_begin()/instrumentation_end() region.Fortunately, the same noinstr annotation lets us dereference @regs in
handle_bug()without unpoisoning them, so we don't have to move theis_valid_bugaddr(regs->ip)check belowinstrumentation_begin().
Imperative mood, please. And capitalize "fix" in the subject.
Quoted 4 lines
Otherwise, looks good.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 9 lines · Alexander Potapenko
Enabling
CONFIG_DEBUG_VIRTUAL=ytogether with KMSAN led to infinite recursion, becausekmsan_get_metadata()ended up calling instrumented__pfn_valid()fromarch/x86/mm/physaddr.c.Prevent it by disabling instrumentation of the whole file.
Reported-by Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Potapenko <[email protected]>
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 5 lines · Alexander Potapenko
Enabling
CONFIG_DEBUG_VIRTUAL=ytogether with KMSAN led to infinite recursion, becausekmsan_get_metadata()ended up calling instrumented__pfn_valid()fromarch/x86/mm/physaddr.c.Prevent it by disabling instrumentation of the whole file.
This does seem rather ad-hoc. It's the same basic reason we have "noinstr": code instrumentation infrastructure uses generally can't be instrumented itself.
How hard would it be to make sure that kmsan_get_metadata() and friends don't call any symbols that were compiled with -fsanitize=kernel-memory?
I do also think I'd much rather see __no_kmsan_checks on the functions than doing whole files. I *guarantee* if code gets moved around that whoever does it will miss the KMSAN_SANITIZE_physaddr.o in the makefile.