Skip to contentSkip to conversation

x86: mm: disable KMSAN instrumentation for physaddr.c

PATCH In review Alexander Potapenko 5 participants 13 messages Jun 21, 2024 – Jul 1, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 x86: mm: disable KMSAN instrumentation for physaddr.c

Enabling CONFIG_DEBUG_VIRTUAL=y together with KMSAN led to infinite recursion, because kmsan_get_metadata() ended up calling instrumented __pfn_valid() from arch/x86/mm/physaddr.c.

Prevent it by disabling instrumentation of the whole file.

---
 arch/x86/mm/Makefile | 1 +
 1 file changed, 1 insertion(+)
arch/x86/mm/Makefile +1 -0
index 8d3a00e5c528e..d3b27a383127d 100644
@@ -17,6 +17,7 @@ KCSAN_SANITIZE := n
 # Avoid recursion by not calling KMSAN hooks for CEA code.
 KMSAN_SANITIZE_cpu_entry_area.o := n
 KMSAN_SANITIZE_mem_encrypt_identity.o := n
+KMSAN_SANITIZE_physaddr.o := n
 
 ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_mem_encrypt.o		= -pg
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
#1 lore raw

PATCH 2/3 lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

At least on x86 KMSAN is seriously slown down by lockdep, as every pfn_valid() call (which is done on every instrumented memory access in the kernel) performs several lockdep checks, all of which, in turn, perform additional memory accesses and call KMSAN instrumentation.

Right now lockdep overflows the stack under KMSAN, but even if we use reentrancy counters to avoid the recursion on the KMSAN side, the slowdown from lockdep remains big enough for the kernel to become unusable.

---
 lib/Kconfig.debug | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/Kconfig.debug +1 -1
index 59b6765d86b8f..036905cf1dbe9 100644
@@ -1339,7 +1339,7 @@ menu "Lock Debugging (spinlocks, mutexes, etc...)"
 
 config LOCK_DEBUGGING_SUPPORT
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y
 
 config PROVE_LOCKING
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
Reviewed-by on v1 2/3
#2 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

Quoted 22 lines · Alexander Potapenko

At least on x86 KMSAN is seriously slown down by lockdep, as every pfn_valid() call (which is done on every instrumented memory access in the kernel) performs several lockdep checks, all of which, in turn, perform additional memory accesses and call KMSAN instrumentation.

Right now lockdep overflows the stack under KMSAN, but even if we use reentrancy counters to avoid the recursion on the KMSAN side, the slowdown from lockdep remains big enough for the kernel to become unusable.

---
 lib/Kconfig.debug | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/Kconfig.debug +0 -0
index 59b6765d86b8f..036905cf1dbe9 100644

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1342

Quoted 5 lines
lib/Kconfig.debug +1 -1
@@ -1339,7 +1339,7 @@ menu "Lock Debugging (spinlocks, mutexes, etc...)"
 
 config LOCK_DEBUGGING_SUPPORT
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN

Nit: no need to pile everything in one line. Add "depends on !KMSAN" on a separate line.

Otherwise, looks sane.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#3 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

In reply to Alexander Potapenko

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT

lib/Kconfig.debug +1 -1
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y

This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.

I'd *MUCH* rather hide KMSAN:

config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT
+ depends on !LOCKDEP

Because, frankly, lockdep is way more important than KMSAN.

But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.

#4 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

Quoted 21 lines · Dave Hansen

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT

lib/Kconfig.debug +1 -1
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y

This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.

I'd *MUCH* rather hide KMSAN:

config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT

(quoted diff) +1 -0
+	depends on !LOCKDEP
Quoted 5 lines

Because, frankly, lockdep is way more important than KMSAN.

But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.

The only locks used in pfn_valid() are rcu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replace rcu_read_lock_sched() with preempt_disable() and rcu_read_unlock_sched() with preempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a special rcu_read_lock_sched() or call lockdep_recursion_inc() in instrumented pfn_valid() to disable lockdep temporarily.

[Cc Paul]

Regards,
Boqun

#5 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT

lib/Kconfig.debug +1 -1
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y

This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.

I'd *MUCH* rather hide KMSAN:

config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT

(quoted diff) +1 -0
+	depends on !LOCKDEP
Quoted 5 lines

Because, frankly, lockdep is way more important than KMSAN.

But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.

The only locks used in pfn_valid() are rcu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replace rcu_read_lock_sched() with preempt_disable() and rcu_read_unlock_sched() with preempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a special rcu_read_lock_sched() or call lockdep_recursion_inc() in instrumented pfn_valid() to disable lockdep temporarily.

Quoted 1 line

[Cc Paul]

Don't tell me what? ;-)

An alternative is to use rcu_read_lock_sched_notrace() and rcu_read_unlock_sched_notrace(). If you really want to use preempt_disable() and preempt_enable() instead, you will likely want the _notrace() variants.

Thanx, Paul

#6 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT

lib/Kconfig.debug +1 -1
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y

This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.

I'd *MUCH* rather hide KMSAN:

config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT

(quoted diff) +1 -0
+	depends on !LOCKDEP
Quoted 5 lines

Because, frankly, lockdep is way more important than KMSAN.

But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.

The only locks used in pfn_valid() are rcu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replace rcu_read_lock_sched() with preempt_disable() and rcu_read_unlock_sched() with preempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a special rcu_read_lock_sched() or call lockdep_recursion_inc() in instrumented pfn_valid() to disable lockdep temporarily.

Quoted 4 lines
Quoted 1 line

[Cc Paul]

Don't tell me what? ;-)

Turn out that telling you is a good idea ;-)

Quoted 2 lines

An alternative is to use rcu_read_lock_sched_notrace() and rcu_read_unlock_sched_notrace(). If you really want to use

Yes, I think this is better than what I proposed.

Regards,
Boqun

Quoted 4 lines

preempt_disable() and preempt_enable() instead, you will likely want the _notrace() variants.

Thanx, Paul

#7 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

Quoted 44 lines · Boqun Feng
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Boqun Feng
Quoted 21 lines · Dave Hansen

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT
bool

lib/Kconfig.debug +1 -1
-       depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+       depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
        default y

This kinda stinks. Practically, it'll mean that anyone turning on KMSAN will accidentally turn off lockdep. That's really nasty, especially for folks who are turning on debug options left and right to track down nasty bugs.

I'd *MUCH* rather hide KMSAN:

config KMSAN
bool "KMSAN: detector of uninitialized values use" depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER depends on DEBUG_KERNEL && !KASAN && !KCSAN depends on !PREEMPT_RT
+ depends on !LOCKDEP

Quoted 5 lines

Because, frankly, lockdep is way more important than KMSAN.

But ideally, we'd allow them to coexist somehow. Have we even discussed the problem with the lockdep folks? For instance, I'd much rather have a relaxed lockdep with no checking in pfn_valid() than no lockdep at all.

The only locks used in pfn_valid() are rcu_read_lock_sched(), right? If so, could you try (don't tell Paul ;-)) replace rcu_read_lock_sched() with preempt_disable() and rcu_read_unlock_sched() with preempt_enable()? That would avoid calling into lockdep. If that works for KMSAN, we can either have a special rcu_read_lock_sched() or call lockdep_recursion_inc() in instrumented pfn_valid() to disable lockdep temporarily.

Quoted 4 lines
Quoted 1 line

[Cc Paul]

Don't tell me what? ;-)

Quoted 6 lines

Turn out that telling you is a good idea ;-)

Quoted 2 lines

An alternative is to use rcu_read_lock_sched_notrace() and rcu_read_unlock_sched_notrace(). If you really want to use

Yes, I think this is better than what I proposed.

Thanks for your comments!
Yes, that's what I was actually looking into after Dave's answer on the other thread
(https://groups.google.com/g/kasan-dev/c/ZBiGzZL36-I/m/WtNuKqP9EQAJ)
I'll still need to rework the code calling virt_to_page() to avoid deadlocks from there though.

#8 lore raw

Re: [PATCH 2/3] lib/Kconfig.debug: disable LOCK_DEBUGGING_SUPPORT under KMSAN

In reply to Alexander Potapenko

lib/Kconfig.debug menu "Lock Debugging (spinlocks, mutexes, etc...)" · line 1343

Quoted 5 lines · Alexander Potapenko

config LOCK_DEBUGGING_SUPPORT

lib/Kconfig.debug +1 -1
 	bool
-	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT
+	depends on TRACE_IRQFLAGS_SUPPORT && STACKTRACE_SUPPORT && LOCKDEP_SUPPORT && !KMSAN
 	default y

This kinda stinks. It ends up doubling the amount of work that ks

#9 lore raw

PATCH 3/3 x86/traps: fix an objtool warning in handle_bug()

In reply to Alexander Potapenko

Because handle_bug() is a noinstr function, call to kmsan_unpoison_entry_regs() should be happening within the instrumentation_begin()/instrumentation_end() region.

Fortunately, the same noinstr annotation lets us dereference @regs in handle_bug() without unpoisoning them, so we don't have to move the is_valid_bugaddr(regs->ip) check below instrumentation_begin().

---
 arch/x86/kernel/traps.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/kernel/traps.c +9 -6
index 4fa0b17e5043a..e8f330d9ba5d4 100644
@@ -217,12 +217,6 @@ static noinstr bool handle_bug(struct pt_regs *regs)
 {
 	bool handled = false;
 
-	/*
-	 * Normally @regs are unpoisoned by irqentry_enter(), but handle_bug()
-	 * is a rare case that uses @regs without passing them to
-	 * irqentry_enter().
-	 */
-	kmsan_unpoison_entry_regs(regs);
 	if (!is_valid_bugaddr(regs->ip))
 		return handled;
 
@@ -230,6 +224,15 @@ static noinstr bool handle_bug(struct pt_regs *regs)
 	 * All lies, just get the WARN/BUG out.
 	 */
 	instrumentation_begin();
+	/*
+	 * Normally @regs are unpoisoned by irqentry_enter(), but handle_bug()
+	 * is a rare case that uses @regs without passing them to
+	 * irqentry_enter().
+	 * Unpoisoning of @regs should be done before the first access to it,
+	 * but because this is a noinstr function it is fine to postpone
+	 * unpoisoning until the call of instrumentation_begin().
+	 */
+	kmsan_unpoison_entry_regs(regs);
 	/*
 	 * Since we're emulating a CALL with exceptions, restore the interrupt
 	 * state to what it was at the exception site.
Signature · 1 line
2.45.2.741.gdbec12cfda-goog
Reviewed-by on v1 3/3
#10 lore raw

Re: [PATCH 3/3] x86/traps: fix an objtool warning in handle_bug()

Quoted 7 lines · Alexander Potapenko

Because handle_bug() is a noinstr function, call to kmsan_unpoison_entry_regs() should be happening within the instrumentation_begin()/instrumentation_end() region.

Fortunately, the same noinstr annotation lets us dereference @regs in handle_bug() without unpoisoning them, so we don't have to move the is_valid_bugaddr(regs->ip) check below instrumentation_begin().

Imperative mood, please. And capitalize "fix" in the subject.

https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/Documentation/process/maintainer-tip.rst#n134

Quoted 4 lines

Otherwise, looks good.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
Reviewed-by on v1 1/3
#11 lore raw
In reply to Alexander Potapenko
Quoted 9 lines · Alexander Potapenko

Enabling CONFIG_DEBUG_VIRTUAL=y together with KMSAN led to infinite recursion, because kmsan_get_metadata() ended up calling instrumented __pfn_valid() from arch/x86/mm/physaddr.c.

Prevent it by disabling instrumentation of the whole file.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#12 lore raw
In reply to Alexander Potapenko
Quoted 5 lines · Alexander Potapenko

Enabling CONFIG_DEBUG_VIRTUAL=y together with KMSAN led to infinite recursion, because kmsan_get_metadata() ended up calling instrumented __pfn_valid() from arch/x86/mm/physaddr.c.

Prevent it by disabling instrumentation of the whole file.

This does seem rather ad-hoc. It's the same basic reason we have "noinstr": code instrumentation infrastructure uses generally can't be instrumented itself.

How hard would it be to make sure that kmsan_get_metadata() and friends don't call any symbols that were compiled with -fsanitize=kernel-memory?

I do also think I'd much rather see __no_kmsan_checks on the functions than doing whole files. I *guarantee* if code gets moved around that whoever does it will miss the KMSAN_SANITIZE_physaddr.o in the makefile.