UBSAN: run-time undefined behavior sanity checker
Subsystems (3)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Linux Kernel
Web: https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4 Commit: c6d308534aef6c99904bf5862066360ae067abc4 Parent: 68920c973254c5b71a684645c5f6f82d6732c5d6 Refname: refs/heads/master Author: Andrey Ryabinin <[email protected]> AuthorDate: Wed Jan 20 15:00:55 2016 -0800 Committer: Linus Torvalds <[email protected]> CommitDate: Wed Jan 20 17:09:18 2016 -0800UBSAN: run-time undefined behavior sanity checkerUBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.
Quoted 7 lines
So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.
GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.
If I enable this and CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.
Compiler is debian unstable's 5.3.1 20160114
I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.
That aside though, neat feature. I look forward to breaking kernels with it :)
Dave
Quoted 25 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web: https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4 Commit: c6d308534aef6c99904bf5862066360ae067abc4 Parent: 68920c973254c5b71a684645c5f6f82d6732c5d6 Refname: refs/heads/master Author: Andrey Ryabinin <[email protected]> AuthorDate: Wed Jan 20 15:00:55 2016 -0800 Committer: Linus Torvalds <[email protected]> CommitDate: Wed Jan 20 17:09:18 2016 -0800UBSAN: run-time undefined behavior sanity checkerUBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.
Quoted 7 lines
So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.
GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.
Quoted 9 lines
If I enable this and
CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.Compiler is debian unstable's 5.3.1 20160114
I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.
Likely caused by unaligned access in very early code, which ends up in too early printk() call. You could try to disable instrumentation (UBSAN_SANITIZE := n) in early code.
Be aware that CONFIG_UBSAN_ALIGNMENT causes a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.
Quoted 36 lines · Andrey Ryabinin
Quoted 25 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web: https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4 Commit: c6d308534aef6c99904bf5862066360ae067abc4 Parent: 68920c973254c5b71a684645c5f6f82d6732c5d6 Refname: refs/heads/master Author: Andrey Ryabinin <[email protected]> AuthorDate: Wed Jan 20 15:00:55 2016 -0800 Committer: Linus Torvalds <[email protected]> CommitDate: Wed Jan 20 17:09:18 2016 -0800UBSAN: run-time undefined behavior sanity checkerUBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.
Quoted 7 lines
So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.
GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.Quoted 9 lines
If I enable this and
CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.Compiler is debian unstable's 5.3.1 20160114
I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.
Quoted 5 lines
Likely caused by unaligned access in very early code, which ends up in too early
printk()call. You could try to disable instrumentation (UBSAN_SANITIZE:= n) in early code.Be aware that
CONFIG_UBSAN_ALIGNMENTcauses a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.
So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?
Dave
Quoted 37 lines · Dave Jones
Quoted 36 lines · Andrey Ryabinin
Quoted 34 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web: https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4 Commit: c6d308534aef6c99904bf5862066360ae067abc4 Parent: 68920c973254c5b71a684645c5f6f82d6732c5d6 Refname: refs/heads/master Author: Andrey Ryabinin <[email protected]> AuthorDate: Wed Jan 20 15:00:55 2016 -0800 Committer: Linus Torvalds <[email protected]> CommitDate: Wed Jan 20 17:09:18 2016 -0800UBSAN: run-time undefined behavior sanity checkerUBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.
Quoted 7 lines
So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.
GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.If I enable this and
CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.Compiler is debian unstable's 5.3.1 20160114
I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.
Quoted 9 lines
Quoted 5 lines
Likely caused by unaligned access in very early code, which ends up in too early
printk()call. You could try to disable instrumentation (UBSAN_SANITIZE:= n) in early code.Be aware that
CONFIG_UBSAN_ALIGNMENTcauses a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?
Try to disable instrumentation for early code, like in the patch bellow. Also send me you .config please. Perhaps I will be able to reproduce this.
arch/x86/kernel/Makefile +2 -0
@@ -20,6 +20,8 @@ KASAN_SANITIZE_head$(BITS).o := n KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n +UBSAN_SANITIZE := n + CFLAGS_irq.o := -I$(src)/../include/asm/trace obj-y := process_$(BITS).o signal.o
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 22 lines · Dave Jones
<off-list because of huge config>
Quoted 18 lines · Andrey Ryabinin
Quoted 3 lines
So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?
Try to disable instrumentation for early code, like in the patch bellow.
arch/x86/kernel/Makefile +2
-0@@ -20,6 +20,8 @@ KASAN_SANITIZE_head$(BITS).o := n KASAN_SANITIZE_dumpstack.o := n KASAN_SANITIZE_dumpstack_$(BITS).o := n +UBSAN_SANITIZE := n + CFLAGS_irq.o := -I$(src)/../include/asm/trace
Quoted 9 lines
arch/x86/kernel/Makefile KASAN_SANITIZE_head$(BITS).o := n · line 27
Quoted 1 line
arch/x86/kernel/Makefile +0
-0obj-y := process_$(BITS).o signal.oThis didn't help.
Quoted 1 line
Also send me you .config please. Perhaps I will be able to reproduce this.
below. Though I diffed a similar config from a machine where UBSAN works, and the only differences seemed to be mostly benign stuff or hw specific drivers.
So after I enabled UBSAN_ALIGNMENT in your config, the kernel didn't boot. That is because unaligned access happens before lockdep_init() so ubsan callback takes the spinlock before locked_init() which is not allowed.
As far as I understood most of your machines doesn't boot even without UBSAN_ALIGNMENT. So I'm guessing it might be similar problem.
Could you try it without CONFIG_DEBUG_LOCKDEP?
Or alternatively with patch like this:
lib/ubsan.c +3 -3
@@ -146,13 +146,13 @@ static bool location_is_valid(struct source_location *loc)return loc->file_name !=NULL ; } -static DEFINE_SPINLOCK (report_lock); +//static DEFINE_SPINLOCK(report_lock); static void ubsan_prologue (struct source_location *location,unsigned long *flags) { current->in_ubsan++; -spin_lock_irqsave (&report_lock, *flags); +// spin_lock_irqsave(&report_lock, *flags); pr_err ("========================================" "========================================\n" );
@@ -164,7 +164,7 @@ static void ubsan_epilogue(unsigned long *flags)dump_stack ();pr_err ("========================================" "========================================\n" ); -spin_unlock_irqrestore (&report_lock, *flags); +// spin_unlock_irqrestore(&report_lock, *flags); current->in_ubsan--; }
Quoted 10 lines · Andrey Ryabinin
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
tried it on hangs the same way, really early. Any thoughts on how to chase this down ?
Try to disable instrumentation for early code, like in the patch bellow.
This didn't help.
So after I enabled
UBSAN_ALIGNMENTin your config, the kernel didn't boot. That is because unaligned access happens beforelockdep_init()so ubsan callback takes the spinlock beforelocked_init()which is not allowed.Could you try it without
CONFIG_DEBUG_LOCKDEP?
Or alternatively with patch like this:
Tried with the patch, seems to work fine now.
thanks,
Dave