Skip to contentSkip to conversation

UBSAN: run-time undefined behavior sanity checker

DISCUSSION Dave Jones 2 participants 6 messages Jan 22, 2016 – Feb 1, 2016 2 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

Dave Jones

author
#1 lore raw
In reply to a message not in the archive
Quoted 16 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4
Commit:     c6d308534aef6c99904bf5862066360ae067abc4
Parent:     68920c973254c5b71a684645c5f6f82d6732c5d6
Refname:    refs/heads/master
Author:     Andrey Ryabinin <[email protected]>
AuthorDate: Wed Jan 20 15:00:55 2016 -0800
Committer:  Linus Torvalds <[email protected]>
CommitDate: Wed Jan 20 17:09:18 2016 -0800
    UBSAN: run-time undefined behavior sanity checker

UBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.

Quoted 7 lines

So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.

GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.

If I enable this and CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.

Compiler is debian unstable's 5.3.1 20160114

I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.

That aside though, neat feature. I look forward to breaking kernels with it :)

Dave

#2 lore raw
Quoted 25 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4
Commit:     c6d308534aef6c99904bf5862066360ae067abc4
Parent:     68920c973254c5b71a684645c5f6f82d6732c5d6
Refname:    refs/heads/master
Author:     Andrey Ryabinin <[email protected]>
AuthorDate: Wed Jan 20 15:00:55 2016 -0800
Committer:  Linus Torvalds <[email protected]>
CommitDate: Wed Jan 20 17:09:18 2016 -0800
    UBSAN: run-time undefined behavior sanity checker

UBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.

Quoted 7 lines

So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.

GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.

Quoted 9 lines

If I enable this and CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.

Compiler is debian unstable's 5.3.1 20160114

I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.

Likely caused by unaligned access in very early code, which ends up in too early printk() call. You could try to disable instrumentation (UBSAN_SANITIZE := n) in early code.

Be aware that CONFIG_UBSAN_ALIGNMENT causes a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.

author
#3 lore raw
Quoted 36 lines · Andrey Ryabinin
Quoted 25 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4
Commit:     c6d308534aef6c99904bf5862066360ae067abc4
Parent:     68920c973254c5b71a684645c5f6f82d6732c5d6
Refname:    refs/heads/master
Author:     Andrey Ryabinin <[email protected]>
AuthorDate: Wed Jan 20 15:00:55 2016 -0800
Committer:  Linus Torvalds <[email protected]>
CommitDate: Wed Jan 20 17:09:18 2016 -0800
    UBSAN: run-time undefined behavior sanity checker

UBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.

Quoted 7 lines

So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.

GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.

Quoted 9 lines

If I enable this and CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.

Compiler is debian unstable's 5.3.1 20160114

I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.

Quoted 5 lines

Likely caused by unaligned access in very early code, which ends up in too early printk() call. You could try to disable instrumentation (UBSAN_SANITIZE := n) in early code.

Be aware that CONFIG_UBSAN_ALIGNMENT causes a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.

So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?

Dave

#4 lore raw
Quoted 37 lines · Dave Jones
Quoted 36 lines · Andrey Ryabinin
Quoted 34 lines · Dave Jones
Quoted 16 lines · Linux Kernel
Web:        https://git.kernel.org/torvalds/c/c6d308534aef6c99904bf5862066360ae067abc4
Commit:     c6d308534aef6c99904bf5862066360ae067abc4
Parent:     68920c973254c5b71a684645c5f6f82d6732c5d6
Refname:    refs/heads/master
Author:     Andrey Ryabinin <[email protected]>
AuthorDate: Wed Jan 20 15:00:55 2016 -0800
Committer:  Linus Torvalds <[email protected]>
CommitDate: Wed Jan 20 17:09:18 2016 -0800
    UBSAN: run-time undefined behavior sanity checker

UBSAN uses compile-time instrumentation to catch undefined behavior (UB). Compiler inserts code that perform certain kinds of checks before operations that could cause UB. If check fails (i.e. UB detected) __ubsan_handle_* function called to print error message.

Quoted 7 lines

So the most of the work is done by compiler. This patch just implements ubsan handlers printing errors.

GCC has this capability since 4.9.x [1] (see -fsanitize=undefined option and its suboptions).
However GCC 5.x has more checkers implemented [2]. Article [3] has a bit more details about UBSAN in the GCC.

If I enable this and CONFIG_UBSAN_ALIGNMENT, the kernel doesn't boot, and hangs really early (pretty much as soon as I hit return in grub) far too early for serial console or even tty output.

Compiler is debian unstable's 5.3.1 20160114

I don't know if this is worth chasing down, I chose to just disable it, but figured I'd post in case other people stumble across the same issue.

Quoted 9 lines
Quoted 5 lines

Likely caused by unaligned access in very early code, which ends up in too early printk() call. You could try to disable instrumentation (UBSAN_SANITIZE := n) in early code.

Be aware that CONFIG_UBSAN_ALIGNMENT causes a *lot* of spam in dmesg. Since x86 supports unaligned accesses, the significant amount of that spam just a false-positive reports.

So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?

Try to disable instrumentation for early code, like in the patch bellow. Also send me you .config please. Perhaps I will be able to reproduce this.

arch/x86/kernel/Makefile +2 -0
index b1b78ff..d39a954 100644
@@ -20,6 +20,8 @@ KASAN_SANITIZE_head$(BITS).o := n
 KASAN_SANITIZE_dumpstack.o := n
 KASAN_SANITIZE_dumpstack_$(BITS).o := n
 
+UBSAN_SANITIZE := n
+
 CFLAGS_irq.o := -I$(src)/../include/asm/trace
 
 obj-y                  := process_$(BITS).o signal.o
#6 lore raw
In reply to a message not in the archive
Quoted 22 lines · Dave Jones

<off-list because of huge config>

Quoted 18 lines · Andrey Ryabinin
Quoted 3 lines

So disabling that option fixed booting on one machine, but every other I've tried it on hangs the same way, really early. Any thoughts on how to chase this down ?

Try to disable instrumentation for early code, like in the patch bellow.

arch/x86/kernel/Makefile +2 -0
index b1b78ff..d39a954 100644
@@ -20,6 +20,8 @@ KASAN_SANITIZE_head$(BITS).o := n
 KASAN_SANITIZE_dumpstack.o := n
 KASAN_SANITIZE_dumpstack_$(BITS).o := n
 
+UBSAN_SANITIZE := n
+
 CFLAGS_irq.o := -I$(src)/../include/asm/trace
 
Quoted 9 lines

arch/x86/kernel/Makefile KASAN_SANITIZE_head$(BITS).o := n · line 27

Quoted 1 line
arch/x86/kernel/Makefile +0 -0
 obj-y                  := process_$(BITS).o signal.o

This didn't help.

Quoted 1 line

Also send me you .config please. Perhaps I will be able to reproduce this.

below. Though I diffed a similar config from a machine where UBSAN works, and the only differences seemed to be mostly benign stuff or hw specific drivers.

So after I enabled UBSAN_ALIGNMENT in your config, the kernel didn't boot. That is because unaligned access happens before lockdep_init() so ubsan callback takes the spinlock before locked_init() which is not allowed.

As far as I understood most of your machines doesn't boot even without UBSAN_ALIGNMENT. So I'm guessing it might be similar problem.

Could you try it without CONFIG_DEBUG_LOCKDEP?
Or alternatively with patch like this:

lib/ubsan.c +3 -3
index 8799ae5..220e9d9 100644
@@ -146,13 +146,13 @@ static bool location_is_valid(struct source_location *loc)
        return loc->file_name != NULL;
 }
 
-static DEFINE_SPINLOCK(report_lock);
+//static DEFINE_SPINLOCK(report_lock);
 
 static void ubsan_prologue(struct source_location *location,
                        unsigned long *flags)
 {
        current->in_ubsan++;
-       spin_lock_irqsave(&report_lock, *flags);
+//     spin_lock_irqsave(&report_lock, *flags);
 
        pr_err("========================================"
                "========================================\n");
@@ -164,7 +164,7 @@ static void ubsan_epilogue(unsigned long *flags)
        dump_stack();
        pr_err("========================================"
                "========================================\n");
-       spin_unlock_irqrestore(&report_lock, *flags);
+//     spin_unlock_irqrestore(&report_lock, *flags);
        current->in_ubsan--;
 }
 
author
#7 lore raw
Quoted 10 lines · Andrey Ryabinin
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

tried it on hangs the same way, really early. Any thoughts on how to chase this down ?

Try to disable instrumentation for early code, like in the patch bellow.

This didn't help.

So after I enabled UBSAN_ALIGNMENT in your config, the kernel didn't boot. That is because unaligned access happens before lockdep_init() so ubsan callback takes the spinlock before locked_init() which is not allowed.

Could you try it without CONFIG_DEBUG_LOCKDEP?
Or alternatively with patch like this:

Tried with the patch, seems to work fine now.

thanks,
Dave