Skip to contentSkip to conversation

stackleak: Disable ftrace for stackleak.c

PATCH In review Alexander Popov 4 participants 10 messages Nov 10, 2018 – Nov 13, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 stackleak: Disable ftrace for stackleak.c

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Let's disable ftrace for stackleak.c to avoid such situations.

---
 kernel/Makefile | 1 +
 1 file changed, 1 insertion(+)
kernel/Makefile +1 -0
index 7343b3a..0906f6d 100644
@@ -18,6 +18,7 @@ obj-$(CONFIG_MULTIUSER) += groups.o
 ifdef CONFIG_FUNCTION_TRACER
 # Do not trace internal ftrace files
 CFLAGS_REMOVE_irq_work.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_stackleak.o = $(CC_FLAGS_FTRACE)
 endif
 
 # Prevents flicker of uninteresting __do_softirq()/__local_bh_disable_ip()
Signature · 1 line
2.7.4
#1 lore raw
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

-- Steve

#2 lore raw
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself. That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Best regards,
Alexander

#3 lore raw
Quoted 13 lines · On Sun, 11 Nov 2018 13:19:45 +0300 Alexander Popov <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

Quoted 8 lines

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself. That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Have you tried adding a "notrace" to stackleak_erase()?

Not tracing the entire file is a bit of overkill. There's no reason ftrace can't trace stack_erasing_sysctl() or perhaps even stackleak_track_stack() as that may be very interesting to trace.

-- Steve

#4 lore raw

Hi Alexander and Steve,

Quoted 22 lines · On Sun, 11 Nov 2018 20:53:51 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 19 lines · On Sun, 11 Nov 2018 13:19:45 +0300 Alexander Popov <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself. That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

Quoted 8 lines
Quoted 2 lines

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Have you tried adding a "notrace" to stackleak_erase()?

Not tracing the entire file is a bit of overkill. There's no reason ftrace can't trace stack_erasing_sysctl() or perhaps even stackleak_track_stack() as that may be very interesting to trace.

I think it is not enough for stopping kprobes. If you want to stop the kprobes (int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase), since kprobes can work without ftrace.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#5 lore raw
Quoted 32 lines · ----- On Nov 11, 2018, at 9:50 PM, Masami Hiramatsu [email protected] wrote:

Hi Alexander and Steve,

Quoted 27 lines · On Sun, 11 Nov 2018 20:53:51 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 13 lines · On Sun, 11 Nov 2018 13:19:45 +0300 Alexander Popov <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

Quoted 8 lines

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself. That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Have you tried adding a "notrace" to stackleak_erase()?

Quoted 8 lines
Quoted 3 lines

Not tracing the entire file is a bit of overkill. There's no reason ftrace can't trace stack_erasing_sysctl() or perhaps even stackleak_track_stack() as that may be very interesting to trace.

I think it is not enough for stopping kprobes. If you want to stop the kprobes (int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase),
since kprobes can work without ftrace.

Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace" annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase), so neither function tracing nor kprobes can hook on that function.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#6 lore raw
Quoted 3 lines · On Mon, 12 Nov 2018 09:53:29 -0500 (EST) Mathieu Desnoyers…

Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace" annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase), so neither function tracing nor kprobes can hook on that function.

Correct.

-- Steve

#7 lore raw
In reply to Masami Hiramatsu

Hello Steven and Masami,

Thanks for your comments.

Quoted 35 lines · Masami Hiramatsu

Hi Alexander and Steve,

Quoted 30 lines · On Sun, 11 Nov 2018 20:53:51 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 13 lines · On Sun, 11 Nov 2018 13:19:45 +0300 Alexander Popov <[email protected]> wrote:
Quoted 11 lines · Steven Rostedt
Quoted 3 lines · On Sun, 11 Nov 2018 01:05:30 +0300 Alexander Popov <[email protected]> wrote:

The stackleak_erase() function is called on the trampoline stack at the end of syscall. This stack is not big enough for ftrace operations, e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().

Is the issue with kprobes or with function tracing? Because this stops function tracing which I only want disabled if function tracing itself is an issue, not for other things that may use the function tracing infrastructure.

Quoted 8 lines

Hello Steven,

I believe that stackleak erasing is not compatible with function tracing itself. That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1

I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4

Have you tried adding a "notrace" to stackleak_erase()?

Not tracing the entire file is a bit of overkill. There's no reason ftrace can't trace stack_erasing_sysctl() or perhaps even stackleak_track_stack() as that may be very interesting to trace.

Yes, thank you. It's much better.

Quoted 3 lines

I think it is not enough for stopping kprobes. If you want to stop the kprobes (int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase), since kprobes can work without ftrace.

Thanks!

I learned how to use kprobes without ftrace and managed to reproduce the problem as well (I modified kprobe_example.c and kretprobe_example.c). NOKPROBE_SYMBOL() allowed to avoid it.

I'll send the patch soon.

By the way, are there any other tracing/instrumentation mechanisms that should be disabled?

Best regards,
Alexander

#8 lore raw
Quoted 2 lines · On Mon, 12 Nov 2018 19:51:00 +0300 Alexander Popov <[email protected]> wrote:

By the way, are there any other tracing/instrumentation mechanisms that should be disabled?

ftrace and kprobes are pretty much the only ones that currently do self modification of code all over the kernel. Kprobes even more so than ftrace.

-- Steve

#9 lore raw
Quoted 9 lines · On Mon, 12 Nov 2018 12:21:48 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 2 lines · On Mon, 12 Nov 2018 19:51:00 +0300 Alexander Popov <[email protected]> wrote:

By the way, are there any other tracing/instrumentation mechanisms that should be disabled?

ftrace and kprobes are pretty much the only ones that currently do self modification of code all over the kernel. Kprobes even more so than ftrace.

Right, since kprobes uses int3 or sw breakpoint exception for hooking into the code, it consumes stack much more.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>