Skip to contentSkip to conversation

Instrumentation and RCU

DISCUSSION Thomas Gleixner 10 participants 44 messages Mar 9, 2020 – Mar 17, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Folks,

I'm starting a new conversation because there are about 20 different threads which look at that problem in various ways and the information is so scattered that creating a coherent picture is pretty much impossible.

There are several problems to solve:

1) Fragile low level entry code

2) Breakpoint utilization

3) RCU idle

4) Callchain protection

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

This is very much related to #3..

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Thoughts?

Thanks,

tglx

maintainer
#1 lore raw
Quoted 24 lines · On Mon, 09 Mar 2020 18:02:32 +0100 Thomas Gleixner <[email protected]> wrote:

Folks,

I'm starting a new conversation because there are about 20 different threads which look at that problem in various ways and the information is so scattered that creating a coherent picture is pretty much impossible.

There are several problems to solve:

1) Fragile low level entry code

2) Breakpoint utilization

3) RCU idle

4) Callchain protection

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

Quoted 6 lines

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

This is very much related to #3..

Now for just entry portions (entering into interrupt context or coming from userspace into normal kernel code) I agree, and I'm fine if we can move things around and not trace these entry points if possible.

Quoted 8 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Quoted 41 lines

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Is this specifically to do with the kernel/trace/trace_preemptirqs.c code that was added by Joel?

Quoted 9 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

Note, if notrace is an issue it shows up pretty quickly, as just enabling function tracing will enable all non notrace locations, and if something shouldn't be traced, it will crash immediately.

I have a RCU option for ftrace ops to set, if it requires RCU to be watching, and in that case, it wont call the callback if RCU is not watching.

Quoted 6 lines

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

Quoted 6 lines

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Thoughts?

This can expand quite a bit. At least when I did something similar with NMIs, as there was a time I wanted to flag all places that could be called from NMI, and found that there's a lot of code that can be.

I can imagine the same for marking nokprobes as well. And I really don't want to make all notrace stop tracing the entire function and all that it can call, as that will go back to removing all callers from NMIs as do_nmi() itself is notrace.

-- Steve

#2 lore raw
Quoted 4 lines · Steven Rostedt

On Mon, 09 Mar 2020 18:02:32 +0100
Thomas Gleixner <[email protected]> wrote:

[...]

Quoted 47 lines
Quoted 40 lines

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Quoted 2 lines

Is this specifically to do with the kernel/trace/trace_preemptirqs.c code that was added by Joel?

Just started a vacation here and will be back on January 12th. Will take a detailed look at Thomas's email at that time.

Adding some more folks (Daniel, Valentin) who have used the preempt/irq tracepoints.

I agree we should reorder things and avoid these circular dependencies, it bothers me too. I am happy to help with any clean ups related to it. Let us definitely discuss more and fix it. Thanks.

- Joel

maintainer
#3 lore raw
Quoted 2 lines · On Mon, 9 Mar 2020 11:42:28 -0700 Joel Fernandes <[email protected]> wrote:

Just started a vacation here and will be back on January 12th. Will take a detailed look at Thomas's email at that time.

January 12th! Wow! Enjoy your sabbatical and see you next year! ;-)

-- Steve

#4 lore raw
Quoted 7 lines · ----- On Mar 9, 2020, at 3:07 PM, rostedt [email protected] wrote:
Quoted 2 lines · On Mon, 9 Mar 2020 11:42:28 -0700 Joel Fernandes <[email protected]> wrote:

Just started a vacation here and will be back on January 12th. Will take a detailed look at Thomas's email at that time.

January 12th! Wow! Enjoy your sabbatical and see you next year! ;-)

If Joel can go through 10 months worth of email backlog, he is my hero. ;-)

On a side-note, I confirmed with Joel that he intends to come back on March 13th.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw
In reply to Steven Rostedt
Quoted 7 lines · Steven Rostedt
Quoted 2 lines · On Mon, 9 Mar 2020 11:42:28 -0700 Joel Fernandes <[email protected]> wrote:

Just started a vacation here and will be back on January 12th. Will take a detailed look at Thomas's email at that time.

January 12th! Wow! Enjoy your sabbatical and see you next year! ;-)

Haha, I am seriously getting picked on for that. But anyway, I am back now because I missed you guys :-D

(Just kidding, obviously I meant March 12th).

thanks,

- Joel

#6 lore raw
In reply to Steven Rostedt
Quoted 14 lines · Steven Rostedt
Quoted 12 lines · On Mon, 09 Mar 2020 18:02:32 +0100 Thomas Gleixner <[email protected]> wrote:

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

Quoted 5 lines
Quoted 1 line

This is very much related to #3..

Now for just entry portions (entering into interrupt context or coming from userspace into normal kernel code) I agree, and I'm fine if we can move things around and not trace these entry points if possible.

I think we can. I spent quite some effort on splitting up the syscall entry/exit stuff so only the low level entry up to the point where RCU is watching and the low level exit where RCU is evtl. going back to idle plus the related ASM maze. It's quite some tedious work, that's why I came up with the section approach.

Quoted 11 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Yes, the break point stuff is unique, but it has nicely demonstrated how much of the code is affected by it.

Quoted 47 lines
Quoted 40 lines

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Quoted 2 lines

Is this specifically to do with the kernel/trace/trace_preemptirqs.c code that was added by Joel?

Don't know who it added, but yes, the above is what it does when it's expanded. I did some profiling with a trivial invalid syscall and trace_hardirqs_off() shows very prominent there, while with the approach I have in my entry series which does:

       lockdep_irq_off();
       enter_from_user_mode();
       __trace_hardirqs_off();

The tracing overhead goes down by ~ factor 2. That's a lot.

Quoted 5 lines
Quoted 5 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Quoted 7 lines
Quoted 3 lines

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

Note, if notrace is an issue it shows up pretty quickly, as just enabling function tracing will enable all non notrace locations, and if something shouldn't be traced, it will crash immediately.

Steven, you're not really serious about this, right? This is tinkering at best.

We have code pathes which are not necessarily covered in regular testing, depend on config options etc.

Have you ever looked at code coverage maps? There are quite some spots which we don't reach in testing.

So how do you explain the user that the blind spot he hit in the weird situation on his server which he wanted to analyze crashed his machine?

Having 'off limit' sections allows us to do proper tool based analysis with full coverage. That's really the only sensible approach.

Quoted 3 lines

I have a RCU option for ftrace ops to set, if it requires RCU to be watching, and in that case, it wont call the callback if RCU is not watching.

That's nice but does not solve the problem.

Quoted 12 lines
Quoted 11 lines

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Thoughts?

Quoted 8 lines

This can expand quite a bit. At least when I did something similar with NMIs, as there was a time I wanted to flag all places that could be called from NMI, and found that there's a lot of code that can be.

I can imagine the same for marking nokprobes as well. And I really don't want to make all notrace stop tracing the entire function and all that it can call, as that will go back to removing all callers from NMIs as do_nmi() itself is notrace.

The point is that you have something like this:

section "text.offlimit"

nmi()

{
        do_fragile_stuff_on_enter();

        offlimit_safecall(do_some_instrumentable_stuff());

        do_fragile_stuff_on_exit();
}

section "text"

do_some_instrumentable_stuff()
{
}

So if someone adds another call

section "text.offlimit"

nmi()

{
        do_fragile_stuff_on_enter();

        offlimit_safecall(do_some_instrumentable_stuff());

        do_some_other_instrumentable_stuff();

        do_fragile_stuff_on_exit();
}

which is also in section "text" then the analysis tool will find the missing offlimit_safecall() - or what ever method we chose to annotate that stuff. Surely not an annotation on the called function itself because that might be safe to call in one context but not in another.

These annotations are halfways easy to monitor for abuse and they should be prominent enough in the code that at least for the people dealing with that kind of code they act as a warning flag.

Thanks,

tglx

maintainer
#7 lore raw

Hi,

Quoted 8 lines · On Mon, 09 Mar 2020 19:59:18 +0100 Thomas Gleixner <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

Quoted 6 lines
Quoted 3 lines

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Yes, the break point stuff is unique, but it has nicely demonstrated how much of the code is affected by it.

I see. I had followed the callchain several times, and always found new function. So I agree with the off-limit section idea. That is a kind of entry code section but more generic one. It is natural to split such sensitive code in different place.

BTW, what about kdb stuffs? (+Cc Jason)

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

Agreed. That's the reason why I haven't add kprobe-fuzzer yet. It is easy to make a fuzzer for kprobes by ftrace (note that we need to enable CONFIG_KPROBE_EVENTS_ON_NOTRACE=y to check notrace functions), but there is no way to kick the target code. In the result, most of the kprobed functions are just not hit. I'm not sure such test code is reasonable or not.

Quoted 80 lines
Quoted 3 lines

Note, if notrace is an issue it shows up pretty quickly, as just enabling function tracing will enable all non notrace locations, and if something shouldn't be traced, it will crash immediately.

Steven, you're not really serious about this, right? This is tinkering at best.

We have code pathes which are not necessarily covered in regular testing, depend on config options etc.

Have you ever looked at code coverage maps? There are quite some spots which we don't reach in testing.

So how do you explain the user that the blind spot he hit in the weird situation on his server which he wanted to analyze crashed his machine?

Having 'off limit' sections allows us to do proper tool based analysis with full coverage. That's really the only sensible approach.

Quoted 3 lines

I have a RCU option for ftrace ops to set, if it requires RCU to be watching, and in that case, it wont call the callback if RCU is not watching.

That's nice but does not solve the problem.

Quoted 12 lines
Quoted 11 lines

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Thoughts?

Quoted 8 lines

This can expand quite a bit. At least when I did something similar with NMIs, as there was a time I wanted to flag all places that could be called from NMI, and found that there's a lot of code that can be.

I can imagine the same for marking nokprobes as well. And I really don't want to make all notrace stop tracing the entire function and all that it can call, as that will go back to removing all callers from NMIs as do_nmi() itself is notrace.

The point is that you have something like this:

section "text.offlimit"

nmi()

{
        do_fragile_stuff_on_enter();

        offlimit_safecall(do_some_instrumentable_stuff());

        do_fragile_stuff_on_exit();
}

section "text"

do_some_instrumentable_stuff()
{
}

So if someone adds another call

section "text.offlimit"

nmi()

{
        do_fragile_stuff_on_enter();

        offlimit_safecall(do_some_instrumentable_stuff());

        do_some_other_instrumentable_stuff();

        do_fragile_stuff_on_exit();
}
Quoted 4 lines

which is also in section "text" then the analysis tool will find the missing offlimit_safecall() - or what ever method we chose to annotate that stuff. Surely not an annotation on the called function itself because that might be safe to call in one context but not in another.

Hmm, what the offlimit_safecall() does? and what happen if the do_fragile_stuff_on_enter() invokes a library code? I think we also need to tweak kbuild to duplicate some library code to the off-limit text area.

Quoted 3 lines

These annotations are halfways easy to monitor for abuse and they should be prominent enough in the code that at least for the people dealing with that kind of code they act as a warning flag.

This off-limit text will be good for entries, but I think we still not able to remove all NOKPROBE_SYMBOLS with this.

For example __die() is marked a NOKPROBE because if we hit a recursive int3, it calls BUG() to dump stacks etc for debug. So that function must NOT probed. (I think we also should mark all backtrace functions in this case, but not yet) Would we move those backtrace related functions (including printk, and console drivers?) into the offlimit text too?

Hmm, if there is a bust_kprobes(), that can be easy to fix this issue.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#8 lore raw

Masami,

Quoted 18 lines · Masami Hiramatsu
Quoted 8 lines · On Mon, 09 Mar 2020 19:59:18 +0100 Thomas Gleixner <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

Quoted 6 lines
Quoted 3 lines

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Yes, the break point stuff is unique, but it has nicely demonstrated how much of the code is affected by it.

Quoted 6 lines

I see. I had followed the callchain several times, and always found new function. So I agree with the off-limit section idea. That is a kind of entry code section but more generic one. It is natural to split such sensitive code in different place.

BTW, what about kdb stuffs? (+Cc Jason)

That's yet another area of wreckage which nobody every looked at.

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

Quoted 6 lines

Agreed. That's the reason why I haven't add kprobe-fuzzer yet. It is easy to make a fuzzer for kprobes by ftrace (note that we need to enable CONFIG_KPROBE_EVENTS_ON_NOTRACE=y to check notrace functions), but there is no way to kick the target code. In the result, most of the kprobed functions are just not hit. I'm not sure such test code is reasonable or not.

Well, test code is always reasonable, but you have to be aware that code coverage is a really hard to solve problem with a code base as complex as the kernel.

Quoted 9 lines
Quoted 4 lines

which is also in section "text" then the analysis tool will find the missing offlimit_safecall() - or what ever method we chose to annotate that stuff. Surely not an annotation on the called function itself because that might be safe to call in one context but not in another.

Hmm, what the offlimit_safecall() does? and what happen if the do_fragile_stuff_on_enter() invokes a library code? I think we also need to tweak kbuild to duplicate some library code to the off-limit text area.

That's why we want the sections and the annotation. If something calls out of a noinstr section into a regular text section and the call is not annotated at the call site, then objtool can complain and tell you. What Peter and I came up with looks like this:

noinstr foo()

	do_protected(); <- Safe because in the noinstr section

	instr_begin();	<- Marks the begin of a safe region, ignored
        		   by objtool

        do_stuff();     <- All good   

        instr_end();    <- End of the safe region. objtool starts
			   looking again

        do_other_stuff();  <- Unsafe because do_other_stuff() is
        		      not protected

and:

noinstr do_protected()

        bar();		<- objtool will complain here

See?

Quoted 7 lines
Quoted 3 lines

These annotations are halfways easy to monitor for abuse and they should be prominent enough in the code that at least for the people dealing with that kind of code they act as a warning flag.

This off-limit text will be good for entries, but I think we still not able to remove all NOKPROBE_SYMBOLS with this.

Quoted 6 lines

For example __die() is marked a NOKPROBE because if we hit a recursive int3, it calls BUG() to dump stacks etc for debug. So that function must NOT probed. (I think we also should mark all backtrace functions in this case, but not yet) Would we move those backtrace related functions (including printk, and console drivers?) into the offlimit text too?

That's something we need to figure out and decide on. Some of this stuff sureley wants to be in the noinstr section. Other things might end up still being explicitely annotated, but that should be the exception not the rule.

Quoted 1 line

Hmm, if there is a bust_kprobes(), that can be easy to fix this issue.

That might help, but is obviously racy as hell.

Thanks,

tglx

#9 lore raw

----- On Mar 10, 2020, at 7:43 AM, Thomas Gleixner [email protected] wrote:

[...]

Quoted 19 lines

That's why we want the sections and the annotation. If something calls out of a noinstr section into a regular text section and the call is not annotated at the call site, then objtool can complain and tell you. What Peter and I came up with looks like this:

noinstr foo()

	do_protected(); <- Safe because in the noinstr section

	instr_begin();	<- Marks the begin of a safe region, ignored
       		   by objtool

       do_stuff();     <- All good

       instr_end();    <- End of the safe region. objtool starts
			   looking again

       do_other_stuff();  <- Unsafe because do_other_stuff() is
       		      not protected
Quoted 6 lines

and:

noinstr do_protected()

       bar();		<- objtool will complain here

See?

I think there are two distinct problems we are trying to solve here, and it would be good to spell them out to see which pieces of technical solution apply to which.

Problem #1) Tracer invoked from partially initialized kernel context

- Moving the early/late entry/exit points into sections invisible from instrumentation seems to make tons of sense for this.

Problem #2) Tracer recursion

- I'm much less convinced that hiding entry points from instrumentation works for this. As an example, with the isntr_begin/end() approach you propose above, as soon as you have a tracer recursing into itself because something below do_stuff() has been instrumented, having hidden the entry point did not help at all.

So I would be tempted to use the "hide entry/exit points" with explicit instr begin/end annotation to solve Problem #1, but I'm still thinking there is value in the per recursion context "in_tracing" flag to prevent tracer recursion.

Thoughts ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#10 lore raw
Quoted 17 lines · On Tue, 10 Mar 2020 11:31:51 -0400 (EDT) Mathieu Desnoyers…

I think there are two distinct problems we are trying to solve here, and it would be good to spell them out to see which pieces of technical solution apply to which.

Problem #1) Tracer invoked from partially initialized kernel context

- Moving the early/late entry/exit points into sections invisible from instrumentation seems to make tons of sense for this.

Problem #2) Tracer recursion

- I'm much less convinced that hiding entry points from instrumentation works for this. As an example, with the isntr_begin/end() approach you propose above, as soon as you have a tracer recursing into itself because something below do_stuff() has been instrumented, having hidden the entry point did not help at all.

Quoted 4 lines

So I would be tempted to use the "hide entry/exit points" with explicit instr begin/end annotation to solve Problem #1, but I'm still thinking there is value in the per recursion context "in_tracing" flag to prevent tracer recursion.

The only recursion issue that I've seen discussed is breakpoints. And that's outside of the tracer infrastructure. Basically, if someone added a breakpoint for a kprobe on something that gets called in the int3 code before kprobes is called we have (let's say rcu_nmi_enter()):

rcu_nmi_enter();
<int3>

     do_int3() {
        rcu_nmi_enter();
          <int3>
             do_int3();
                [..]

Where would a "in_tracer" flag help here? Perhaps a "in_breakpoint" could?

-- Steve

#11 lore raw
Quoted 33 lines · ----- On Mar 10, 2020, at 11:46 AM, rostedt [email protected] wrote:
Quoted 17 lines · On Tue, 10 Mar 2020 11:31:51 -0400 (EDT) Mathieu Desnoyers…

I think there are two distinct problems we are trying to solve here, and it would be good to spell them out to see which pieces of technical solution apply to which.

Problem #1) Tracer invoked from partially initialized kernel context

- Moving the early/late entry/exit points into sections invisible from instrumentation seems to make tons of sense for this.

Problem #2) Tracer recursion

- I'm much less convinced that hiding entry points from instrumentation works for this. As an example, with the isntr_begin/end() approach you propose above, as soon as you have a tracer recursing into itself because something below do_stuff() has been instrumented, having hidden the entry point did not help at all.

Quoted 4 lines

So I would be tempted to use the "hide entry/exit points" with explicit instr begin/end annotation to solve Problem #1, but I'm still thinking there is value in the per recursion context "in_tracing" flag to prevent tracer recursion.

The only recursion issue that I've seen discussed is breakpoints. And that's outside of the tracer infrastructure. Basically, if someone added a breakpoint for a kprobe on something that gets called in the int3 code before kprobes is called we have (let's say rcu_nmi_enter()):

rcu_nmi_enter();
<int3>

Quoted 7 lines
    do_int3() {
       rcu_nmi_enter();
         <int3>
            do_int3();
               [..]

Where would a "in_tracer" flag help here? Perhaps a "in_breakpoint" could?

An approach where the "in_tracer" flag is tested and set by the instrumentation (function tracer, kprobes, tracepoints) would work here. Let's say the beginning of the int3 ISR is part of the code which is invisible to instrumentation, and before we issue rcu_nmi_enter(), we handle the in_tracer flag:

rcu_nmi_enter();
<int3>

    (recursion_ctx->in_tracer == false)
    set recursion_ctx->in_tracer = true
    do_int3() {
       rcu_nmi_enter();
         <int3>
            if (recursion_ctx->in_tracer == true)
                iret

We can change "in_tracer" for "in_breakpoint", "in_tracepoint" and "in_function_trace" if we ever want to allow different types of instrumentation to nest. I'm not sure whether this is useful or not through.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#12 lore raw

Hi Mathieu,

Quoted 58 lines · On Tue, 10 Mar 2020 12:21:31 -0400 (EDT) Mathieu Desnoyers…
Quoted 33 lines · ----- On Mar 10, 2020, at 11:46 AM, rostedt [email protected] wrote:
Quoted 17 lines · On Tue, 10 Mar 2020 11:31:51 -0400 (EDT) Mathieu Desnoyers…

I think there are two distinct problems we are trying to solve here, and it would be good to spell them out to see which pieces of technical solution apply to which.

Problem #1) Tracer invoked from partially initialized kernel context

- Moving the early/late entry/exit points into sections invisible from instrumentation seems to make tons of sense for this.

Problem #2) Tracer recursion

- I'm much less convinced that hiding entry points from instrumentation works for this. As an example, with the isntr_begin/end() approach you propose above, as soon as you have a tracer recursing into itself because something below do_stuff() has been instrumented, having hidden the entry point did not help at all.

Quoted 4 lines

So I would be tempted to use the "hide entry/exit points" with explicit instr begin/end annotation to solve Problem #1, but I'm still thinking there is value in the per recursion context "in_tracing" flag to prevent tracer recursion.

The only recursion issue that I've seen discussed is breakpoints. And that's outside of the tracer infrastructure. Basically, if someone added a breakpoint for a kprobe on something that gets called in the int3 code before kprobes is called we have (let's say rcu_nmi_enter()):

rcu_nmi_enter();
<int3>

Quoted 7 lines
    do_int3() {
       rcu_nmi_enter();
         <int3>
            do_int3();
               [..]

Where would a "in_tracer" flag help here? Perhaps a "in_breakpoint" could?

An approach where the "in_tracer" flag is tested and set by the instrumentation (function tracer, kprobes, tracepoints) would work here. Let's say the beginning of the int3 ISR is part of the code which is invisible to instrumentation, and before we issue rcu_nmi_enter(), we handle the in_tracer flag:

rcu_nmi_enter();
<int3>

    (recursion_ctx->in_tracer == false)
    set recursion_ctx->in_tracer = true
    do_int3() {
       rcu_nmi_enter();
         <int3>
            if (recursion_ctx->in_tracer == true)
                iret
Quoted 3 lines

We can change "in_tracer" for "in_breakpoint", "in_tracepoint" and "in_function_trace" if we ever want to allow different types of instrumentation to nest. I'm not sure whether this is useful or not through.

Kprobes already has its own "in_kprobe" flag, and the recursion path is not so simple. Since the int3 replaces the original instruction, we have to execute the original instruction with single-step and fixup.

This means it involves do_debug() too. Thus, we can not do iret directly from do_int3 like above, but if recursion happens, we have no way to recover to origonal execution path (and call BUG()).

As my previous email, I showed a patch which is something like "bust_kprobes()" for oops path. That is not safe but no other way to escape from this recursion hell. (Maybe we can try to call it instead of calling BUG() so that the kernel can continue to run, but I'm not sure we can safely make the pagetable to readonly again.)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#13 lore raw

----- On Mar 10, 2020, at 8:18 PM, Masami Hiramatsu [email protected] wrote:
[...]

Quoted 19 lines
Quoted 15 lines

An approach where the "in_tracer" flag is tested and set by the instrumentation (function tracer, kprobes, tracepoints) would work here. Let's say the beginning of the int3 ISR is part of the code which is invisible to instrumentation, and before we issue rcu_nmi_enter(), we handle the in_tracer flag:

rcu_nmi_enter();
<int3>

    (recursion_ctx->in_tracer == false)
    set recursion_ctx->in_tracer = true
    do_int3() {
       rcu_nmi_enter();
         <int3>
            if (recursion_ctx->in_tracer == true)
                iret
Quoted 3 lines

We can change "in_tracer" for "in_breakpoint", "in_tracepoint" and "in_function_trace" if we ever want to allow different types of instrumentation to nest. I'm not sure whether this is useful or not through.

Quoted 7 lines

Kprobes already has its own "in_kprobe" flag, and the recursion path is not so simple. Since the int3 replaces the original instruction, we have to execute the original instruction with single-step and fixup.

This means it involves do_debug() too. Thus, we can not do iret directly from do_int3 like above, but if recursion happens, we have no way to recover to origonal execution path (and call BUG()).

I think that all the code involved when hitting a breakpoint which would be the minimal subset required to act as if the kprobe was not there in the first place (single-step, fixup) should be hidden from kprobes instrumentation. I suspect this is the current intent today with noprobe annotations, but Thomas' proposal brings this a step further.

However, any other kprobe code (and tracer callbacks) beyond that minimalistic "effect-less" kprobe could be protected by a per-recursion-context in_kprobe flag.

Quoted 5 lines

As my previous email, I showed a patch which is something like "bust_kprobes()" for oops path. That is not safe but no other way to escape from this recursion hell. (Maybe we can try to call it instead of calling BUG() so that the kernel can continue to run, but I'm not sure we can safely make the pagetable to readonly again.)

As long as we provide a minimalistic "effect-less" kprobe implementation in a non-instrumentable section which can be used whenever we are in a recursion scenario, I think we could achieve something recursion-free without requiring a bust_kprobes() work-around.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#14 lore raw
Quoted 36 lines · On Tue, 10 Mar 2020 20:37:41 -0400 (EDT) Mathieu Desnoyers…

----- On Mar 10, 2020, at 8:18 PM, Masami Hiramatsu [email protected] wrote:
[...]

Quoted 19 lines
Quoted 15 lines

An approach where the "in_tracer" flag is tested and set by the instrumentation (function tracer, kprobes, tracepoints) would work here. Let's say the beginning of the int3 ISR is part of the code which is invisible to instrumentation, and before we issue rcu_nmi_enter(), we handle the in_tracer flag:

rcu_nmi_enter();
<int3>

    (recursion_ctx->in_tracer == false)
    set recursion_ctx->in_tracer = true
    do_int3() {
       rcu_nmi_enter();
         <int3>
            if (recursion_ctx->in_tracer == true)
                iret
Quoted 3 lines

We can change "in_tracer" for "in_breakpoint", "in_tracepoint" and "in_function_trace" if we ever want to allow different types of instrumentation to nest. I'm not sure whether this is useful or not through.

Quoted 7 lines

Kprobes already has its own "in_kprobe" flag, and the recursion path is not so simple. Since the int3 replaces the original instruction, we have to execute the original instruction with single-step and fixup.

This means it involves do_debug() too. Thus, we can not do iret directly from do_int3 like above, but if recursion happens, we have no way to recover to origonal execution path (and call BUG()).

I think that all the code involved when hitting a breakpoint which would be the minimal subset required to act as if the kprobe was not there in the first place (single-step, fixup) should be hidden from kprobes instrumentation. I suspect this is the current intent today with noprobe annotations, but Thomas' proposal brings this a step further.

Quoted 3 lines

However, any other kprobe code (and tracer callbacks) beyond that minimalistic "effect-less" kprobe could be protected by a per-recursion-context in_kprobe flag.

Would you mean "in_kprobe" flag will prevent recursive execution of kprobes but not prevent other tracer like tracepoint? If so, it is already done I think. As I pointed, kprobe itself has in_kprobe like flag for checking re-entrance. Thus the kprobe handler can call the function which has a tracepoint safely.

Anyway, I agree with you to port all kprobe int3/debug handling parts to the effect-less (offlimit) area, except for its pre/post handlers.

Quoted 10 lines
Quoted 5 lines

As my previous email, I showed a patch which is something like "bust_kprobes()" for oops path. That is not safe but no other way to escape from this recursion hell. (Maybe we can try to call it instead of calling BUG() so that the kernel can continue to run, but I'm not sure we can safely make the pagetable to readonly again.)

As long as we provide a minimalistic "effect-less" kprobe implementation in a non-instrumentable section which can be used whenever we are in a recursion scenario, I think we could achieve something recursion-free without requiring a bust_kprobes() work-around.

Yeah, I hope so. The bust_kprobes() is something like an emergency escape hammer which everyone hopes never be used :)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#15 lore raw
In reply to Thomas Gleixner
Quoted 46 lines · On Tue, 10 Mar 2020 12:43:27 +0100 Thomas Gleixner <[email protected]> wrote:

Masami,

Quoted 18 lines · Masami Hiramatsu
Quoted 8 lines · On Mon, 09 Mar 2020 19:59:18 +0100 Thomas Gleixner <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

Quoted 6 lines
Quoted 3 lines

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Yes, the break point stuff is unique, but it has nicely demonstrated how much of the code is affected by it.

Quoted 6 lines

I see. I had followed the callchain several times, and always found new function. So I agree with the off-limit section idea. That is a kind of entry code section but more generic one. It is natural to split such sensitive code in different place.

BTW, what about kdb stuffs? (+Cc Jason)

That's yet another area of wreckage which nobody every looked at.

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

Quoted 6 lines

Agreed. That's the reason why I haven't add kprobe-fuzzer yet. It is easy to make a fuzzer for kprobes by ftrace (note that we need to enable CONFIG_KPROBE_EVENTS_ON_NOTRACE=y to check notrace functions), but there is no way to kick the target code. In the result, most of the kprobed functions are just not hit. I'm not sure such test code is reasonable or not.

Quoted 3 lines

Well, test code is always reasonable, but you have to be aware that code coverage is a really hard to solve problem with a code base as complex as the kernel.

Yes, especially, the corner case which we need to find is hard to cover.

Quoted 28 lines
Quoted 9 lines
Quoted 4 lines

which is also in section "text" then the analysis tool will find the missing offlimit_safecall() - or what ever method we chose to annotate that stuff. Surely not an annotation on the called function itself because that might be safe to call in one context but not in another.

Hmm, what the offlimit_safecall() does? and what happen if the do_fragile_stuff_on_enter() invokes a library code? I think we also need to tweak kbuild to duplicate some library code to the off-limit text area.

That's why we want the sections and the annotation. If something calls out of a noinstr section into a regular text section and the call is not annotated at the call site, then objtool can complain and tell you. What Peter and I came up with looks like this:

noinstr foo()

	do_protected(); <- Safe because in the noinstr section

	instr_begin();	<- Marks the begin of a safe region, ignored
        		   by objtool

        do_stuff();     <- All good   

        instr_end();    <- End of the safe region. objtool starts
			   looking again

        do_other_stuff();  <- Unsafe because do_other_stuff() is
        		      not protected
Quoted 6 lines

and:

noinstr do_protected()

        bar();		<- objtool will complain here

See?

OK, so this is for what the instr_begin() and instr_end() ensure the instrumentation safeness. Would you think this will also applied to notrace functons? I mean, how can large this section be.

It seems there are several different aspect (or level), RCU idle, kprobes recursive call(NOKPROBE), and ftrace recursive call(notrace). Would the offlimit section include all of them or some specific cases?

Quoted 14 lines
Quoted 7 lines
Quoted 3 lines

These annotations are halfways easy to monitor for abuse and they should be prominent enough in the code that at least for the people dealing with that kind of code they act as a warning flag.

This off-limit text will be good for entries, but I think we still not able to remove all NOKPROBE_SYMBOLS with this.

Quoted 6 lines

For example __die() is marked a NOKPROBE because if we hit a recursive int3, it calls BUG() to dump stacks etc for debug. So that function must NOT probed. (I think we also should mark all backtrace functions in this case, but not yet) Would we move those backtrace related functions (including printk, and console drivers?) into the offlimit text too?

Quoted 4 lines

That's something we need to figure out and decide on. Some of this stuff sureley wants to be in the noinstr section. Other things might end up still being explicitely annotated, but that should be the exception not the rule.

I think some library code can be duplicated in the offlimit section if needed. And stacktrace can also be a part of offlimit as like as a "service" call from normal world, since usually, stacktrace is for debugging.

Quoted 3 lines
Quoted 1 line

Hmm, if there is a bust_kprobes(), that can be easy to fix this issue.

That might help, but is obviously racy as hell.

I tried something like below. It needs a new set_memory_rw() like function which can be called from int3 context.

arch/x86/kernel/kprobes/core.c +16 -0
index 4d7022a740ab..7ab5bbc69cc9 100644
@@ -619,6 +619,9 @@ static void setup_singlestep(struct kprobe *p, struct pt_regs *regs,
 }
 NOKPROBE_SYMBOL(setup_singlestep);
 
+int __read_mostly kprobe_oops_in_progress;
+extern int __set_page_rw_raw(unsigned long addr);
+
 /*
  * We have reentered the kprobe_handler(), since another probe was hit while
  * within the handler. We save the original kprobes variables and just single
@@ -635,6 +638,17 @@ static int reenter_kprobe(struct kprobe *p, struct pt_regs *regs,
 		setup_singlestep(p, regs, kcb, 1);
 		break;
 	case KPROBE_REENTER:
+		if (unlikely(kprobe_oops_in_progress)) {
+			/*
+			 * We do not sync cores but there is no way to send
+			 * IPI from here. If other core hits int3, they can
+			 * recover by themselves.
+			 */
+			__set_page_rw_raw((unsigned long)p->addr);
+			*(p->addr) = p->ainsn.insn[0];
+			return 1;
+		}
+
 		/* A probe has been hit in the codepath leading up to, or just
 		 * after, single-stepping of a probed instruction. This entire
 		 * codepath should strictly reside in .kprobes.text section.
@@ -643,6 +657,8 @@ static int reenter_kprobe(struct kprobe *p, struct pt_regs *regs,
 		 */
 		pr_err("Unrecoverable kprobe detected.\n");
 		dump_kprobe(p);
+		/* Any further non-optimized kprobes are disabled forcibly */
+		kprobe_oops_in_progress++;
 		BUG();
 	default:
 		/* impossible cases */
arch/x86/mm/init_64.c +55 -0

67 lines not on this page · raw message

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#16 lore raw
In reply to Thomas Gleixner
Quoted 18 lines · Thomas Gleixner

That's why we want the sections and the annotation. If something calls out of a noinstr section into a regular text section and the call is not annotated at the call site, then objtool can complain and tell you. What Peter and I came up with looks like this:

noinstr foo()

	do_protected(); <- Safe because in the noinstr section

	instr_begin();	<- Marks the begin of a safe region, ignored
        		   by objtool

        do_stuff();     <- All good   

        instr_end();    <- End of the safe region. objtool starts
			   looking again

        do_other_stuff();  <- Unsafe because do_other_stuff() is
        		      not protected
Quoted 6 lines

and:

noinstr do_protected()

        bar();		<- objtool will complain here

See?

Find here:

https://lkml.kernel.org/r/[email protected]

#17 lore raw
In reply to Masami Hiramatsu
Quoted 5 lines · ----- On Mar 10, 2020, at 4:09 AM, Masami Hiramatsu [email protected] wrote:

Hi,

On Mon, 09 Mar 2020 19:59:18 +0100
Thomas Gleixner <[email protected]> wrote:

[...]

Quoted 16 lines
Quoted 4 lines

which is also in section "text" then the analysis tool will find the missing offlimit_safecall() - or what ever method we chose to annotate that stuff. Surely not an annotation on the called function itself because that might be safe to call in one context but not in another.

Hmm, what the offlimit_safecall() does? and what happen if the do_fragile_stuff_on_enter() invokes a library code? I think we also need to tweak kbuild to duplicate some library code to the off-limit text area.

Quoted 3 lines

These annotations are halfways easy to monitor for abuse and they should be prominent enough in the code that at least for the people dealing with that kind of code they act as a warning flag.

This off-limit text will be good for entries, but I think we still not able to remove all NOKPROBE_SYMBOLS with this.

Quoted 8 lines

For example __die() is marked a NOKPROBE because if we hit a recursive int3, it calls BUG() to dump stacks etc for debug. So that function must NOT probed. (I think we also should mark all backtrace functions in this case, but not yet) Would we move those backtrace related functions (including printk, and console drivers?) into the offlimit text too?

Hmm, if there is a bust_kprobes(), that can be easy to fix this issue.

In order to solve the recursion issue without losing instrumentation coverage of traps and significant system events, I really think we should look into adding some kind of "in_tracing" flag near each stack. We could name this "struct recursion_context" or something similar. We could then add this structure alongside each thread and ISR stack, but make sure traps (including breakpoints) still point to the recursion context of whatever caused the trap.

This should allow us to detect and prevent recursion from tracers directly at the instrumentation level.

Thoughts ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#18 lore raw
In reply to Masami Hiramatsu
Quoted 20 lines · Masami Hiramatsu

Hi,

Quoted 8 lines · On Mon, 09 Mar 2020 19:59:18 +0100 Thomas Gleixner <[email protected]> wrote:
Quoted 8 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

Quoted 6 lines
Quoted 3 lines

This is rather unique, and I agree that its best to at least get to a point where we limit the tracing within breakpoint code. I'm fine with making rcu_nmi_exit() nokprobe too.

Yes, the break point stuff is unique, but it has nicely demonstrated how much of the code is affected by it.

Quoted 6 lines

I see. I had followed the callchain several times, and always found new function. So I agree with the off-limit section idea. That is a kind of entry code section but more generic one. It is natural to split such sensitive code in different place.

BTW, what about kdb stuffs? (+Cc Jason)

There is some double breakpoint detection code but IIRC this merely retrospectively warns the user that they have their hurt their system... and whether the system would run long enough to reach that logic is relatively unlikely.

For both kdb and kgdb, the main "defence" is the use case. Neither kdb or kgdb faces the userspace (except via a SysRq, which can be disabled) and triggering either already implies the user is not especially concerned about things like availability guarantees since they are happy for everything running on the system to be halted indefinitely.

Additionally breakpoints in kgdb/kdb are not wildcarded so there are no need to worry about a user selecting a bad pattern! Setting a breakpoint with kgdb/kdb needs a user (who is assumed to have kernel knowledge) to have explicitly chose to study the dynamic behaviour of that particular bit of code.

I'm not saying kgdb/kdb would not benefit from additional safety barriers (it would), simply that the problem is less acute.

Daniel.

#19 lore raw
In reply to Thomas Gleixner
Quoted 24 lines · ----- On Mar 9, 2020, at 1:02 PM, Thomas Gleixner [email protected] wrote:

Folks,

I'm starting a new conversation because there are about 20 different threads which look at that problem in various ways and the information is so scattered that creating a coherent picture is pretty much impossible.

There are several problems to solve:

1) Fragile low level entry code

2) Breakpoint utilization

3) RCU idle

4) Callchain protection

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

Quoted 6 lines

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

This is very much related to #3..

Do I understand correctly that you intend on moving all kernel low level entry/exit code into sections which cannot be instrumented by kprobes nor the function tracer, and require explicit whitelisting, either through annotations or use of explicit tracepoints ?

Quoted 8 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

I think there is another question that arises here: the lack of automated continuous testing of the kprobes coverage. We have performed some testing of various random permutations of kprobes instrumentation, and have succeeded in crashing the kernel in various ways. Unfortunately, that testing is not done on a continuous basis, and maintainers understandably have little spare time to play the whack-a-mole game of adding missing nokprobes annotations as the kernel code evolves.

Quoted 32 lines

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

   trace_hardirqs_off()
      idx = srcu_read_lock()
      rcu_irq_enter_irqson();
      ...
      rcu_irq_exit_irqson();
      srcu_read_unlock(idx);

   if (user_mode)
      user_exit_irqsoff();
   else
      rcu_irq_enter();
Quoted 8 lines

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

I strongly suspect that most of the overhead we've been trying to avoid when introducing use of SRCU in rcuidle tracepoints was actually caused by callsites which use rcuidle tracepoints while having RCU watching, just because there is a handful of callsites which don't have RCU watching. This is confirmed by the commit message of commit e6753f23d9 "tracepoint: Make rcuidle tracepoint callers use SRCU":

"In recent tests with IRQ on/off tracepoints, a large performance
overhead ~10% is noticed when running hackbench. This is root caused to calls to rcu_irq_enter_irqson and rcu_irq_exit_irqson from the tracepoint code. Following a long discussion on the list [1] about this, we concluded that srcu is a better alternative for use during rcu idle. Although it does involve extra barriers, its lighter than the sched-rcu version which has to do additional RCU calls to notify RCU idle about entry into RCU sections.
[...]
Test: Tested idle and preempt/irq tracepoints."

So I think we could go back to plain RCU for rcuidle tracepoints if we do the cheaper "rcu_is_watching()" check rather than invoking rcu_irq_{enter,exit}_irqson() unconditionally.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Part of the issue here is the current overhead of SRCU read-side lock, which contains memory barriers. The other part of the issue is the fact that rcu_irq_{enter,exit}_irqson() contains an atomic_add_return atomic instruction.

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable RCU for those cases. We could then simply go back on using regular RCU like so:

#define __DO_TRACE(tp, proto, args, cond, rcuidle)                      \
        do {                                                            \
                struct tracepoint_func *it_func_ptr;                    \
                void *it_func;                                          \
                void *__data;                                           \
                bool exit_rcu = false;                                  \
                                                                        \
                if (!(cond))                                            \
                        return;                                         \
                                                                        \
                if (rcuidle && !rcu_is_watching()) {                    \
                        rcu_irq_enter_irqson();                         \
                        exit_rcu = true;                                \
                }                                                       \
                preempt_disable_notrace();                              \
                it_func_ptr = rcu_dereference_raw((tp)->funcs);         \
                if (it_func_ptr) {                                      \
                        do {                                            \
                                it_func = (it_func_ptr)->func;          \
                                __data = (it_func_ptr)->data;           \
                                ((void(*)(proto))(it_func))(args);      \
                        } while ((++it_func_ptr)->func);                \
                }                                                       \
                preempt_enable_notrace();                               \
                if (exit_rcu)                                           \
                        rcu_irq_exit_irqson();                          \
        } while (0)
Quoted 14 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

Quoted 4 lines

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

If we go all the way into that direction, I suspect it might even make sense to duplicate some kernel functions so they can still be part of the code which can be instrumented, but provide tracing-specific copy which would be hidden from instrumentation. I wonder what would be the size cost of this duplication.

In addition to splitting tracing code into a separate section, which I think makes sense, I can think of another alternative way to provide call chains
protection: adding a "in_tracing" flag somewhere alongside each kernel stack. Each thread and interrupt stack would have its own flag. However, trap handlers should share the "in_tracing" flag with the context which triggers the trap.

If a tracer recurses, or if a tracer attempts to trace another tracer, the instrumentation would break the recursion chain by preventing instrumentation from firing. If we end up caring about tracers tracing other tracers, we could have one distinct flag per tracer and let each tracer break the recursion chain.

Having this flag per kernel stack rather than per CPU or per thread would allow tracing of nested interrupt handlers (and NMIs), but would break call chains both within the same stack or going through a trap. I think it could be a nice complementary safety net to handle mishaps in a non-fatal way.

Thanks,

Mathieu

Quoted 6 lines

Thoughts?

Thanks,

tglx

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#20 lore raw
Quoted 3 lines · On Mon, 9 Mar 2020 14:37:40 -0400 (EDT) Mathieu Desnoyers…

So I think we could go back to plain RCU for rcuidle tracepoints if we do the cheaper "rcu_is_watching()" check rather than invoking rcu_irq_{enter,exit}_irqson() unconditionally.

You mean something like this?

-- Steve

include/linux/tracepoint.h +4 -4
index 5f4de82ffa0f..1904dbb3a921 100644
@@ -164,7 +164,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 		struct tracepoint_func *it_func_ptr;			\
 		void *it_func;						\
 		void *__data;						\
-		int __maybe_unused __idx = 0;				\
+		int __maybe_unused __idx = -1;				\
 									\
 		if (!(cond))						\
 			return;						\
@@ -179,8 +179,8 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 		 * For rcuidle callers, use srcu since sched-rcu	\
 		 * doesn't work from the idle path.			\
 		 */							\
-		if (rcuidle)						\
-			__idx = srcu_read_lock_notrace(&tracepoint_srcu);\
+		if (rcuidle && !rcu_is_watching())			\
+			__idx = srcu_read_lock_notrace(&tracepoint_srcu); \
 									\
 		it_func_ptr = rcu_dereference_raw((tp)->funcs);		\
 									\
@@ -199,7 +199,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 			} while ((++it_func_ptr)->func);		\
 		}							\
 									\
-		if (rcuidle)						\
+		if (rcuidle && __idx != -1)				\
 			rcu_irq_exit_irqson();				\
 									\
 		preempt_enable_notrace();				\
#21 lore raw
Quoted 8 lines · ----- On Mar 9, 2020, at 2:44 PM, rostedt [email protected] wrote:
Quoted 3 lines · On Mon, 9 Mar 2020 14:37:40 -0400 (EDT) Mathieu Desnoyers…

So I think we could go back to plain RCU for rcuidle tracepoints if we do the cheaper "rcu_is_watching()" check rather than invoking rcu_irq_{enter,exit}_irqson() unconditionally.

You mean something like this?

I have a hard time applying this patch to any tree I can fetch, so I will use caution and say: probably not. ;-)

And when I say "go back to plain RCU", I really mean removing use of SRCU from the tracepoints until we have other purposes for it (e.g. taking faults within specific tracepoint probes such as syscall enter/exit).

Thanks,

Mathieu

Quoted 39 lines

-- Steve

include/linux/tracepoint.h +1 -1
index 5f4de82ffa0f..1904dbb3a921 100644
@@ -164,7 +164,7 @@ static inline struct tracepoint
 *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 		struct tracepoint_func *it_func_ptr;			\
 		void *it_func;						\
 		void *__data;						\
-		int __maybe_unused __idx = 0;				\
+		int __maybe_unused __idx = -1;				\
 									\
 		if (!(cond))						\
			return;						\
include/linux/tracepoint.h +2 -2
@@ -179,8 +179,8 @@ static inline struct tracepoint
 *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 		 * For rcuidle callers, use srcu since sched-rcu	\
 		 * doesn't work from the idle path.			\
 		 */							\
-		if (rcuidle)						\
-			__idx = srcu_read_lock_notrace(&tracepoint_srcu);\
+		if (rcuidle && !rcu_is_watching())			\
+			__idx = srcu_read_lock_notrace(&tracepoint_srcu); \
 									\
 		it_func_ptr = rcu_dereference_raw((tp)->funcs);		\
									\
include/linux/tracepoint.h +1 -1
@@ -199,7 +199,7 @@ static inline struct tracepoint
 *tracepoint_ptr_deref(tracepoint_ptr_t *p)
 			} while ((++it_func_ptr)->func);		\
 		}							\
 									\
-		if (rcuidle)						\
+		if (rcuidle && __idx != -1)				\
 			rcu_irq_exit_irqson();				\
 									\

preempt_enable_notrace(); \

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
maintainer
#22 lore raw
Quoted 3 lines · On Mon, 9 Mar 2020 14:52:40 -0400 (EDT) Mathieu Desnoyers…

And when I say "go back to plain RCU", I really mean removing use of SRCU from the tracepoints until we have other purposes for it (e.g. taking faults within specific tracepoint probes such as syscall enter/exit).

Actually, with both you and Alexei talking about having a sleeping tracepoint callback, where we can add a can sleep check (but not in the DO_TRACE macro, I would think that registered sleeping callbacks would be its own callback), I would think we do not want to remove the SRCU usage.

-- Steve

#23 lore raw
Quoted 11 lines · ----- On Mar 9, 2020, at 3:09 PM, rostedt [email protected] wrote:
Quoted 3 lines · On Mon, 9 Mar 2020 14:52:40 -0400 (EDT) Mathieu Desnoyers…

And when I say "go back to plain RCU", I really mean removing use of SRCU from the tracepoints until we have other purposes for it (e.g. taking faults within specific tracepoint probes such as syscall enter/exit).

Actually, with both you and Alexei talking about having a sleeping tracepoint callback, where we can add a can sleep check (but not in the DO_TRACE macro, I would think that registered sleeping callbacks would be its own callback), I would think we do not want to remove the SRCU usage.

Whether we keep it or add it back later when needed should not make much difference.

In any case, considering that overhead which motivated use of SRCU for the rcuidle case could instead be handled by using is_rcu_watching() and normal RCU, I would prefer removing it from the rcuidle tracepoints for now, and add it back when we add a new kind of "sleepable" tracepoints later on.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#24 lore raw
In reply to Mathieu Desnoyers
Quoted 13 lines · Mathieu Desnoyers
Quoted 12 lines · ----- On Mar 9, 2020, at 1:02 PM, Thomas Gleixner [email protected] wrote:

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

Quoted 6 lines
Quoted 1 line

This is very much related to #3..

Do I understand correctly that you intend on moving all kernel low level entry/exit code into sections which cannot be instrumented by kprobes nor the function tracer, and require explicit whitelisting, either through annotations or use of explicit tracepoints ?

Pretty much so.

Quoted 8 lines
Quoted 7 lines

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

Quoted 7 lines

I think there is another question that arises here: the lack of automated continuous testing of the kprobes coverage. We have performed some testing of various random permutations of kprobes instrumentation, and have succeeded in crashing the kernel in various ways. Unfortunately, that testing is not done on a continuous basis, and maintainers understandably have little spare time to play the whack-a-mole game of adding missing nokprobes annotations as the kernel code evolves.

That's why I think a section approach is more practicable.

Quoted 5 lines
Quoted 5 lines

#3) RCU idle
are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

Quoted 8 lines
Quoted 4 lines

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

That and in some places in the entry code we do the heavy weight operations to cover 100 instructions which turn on RCU anyway. That does not make any sense at all.

Quoted 7 lines

I strongly suspect that most of the overhead we've been trying to avoid when introducing use of SRCU in rcuidle tracepoints was actually caused by callsites which use rcuidle tracepoints while having RCU watching, just because there is a handful of callsites which don't have RCU watching. This is confirmed by the commit message of commit e6753f23d9 "tracepoint: Make rcuidle tracepoint callers use SRCU":

Quoted 10 lines

"In recent tests with IRQ on/off tracepoints, a large performance
overhead ~10% is noticed when running hackbench. This is root caused to calls to rcu_irq_enter_irqson and rcu_irq_exit_irqson from the tracepoint code. Following a long discussion on the list [1] about this, we concluded that srcu is a better alternative for use during rcu idle. Although it does involve extra barriers, its lighter than the sched-rcu version which has to do additional RCU calls to notify RCU idle about entry into RCU sections.
[...]
Test: Tested idle and preempt/irq tracepoints."

In a quick test I did with a invalid syscall number with profiling the trace_hardirqs_off() is pretty prominent and goes down by roughly a factor of 2 when I move it past enter_from_user_mode() and use just the non RCU idle variant.

Quoted 3 lines

So I think we could go back to plain RCU for rcuidle tracepoints if we do the cheaper "rcu_is_watching()" check rather than invoking rcu_irq_{enter,exit}_irqson() unconditionally.

Haven't tried that yet for the overall usage of trace_hardirqs_off(), but yes, it's going to be a measurable difference.

Quoted 7 lines
Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Quoted 8 lines

Part of the issue here is the current overhead of SRCU read-side lock, which contains memory barriers. The other part of the issue is the fact that rcu_irq_{enter,exit}_irqson() contains an atomic_add_return atomic instruction.

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable RCU for those cases. We could then simply go back on using regular RCU like so:

Right, but that still does the whole rcu_irq dance especially in the entry code just to trace 50 or 100 instructions which are turning on RCU anyway.

Quoted 19 lines
Quoted 14 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

Quoted 4 lines

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Quoted 5 lines

If we go all the way into that direction, I suspect it might even make sense to duplicate some kernel functions so they can still be part of the code which can be instrumented, but provide tracing-specific copy which would be hidden from instrumentation. I wonder what would be the size cost of this duplication.

That might happen, but we'll see how much of that is truly requried. It's not horrible large. For the int3 isolation most of it was fixing up code which should have been protected in the first place and the only copied code was bsearch which is tiny.

Quoted 11 lines

In addition to splitting tracing code into a separate section, which I think makes sense, I can think of another alternative way to provide call chains
protection: adding a "in_tracing" flag somewhere alongside each kernel stack. Each thread and interrupt stack would have its own flag. However, trap handlers should share the "in_tracing" flag with the context which triggers the trap.

If a tracer recurses, or if a tracer attempts to trace another tracer, the instrumentation would break the recursion chain by preventing instrumentation from firing. If we end up caring about tracers tracing other tracers, we could have one distinct flag per tracer and let each tracer break the recursion chain.

Quoted 5 lines

Having this flag per kernel stack rather than per CPU or per thread would allow tracing of nested interrupt handlers (and NMIs), but would break call chains both within the same stack or going through a trap. I think it could be a nice complementary safety net to handle mishaps in a non-fatal way.

That works as long as none of this uses breakpoint based patching to dynamically disable/enable stuff.

Thanks,

tglx

#25 lore raw
Quoted 2 lines · ----- On Mar 9, 2020, at 3:52 PM, Thomas Gleixner [email protected] wrote:
Quoted 1 line · Mathieu Desnoyers

----- On Mar 9, 2020, at 1:02 PM, Thomas Gleixner [email protected] wrote:

[...]

Quoted 37 lines
Quoted 6 lines
Quoted 6 lines

#3) RCU idle
are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

Quoted 8 lines
Quoted 4 lines

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

That and in some places in the entry code we do the heavy weight operations to cover 100 instructions which turn on RCU anyway. That does not make any sense at all.

Quoted 7 lines

I strongly suspect that most of the overhead we've been trying to avoid when introducing use of SRCU in rcuidle tracepoints was actually caused by callsites which use rcuidle tracepoints while having RCU watching, just because there is a handful of callsites which don't have RCU watching. This is confirmed by the commit message of commit e6753f23d9 "tracepoint: Make rcuidle tracepoint callers use SRCU":

Quoted 10 lines

"In recent tests with IRQ on/off tracepoints, a large performance
overhead ~10% is noticed when running hackbench. This is root caused to calls to rcu_irq_enter_irqson and rcu_irq_exit_irqson from the tracepoint code. Following a long discussion on the list [1] about this, we concluded that srcu is a better alternative for use during rcu idle. Although it does involve extra barriers, its lighter than the sched-rcu version which has to do additional RCU calls to notify RCU idle about entry into RCU sections.
[...]
Test: Tested idle and preempt/irq tracepoints."

Quoted 4 lines

In a quick test I did with a invalid syscall number with profiling the trace_hardirqs_off() is pretty prominent and goes down by roughly a factor of 2 when I move it past enter_from_user_mode() and use just the non RCU idle variant.

I think one issue here is that trace_hardirqs_off() is now shared between lockdep and tracing. For lockdep, we have the following comment:

/*
* IRQ from user mode.
*
* We need to tell lockdep that IRQs are off. We can't do this until
* we fix gsbase, and we should do it before enter_from_user_mode
* (which can take locks). Since TRACE_IRQS_OFF is idempotent,
* the simplest way to handle it is to just call it twice if
* we enter from user mode. There's no reason to optimize this since
* TRACE_IRQS_OFF is a no-op if lockdep is off.
*/
TRACE_IRQS_OFF

CALL_enter_from_user_mode

1:
ENTER_IRQ_STACK old_rsp=%rdi save_ret=1
/* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF

which seems to imply that lockdep requires TRACE_IRQS_OFF to be performed _before_ entering from usermode. I don't expect this to be useful at all for other tracers though. I think this should be replaced by a new e.g. LOCKDEP_ENTER_FROM_USER_MODE or such which would call into lockdep without calling other tracers.

Quoted 7 lines
Quoted 4 lines

So I think we could go back to plain RCU for rcuidle tracepoints if we do the cheaper "rcu_is_watching()" check rather than invoking rcu_irq_{enter,exit}_irqson() unconditionally.

Haven't tried that yet for the overall usage of trace_hardirqs_off(), but yes, it's going to be a measurable difference.

Ideally we should do both: change the TRACE_IRQS_OFF prior entering from usermode for a lockdep-specific call, which would remove the frequent case which requires temporarily enabling RCU, *and* change tracepoints to use rcu_is_watching(), temporarily enable RCU, and use standard RCU for rcuidle cases.

Quoted 18 lines
Quoted 12 lines
Quoted 7 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Part of the issue here is the current overhead of SRCU read-side lock, which contains memory barriers. The other part of the issue is the fact that rcu_irq_{enter,exit}_irqson() contains an atomic_add_return atomic instruction.

Quoted 5 lines

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable
RCU for those cases. We could then simply go back on using regular RCU like so:

Quoted 3 lines

Right, but that still does the whole rcu_irq dance especially in the entry code just to trace 50 or 100 instructions which are turning on RCU anyway.

Agreed. Would changing this to a lockdep-specific call as I suggest above solve this ?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

#4 Protecting call chains

[...]

Quoted 12 lines
Quoted 12 lines

In addition to splitting tracing code into a separate section, which I think makes sense, I can think of another alternative way to provide call chains
protection: adding a "in_tracing" flag somewhere alongside each kernel stack. Each thread and interrupt stack would have its own flag. However, trap handlers should share the "in_tracing" flag with the context which triggers the trap.

If a tracer recurses, or if a tracer attempts to trace another tracer, the instrumentation would break the recursion chain by preventing instrumentation from firing. If we end up caring about tracers tracing other tracers, we could have one distinct flag per tracer and let each tracer break the recursion chain.

Quoted 8 lines
Quoted 5 lines

Having this flag per kernel stack rather than per CPU or per thread would allow tracing of nested interrupt handlers (and NMIs), but would break call chains both within the same stack or going through a trap. I think it could be a nice complementary safety net to handle mishaps in a non-fatal way.

That works as long as none of this uses breakpoint based patching to dynamically disable/enable stuff.

I'm clearly missing something here. I was expecting the "in_tracing" flag trick to be able to fix the breakpoint recursion issue. What is the problem I'm missing here ?

An additional point about splitting core code into separate non-instrumentable
sections: The part I'm concerned about is instrumentation of trap handlers. _That_ will likely need to remain shared between the kernel and the tracers. This is where I think the "in_tracing" flag approach would help us solve the recursion issue without losing that important instrumentation coverage.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#26 lore raw
Quoted 28 lines · Mathieu Desnoyers
Quoted 4 lines · ----- On Mar 9, 2020, at 3:52 PM, Thomas Gleixner [email protected] wrote:

In a quick test I did with a invalid syscall number with profiling the trace_hardirqs_off() is pretty prominent and goes down by roughly a factor of 2 when I move it past enter_from_user_mode() and use just the non RCU idle variant.

I think one issue here is that trace_hardirqs_off() is now shared between lockdep and tracing. For lockdep, we have the following comment:

/*
* IRQ from user mode.
*
* We need to tell lockdep that IRQs are off. We can't do this until
* we fix gsbase, and we should do it before enter_from_user_mode
* (which can take locks). Since TRACE_IRQS_OFF is idempotent,
* the simplest way to handle it is to just call it twice if
* we enter from user mode. There's no reason to optimize this since
* TRACE_IRQS_OFF is a no-op if lockdep is off.
*/
TRACE_IRQS_OFF

CALL_enter_from_user_mode

1:
ENTER_IRQ_STACK old_rsp=%rdi save_ret=1
/* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF

Quoted 5 lines

which seems to imply that lockdep requires TRACE_IRQS_OFF to be performed _before_ entering from usermode. I don't expect this to be useful at all for other tracers though. I think this should be replaced by a new e.g. LOCKDEP_ENTER_FROM_USER_MODE or such which would call into lockdep without calling other tracers.

See the entry series I'm working on. Aside of moving all this nonsense into C-code it splits lockdep and tracing so it looks like this:

            lockdep_hardirqs_off();
            user_exit_irqsoff();
            __trace_hardirqs_off();

The latter uses regular RCU and not the scru/rcu_irq dance.

Quoted 6 lines
Quoted 3 lines

Right, but that still does the whole rcu_irq dance especially in the entry code just to trace 50 or 100 instructions which are turning on RCU anyway.

Agreed. Would changing this to a lockdep-specific call as I suggest above solve this ?

That split exist for a few weeks now at least in my patches :)

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

If a tracer recurses, or if a tracer attempts to trace another tracer, the instrumentation would break the recursion chain by preventing instrumentation from firing. If we end up caring about tracers tracing other tracers, we could have one distinct flag per tracer and let each tracer break the recursion chain.

Having this flag per kernel stack rather than per CPU or per thread would allow tracing of nested interrupt handlers (and NMIs), but would break call chains both within the same stack or going through a trap. I think it could be a nice complementary safety net to handle mishaps in a non-fatal way.

Quoted 6 lines
Quoted 2 lines

That works as long as none of this uses breakpoint based patching to dynamically disable/enable stuff.

I'm clearly missing something here. I was expecting the "in_tracing" flag trick to be able to fix the breakpoint recursion issue. What is the problem I'm missing here ?

How do you "fix" that when you can't reach the tracepoint because you trip over a breakpoint and then while trying to fixup that stuff you hit another one?

Thanks,

tglx

#27 lore raw
Quoted 1 line · ----- On Mar 10, 2020, at 12:48 PM, Thomas Gleixner [email protected] wrote:

Mathieu Desnoyers <[email protected]> writes:

[...]

Quoted 9 lines

See the entry series I'm working on. Aside of moving all this nonsense into C-code it splits lockdep and tracing so it looks like this:

           lockdep_hardirqs_off();
           user_exit_irqsoff();
           __trace_hardirqs_off();

The latter uses regular RCU and not the scru/rcu_irq dance.

Awesome :)

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 11 lines

If a tracer recurses, or if a tracer attempts to trace another tracer, the instrumentation would break the recursion chain by preventing instrumentation from firing. If we end up caring about tracers tracing other tracers, we could have one distinct flag per tracer and let each tracer break the recursion chain.

Having this flag per kernel stack rather than per CPU or per thread would allow tracing of nested interrupt handlers (and NMIs), but would break call chains both within the same stack or going through a trap. I think it could be a nice complementary safety net to handle mishaps in a non-fatal way.

That works as long as none of this uses breakpoint based patching to dynamically disable/enable stuff.

Quoted 8 lines
Quoted 4 lines

I'm clearly missing something here. I was expecting the "in_tracing" flag trick to be able to fix the breakpoint recursion issue. What is the problem I'm missing
here ?

How do you "fix" that when you can't reach the tracepoint because you trip over a breakpoint and then while trying to fixup that stuff you hit another one?

I may still be missing something, but if the fixup code (AFAIU the code performing the out-of-line single-stepping of the original instruction) belongs to a section hidden from instrumentation, it should not be an issue.

The basic idea would be, e.g. pseudo-code for int3:

<int3> <---- in section which cannot be instrumented if (recursion_ctx->in_tracer) {
single-step original instruction
iret
}
[...] prepare stuff
recursion_ctx->in_tracer = true; instr_allowed()

call external kernel functions (which can be instrumented)

instr_disallowed()
recursion_ctx->in_tracer = false;
single-step original instruction
iret

The purpose of the "in_tracer" flag is to protect whatever is done within external kernel functions (which can be instrumented) from triggering tracer recursion. It needs to be combined with hiding of early/late low-level entry/exit functions from instrumentation (as you propose) to work.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#28 lore raw
Quoted 8 lines · Mathieu Desnoyers
Quoted 3 lines · ----- On Mar 10, 2020, at 12:48 PM, Thomas Gleixner [email protected] wrote:

How do you "fix" that when you can't reach the tracepoint because you trip over a breakpoint and then while trying to fixup that stuff you hit another one?

I may still be missing something, but if the fixup code (AFAIU the code performing the out-of-line single-stepping of the original instruction) belongs to a section hidden from instrumentation, it should not be an issue.

Sure, but what guarantees that on the way there is nothing which might call into instrumentable code? Nothing, really.

That's why I want the explicit sections which can be analyzed by tools. Humans (including me) are really bad at it was demonstrated several times.

Thanks,

tglx

#29 lore raw
Quoted 12 lines · ----- On Mar 10, 2020, at 2:31 PM, Thomas Gleixner [email protected] wrote:
Quoted 10 lines · Mathieu Desnoyers
Quoted 3 lines · ----- On Mar 10, 2020, at 12:48 PM, Thomas Gleixner [email protected] wrote:

How do you "fix" that when you can't reach the tracepoint because you trip over a breakpoint and then while trying to fixup that stuff you hit another one?

I may still be missing something, but if the fixup code (AFAIU the code performing
the out-of-line single-stepping of the original instruction) belongs to a section
hidden from instrumentation, it should not be an issue.

Quoted 6 lines

Sure, but what guarantees that on the way there is nothing which might call into instrumentable code? Nothing, really.

That's why I want the explicit sections which can be analyzed by tools. Humans (including me) are really bad at it was demonstrated several times.

AFAIU, my in_tracer flag proposal complements yours.

The explicit sections thingy is required for early/late entry/exit code, but does nothing to prevent "explicitly marked" safe-to-instrument kernel code from triggering infinite recursion.

My flag proposal handles that second issue.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#30 lore raw
In reply to Mathieu Desnoyers
Quoted 9 lines · Mathieu Desnoyers
Quoted 5 lines

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

I think kernel/trace/trace_preemptirq.c created too many problems for the kernel without providing tangible benefits. My understanding no one is using it in production. It's a tool to understand how kernel works. And such debugging tool can and should be removed.

One of Thomas's patches mentioned that bpf can be invoked from hardirq and preempt tracers. This connection doesn't exist in a direct way, but theoretically it's possible. There is no practical use though and I would be happy to blacklist such bpf usage at a minimum.

Quoted 23 lines

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable RCU for those cases. We could then simply go back on using regular RCU like so:

#define __DO_TRACE(tp, proto, args, cond, rcuidle)                      \
        do {                                                            \
                struct tracepoint_func *it_func_ptr;                    \
                void *it_func;                                          \
                void *__data;                                           \
                bool exit_rcu = false;                                  \
                                                                        \
                if (!(cond))                                            \
                        return;                                         \
                                                                        \
                if (rcuidle && !rcu_is_watching()) {                    \
                        rcu_irq_enter_irqson();                         \
                        exit_rcu = true;                                \
                }                                                       \
                preempt_disable_notrace();                              \
                it_func_ptr = rcu_dereference_raw((tp)->funcs);         \
                if (it_func_ptr) {                                      \
                        do {                                            \
                                it_func = (it_func_ptr)->func;          \
Quoted 8 lines
                                __data = (it_func_ptr)->data;           \
                                ((void(*)(proto))(it_func))(args);      \
                        } while ((++it_func_ptr)->func);                \
                }                                                       \
                preempt_enable_notrace();                               \
                if (exit_rcu)                                           \
                        rcu_irq_exit_irqson();                          \
        } while (0)

I think it's a fine approach interim.

Long term sounds like Paul is going to provide sleepable and low overhead rcu_read_lock_for_tracers() that will include bpf.
My understanding that this new rcu flavor won't have "idle" issues, so rcu_is_watching() checks will not be necessary.
And if we remove trace_preemptirq.c the only thing left will be Thomas's points 1 (low level entry) and 2 (breakpoints) that can be addressed without creating fancy .text annotations and teach objtool about it.

In the mean time I've benchmarked srcu for sleepable bpf and it's quite heavy. srcu_read_lock+unlock roughly adds 10x execution cost to trivial bpf prog. I'm proceeding with it anyway, but really hoping that rcu_read_lock_for_tracers() will materialize soon.

In general I'm sceptical that .text annotations will work. Let's say all of idle is a red zone. But a ton of normal functions are called when idle. So objtool will go and mark them as red zone too. This way large percent of the kernel will be off limits for tracers. Which is imo not a good trade off. I think addressing 1 and 2 with explicit notrace/nokprobe annotations will cover all practical cases where people can shot themselves in a foot with a tracer. I realize that there will be forever whack-a-mole game and these annotations will never reach 100%. I think it's a fine trade off. Security is never 100% either. Tracing is never going to be 100% safe too.

#31 lore raw
Quoted 3 lines · Alexei Starovoitov

In general I'm sceptical that .text annotations will work. Let's say all of idle is a red zone. But a ton of normal functions are called when idle. So objtool will go and mark them as red zone too.

No. If you carefully read what I proposed its:

noinst foo()

{
        protected_work();
        
        instr_begin();

        invoke_otherstuff();

        instr_end();

        moar_protected_work();

}

objtool will not mark anything. It will check that invocations out of the protected section are marked as safe, i.e. inside a instr_begin/end() pair.

So if you fail to mark protected_work() as noinstr then it will complain. If you forget to put instr_begin/end() around the safe area it will complain about invoke_otherstuff().

So it's a very targeted approach. objtool is there to verify that it's consistent nothing else.

Quoted 5 lines

This way large percent of the
kernel will be off limits for tracers. Which is imo not a good trade off. I think addressing 1 and 2 with explicit notrace/nokprobe annotations will cover all practical cases where people can shot themselves in a foot with a tracer.

That's simply wishful thinking. The discussions in the last weeks have clearly demonstrated that this is not the case. People were truly convinced that e.g. probing rcu_idle_exit() is safe, but it was not. Read the thread how long this went on.

Quoted 4 lines

I realize that there will be forever whack-a-mole game and these annotations will never reach 100%. I think it's a fine trade off. Security is never 100% either. Tracing is never going to be 100% safe too.

I disagree. Whack a mole games are horrible and have a guaranteed high failure rate. Otherwise we would not discuss this at all.

And no, it's not a fine trade off.

If we can have technical means to prevent the wreckage, then not using them for handwaving reasons is just violating the only sane engineering principle:

Correctness first

I spent the last 20 years mopping up the violations of this principle.

We have to stop the "features first, performance first" and "good enough" mentality if we want to master the ever increasing complexity of hardware and software in the long run.

From my experience of cleaning up stuff, I can tell you, that correctness first neither hurts performance nor does it prevent features, except those which are wrong to begin with.

As quite some people do not care about or even willfully ignore "correctness first", we have to force them to adhere by technical means, which spares us to mop up the mess they'd create otherwise.

And even for those who deeply care tooling support is a great help to prevent the accidental slip up. I wish I could have spared chasing call chains manually and then figure out two days later that I missed something.

Thanks,

tglx

#32 lore raw
In reply to Alexei Starovoitov
Quoted 53 lines · Alexei Starovoitov
Quoted 9 lines · Mathieu Desnoyers
Quoted 5 lines

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

I think kernel/trace/trace_preemptirq.c created too many problems for the kernel without providing tangible benefits. My understanding no one is using it in production. It's a tool to understand how kernel works. And such debugging tool can and should be removed.

One of Thomas's patches mentioned that bpf can be invoked from hardirq and preempt tracers. This connection doesn't exist in a direct way, but theoretically it's possible. There is no practical use though and I would be happy to blacklist such bpf usage at a minimum.

Quoted 23 lines

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable RCU for those cases. We could then simply go back on using regular RCU like so:

#define __DO_TRACE(tp, proto, args, cond, rcuidle)                      \
        do {                                                            \
                struct tracepoint_func *it_func_ptr;                    \
                void *it_func;                                          \
                void *__data;                                           \
                bool exit_rcu = false;                                  \
                                                                        \
                if (!(cond))                                            \
                        return;                                         \
                                                                        \
                if (rcuidle && !rcu_is_watching()) {                    \
                        rcu_irq_enter_irqson();                         \
                        exit_rcu = true;                                \
                }                                                       \
                preempt_disable_notrace();                              \
                it_func_ptr = rcu_dereference_raw((tp)->funcs);         \
                if (it_func_ptr) {                                      \
                        do {                                            \
                                it_func = (it_func_ptr)->func;          \
Quoted 8 lines
                                __data = (it_func_ptr)->data;           \
                                ((void(*)(proto))(it_func))(args);      \
                        } while ((++it_func_ptr)->func);                \
                }                                                       \
                preempt_enable_notrace();                               \
                if (exit_rcu)                                           \
                        rcu_irq_exit_irqson();                          \
        } while (0)
Quoted 4 lines

I think it's a fine approach interim.

Long term sounds like Paul is going to provide sleepable and low overhead rcu_read_lock_for_tracers() that will include bpf.

It now builds without errors, so the obvious problems are taken care of...

Working on the less-obvious errors as rcutorture encounters them.

Quoted 2 lines

My understanding that this new rcu flavor won't have "idle" issues, so rcu_is_watching() checks will not be necessary.

True. However, if the from-idle code invokes other code relying on vanilla RCU, such checks are still required. But I must let others weigh in on this.

Quoted 3 lines

And if we remove trace_preemptirq.c the only thing left will be Thomas's points 1 (low level entry) and 2 (breakpoints) that can be addressed without creating fancy .text annotations and teach objtool about it.

And the intent is to cover these cases as well. Of course, we all know which road is paved with good intentions. ;-)

Quoted 4 lines

In the mean time I've benchmarked srcu for sleepable bpf and it's quite heavy. srcu_read_lock+unlock roughly adds 10x execution cost to trivial bpf prog. I'm proceeding with it anyway, but really hoping that rcu_read_lock_for_tracers() will materialize soon.

OK, 10x is a bit on the painful side!

Thanx, Paul

Quoted 9 lines

In general I'm sceptical that .text annotations will work. Let's say all of idle is a red zone. But a ton of normal functions are called when idle. So objtool will go and mark them as red zone too. This way large percent of the kernel will be off limits for tracers. Which is imo not a good trade off. I think addressing 1 and 2 with explicit notrace/nokprobe annotations will cover all practical cases where people can shot themselves in a foot with a tracer. I realize that there will be forever whack-a-mole game and these annotations will never reach 100%. I think it's a fine trade off. Security is never 100% either. Tracing is never going to be 100% safe too.

#33 lore raw
In reply to Alexei Starovoitov
Quoted 7 lines · Alexei Starovoitov
Quoted 6 lines · Mathieu Desnoyers
Quoted 5 lines

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 7 lines
Quoted 3 lines

I think one of the big issues here is that most of the uses of trace_hardirqs_off() are from sites which already have RCU watching, so we are doing heavy-weight operations for nothing.

I think kernel/trace/trace_preemptirq.c created too many problems for the kernel without providing tangible benefits. My understanding no one is using it in production.

Hi Alexei,
There are various people use the preempt/irq disable tracepoints for last 2 years at Google and ARM. There's also a BPF tool (in BCC) that uses those for tracing critical sections. Also Daniel Bristot's entire Preempt-IRQ formal verification stuff depends on it.

Quoted 2 lines

It's a tool to understand how kernel works. And such debugging tool can and should be removed.

If we go by that line of reasoning, then function tracing also should be removed from the kernel.

I am glad Thomas and Peter are working on it and looking forward to seeing the patches,

thanks,

- Joel

Quoted 61 lines

One of Thomas's patches mentioned that bpf can be invoked from hardirq and preempt tracers. This connection doesn't exist in a direct way, but theoretically it's possible. There is no practical use though and I would be happy to blacklist such bpf usage at a minimum.

Quoted 23 lines

We could use the approach proposed by Peterz's and Steven's patches to basically do a lightweight "is_rcu_watching()" check for rcuidle tracepoint, and only enable RCU for those cases. We could then simply go back on using regular RCU like so:

#define __DO_TRACE(tp, proto, args, cond, rcuidle)                      \
        do {                                                            \
                struct tracepoint_func *it_func_ptr;                    \
                void *it_func;                                          \
                void *__data;                                           \
                bool exit_rcu = false;                                  \
                                                                        \
                if (!(cond))                                            \
                        return;                                         \
                                                                        \
                if (rcuidle && !rcu_is_watching()) {                    \
                        rcu_irq_enter_irqson();                         \
                        exit_rcu = true;                                \
                }                                                       \
                preempt_disable_notrace();                              \
                it_func_ptr = rcu_dereference_raw((tp)->funcs);         \
                if (it_func_ptr) {                                      \
                        do {                                            \
                                it_func = (it_func_ptr)->func;          \
Quoted 8 lines
                                __data = (it_func_ptr)->data;           \
                                ((void(*)(proto))(it_func))(args);      \
                        } while ((++it_func_ptr)->func);                \
                }                                                       \
                preempt_enable_notrace();                               \
                if (exit_rcu)                                           \
                        rcu_irq_exit_irqson();                          \
        } while (0)

I think it's a fine approach interim.

Long term sounds like Paul is going to provide sleepable and low overhead rcu_read_lock_for_tracers() that will include bpf.
My understanding that this new rcu flavor won't have "idle" issues, so rcu_is_watching() checks will not be necessary.
And if we remove trace_preemptirq.c the only thing left will be Thomas's points 1 (low level entry) and 2 (breakpoints) that can be addressed without creating fancy .text annotations and teach objtool about it.

In the mean time I've benchmarked srcu for sleepable bpf and it's quite heavy. srcu_read_lock+unlock roughly adds 10x execution cost to trivial bpf prog. I'm proceeding with it anyway, but really hoping that rcu_read_lock_for_tracers() will materialize soon.

In general I'm sceptical that .text annotations will work. Let's say all of idle is a red zone. But a ton of normal functions are called when idle. So objtool will go and mark them as red zone too. This way large percent of the kernel will be off limits for tracers. Which is imo not a good trade off. I think addressing 1 and 2 with explicit notrace/nokprobe annotations will cover all practical cases where people can shot themselves in a foot with a tracer. I realize that there will be forever whack-a-mole game and these annotations will never reach 100%. I think it's a fine trade off. Security is never 100% either. Tracing is never going to be 100% safe too.

maintainer
#34 lore raw
In reply to Thomas Gleixner
Quoted 14 lines · Thomas Gleixner

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

Quoted 4 lines

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

So the only tool I know of that does full callchains is smatch. And in one of my series I did prod Dan about this.

The alternative is that we bite the bullet and add a vmlinux objtool pass. This keeps getting mentioned, so maybe it is time :/ I'd hate it, because it will increase build time at the slowest point, but it'd get us the coverage we need here.

#35 lore raw
In reply to Thomas Gleixner
Quoted 79 lines · Thomas Gleixner

Folks,

I'm starting a new conversation because there are about 20 different threads which look at that problem in various ways and the information is so scattered that creating a coherent picture is pretty much impossible.

There are several problems to solve:

1) Fragile low level entry code

2) Breakpoint utilization

3) RCU idle

4) Callchain protection

#1 Fragile low level entry code

While I understand the desire of instrumentation to observe everything we really have to ask the question whether it is worth the trouble especially with entry trainwrecks like x86, PTI and other horrors in that area.

I don't think so and we really should just bite the bullet and forbid any instrumentation in that code unless it is explicitly designed for that case, makes sense and has a real value from an observation perspective.

This is very much related to #3..

#2) Breakpoint utilization

As recent findings have shown, breakpoint utilization needs to be extremly careful about not creating infinite breakpoint recursions.

I think that's pretty much obvious, but falls into the overall question of how to protect callchains.

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Thanx, Paul

Quoted 32 lines

#4 Protecting call chains

Our current approach of annotating functions with notrace/noprobe is pretty much broken.

Functions which are marked NOPROBE or notrace call out into functions which are not marked and while this might be ok, there are enough places where it is not. But we have no way to verify that.

That's just a recipe for disaster. We really cannot request from sysadmins who want to use instrumentation to stare at the code first whether they can place/enable an instrumentation point somewhere. That'd be just a bad joke.

I really think we need to have proper text sections which are off limit for any form of instrumentation and have tooling to analyze the calls into other sections. These calls need to be annotated as safe and intentional.

Thoughts?

Thanks,

tglx

maintainer
#36 lore raw
Quoted 2 lines · On Mon, 9 Mar 2020 13:47:10 -0700 "Paul E. McKenney" <[email protected]> wrote:

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

And call this variant: yarcu (Yet Another RCU) (pronounced yar-ku ?)

-- Steve

#37 lore raw
Quoted 7 lines · Steven Rostedt
Quoted 2 lines · On Mon, 9 Mar 2020 13:47:10 -0700 "Paul E. McKenney" <[email protected]> wrote:

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

And call this variant: yarcu (Yet Another RCU) (pronounced yar-ku ?)

I was thinking in terms of rcu_read_lock_trace(), but hey!

Thanx, Paul

#38 lore raw
In reply to Paul E. McKenney
Quoted 54 lines · Paul E. McKenney
Quoted 40 lines · Thomas Gleixner

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 8 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

I was thinking about having a tracing-specific implementation of RCU. Last week Steve told me that the tracing ring buffer has its own ad-hoc RCU implementation which schedule a thread on each CPU to complete a grace period (did I understand it right?). Of course such a flavour of RCU wouldn't be nice to nohz_full but surely we can arrange some tweaks for those who require strong isolation. I'm sure you're having a much better idea though.

#39 lore raw
Quoted 63 lines · Frederic Weisbecker
Quoted 54 lines · Paul E. McKenney
Quoted 40 lines · Thomas Gleixner

#3) RCU idle

Being able to trace code inside RCU idle sections is very similar to the question raised in #1.

Assume all of the instrumentation would be doing conditional RCU schemes, i.e.:

if (rcuidle)
....
else
rcu_read_lock_sched()

before invoking the actual instrumentation functions and of course undoing that right after it, that really begs the question whether it's worth it.

Especially constructs like:

trace_hardirqs_off()
idx = srcu_read_lock()
rcu_irq_enter_irqson();
...
rcu_irq_exit_irqson();
srcu_read_unlock(idx);

if (user_mode)
user_exit_irqsoff();
else
rcu_irq_enter();

are really more than questionable. For 99.9999% of instrumentation users it's absolutely irrelevant whether this traces the interrupt disabled time of user_exit_irqsoff() or rcu_irq_enter() or not.

But what's relevant is the tracer overhead which is e.g. inflicted with todays trace_hardirqs_off/on() implementation because that unconditionally uses the rcuidle variant with the scru/rcu_irq dance around every tracepoint.

Quoted 6 lines

Even if the tracepoint sits in the ASM code it just covers about ~20 low level ASM instructions more. The tracer invocation, which is even done twice when coming from user space on x86 (the second call is optimized in the tracer C-code), costs definitely way more cycles. When you take the scru/rcu_irq dance into account it's a complete disaster performance wise.

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 7 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Quoted 6 lines

I was thinking about having a tracing-specific implementation of RCU. Last week Steve told me that the tracing ring buffer has its own ad-hoc RCU implementation which schedule a thread on each CPU to complete a grace period (did I understand it right?). Of course such a flavour of RCU wouldn't be nice to nohz_full but surely we can arrange some tweaks for those who require strong isolation. I'm sure you're having a much better idea though.

Well, that too. Please see CONFIG_TASKS_RCU_RUDE in current "dev" on -rcu. But yes, another is on its way...

Hey, it compiled, so it much be perfect, right? :-/

Thanx, Paul

#40 lore raw
In reply to Paul E. McKenney

----- On Mar 9, 2020, at 4:47 PM, paulmck [email protected] wrote:
[...]

Quoted 7 lines

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 7 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Hi Paul,

I think that before introducing yet another RCU flavor, it's important to take a step back and look at the tracer requirements first. If those end up being covered by currently available RCU flavors, then why add another ?

I can start with a few use-cases I have in mind. Others should feel free to pitch in:

Tracing callsite context:

1) Thread context

1.1) Preemption enabled

One tracepoint in this category is syscall enter/exit. We should introduce a variant of tracepoints relying on SRCU for this use-case so we can take page faults when fetching userspace data.

1.2) Preemption disabled

Tree-RCU works fine.

1.3) IRQs disabled

Tree-RCU works fine.

2) IRQ handler context

Tree-RCU works fine.

3) NMI context

Tree-RCU works fine.

4) cpuidle context (!rcu_is_watching())

- By all means, we should not have tracepoints requiring to temporarily enable RCU in frequent code-paths. It appears that we should be able to remove the few offenders we currently have (e.g. enter from usermode),
- For tracepoints which are infrequently called from !rcu_is_watching context, checking whether RCU is watching and only enabling when needed should be fast enough.

Are there other use-cases am I missing that would justify adding another flavor of RCU ?

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#41 lore raw
Quoted 20 lines · Mathieu Desnoyers

----- On Mar 9, 2020, at 4:47 PM, paulmck [email protected] wrote:
[...]

Quoted 7 lines

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 7 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Quoted 6 lines

Hi Paul,

I think that before introducing yet another RCU flavor, it's important to take a step back and look at the tracer requirements first. If those end up being covered by currently available RCU flavors, then why add another ?

Well, we have BPF requirements as well.

Quoted 5 lines

I can start with a few use-cases I have in mind. Others should feel free to pitch in:

Tracing callsite context:

Quoted 7 lines

1) Thread context

1.1) Preemption enabled

One tracepoint in this category is syscall enter/exit. We should introduce a variant of tracepoints relying on SRCU for this use-case so we can take page faults when fetching userspace data.

Agreed, SRCU works fine for the page-fault case, as the read-side memory barriers are in the noise compared to page-fault overhead. Back in the day, there were light-weight system calls. Are all of these now converted to VDSO or similar?

Quoted 18 lines

1.2) Preemption disabled

Tree-RCU works fine.

1.3) IRQs disabled

Tree-RCU works fine.

2) IRQ handler context

Tree-RCU works fine.

3) NMI context

Tree-RCU works fine.

4) cpuidle context (!rcu_is_watching())

Quoted 7 lines

- By all means, we should not have tracepoints requiring to temporarily enable RCU in frequent code-paths. It appears that we should be able to remove the few offenders we currently have (e.g. enter from usermode),
- For tracepoints which are infrequently called from !rcu_is_watching context, checking whether RCU is watching and only enabling when needed should be fast enough.

Are there other use-cases am I missing that would justify adding another flavor of RCU ?

BPF programs that might sometimes sleep, but are usually lightweight.

I will be double-checking this, of course.

Thanx, Paul

#42 lore raw
Quoted 39 lines · ----- On Mar 10, 2020, at 12:49 PM, paulmck [email protected] wrote:
Quoted 20 lines · Mathieu Desnoyers

----- On Mar 9, 2020, at 4:47 PM, paulmck [email protected] wrote:
[...]

Quoted 7 lines

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 7 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Quoted 6 lines

Hi Paul,

I think that before introducing yet another RCU flavor, it's important to take a step back and look at the tracer requirements first. If those end up being covered by currently available RCU flavors, then why add another ?

Well, we have BPF requirements as well.

Quoted 9 lines

I can start with a few use-cases I have in mind. Others should feel free to pitch in:

Tracing callsite context:

1) Thread context

1.1) Preemption enabled

Quoted 8 lines
Quoted 3 lines

One tracepoint in this category is syscall enter/exit. We should introduce a variant of tracepoints relying on SRCU for this use-case so we can take page faults when fetching userspace data.

Agreed, SRCU works fine for the page-fault case, as the read-side memory barriers are in the noise compared to page-fault overhead. Back in the day, there were light-weight system calls. Are all of these now converted to VDSO or similar?

There is a big difference between allowing page faults to happen, and expecting page faults to happen every time. I suspect many use-cases will end up having a fast-path which touches user-space data which is in the page cache, but may end up triggering page faults in rare occasions.

Therefore, this might justify an SRCU which has low-overhead read-side.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#43 lore raw
Quoted 50 lines · Mathieu Desnoyers
Quoted 39 lines · ----- On Mar 10, 2020, at 12:49 PM, paulmck [email protected] wrote:
Quoted 20 lines · Mathieu Desnoyers

----- On Mar 9, 2020, at 4:47 PM, paulmck [email protected] wrote:
[...]

Quoted 7 lines

Suppose that we had a variant of RCU that had about the same read-side overhead as Preempt-RCU, but which could be used from idle as well as from CPUs in the process of coming online or going offline? I have not thought through the irq/NMI/exception entry/exit cases, but I don't see why that would be problem.

Quoted 7 lines

This would have explicit critical-section entry/exit code, so it would not be any help for trampolines.

Would such a variant of RCU help?

Yeah, I know. Just what the kernel doesn't need, yet another variant of RCU...

Quoted 6 lines

Hi Paul,

I think that before introducing yet another RCU flavor, it's important to take a step back and look at the tracer requirements first. If those end up being covered by currently available RCU flavors, then why add another ?

Well, we have BPF requirements as well.

Quoted 9 lines

I can start with a few use-cases I have in mind. Others should feel free to pitch in:

Tracing callsite context:

1) Thread context

1.1) Preemption enabled

Quoted 8 lines
Quoted 3 lines

One tracepoint in this category is syscall enter/exit. We should introduce a variant of tracepoints relying on SRCU for this use-case so we can take page faults when fetching userspace data.

Agreed, SRCU works fine for the page-fault case, as the read-side memory barriers are in the noise compared to page-fault overhead. Back in the day, there were light-weight system calls. Are all of these now converted to VDSO or similar?

Quoted 6 lines

There is a big difference between allowing page faults to happen, and expecting page faults to happen every time. I suspect many use-cases will end up having a fast-path which touches user-space data which is in the page cache, but may end up triggering page faults in rare occasions.

Therefore, this might justify an SRCU which has low-overhead read-side.

OK, good to know, thank you!

Thanx, Paul