Skip to contentSkip to conversation

using --function/--probe

DISCUSSION Mathieu Desnoyers 2 participants 4 messages Jun 5, 2012 – May 3, 2013 4 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20130502195431.GA15675@Krystal

A message that is not in the archive (sent off-list or not yet received): 20120605154950.GA21955@Krystal

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 10 lines · * Bill Trost ([email protected]) wrote:

Hi,

I'm trying to investigate a problem under Linux 2.6.39.4 involving kworker threads and am pretty much stymied. I've tried enabling various events using --function and --probe and it's been very hit or miss. Very often, "lttng enable-event" comes back with nothing more specific than "bad ioctl", but every once in a while I'll find a symbol I can add a probe point for.

So: What does "--probe" mean?

--probe allows to put a kprobe (breakpoint) at a kernel text address or symbol.

Quoted 1 line

What does "--function" mean,

--function allows to put a kretprobes at a function entry site (using its symbol name), which will trigger events at function entry return.

Quoted 3 lines

and how does
it differ from "--probe"? How do I determine what symbols are valid for each of these options?

It entirely depends on which functions are blacklisted in the kernel (this is an attribute added to the functions specifically for kprobes). The keyword is "__kprobes".

Quoted 2 lines

And, maybe as a worked example, how do I
trace what work is being enqueued and run by the kworker threads?

For this level of details, I think kprobes/kretprobes will not currently allow you to fetch it. The two options we have are:

- use static tracepoints. Is there a tracepoint that targets the information you are looking for ? Try "lttng list -k".
- or extend the dynamic probing to allow fetching variables from debug information,

Thanks,

Mathieu

Quoted 8 lines

Thanks,
Bill

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#4 lore raw
In reply to a message not in the archive

Mathieu wrote:
* Bill Trost ([email protected]) wrote:
> and how does it differ from "--probe"? How do I determine > what symbols are valid for each of these options?

It entirely depends on which functions are blacklisted in the kernel (this is an attribute added to the functions specifically for kprobes). The keyword is "__kprobes".

Is that the only basis? For example, I can add a dynamic tracepoint (of either flawor) for queue_work_on(), but not work __queue_work(), yet neither of those functions have any apparent annotation. Or does EXPORT_SYMBOL_GPL imply a probe point?

Quoted 2 lines

...how do I trace what work is being
enqueued and run by the kworker threads?

For this level of details, I think kprobes/kretprobes will not currently allow you to fetch it. The two options we have are:

- use static tracepoints. Is there a tracepoint that targets the information you are looking for ? Try "lttng list -k".

No, I've been capturing all kernel events and haven't seen anything resembling the kworker tracepoints of lttng 0.x.

- or extend the dynamic probing to allow fetching variables from debug information,

Hmm. Well, I did a bit more poking and got a bit closer -- at least now I know what dynamic points to use. (eg, "queue_work_on" and friends to see what is being queued). I take it that LTT can't capture function call arguments as a form of additional context? That would be an ideal bit of information in this case.

Thanks again,
Bill

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 5 lines · * Bill Trost ([email protected]) wrote:

Mathieu wrote:
* Bill Trost ([email protected]) wrote:
> and how does it differ from "--probe"? How do I determine > what symbols are valid for each of these options?

Quoted 8 lines

It entirely depends on which functions are blacklisted in the kernel (this is an attribute added to the functions specifically for kprobes). The keyword is "__kprobes".

Is that the only basis? For example, I can add a dynamic tracepoint (of either flawor) for queue_work_on(), but not work __queue_work(), yet neither of those functions have any apparent annotation. Or does EXPORT_SYMBOL_GPL imply a probe point?

No, it doesn't.

Quoted 4 lines
Quoted 3 lines

...how do I trace what work is being
enqueued and run by the kworker threads?

Quoted 8 lines

For this level of details, I think kprobes/kretprobes will not currently allow you to fetch it. The two options we have are:

- use static tracepoints. Is there a tracepoint that targets the information you are looking for ? Try "lttng list -k".

No, I've been capturing all kernel events and haven't seen anything resembling the kworker tracepoints of lttng 0.x.

LTTng 2.2-rc has a new workqueue instrumentation. Maybe this can fit your use-case ?

Quoted 9 lines

- or extend the dynamic probing to allow fetching variables from debug information,

Hmm. Well, I did a bit more poking and got a bit closer -- at least now I know what dynamic points to use. (eg, "queue_work_on" and friends to see what is being queued). I take it that LTT can't capture function call arguments as a form of additional context? That would be an ideal bit of information in this case.

Currently it can't.

(sorry for late reply)

Thanks,

Mathieu

Quoted 3 lines

Thanks again,
Bill

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#7 lore raw
In reply to a message not in the archive

Mathieu wrote:
* Bill Trost ([email protected]) wrote:
> Mathieu wrote:
> * Bill Trost ([email protected]) wrote:
> > ...how do I trace what work is being
> > enqueued and run by the kworker threads?
>
> For this level of details, I think kprobes/kretprobes will not
> currently allow you to fetch it. The two options we have are:
>
> - use static tracepoints. Is there a tracepoint that targets the
> information you are looking for ? Try "lttng list -k".
>
> No, I've been capturing all kernel events and haven't seen > anything resembling the kworker tracepoints of lttng 0.x.

LTTng 2.2-rc has a new workqueue instrumentation. Maybe this can fit your use-case ?

Hi, Mathieu,

I'd forgotten all about this. We eventually discovered, as part of investigating the kworkers, that we had a misconfigured sysctl. When we fixed that, the offending kwork then disappeared completely -- problem solved.

Thanks for the "blast from the past",
Bill