Skip to contentSkip to conversation

sdt.h tracepoints with unicode data and/or structs

DISCUSSION Mathieu Desnoyers 3 participants 13 messages Sep 12, 2016 – Oct 5, 2016 12 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 133954640.uMaX5Hy6f7@agathebauer

#1 lore raw
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:

Hey all,

where can I find more documentation on how to use sdt.h to add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.

Hi Milian,

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 4 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.

Quoted 2 lines

So my questions:

My answers will be about lttng-ust tracepoints.

Quoted 2 lines

What data types can I use for trace point arguments?

See http://lttng.org/docs/#doc-c-application

Quoted 2 lines

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is
not required to be null-terminated. See http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Quoted 3 lines

Can I pass structs? Or do I need to pass each member as an individual argument?

You can pass pointers to structures to the tracepoint() call from your application source code, and then you need to express each individual field you want to serialize into the trace within the TP_FIELDS() macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Thanks,

Mathieu

Quoted 12 lines

Thanks

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list [email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 8 lines · CEST Mathieu Desnoyers
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:

Hey all,

where can I find more documentation on how to use sdt.h to add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 7 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

Quoted 7 lines

The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.

Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

Quoted 11 lines
Quoted 1 line

What data types can I use for trace point arguments?

See http://lttng.org/docs/#doc-c-application

Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.

The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.

Quoted 7 lines
Quoted 2 lines

Can I pass structs? Or do I need to pass each member as an individual argument?

You can pass pointers to structures to the tracepoint() call from your application source code, and then you need to express each individual field you want to serialize into the trace within the TP_FIELDS() macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Thank you.

Bye

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 3340010.kaCWG37SKB@milian-kdab2

#5 lore raw
Quoted 24 lines · ----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected] wrote:
Quoted 8 lines · CEST Mathieu Desnoyers
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:

Hey all,

where can I find more documentation on how to use sdt.h to add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 12 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:

The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.

Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.

If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.

The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.

Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

What data types can I use for trace point arguments?

See http://lttng.org/docs/#doc-c-application

Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

Quoted 8 lines
Quoted 5 lines

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.

Quoted 8 lines

Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.

The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.

As a short term solution this appears to be fine.

In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.

Thanks!

Mathieu

Quoted 18 lines
Quoted 8 lines
Quoted 3 lines

Can I pass structs? Or do I need to pass each member as an individual argument?

You can pass pointers to structures to the tracepoint() call from your application source code, and then you need to express each individual field you want to serialize into the trace within the TP_FIELDS() macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Thank you.

Bye

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:

----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]

Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line

----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]

Quoted 60 lines · wrote:
Quoted 24 lines
Quoted 8 lines
Quoted 6 lines

Hey all,

where can I find more documentation on how to use sdt.h to add static tracepoints to user-land applications? If this is not the right place to
ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 13 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:

The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.

Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.

If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.

Quoted 7 lines

The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.

Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.

Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather, sdt.h does allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for lttng-ust tracepoints?

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 1 line

What data types can I use for trace point arguments?

See http://lttng.org/docs/#doc-c-application

Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Quoted 7 lines
Quoted 3 lines

This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.

When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.

Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.

At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.

Quoted 8 lines
Quoted 7 lines

Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.

The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.

Quoted 5 lines

As a short term solution this appears to be fine.

In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.

Thanks again!

Bye

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 1806825.5qdYm7EQhg@agathebauer

#9 lore raw
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Quoted 7 lines
Quoted 3 lines

This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.

Quoted 5 lines

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.

As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the tracepoint_enabled() and do_tracepoint() macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.

Quoted 5 lines

When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.

There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.

However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.

Quoted 3 lines

Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.

Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.

You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.

Quoted 8 lines

At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.

I leave this part for Mathieu ;-).

Phil

Quoted 25 lines
Quoted 9 lines
Quoted 8 lines

Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.

The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.

Quoted 5 lines

As a short term solution this appears to be fine.

In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.

Thanks again!

Bye

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAB4xu_2r+eNS6s5CWTGoNGPRJwp8H_gr6TDTG+6LJCgvDtO5gg@mail.gmail.com

#11 lore raw
Quoted 61 lines · ----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]…
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

Quoted 7 lines
Quoted 3 lines

This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.

Quoted 5 lines

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.

As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the tracepoint_enabled() and do_tracepoint() macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.

Quoted 5 lines

When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.

There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.

However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.

Quoted 3 lines

Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.

Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.

You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.

Quoted 8 lines

At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.

Quoted 1 line

I leave this part for Mathieu ;-).

This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.

Thanks,

Mathieu

Quoted 37 lines

Phil

Quoted 25 lines
Quoted 9 lines
Quoted 8 lines

Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.

The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.

Quoted 5 lines

As a short term solution this appears to be fine.

In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.

Thanks again!

Bye

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw
Quoted 1 line · CEST Mathieu Desnoyers

----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]

Quoted 62 lines · wrote:
Quoted 62 lines
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

Quoted 8 lines
Quoted 3 lines

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

This sounds like tracing would then incur a huge runtime overhead, when I
need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Quoted 5 lines

Are you concerned about the runtime overhead when tracing is disabled, or
enabled ? It would be good to gather some metrics on the overhead of this
conversion.

Quoted 5 lines

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.

As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the tracepoint_enabled() and do_tracepoint()
macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.

Quoted 4 lines

When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.

There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.

However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.

Quoted 2 lines

Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.

Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.

You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.

Quoted 7 lines

At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.

Quoted 4 lines
Quoted 1 line

I leave this part for Mathieu ;-).

This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.

This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:

	if (tracepoint_enabled(ust_tests_hello, tptest)) {
		/* prepare arguments */
		do_tracepoint(ust_tests_hello, tptest, i, netint, values,
			text, strlen(text), dbl, flt);
	}

If I understood you correctly, then I could do something like

	if (tracepoint_enabled(ust_tests_hello, tptest)) {
		/* don't prepare arguments */
		do_tracepoint(ust_tests_hello, my_complex_data);
	}

And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?

Thanks

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 12621392.supdKG9qFk@milian-kdab2

#15 lore raw
Quoted 79 lines · ----- On Sep 22, 2016, at 8:56 AM, Milian Wolff [email protected] wrote:
Quoted 1 line · CEST Mathieu Desnoyers

----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]

Quoted 62 lines · wrote:
Quoted 62 lines
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Can I pass UTF16 strings? Do they need to be null-terminated?

You should convert them to UTF8.

Quoted 8 lines
Quoted 3 lines

A ctf_string() needs to be null-terminated. A ctf_sequence_text() is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields

This sounds like tracing would then incur a huge runtime overhead, when I
need to convert all my UTF-16 strings to UTF-8. How does one deal with that?

Quoted 5 lines

Are you concerned about the runtime overhead when tracing is disabled, or
enabled ? It would be good to gather some metrics on the overhead of this
conversion.

Quoted 5 lines

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.

As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the tracepoint_enabled() and do_tracepoint()
macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.

Quoted 4 lines

When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.

There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.

However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.

Quoted 2 lines

Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.

Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.

You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.

Quoted 7 lines

At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.

Quoted 4 lines
Quoted 1 line

I leave this part for Mathieu ;-).

This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.

This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:

	if (tracepoint_enabled(ust_tests_hello, tptest)) {
		/* prepare arguments */
		do_tracepoint(ust_tests_hello, tptest, i, netint, values,
			text, strlen(text), dbl, flt);
	}
Quoted 9 lines

If I understood you correctly, then I could do something like

	if (tracepoint_enabled(ust_tests_hello, tptest)) {
		/* don't prepare arguments */
		do_tracepoint(ust_tests_hello, my_complex_data);
	}

And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?

Yes. Both approaches can be used.

Note that the second approach you refer to is the same as using a plain tracepoint() macro and doing the preparation within the TRACEPOINT_EVENT() macro.

The preparation within TRACEPOINT_EVENT() can currently only be done as expression evaluation in the TP_FIELDS. LTTng-modules has more flexibility in that respect, but not lttng-ust yet.

A patch contributing such example to lttng-ust would be welcome :)

Thanks,

Mathieu

Quoted 6 lines

Thanks
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw
Quoted 16 lines
Quoted 16 lines

This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:

      if (tracepoint_enabled(ust_tests_hello, tptest)) {
              /* prepare arguments */
              do_tracepoint(ust_tests_hello, tptest, i, netint, values,
                      text, strlen(text), dbl, flt);
      }

If I understood you correctly, then I could do something like

      if (tracepoint_enabled(ust_tests_hello, tptest)) {
              /* don't prepare arguments */
              do_tracepoint(ust_tests_hello, my_complex_data);
      }
Quoted 8 lines
Quoted 2 lines

And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?

Yes. Both approaches can be used.

Note that the second approach you refer to is the same as using a plain tracepoint() macro and doing the preparation within the TRACEPOINT_EVENT() macro.

Here's an example (C++):

    #define _my_enum_evaluation(_enum, _field, _member)                     \
      ctf_enum(my_provider, _enum, int, _field,                             \
               (_member).is_bound() ?                                       \
                 ((SomeApi::_enum::enum_type) (_member) <
SomeApi::_enum::UNKNOWN_VALUE ? \
                   (int) ((SomeApi::_enum::enum_type) (_member)) : INT_MAX  \
                 ) : INT_MAX)

    TRACEPOINT_ENUM(
      my_provider,
      my_enum,
      TP_ENUM_VALUES(
        ctf_enum_value("unknown", 0)
        ctf_enum_value("apple", 1)
        ctf_enum_value("banana", 2)
        ctf_enum_value("orange", 3)
        ctf_enum_value("strawberry", 4)
      )
    )

    TRACEPOINT_EVENT(
      my_provider,
      my_tracepoint,
      TP_ARGS(
        int something,
        const SomeApi::SomeObject&, object
      ),
      TP_FIELDS(
      ctf_integer(int, int_field, something)
        _my_enum_evaluation(my_enum, enum_field, object.someMember())
      )
    )

You can call functions in there, evaluate conditions using the ternary operator, etc.

Phil

Quoted 19 lines

The preparation within TRACEPOINT_EVENT() can currently only be done as expression evaluation in the TP_FIELDS. LTTng-modules has more flexibility in that respect, but not lttng-ust yet.

A patch contributing such example to lttng-ust would be welcome :)

Thanks,

Mathieu

Quoted 6 lines

Thanks
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Quoted 4 lines

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAB4xu_1JBA_d3w8ZU3Dbh1wUSCoVK0CnM6n-+9uHg13LZpWh-w@mail.gmail.com

#19 lore raw
Quoted 64 lines · CEST Philippe Proulx
Quoted 21 lines
Quoted 21 lines

This sounds excellent. Can you tell me how? Could you maybe add an example to>>
lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:

      if (tracepoint_enabled(ust_tests_hello, tptest)) {
      
              /* prepare arguments */
              do_tracepoint(ust_tests_hello, tptest, i, netint, values,
              
                      text, strlen(text), dbl, flt);
      
      }

If I understood you correctly, then I could do something like

      if (tracepoint_enabled(ust_tests_hello, tptest)) {
      
              /* don't prepare arguments */
              do_tracepoint(ust_tests_hello, my_complex_data);
      
      }
Quoted 7 lines
Quoted 1 line

And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?

Yes. Both approaches can be used.

Note that the second approach you refer to is the same as using a plain tracepoint() macro and doing the preparation within the TRACEPOINT_EVENT() macro.

Here's an example (C++):

    #define _my_enum_evaluation(_enum, _field, _member)                    
\ ctf_enum(my_provider, _enum, int, _field,                             \
(_member).is_bound() ?                                       \
((SomeApi::_enum::enum_type) (_member) <
SomeApi::_enum::UNKNOWN_VALUE ? \
                   (int) ((SomeApi::_enum::enum_type) (_member)) : INT_MAX 

\ ) : INT_MAX)

    TRACEPOINT_ENUM(
      my_provider,
      my_enum,
      TP_ENUM_VALUES(
        ctf_enum_value("unknown", 0)
        ctf_enum_value("apple", 1)
        ctf_enum_value("banana", 2)
        ctf_enum_value("orange", 3)
        ctf_enum_value("strawberry", 4)
      )
    )

    TRACEPOINT_EVENT(
      my_provider,
      my_tracepoint,
      TP_ARGS(
        int something,
        const SomeApi::SomeObject&, object
      ),
      TP_FIELDS(
      ctf_integer(int, int_field, something)
        _my_enum_evaluation(my_enum, enum_field, object.someMember())
      )
    )
Quoted 2 lines

You can call functions in there, evaluate conditions using the ternary operator, etc.

Thank you, that is exactly the example that I was looking for.

Cheers

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#20 lore raw
In reply to a message not in the archive
Quoted 74 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:

----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]

Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line

----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]

Quoted 60 lines · wrote:
Quoted 24 lines
Quoted 8 lines
Quoted 6 lines

Hey all,

where can I find more documentation on how to use sdt.h to add static tracepoints to user-land applications? If this is not the right place to
ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 13 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.

Quoted 7 lines
Quoted 2 lines
Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:

The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.

Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.

If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.

Quoted 7 lines

The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.

Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.

Quoted 6 lines

Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather, sdt.h does allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for lttng-ust tracepoints?

Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?

Thanks,

Mathieu

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): [email protected]

#22 lore raw
Quoted 88 lines · CEST Mathieu Desnoyers
Quoted 81 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:

----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]

Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line

----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]

Quoted 65 lines · wrote:
Quoted 28 lines
Quoted 10 lines
Quoted 8 lines

Hey all,

where can I find more documentation on how to use sdt.h to add static
tracepoints to user-land applications? If this is not the right place
to
ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 15 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be
usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you
can use sdt.h instrumentation.

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero-
overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:

The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.

Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.

If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.

Quoted 7 lines

The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.

Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.

Quoted 5 lines

Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather, sdt.h does allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for lttng-ust tracepoints?

Quoted 3 lines

Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?

The fast path to check whether to trace or not:

$ cat probes.d
provider milian {
    probe test();                                                                                                                                                                                                 
};                                                                                                                                                                                                                
$ dtrace -C -h -s probes.d -o probes.h                                                                                                                                                    
$ cat probes.h                                                                                                                                                  
/* Generated by the Systemtap dtrace wrapper */                                                                                                                                                                   

#define _SDT_HAS_SEMAPHORES 1

#define STAP_HAS_SEMAPHORES 1 /* deprecated */

#include <sys/sdt.h>

/* MILIAN_TEST ( ) */
#if defined STAP_SDT_V1
#define MILIAN_TEST_ENABLED() __builtin_expect (test_semaphore, 0) #define milian_test_semaphore test_semaphore
#else
#define MILIAN_TEST_ENABLED() __builtin_expect (milian_test_semaphore, 0) #endif
__extension__ extern unsigned short milian_test_semaphore __attribute__ ((unused)) __attribute__ ((section (".probes")));
#define MILIAN_TEST() \
DTRACE_PROBE (milian, test)

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): 4486738.j9zaXj12nX@milian-kdab2

#24 lore raw
Quoted 112 lines · ----- On Sep 22, 2016, at 8:47 AM, Milian Wolff [email protected] wrote:
Quoted 88 lines · CEST Mathieu Desnoyers
Quoted 81 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:

----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]

Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line

----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]

Quoted 65 lines · wrote:
Quoted 28 lines
Quoted 10 lines
Quoted 8 lines

Hey all,

where can I find more documentation on how to use sdt.h to add static
tracepoints to user-land applications? If this is not the right place
to
ask, please refer to to elsewhere.

Hi Milian,

Hey Mathieu,

Quoted 15 lines

LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.

Quoted 3 lines

I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.

The main question here would be: do you want your instrumentation to be
usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you
can use sdt.h instrumentation.

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

So my questions:

My answers will be about lttng-ust tracepoints.

Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero-
overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.

The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:

The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.

Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.

If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.

Quoted 7 lines

The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.

Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.

Quoted 5 lines

Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather, sdt.h does allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for lttng-ust tracepoints?

Quoted 3 lines

Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?

The fast path to check whether to trace or not:

$ cat probes.d
provider milian {
   probe test();
};
$ dtrace -C -h -s probes.d -o probes.h
$ cat probes.h
/* Generated by the Systemtap dtrace wrapper */

#define _SDT_HAS_SEMAPHORES 1

#define STAP_HAS_SEMAPHORES 1 /* deprecated */

#include <sys/sdt.h>

Quoted 11 lines

/* MILIAN_TEST ( ) */
#if defined STAP_SDT_V1
#define MILIAN_TEST_ENABLED() __builtin_expect (test_semaphore, 0) #define milian_test_semaphore test_semaphore
#else
#define MILIAN_TEST_ENABLED() __builtin_expect (milian_test_semaphore, 0) #endif
__extension__ extern unsigned short milian_test_semaphore __attribute__ ((unused)) __attribute__ ((section (".probes")));
#define MILIAN_TEST() \
DTRACE_PROBE (milian, test)

Got it. The performance difference between lttng-ust tracepoint_enabled() check and the sdt.h semaphore check will be none, because both perform an unlikely branch (using builtin expect 0).

Thanks,

Mathieu

Quoted 11 lines

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list [email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 7 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev