sdt.h tracepoints with unicode data and/or structs
A message that is not in the archive (sent off-list or not yet received): 133954640.uMaX5Hy6f7@agathebauer
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:
Hey all,
where can I find more documentation on how to use
sdt.hto add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.
Hi Milian,
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "sdt.h" probe points too for compatibility with SystemTAP.
Quoted 4 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use sdt.h instrumentation.
Quoted 2 lines
So my questions:
My answers will be about lttng-ust tracepoints.
Quoted 2 lines
What data types can I use for trace point arguments?
See http://lttng.org/docs/#doc-c-application
Quoted 2 lines
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
A ctf_string() needs to be null-terminated. A ctf_sequence_text() is
not required to be null-terminated. See http://lttng.org/docs/#doc-liblttng-ust-tp-fields
Quoted 3 lines
Can I pass structs? Or do I need to pass each member as an individual argument?
You can pass pointers to structures to the tracepoint() call from your application source code, and then you need to express each individual field you want to serialize into the trace within the TP_FIELDS() macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fields
Thanks,
Mathieu
Quoted 12 lines
Thanks
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list [email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · CEST Mathieu Desnoyers
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:
Hey all,
where can I find more documentation on how to use
sdt.hto add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 7 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
Quoted 7 lines
The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use
sdt.hinstrumentation.Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.
Quoted 11 lines
Quoted 1 line
What data types can I use for trace point arguments?
See http://lttng.org/docs/#doc-c-application
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields
This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?
Looking at the API documentation, using ctf_array instead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.
The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.
Quoted 7 lines
Quoted 2 lines
Can I pass structs? Or do I need to pass each member as an individual argument?
You can pass pointers to structures to the
tracepoint()call from your application source code, and then you need to express each individual field you want to serialize into the trace within theTP_FIELDS()macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fields
Thank you.
Bye
Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 3340010.kaCWG37SKB@milian-kdab2
Quoted 24 lines · ----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected] wrote:
Quoted 8 lines · CEST Mathieu Desnoyers
Quoted 5 lines · ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected] wrote:
Hey all,
where can I find more documentation on how to use
sdt.hto add static tracepoints to user-land applications? If this is not the right place to ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 12 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use
sdt.hinstrumentation.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.
The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:
The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.
Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.
If we look at the code generated by SystemTAP sdt.h probes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.
The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.
Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
What data types can I use for trace point arguments?
See http://lttng.org/docs/#doc-c-application
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
Quoted 8 lines
Quoted 5 lines
You should convert them to UTF8.
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsThis sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?
Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.
Quoted 8 lines
Looking at the API documentation, using
ctf_arrayinstead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.
As a short term solution this appears to be fine.
In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.
Thanks!
Mathieu
Quoted 18 lines
Quoted 8 lines
Quoted 3 lines
Can I pass structs? Or do I need to pass each member as an individual argument?
You can pass pointers to structures to the
tracepoint()call from your application source code, and then you need to express each individual field you want to serialize into the trace within theTP_FIELDS()macro. Ref. http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsThank you.
Bye
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:
----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]
Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line
----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]
Quoted 60 lines · wrote:
Quoted 24 lines
Quoted 8 lines
Quoted 6 lines
Hey all,
where can I find more documentation on how to use
sdt.hto add static tracepoints to user-land applications? If this is not the right place to
ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 13 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use
sdt.hinstrumentation.Quoted 7 lines
Quoted 2 lines
Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.
The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:
The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.
Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.
If we look at the code generated by SystemTAP
sdt.hprobes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.
Quoted 7 lines
The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.
Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.
Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather, sdt.h does allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for lttng-ust tracepoints?
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 1 line
What data types can I use for trace point arguments?
See http://lttng.org/docs/#doc-c-application
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fields
Quoted 7 lines
Quoted 3 lines
This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?
Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.
I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.
When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.
Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.
At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.
Quoted 8 lines
Quoted 7 lines
Looking at the API documentation, using
ctf_arrayinstead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.
Quoted 5 lines
As a short term solution this appears to be fine.
In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.
Thanks again!
Bye
Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 1806825.5qdYm7EQhg@agathebauer
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsQuoted 7 lines
Quoted 3 lines
This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?
Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.
Quoted 5 lines
I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.
As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the tracepoint_enabled() and do_tracepoint() macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.
Quoted 5 lines
When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.
There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.
However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.
Quoted 3 lines
Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.
Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.
You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.
Quoted 8 lines
At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.
I leave this part for Mathieu ;-).
Phil
Quoted 25 lines
Quoted 9 lines
Quoted 8 lines
Looking at the API documentation, using
ctf_arrayinstead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.
Quoted 5 lines
As a short term solution this appears to be fine.
In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.
Thanks again!
Bye
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
Quoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CAB4xu_2r+eNS6s5CWTGoNGPRJwp8H_gr6TDTG+6LJCgvDtO5gg@mail.gmail.com
Quoted 61 lines · ----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]…
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsQuoted 7 lines
Quoted 3 lines
This sounds like tracing would then incur a huge runtime overhead, when I need to convert all my UTF-16 strings to UTF-8. How does one deal with that?
Are you concerned about the runtime overhead when tracing is disabled, or enabled ? It would be good to gather some metrics on the overhead of this conversion.
Quoted 5 lines
I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.
As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the
tracepoint_enabled()anddo_tracepoint()macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.Quoted 5 lines
When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.
There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.
However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.
Quoted 3 lines
Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.
Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.
Quoted 8 lines
At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.
Quoted 1 line
I leave this part for Mathieu ;-).
This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.
Thanks,
Mathieu
Quoted 37 lines
Phil
Quoted 25 lines
Quoted 9 lines
Quoted 8 lines
Looking at the API documentation, using
ctf_arrayinstead with ushort as C type sounds like a better approach. In a custom babeltrace consumer script, I could then convert it to UTF8 offline, if needed.The advantage here is that I have "zero" overhead while tracing, which is crucial for me to get the tracepoints accepted upstream in Qt, without hiding them behind a compiler switch.
Quoted 5 lines
As a short term solution this appears to be fine.
In the long run, we're currently designing CTF 2.0, and we're been wondering whether we keep it limited to UTF8, or extend it somehow. I'm CCing Philippe Proulx who is driving this effort.
Thanks again!
Bye
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt ExpertsQuoted 5 lines
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · CEST Mathieu Desnoyers
----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]
Quoted 62 lines · wrote:
Quoted 62 lines
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
Quoted 8 lines
Quoted 3 lines
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsThis sounds like tracing would then incur a huge runtime overhead, when I
need to convert all my UTF-16 strings to UTF-8. How does one deal with that?Quoted 5 lines
Are you concerned about the runtime overhead when tracing is disabled, or
enabled ? It would be good to gather some metrics on the overhead of this
conversion.Quoted 5 lines
I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.
As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the
tracepoint_enabled()anddo_tracepoint()
macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.Quoted 4 lines
When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.
There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.
However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.
Quoted 2 lines
Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.
Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.
Quoted 7 lines
At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.
Quoted 4 lines
Quoted 1 line
I leave this part for Mathieu ;-).
This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.
This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:
if (tracepoint_enabled(ust_tests_hello, tptest)) {
/* prepare arguments */
do_tracepoint(ust_tests_hello, tptest, i, netint, values,
text, strlen(text), dbl, flt);
}If I understood you correctly, then I could do something like
if (tracepoint_enabled(ust_tests_hello, tptest)) {
/* don't prepare arguments */
do_tracepoint(ust_tests_hello, my_complex_data);
}And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?
Thanks
Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 12621392.supdKG9qFk@milian-kdab2
Quoted 79 lines · ----- On Sep 22, 2016, at 8:56 AM, Milian Wolff [email protected] wrote:
Quoted 1 line · CEST Mathieu Desnoyers
----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx [email protected]
Quoted 62 lines · wrote:
Quoted 62 lines
Quoted 19 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Can I pass UTF16 strings? Do they need to be null-terminated?
You should convert them to UTF8.
Quoted 8 lines
Quoted 3 lines
A
ctf_string()needs to be null-terminated. Actf_sequence_text()is not required to be null-terminated. See
http://lttng.org/docs/#doc-liblttng-ust-tp-fieldsThis sounds like tracing would then incur a huge runtime overhead, when I
need to convert all my UTF-16 strings to UTF-8. How does one deal with that?Quoted 5 lines
Are you concerned about the runtime overhead when tracing is disabled, or
enabled ? It would be good to gather some metrics on the overhead of this
conversion.Quoted 5 lines
I'm mostly concerned about the overhead when tracing is disabled. Ideally, I would like to see these tracepoints unconditionally compiled into Qt. But if the overhead is noticeable when they are not in use, I may have a hard time achieving this goal. Instead, one will then probably need to recompile Qt with the tracepoints enabled.
As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty much unnoticeable. You can use the
tracepoint_enabled()anddo_tracepoint()
macros to perform some computation, conversions, and prepare stuff specifically for the payload of the event.Quoted 4 lines
When in use, the overhead of the tracepoints should also be minimal. Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large overhead, and then converting the data to UTF-8 also adds on top of that. Thus, if at all possible, I would like to prevent that.
There's no way to support UTF-16 as of the current versions of LTTng and CTF. Even if you find a way to store the string as is, for example using a sequence of bytes, the existing CTF viewers and analyzers won't recognize the field as a string.
However we could think about a way to support UTF-16 in the future, and possibly other Unicode encodings too.
Quoted 2 lines
Similarly, I am looking for a way to put URLs into tracepoints, and converting a QUrl to string data is far from cheap.
Yes, I see that this code is executed:
<https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.You could allocate one tracepoint field for each individual attribute of the QUrl object, for example the scheme, the path, the query string, the fragment, etc. But I guess you'd still have the UTF-16 issue.
Quoted 7 lines
At this point, I don't see a way around only enabling tracepoints optionally. Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust or sdt). The conditional to check whether tracing is enabled may not be too bad. But then in the conditional it looks like $some code will be required to massage the data into a form that the tracepoints accept them. This increase in code size negatively influences code caches and I don't see any way around that.
Quoted 4 lines
Quoted 1 line
I leave this part for Mathieu ;-).
This "extra code" can be implemented within the tracepoint provider, which is a cache cold function, not used at all when tracing is disabled.
This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:
if (tracepoint_enabled(ust_tests_hello, tptest)) { /* prepare arguments */ do_tracepoint(ust_tests_hello, tptest, i, netint, values, text, strlen(text), dbl, flt); }
Quoted 9 lines
If I understood you correctly, then I could do something like
if (tracepoint_enabled(ust_tests_hello, tptest)) { /* don't prepare arguments */ do_tracepoint(ust_tests_hello, my_complex_data); }And then have the "prepare" code somewhere in my
TRACEPOINT_EVENT?
Yes. Both approaches can be used.
Note that the second approach you refer to is the same as using a plain tracepoint() macro and doing the preparation within the TRACEPOINT_EVENT() macro.
The preparation within TRACEPOINT_EVENT() can currently only be done as expression evaluation in the TP_FIELDS. LTTng-modules has more flexibility in that respect, but not lttng-ust yet.
A patch contributing such example to lttng-ust would be welcome :)
Thanks,
Mathieu
Quoted 6 lines
Thanks
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines
Quoted 16 lines
This sounds excellent. Can you tell me how? Could you maybe add an example to lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:
if (tracepoint_enabled(ust_tests_hello, tptest)) { /* prepare arguments */ do_tracepoint(ust_tests_hello, tptest, i, netint, values, text, strlen(text), dbl, flt); }If I understood you correctly, then I could do something like
if (tracepoint_enabled(ust_tests_hello, tptest)) { /* don't prepare arguments */ do_tracepoint(ust_tests_hello, my_complex_data); }
Quoted 8 lines
Quoted 2 lines
And then have the "prepare" code somewhere in my
TRACEPOINT_EVENT?Yes. Both approaches can be used.
Note that the second approach you refer to is the same as using a plain
tracepoint()macro and doing the preparation within theTRACEPOINT_EVENT()macro.
Here's an example (C++):
#define _my_enum_evaluation(_enum, _field, _member) \
ctf_enum(my_provider, _enum, int, _field, \
(_member).is_bound() ? \
((SomeApi::_enum::enum_type) (_member) <
SomeApi::_enum::UNKNOWN_VALUE ? \
(int) ((SomeApi::_enum::enum_type) (_member)) : INT_MAX \
) : INT_MAX)
TRACEPOINT_ENUM(
my_provider,
my_enum,
TP_ENUM_VALUES(
ctf_enum_value("unknown", 0)
ctf_enum_value("apple", 1)
ctf_enum_value("banana", 2)
ctf_enum_value("orange", 3)
ctf_enum_value("strawberry", 4)
)
)
TRACEPOINT_EVENT(
my_provider,
my_tracepoint,
TP_ARGS(
int something,
const SomeApi::SomeObject&, object
),
TP_FIELDS(
ctf_integer(int, int_field, something)
_my_enum_evaluation(my_enum, enum_field, object.someMember())
)
)You can call functions in there, evaluate conditions using the ternary operator, etc.
Phil
Quoted 19 lines
The preparation within
TRACEPOINT_EVENT()can currently only be done as expression evaluation in theTP_FIELDS. LTTng-modules has more flexibility in that respect, but not lttng-ust yet.A patch contributing such example to lttng-ust would be welcome :)
Thanks,
Mathieu
Quoted 6 lines
Thanks
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
Quoted 4 lines
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): CAB4xu_1JBA_d3w8ZU3Dbh1wUSCoVK0CnM6n-+9uHg13LZpWh-w@mail.gmail.com
Quoted 64 lines · CEST Philippe Proulx
Quoted 21 lines
Quoted 21 lines
This sounds excellent. Can you tell me how? Could you maybe add an example to>>
lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:if (tracepoint_enabled(ust_tests_hello, tptest)) { /* prepare arguments */ do_tracepoint(ust_tests_hello, tptest, i, netint, values, text, strlen(text), dbl, flt); }If I understood you correctly, then I could do something like
if (tracepoint_enabled(ust_tests_hello, tptest)) { /* don't prepare arguments */ do_tracepoint(ust_tests_hello, my_complex_data); }Quoted 7 lines
Quoted 1 line
And then have the "prepare" code somewhere in my
TRACEPOINT_EVENT?Yes. Both approaches can be used.
Note that the second approach you refer to is the same as using a plain
tracepoint()macro and doing the preparation within theTRACEPOINT_EVENT()macro.Here's an example (C++):
#define _my_enum_evaluation(_enum, _field, _member) \ ctf_enum(my_provider, _enum, int, _field, \ (_member).is_bound() ? \ ((SomeApi::_enum::enum_type) (_member) < SomeApi::_enum::UNKNOWN_VALUE ? \ (int) ((SomeApi::_enum::enum_type) (_member)) : INT_MAX\ ) :
INT_MAX)TRACEPOINT_ENUM( my_provider, my_enum, TP_ENUM_VALUES( ctf_enum_value("unknown", 0) ctf_enum_value("apple", 1) ctf_enum_value("banana", 2) ctf_enum_value("orange", 3) ctf_enum_value("strawberry", 4) ) ) TRACEPOINT_EVENT( my_provider, my_tracepoint, TP_ARGS( int something, const SomeApi::SomeObject&, object ), TP_FIELDS( ctf_integer(int, int_field, something) _my_enum_evaluation(my_enum, enum_field, object.someMember()) ) )
Quoted 2 lines
You can call functions in there, evaluate conditions using the ternary operator, etc.
Thank you, that is exactly the example that I was looking for.
Cheers
Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Quoted 74 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:
----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]
Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line
----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]
Quoted 60 lines · wrote:
Quoted 24 lines
Quoted 8 lines
Quoted 6 lines
Hey all,
where can I find more documentation on how to use
sdt.hto add static tracepoints to user-land applications? If this is not the right place to
ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 13 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you can use
sdt.hinstrumentation.Quoted 7 lines
Quoted 2 lines
Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero- overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.
The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:
The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.
Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.
If we look at the code generated by SystemTAP
sdt.hprobes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.Quoted 7 lines
The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.
Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.
Quoted 6 lines
Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather,
sdt.hdoes allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of asdt.htracepoint vs. doing the analogous check for lttng-ust tracepoints?
Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?
Thanks,
Mathieu
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 88 lines · CEST Mathieu Desnoyers
Quoted 81 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:
----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]
Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line
----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]
Quoted 65 lines · wrote:
Quoted 28 lines
Quoted 10 lines
Quoted 8 lines
Hey all,
where can I find more documentation on how to use
sdt.hto add static
tracepoints to user-land applications? If this is not the right place
to
ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 15 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be
usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you
can usesdt.hinstrumentation.Quoted 8 lines
Quoted 2 lines
Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero-
overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:
The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.
Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.
If we look at the code generated by SystemTAP
sdt.hprobes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.Quoted 7 lines
The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.
Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.
Quoted 5 lines
Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather,
sdt.hdoes allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of asdt.htracepoint vs. doing the analogous check for lttng-ust tracepoints?
Quoted 3 lines
Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?
The fast path to check whether to trace or not:
$ cat probes.d
provider milian {
probe test();
};
$ dtrace -C -h -s probes.d -o probes.h
$ cat probes.h
/* Generated by the Systemtap dtrace wrapper */ #define _SDT_HAS_SEMAPHORES 1
#define STAP_HAS_SEMAPHORES 1 /* deprecated */
#include <sys/sdt.h>
/* MILIAN_TEST ( ) */
#if defined STAP_SDT_V1
#define MILIAN_TEST_ENABLED() __builtin_expect (test_semaphore, 0) #define milian_test_semaphore test_semaphore
#else
#define MILIAN_TEST_ENABLED() __builtin_expect (milian_test_semaphore, 0) #endif__extension__ extern unsigned short milian_test_semaphore __attribute__ ((unused)) __attribute__ ((section (".probes")));
#define MILIAN_TEST() \DTRACE_PROBE (milian, test)
Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
A message that is not in the archive (sent off-list or not yet received): 4486738.j9zaXj12nX@milian-kdab2
Quoted 112 lines · ----- On Sep 22, 2016, at 8:47 AM, Milian Wolff [email protected] wrote:
Quoted 88 lines · CEST Mathieu Desnoyers
Quoted 81 lines · ----- On Sep 20, 2016, at 4:34 PM, Milian Wolff [email protected] wrote:
Quoted 1 line · On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:
----- On Sep 12, 2016, at 11:40 AM, Milian Wolff [email protected]
Quoted 2 lines · wrote:
Quoted 2 lines · CEST Mathieu Desnoyers
Quoted 1 line
----- On Sep 6, 2016, at 3:00 PM, Milian Wolff [email protected]
Quoted 65 lines · wrote:
Quoted 28 lines
Quoted 10 lines
Quoted 8 lines
Hey all,
where can I find more documentation on how to use
sdt.hto add static
tracepoints to user-land applications? If this is not the right place
to
ask, please refer to to elsewhere.Hi Milian,
Hey Mathieu,
Quoted 15 lines
LTTng-UST offers a "tracepoint" instrumentation facility, which can optionally emit "
sdt.h" probe points too for compatibility with SystemTAP.Quoted 3 lines
I plan to upstream a collection of tracepoints to Qt, and possibly elsewhere. One problem I'm having right now is figuring out how to "design" the tracepoints such that they have minimal overhead.
The main question here would be: do you want your instrumentation to be
usable with LTTng-UST ? If yes, then you want the tracepoint instrumentation facility. Else, if you only plan on using SystemTAP, you
can usesdt.hinstrumentation.Quoted 8 lines
Quoted 2 lines
Quoted 1 line
So my questions:
My answers will be about lttng-ust tracepoints.
Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero-
overhead" like the SystemTap once, i.e. put into a separate section and "only" one nop is added when tracing is disabled? I could not find such an information anywhere yet.The lttng-ust tracepoint mechanism comes from the design of the Linux kernel tracepoints, which have proven to be unnoticeable when disabled. There is one slight technicality though:
The linux kernel tracepoints use the asm goto mechanism and code patching to flip between a no-op and a jump to dynamically enable each tracepoint. This is all very fine for the kernel.
Now for user-space, the lttng-ust tracepoints rather use a conditional branch to skip over the entire stack setup and the function call. There are a few reasons for using a old-fashioned conditional branch: first, there are no widely adopted multi-core, live code patching library available in user-space that am I aware of. Second, even if we would have one, we may not want to send SIGSTOP/SIGCONT to the traced process, due to debugger interactions. Thirdly, doing code patching may not interact will security features such as read-only code sections and code checksumming. Finally, doing code patching of library and executable code will trigger copy-on-write of the touched pages, which will prevent sharing those pages between processes running the same apps/libraries. All in all, the conditional branch does not seem like a too bad alternative after all.
If we look at the code generated by SystemTAP
sdt.hprobes (as well as DTrace), they are actually more heavyweight than what has been claimed by Sun's marketing department back in the days: it does cost a stack setup of all the variables passed to the the probe, and indeed the function call is no-op'd.Quoted 7 lines
The downside here is that all side-effects, and layout of the arguments for the no-op'd call, are taken by the instrumented application, even though tracing is not dynamically enabled.
Therefore, the SDT mechanism is not a lightweight as is generally claimed. It is not "just a single no-op per site", but rather a function call stack setup and a no-op.
Quoted 5 lines
Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what I gather,
sdt.hdoes allow to check whether a given trace point is enabled or not, no? I.e. via the semaphores. Is there any performance difference between checking the semaphore of asdt.htracepoint vs. doing the analogous check for lttng-ust tracepoints?Quoted 3 lines
Not sure what you refer to about "semaphores". And it's unclear how you define "checking if active": is it to decide in the fast path whether to trace or not, or in a slow path to query what is currently active ?
The fast path to check whether to trace or not:
$ cat probes.d provider milian { probe test(); }; $ dtrace -C -h -s probes.d -o probes.h $ cat probes.h /* Generated by the Systemtap dtrace wrapper */#define
_SDT_HAS_SEMAPHORES1#define
STAP_HAS_SEMAPHORES1 /* deprecated */#include <
sys/sdt.h>
Quoted 11 lines
/*
MILIAN_TEST( ) */
#if definedSTAP_SDT_V1
#defineMILIAN_TEST_ENABLED()__builtin_expect(test_semaphore, 0) #definemilian_test_semaphoretest_semaphore
#else
#defineMILIAN_TEST_ENABLED()__builtin_expect(milian_test_semaphore, 0) #endif__extension__extern unsigned shortmilian_test_semaphore__attribute__((unused))__attribute__((section (".probes")));
#defineMILIAN_TEST()\DTRACE_PROBE(milian, test)
Got it. The performance difference between lttng-ust tracepoint_enabled() check and the sdt.h semaphore check will be none, because both perform an unlikely branch (using builtin expect 0).
Thanks,
Mathieu
Quoted 11 lines
--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
_______________________________________________
lttng-dev mailing list [email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 7 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev