UST tracing problem on ARM/Android
So I just found out why this was hapenning and thought I would share my findings.
The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The function add_calliste will populate the callsite_table which is local to the dl_open(ed) library while tracepoint_sync_callsites will try to read an unitialized one, local to it's own symbols.
That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?
Thanks,
Charles
Quoted 30 lines · Charles Briere
Hello world,
I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.
lttng create
lttng enable-event -uaAlthough, when a tracepoint is called, its state ( struct
lttng_ust_tracepoint-> state) is 0.My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.
I found out, that when registering the library (
tracepoint_register_lib),callsite_tableis populated. But later on when it comes to register the probes (lttng_prove_register),callsite_table's address is different and the list is empty. As a result, whentracepoint_sync_callsite()is called,set_tracepoint()is never executed as looping through an empty list.What I don't understand in here, is that the
callsite_tablepointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.Thanks,
Charles
----- Original Message -----
Quoted 17 lines
From: "Charles Briere" <[email protected]>
To: [email protected]
Sent: Friday, January 23, 2015 2:15:40 PM
Subject: Re: [lttng-dev] UST tracing problem on ARM/AndroidSo I just found out why this was hapenning and thought I would share my findings.
The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when
dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The functionadd_callistewill populate thecallsite_tablewhich is local to thedl_open(ed)library whiletracepoint_sync_callsiteswill try to read an unitialized one, local to it's own symbols.
Quoted 3 lines
That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to
dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?
Versioning would allow us to change the ABI if we really, really need to do it in the future. Ideally we would never change it, but you never know what the future has in store.
According to the opengroup POSIX spec:
http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html
"Only a single copy of an object file is brought into the address space, even if dlopen() is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file."
So it appears to be a bug in Android's implementation of their libc. You might want to report it to them.
One possible work-around we could implement in lttng-ust would be do to a wrapper for Android dlopen (let's call it lttng_ust_dlopen_wrapper) that first calls realpath(3) to first resolve the symlinks, and call dlopen on the result. This wrapper could be directly built into liblttng-ust.
Thoughts ?
Thanks,
Mathieu
Quoted 41 lines
Thanks,
CharlesQuoted 22 lines · Charles Briere
Hello world,
I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.
lttng create
lttng enable-event -uaAlthough, when a tracepoint is called, its state ( struct
lttng_ust_tracepoint-> state) is 0.My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.
I found out, that when registering the library (
tracepoint_register_lib),callsite_tableis populated. But later on when it comes to register the probes (lttng_prove_register),callsite_table's address is different and the list is empty. As a result, whentracepoint_sync_callsite()is called,set_tracepoint()is never executed as looping through an empty list.Quoted 8 lines
What I don't understand in here, is that the
callsite_tablepointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.Thanks,
Charles_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 26 lines · Mathieu Desnoyers
----- Original Message -----
Quoted 17 lines
From: "Charles Briere" <[email protected]>
To: [email protected]
Sent: Friday, January 23, 2015 2:15:40 PM
Subject: Re: [lttng-dev] UST tracing problem on ARM/AndroidSo I just found out why this was hapenning and thought I would share my findings.
The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when
dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The functionadd_callistewill populate thecallsite_tablewhich is local to thedl_open(ed)library whiletracepoint_sync_callsiteswill try to read an unitialized one, local to it's own symbols.Quoted 3 lines
That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to
dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?Versioning would allow us to change the ABI if we really, really need to do it in the future. Ideally we would never change it, but you never know what the future has in store.
Quoted 8 lines
According to the opengroup POSIX spec:
http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html
"Only a single copy of an object file is brought into the address space, even if
dlopen()is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file."So it appears to be a bug in Android's implementation of their libc. You might want to report it to them.
Thanks for pointing that out and confirming it's a bug, I'll report it.
Quoted 8 lines
One possible work-around we could implement in lttng-ust would be do to a wrapper for Android dlopen (let's call it
lttng_ust_dlopen_wrapper) that first calls realpath(3) to first resolve the symlinks, and call dlopen on the result. This wrapper could be directly built into liblttng-ust.Thoughts ?
Agreed with this solution.
Thanks for the feedback,
Charles
Quoted 47 lines
Thanks,
Mathieu
Quoted 41 lines
Thanks,
CharlesQuoted 22 lines · Charles Briere
Hello world,
I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.
lttng create
lttng enable-event -uaAlthough, when a tracepoint is called, its state ( struct
lttng_ust_tracepoint-> state) is 0.My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.
I found out, that when registering the library (
tracepoint_register_lib),callsite_tableis populated. But later on when it comes to register the probes (lttng_prove_register),callsite_table's address is different and the list is empty. As a result, whentracepoint_sync_callsite()is called,set_tracepoint()is never executed as looping through an empty list.Quoted 8 lines
What I don't understand in here, is that the
callsite_tablepointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.Thanks,
Charles_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev