Skip to contentSkip to conversation

UST tracing problem on ARM/Android

DISCUSSION Charles Briere 2 participants 3 messages Jan 23, 2015 – Jan 26, 2015 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

So I just found out why this was hapenning and thought I would share my findings.

The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The function add_calliste will populate the callsite_table which is local to the dl_open(ed) library while tracepoint_sync_callsites will try to read an unitialized one, local to it's own symbols.

That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?

Thanks,
Charles

Quoted 30 lines · Charles Briere

Hello world,

I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.

lttng create
lttng enable-event -ua

Although, when a tracepoint is called, its state ( struct lttng_ust_tracepoint -> state) is 0.

My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.

I found out, that when registering the library (tracepoint_register_lib), callsite_table is populated. But later on when it comes to register the probes (lttng_prove_register), callsite_table 's address is different and the list is empty. As a result, when tracepoint_sync_callsite() is called, set_tracepoint() is never executed as looping through an empty list.

What I don't understand in here, is that the callsite_table pointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.

Thanks,
Charles

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

----- Original Message -----

Quoted 17 lines

From: "Charles Briere" <[email protected]>
To: [email protected]
Sent: Friday, January 23, 2015 2:15:40 PM
Subject: Re: [lttng-dev] UST tracing problem on ARM/Android

So I just found out why this was hapenning and thought I would share my findings.

The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The function add_calliste will populate the callsite_table which is local to the dl_open(ed) library while tracepoint_sync_callsites will try to read an unitialized one, local to it's own symbols.

Quoted 3 lines

That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?

Versioning would allow us to change the ABI if we really, really need to do it in the future. Ideally we would never change it, but you never know what the future has in store.

According to the opengroup POSIX spec:

http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html

"Only a single copy of an object file is brought into the address space, even if dlopen() is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file."

So it appears to be a bug in Android's implementation of their libc. You might want to report it to them.

One possible work-around we could implement in lttng-ust would be do to a wrapper for Android dlopen (let's call it lttng_ust_dlopen_wrapper) that first calls realpath(3) to first resolve the symlinks, and call dlopen on the result. This wrapper could be directly built into liblttng-ust.

Thoughts ?

Thanks,

Mathieu

Quoted 41 lines

Thanks,
Charles

Quoted 22 lines · Charles Briere

Hello world,

I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.

lttng create
lttng enable-event -ua

Although, when a tracepoint is called, its state ( struct lttng_ust_tracepoint -> state) is 0.

My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.

I found out, that when registering the library (tracepoint_register_lib), callsite_table is populated. But later on when it comes to register the probes (lttng_prove_register), callsite_table 's address is different and the list is empty. As a result, when tracepoint_sync_callsite() is called, set_tracepoint() is never executed as looping through an empty list.

Quoted 8 lines

What I don't understand in here, is that the callsite_table pointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.

Thanks,
Charles

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 26 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 17 lines

From: "Charles Briere" <[email protected]>
To: [email protected]
Sent: Friday, January 23, 2015 2:15:40 PM
Subject: Re: [lttng-dev] UST tracing problem on ARM/Android

So I just found out why this was hapenning and thought I would share my findings.

The Android linker will load a shared library twice in memory if it has a different path (even symbolic link). Therefore when dl_opening("lttng-ust-tracepoint.so.0", ...), even if that library is already loaded as it has been defined at link time, it will duplicate all it's symbols. The function add_calliste will populate the callsite_table which is local to the dl_open(ed) library while tracepoint_sync_callsites will try to read an unitialized one, local to it's own symbols.

Quoted 3 lines

That brings me to ask, why not passing "liblttng-ust-tracepoint.so" to dl_open()? What is the reason for versioning here as anyway at link time, liblttng-ust-tracepoint.so is the one beeing used?

Versioning would allow us to change the ABI if we really, really need to do it in the future. Ideally we would never change it, but you never know what the future has in store.

Quoted 8 lines

According to the opengroup POSIX spec:

http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html

"Only a single copy of an object file is brought into the address space, even if dlopen() is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file."

So it appears to be a bug in Android's implementation of their libc. You might want to report it to them.

Thanks for pointing that out and confirming it's a bug, I'll report it.

Quoted 8 lines

One possible work-around we could implement in lttng-ust would be do to a wrapper for Android dlopen (let's call it lttng_ust_dlopen_wrapper) that first calls realpath(3) to first resolve the symlinks, and call dlopen on the result. This wrapper could be directly built into liblttng-ust.

Thoughts ?

Agreed with this solution.

Thanks for the feedback,
Charles

Quoted 47 lines

Thanks,

Mathieu

Quoted 41 lines

Thanks,
Charles

Quoted 22 lines · Charles Briere

Hello world,

I've been running in some inssues lately with UST on ARM/Android. Using the following to configure my session, I enable every ust event.

lttng create
lttng enable-event -ua

Although, when a tracepoint is called, its state ( struct lttng_ust_tracepoint -> state) is 0.

My understanding of the whole registration process is limited, so the following reasoning is probably flawed, so please correct me if that happens.

I found out, that when registering the library (tracepoint_register_lib), callsite_table is populated. But later on when it comes to register the probes (lttng_prove_register), callsite_table 's address is different and the list is empty. As a result, when tracepoint_sync_callsite() is called, set_tracepoint() is never executed as looping through an empty list.

Quoted 8 lines

What I don't understand in here, is that the callsite_table pointer is not the same during the lib registration and probe registration. Both are ran sequentially in the same thread and I can't figure what other operation could affect the variable. Any hint/help would be much appreciated.

Thanks,
Charles

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev