Skip to contentSkip to conversation

Bug in LTTng-UST

DISCUSSION Simon Marchi 2 participants 3 messages Dec 13, 2014 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1418432075.9766.6.camel@moya

#1 lore raw

Actually, I think the problem comes from UST not handling when a program dlcloses a lib. There is a simpler way to show the problem. Just have an app dlopen a .so that has a tracepoint and dlclose it. When the app quits, UST will try to unregister the tracepoint and read/write in the memory previously mapped to the library, causing a segfault.

I have made a small example. You just need to click on "Download gist" in [1] to download a tarball, and a then simple make should be enough. Start a UST session with -u -a and run ./app. Segfault should ensue.

[1] https://gist.github.com/simark/9fc0e2395899fcd8d10d

Quoted 25 lines · Matthew Khouzam

Hi, it appears as though if a tracepoint is in a so, loading it, unloading it and reloading it makes it no longer visible.

To confirm this you can try out this repo :
https://github.com/TheMatthew/yquake2
(I know, it's not a serious program... but the bug is serious, I think) You can get the demo files here
http://www.fileplanet.com/6584/0/fileinfo/Quake-2-Demo-v3.14
make the program (make from the git dir)
copy the contents of the baseq2 in the demo into the release/baseq2 directory

start a ust session
run the program.
Ignore the baddr deadlocks :)
The tracepoint is not visible, it is located in the game.so file.

Chances are though that the problem is on my side, but the evidence (single registration ) seems to indicate that this is an actual problem.

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAFXXi0nPMbLj-nAT281+7P=iUK+pg_PdMOP52XabquvVUikHSQ@mail.gmail.com

#3 lore raw
Quoted 7 lines · Simon Marchi

Actually, I think the problem comes from UST not handling when a program dlcloses a lib. There is a simpler way to show the problem. Just have an app dlopen a .so that has a tracepoint and dlclose it. When the app quits, UST will try to unregister the tracepoint and read/write in the memory previously mapped to the library, causing a segfault.

Quoted 5 lines

I have made a small example. You just need to click on "Download gist" in [1] to download a tarball, and a then simple make should be enough. Start a UST session with -u -a and run ./app. Segfault should ensue.

[1] https://gist.github.com/simark/9fc0e2395899fcd8d10d

Don't you have to _preload_ the tracepoint provider in this context? Either by dlopen()ing the tracepoint provider shared object manually, or by using LD_PRELOAD?

Try this instead <https://gist.github.com/eepp/d280a27f5e7d0a64abaa>.
Look at the changes to Makefile, tracepoints.c and truc.c. Make, then launch the app while tracing is started:

    ./app

Nothing is recorded: the tracepoint provider shared object is not loaded. Now try preloading the tracepoint provider and launching the app:

LD_PRELOAD=./tracepoints.so ./app

I get this:

    lttng stop
    lttng view
    ...
    [23:55:36.393480541] (+0.000074994) eeppdesk

sample_tracepoint:message: { cpu_id = 0 }, { intfield = 4 }

and no segfault.

See: <http://lttng.org/docs/#doc-building-tracepoint-providers-and-user-application>,
"Dynamic linking" section.

Phil

Quoted 35 lines
Quoted 17 lines · Matthew Khouzam

Hi, it appears as though if a tracepoint is in a so, loading it, unloading it and reloading it makes it no longer visible.

To confirm this you can try out this repo :
https://github.com/TheMatthew/yquake2
(I know, it's not a serious program... but the bug is serious, I think) You can get the demo files here
http://www.fileplanet.com/6584/0/fileinfo/Quake-2-Demo-v3.14
make the program (make from the git dir)
copy the contents of the baseq2 in the demo into the release/baseq2 directory

start a ust session
run the program.
Ignore the baddr deadlocks :)
The tracepoint is not visible, it is located in the game.so file.

Quoted 8 lines

Chances are though that the problem is on my side, but the evidence (single registration ) seems to indicate that this is an actual problem.

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

A message that is not in the archive (sent off-list or not yet received): CAB4xu_0abUHmaRSYSx1ehSx2jH1s3Oqm++mw6rNOXarbP31ZKg@mail.gmail.com

#5 lore raw
Quoted 35 lines · Philippe Proulx
Quoted 7 lines · Simon Marchi

Actually, I think the problem comes from UST not handling when a program dlcloses a lib. There is a simpler way to show the problem. Just have an app dlopen a .so that has a tracepoint and dlclose it. When the app quits, UST will try to unregister the tracepoint and read/write in the memory previously mapped to the library, causing a segfault.

Quoted 5 lines

I have made a small example. You just need to click on "Download gist" in [1] to download a tarball, and a then simple make should be enough. Start a UST session with -u -a and run ./app. Segfault should ensue.

[1] https://gist.github.com/simark/9fc0e2395899fcd8d10d

Don't you have to _preload_ the tracepoint provider in this context? Either by dlopen()ing the tracepoint provider shared object manually, or by using LD_PRELOAD?

Try this instead <https://gist.github.com/eepp/d280a27f5e7d0a64abaa>.
Look at the changes to Makefile, tracepoints.c and truc.c. Make, then launch the app while tracing is started:

    ./app

Nothing is recorded: the tracepoint provider shared object is not loaded. Now try preloading the tracepoint provider and launching the app:

LD_PRELOAD=./tracepoints.so ./app

I get this:

    lttng stop
    lttng view
    ...
    [23:55:36.393480541] (+0.000074994) eeppdesk
Quoted 8 lines

sample_tracepoint:message: { cpu_id = 0 }, { intfield = 4 }

and no segfault.

See: <http://lttng.org/docs/#doc-building-tracepoint-providers-and-user-application>,
"Dynamic linking" section.

Phil

Ok so as Philippe noticed, the man page is quite clear about this.

- Note about dlclose() usage: it is not safe to use dlclose on a provider shared object that is being actively used for tracing due to a lack of reference counting from lttng-ust to the used shared object.

Given that game.so *is* the trace provider, we can't dlclose it. I suppose the solution would be to put the tracepoints in a separate library (game-tracepoints.so) and LD_PRELOAD that library whenever you want to trace. I'll work on that.