baddr feature triggers deadlock in lttng-ust
Could it be hidden behind a non-default configure option, instead of completely removed?
Alex
Quoted 13 lines · Mathieu Desnoyers
Hi Paul,
Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.
We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.
Thanks,
Mathieu
As fixing it will very likely require a major rethinking of the locking strategy for the entire UST, I'm not very keen to make that kind of change within stable versions.
Unless an elegant solution shows up by tomorrow, I plan to disable the feature (leaving the code there), and think up a solution for 2.5.
Thoughts ?
Thanks,
Mathieu
----- Original Message -----
Quoted 28 lines
From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Paul Woegerer" <[email protected]>, [email protected]
Sent: Thursday, February 27, 2014 6:37:23 PM
Subject: Re: [lttng-dev] baddr feature triggers deadlock in lttng-ustCould it be hidden behind a non-default configure option, instead of completely removed?
Alex
Quoted 13 lines · Mathieu Desnoyers
Hi Paul,
Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.
We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Hi Mathieu,
this is very unfortunate. I tested forks as well but couldn’t find any issues on x86_64. Ironically, last week we found an issue that matches your description on PowerPC but since we could not reproduce it on x86 I blamed it on the peculiarities of that platform and worked on seemingly higher priority issues instead.
For the sake of damage containment I just posted:
http://lists.lttng.org/pipermail/lttng-dev/2014-February/022494.html
to the list.
Not having that feature in 2.4 (at least in the experimental form suggested above) would be a disaster for us.
Meanwhile I'm trying to think about a more fundamental solution to the problem. But I'm afraid your comment about rethinking of the locking strategy for the entire UST might come true.
Thanks,
Paul
Quoted 13 lines · Mathieu Desnoyers
Hi Paul,
Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.
We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.
Thanks,
Mathieu
Signature · 8 lines
Paul Woegerer, SW Development Engineer Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer> Mentor Graphics, Embedded Software Division _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
I have looked a bit into the issue...
The ust_locking inside the dl_iterate_phdr triggers the deadlock.
If I just collect the base address info inside dl_iterate_phdr and dump the collected data with trace_baddr afterwards (outside the dl_iterate_phdr iteration) the deadlock will be prevented.
I'm now going to create a patch that implements this approach.
Thanks,
Paul
Quoted 13 lines · Mathieu Desnoyers
Hi Paul,
Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.
We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.
Thanks,
Mathieu
Signature · 3 lines
Paul Woegerer, SW Development Engineer Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer> Mentor Graphics, Embedded Software Division
----- Original Message -----
Quoted 8 lines
From: "Paul Woegerer" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, [email protected], "CS ESD-Analyzer" <[email protected]>
Sent: Friday, February 28, 2014 7:38:55 AM
Subject: Re: baddr feature triggers deadlock in lttng-ustI have looked a bit into the issue...
Quoted 7 lines
The
ust_lockinginside thedl_iterate_phdrtriggers the deadlock.If I just collect the base address info inside
dl_iterate_phdrand dump the collected data withtrace_baddrafterwards (outside thedl_iterate_phdriteration) the deadlock will be prevented.I'm now going to create a patch that implements this approach.
I suspect that having the UST lock nested inside the dynamic loader lock only makes the problem happen more consistently, but that the basic issue is that the dynamic loader lock should not be held while another thread concurrently calls fork/clone/daemon. It seems to leave the child process in a state where the lock state is stale (taken).
Thanks,
Mathieu
Quoted 26 lines
Thanks,
PaulQuoted 5 lines · Mathieu Desnoyers
Hi Paul,
Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.
Quoted 8 lines
We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.
Thanks,
Mathieu
--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com