Skip to contentSkip to conversation

baddr feature triggers deadlock in lttng-ust

DISCUSSION Alexandre Montplaisir 3 participants 5 messages Feb 27, 2014 – Feb 28, 2014 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Could it be hidden behind a non-default configure option, instead of completely removed?

Alex

Quoted 13 lines · Mathieu Desnoyers

Hi Paul,

Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.

We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.

Thanks,

Mathieu

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

As fixing it will very likely require a major rethinking of the locking strategy for the entire UST, I'm not very keen to make that kind of change within stable versions.

Unless an elegant solution shows up by tomorrow, I plan to disable the feature (leaving the code there), and think up a solution for 2.5.

Thoughts ?

Thanks,

Mathieu

----- Original Message -----

Quoted 28 lines

From: "Alexandre Montplaisir" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: "Paul Woegerer" <[email protected]>, [email protected]
Sent: Thursday, February 27, 2014 6:37:23 PM
Subject: Re: [lttng-dev] baddr feature triggers deadlock in lttng-ust

Could it be hidden behind a non-default configure option, instead of completely removed?

Alex

Quoted 13 lines · Mathieu Desnoyers

Hi Paul,

Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.

We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#4 lore raw
In reply to a message not in the archive

Hi Mathieu,

this is very unfortunate. I tested forks as well but couldn’t find any issues on x86_64. Ironically, last week we found an issue that matches your description on PowerPC but since we could not reproduce it on x86 I blamed it on the peculiarities of that platform and worked on seemingly higher priority issues instead.

For the sake of damage containment I just posted:
http://lists.lttng.org/pipermail/lttng-dev/2014-February/022494.html
to the list.

Not having that feature in 2.4 (at least in the experimental form suggested above) would be a disaster for us.

Meanwhile I'm trying to think about a more fundamental solution to the problem. But I'm afraid your comment about rethinking of the locking strategy for the entire UST might come true.

Thanks,
Paul

Quoted 13 lines · Mathieu Desnoyers

Hi Paul,

Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.

We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.

Thanks,

Mathieu

Signature · 8 lines
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
#5 lore raw
In reply to a message not in the archive

I have looked a bit into the issue...

The ust_locking inside the dl_iterate_phdr triggers the deadlock.

If I just collect the base address info inside dl_iterate_phdr and dump the collected data with trace_baddr afterwards (outside the dl_iterate_phdr iteration) the deadlock will be prevented.

I'm now going to create a patch that implements this approach.

Thanks,
Paul

Quoted 13 lines · Mathieu Desnoyers

Hi Paul,

Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.

We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.

Thanks,

Mathieu

Signature · 3 lines
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

----- Original Message -----

Quoted 8 lines

From: "Paul Woegerer" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>, [email protected], "CS ESD-Analyzer" <[email protected]>
Sent: Friday, February 28, 2014 7:38:55 AM
Subject: Re: baddr feature triggers deadlock in lttng-ust

I have looked a bit into the issue...

Quoted 7 lines

The ust_locking inside the dl_iterate_phdr triggers the deadlock.

If I just collect the base address info inside dl_iterate_phdr and dump the collected data with trace_baddr afterwards (outside the dl_iterate_phdr iteration) the deadlock will be prevented.

I'm now going to create a patch that implements this approach.

I suspect that having the UST lock nested inside the dynamic loader lock only makes the problem happen more consistently, but that the basic issue is that the dynamic loader lock should not be held while another thread concurrently calls fork/clone/daemon. It seems to leave the child process in a state where the lock state is stale (taken).

Thanks,

Mathieu

Quoted 26 lines

Thanks,
Paul

Quoted 5 lines · Mathieu Desnoyers

Hi Paul,

Please see http://bugs.lttng.org/issues/745, which explains a deadlock we just found out.

Quoted 8 lines

We might have to disable the baddr dump feature, since our release is tomorrow. We'll have to re-think our locking to protect operations using the dynamic loader lock against fork, clone, and daemon.

Thanks,

Mathieu

--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com