Skip to contentSkip to conversation

perf tools: Fix buildid processing

PATCH In review Adrian Hunter 4 participants 10 messages Aug 19, 2015 – Aug 20, 2015 perf events

lore
Share & tools
lore

Adrian Hunter

author reviewer
#0 lore raw

cover perf tools: Fix buildid processing

Hi

Here are 3 patches related to the "getting stuck at end" problem.

The first patch dodges the deadlock.

The second patch stops fork from messing up i.e.
prevents the deadlock condition from arising in
the first place.

The third patch processes the data in order i.e.
prevents fork from having to deal with weird data
anyway.

Adrian Hunter (3):
perf tools: Avoid deadlock when map_groups are broken perf tools: Make fork event processing more resilient perf tools: Fix buildid processing

 tools/perf/builtin-record.c | 11 +++++++++++
 tools/perf/util/machine.c   | 20 ++++++++++++++++++--
 tools/perf/util/thread.c    |  6 ++++++
 3 files changed, 35 insertions(+), 2 deletions(-)

Regards
Adrian

author reviewer
#1 lore raw

PATCH 1/3 perf tools: Avoid deadlock when map_groups are broken

Attempting to clone map groups onto themselves will deadlock. It only happens because of other bugs, but the code should protect itself anyway.

---
 tools/perf/util/thread.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/util/thread.c +6 -0
index 28c4b746baa1..471102554a78 100644
@@ -191,6 +191,12 @@ static int thread__clone_map_groups(struct thread *thread,
 	if (thread->pid_ == parent->pid_)
 		return 0;
 
+	if (thread->mg == parent->mg) {
+		dump_printf("broken map groups on thread %d/%d parent %d/%d\n",
+			    thread->pid_, thread->tid, parent->pid_, parent->tid);
+		return 0;
+	}
+
 	/* But this one is new process, copy maps. */
 	for (i = 0; i < MAP__NR_TYPES; ++i)
 		if (map_groups__clone(thread->mg, parent->mg, i) < 0)
Signature · 1 line
1.9.1
#2 lore raw

[tip:perf/core] perf tools: Avoid deadlock when map_groups are broken

perf tools: Avoid deadlock when map_groups are broken
Commit-ID:  0d7e7acc47133e82911e24758178815cfee309e9
Gitweb:     http://git.kernel.org/tip/0d7e7acc47133e82911e24758178815cfee309e9
Author:     Adrian Hunter <[email protected]>
AuthorDate: Wed, 19 Aug 2015 17:29:19 +0300
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Wed, 19 Aug 2015 14:15:09 -0300

perf tools: Avoid deadlock when map_groups are broken

Attempting to clone map groups onto themselves will deadlock.

It only happens because of other bugs, but the code should protect itself anyway.

---
 tools/perf/util/thread.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/util/thread.c +6 -0
index 28c4b74..0a9ae80 100644
@@ -191,6 +191,12 @@ static int thread__clone_map_groups(struct thread *thread,
 	if (thread->pid_ == parent->pid_)
 		return 0;
 
+	if (thread->mg == parent->mg) {
+		pr_debug("broken map groups on thread %d/%d parent %d/%d\n",
+			 thread->pid_, thread->tid, parent->pid_, parent->tid);
+		return 0;
+	}
+
 	/* But this one is new process, copy maps. */
 	for (i = 0; i < MAP__NR_TYPES; ++i)
 		if (map_groups__clone(thread->mg, parent->mg, i) < 0)
author reviewer
#3 lore raw

PATCH 2/3 perf tools: Make fork event processing more resilient

In reply to Adrian Hunter

When processing a fork event, the tools lookup the parent thread by its tid. In a couple of cases, it is possible for that thread to have the wrong pid. That can happen if the data is being processed out of order, or if the (fork) event that would have removed the erroneous thread was lost. Assume the latter case, print a dump message, remove the erroneous thread, create a new one with the correct pid, and keep going.

---
 tools/perf/util/machine.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
tools/perf/util/machine.c +18 -2
index 7ff682770fdb..f1a4c833121e 100644
@@ -1387,6 +1387,24 @@ int machine__process_fork_event(struct machine *machine, union perf_event *event
 							event->fork.ptid);
 	int err = 0;
 
+	if (dump_trace)
+		perf_event__fprintf_task(event, stdout);
+
+	/*
+	 * There may be an existing thread that is not actually the parent,
+	 * either because we are processing events out of order, or because the
+	 * (fork) event that would have removed the thread was lost. Assume the
+	 * latter case and continue on as best we can.
+	 */
+	if (parent->pid_ != (pid_t)event->fork.ppid) {
+		dump_printf("removing erroneous parent thread %d/%d\n",
+			    parent->pid_, parent->tid);
+		machine__remove_thread(machine, parent);
+		thread__put(parent);
+		parent = machine__findnew_thread(machine, event->fork.ppid,
+						 event->fork.ptid);
+	}
+
 	/* if a thread currently exists for the thread id remove it */
 	if (thread != NULL) {
 		machine__remove_thread(machine, thread);
@@ -1395,8 +1413,6 @@ int machine__process_fork_event(struct machine *machine, union perf_event *event
 
 	thread = machine__findnew_thread(machine, event->fork.pid,
 					 event->fork.tid);
-	if (dump_trace)
-		perf_event__fprintf_task(event, stdout);
 
 	if (thread == NULL || parent == NULL ||
 	    thread__fork(thread, parent, sample->time) < 0) {
Signature · 1 line
1.9.1
#4 lore raw

[tip:perf/core] perf tools: Make fork event processing more resilient

perf tools: Make fork event processing more resilient
Commit-ID:  5cb73340d92a716fd2776700742c3558206ae298
Gitweb:     http://git.kernel.org/tip/5cb73340d92a716fd2776700742c3558206ae298
Author:     Adrian Hunter <[email protected]>
AuthorDate: Wed, 19 Aug 2015 17:29:20 +0300
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Wed, 19 Aug 2015 14:15:25 -0300

perf tools: Make fork event processing more resilient

When processing a fork event, the tools lookup the parent thread by its tid. In a couple of cases, it is possible for that thread to have the wrong pid.

That can happen if the data is being processed out of order, or if the (fork) event that would have removed the erroneous thread was lost.

Assume the latter case, print a dump message, remove the erroneous thread, create a new one with the correct pid, and keep going.

---
 tools/perf/util/machine.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
tools/perf/util/machine.c +18 -2
index 7ff6827..f1a4c83 100644
@@ -1387,6 +1387,24 @@ int machine__process_fork_event(struct machine *machine, union perf_event *event
 							event->fork.ptid);
 	int err = 0;
 
+	if (dump_trace)
+		perf_event__fprintf_task(event, stdout);
+
+	/*
+	 * There may be an existing thread that is not actually the parent,
+	 * either because we are processing events out of order, or because the
+	 * (fork) event that would have removed the thread was lost. Assume the
+	 * latter case and continue on as best we can.
+	 */
+	if (parent->pid_ != (pid_t)event->fork.ppid) {
+		dump_printf("removing erroneous parent thread %d/%d\n",
+			    parent->pid_, parent->tid);
+		machine__remove_thread(machine, parent);
+		thread__put(parent);
+		parent = machine__findnew_thread(machine, event->fork.ppid,
+						 event->fork.ptid);
+	}
+
 	/* if a thread currently exists for the thread id remove it */
 	if (thread != NULL) {
 		machine__remove_thread(machine, thread);
@@ -1395,8 +1413,6 @@ int machine__process_fork_event(struct machine *machine, union perf_event *event
 
 	thread = machine__findnew_thread(machine, event->fork.pid,
 					 event->fork.tid);
-	if (dump_trace)
-		perf_event__fprintf_task(event, stdout);
 
 	if (thread == NULL || parent == NULL ||
 	    thread__fork(thread, parent, sample->time) < 0) {
author reviewer
#5 lore raw

PATCH 3/3 perf tools: Fix buildid processing

In reply to Adrian Hunter

After recording, 'perf record' post-processes the data to determine which buildids are needed. That processing must process the data in time order, if possible, because otherwise dependent events, like forks and mmaps, will not make sense.

---
 tools/perf/builtin-record.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/perf/builtin-record.c +11 -0
index de165a1b9240..f36b88938499 100644
@@ -500,6 +500,15 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		return -1;
 	}
 
+	/*
+	 * Normally perf_session__new would do this, but it doesn't have the
+	 * evlist.
+	 */
+	if (rec->tool.ordered_events && !perf_evlist__sample_id_all(rec->evlist)) {
+		dump_printf("WARNING: No sample_id_all support, falling back to unordered processing\n");
+		rec->tool.ordered_events = false;
+	}
+
 	fd = perf_data_file__fd(file);
 	rec->session = session;
 
@@ -965,9 +974,11 @@ static struct record record = {
 	.tool = {
 		.sample		= process_sample_event,
 		.fork		= perf_event__process_fork,
+		.exit		= perf_event__process_exit,
 		.comm		= perf_event__process_comm,
 		.mmap		= perf_event__process_mmap,
 		.mmap2		= perf_event__process_mmap2,
+		.ordered_events	= true,
 	},
 };
 
Signature · 1 line
1.9.1
author reviewer
#6 lore raw
Quoted 18 lines · Adrian Hunter

After recording, 'perf record' post-processes the data to determine which buildids are needed. That processing must process the data in time order, if possible, because otherwise dependent events, like forks and mmaps, will not make sense.

---
  tools/perf/builtin-record.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
tools/perf/builtin-record.c +0 -0
index de165a1b9240..f36b88938499 100644
@@ -500,6 +500,15 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
  		return -1;
  	}

tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 511

Quoted 10 lines
tools/perf/builtin-record.c +9 -0
 
+	/*
+	 * Normally perf_session__new would do this, but it doesn't have the
+	 * evlist.
+	 */
+	if (rec->tool.ordered_events && !perf_evlist__sample_id_all(rec->evlist)) {
+		dump_printf("WARNING: No sample_id_all support, falling back to unordered processing\n");
+		rec->tool.ordered_events = false;
+	}
+

In fact this chunk is misplaced, it needs to be below the call to record__open() after which the attributes are correctly configured. And dump_printf() is no good for 'perf record', pr_debug would be better. Don't have time to do that right now, I'll do it tommorow unless Arnaldo fixes it up.

Quoted 16 lines
tools/perf/builtin-record.c +2 -0
  	fd = perf_data_file__fd(file);
  	rec->session = session;
 
@@ -965,9 +974,11 @@ static struct record record = {
  	.tool = {
  		.sample		= process_sample_event,
  		.fork		= perf_event__process_fork,
+		.exit		= perf_event__process_exit,
  		.comm		= perf_event__process_comm,
  		.mmap		= perf_event__process_mmap,
  		.mmap2		= perf_event__process_mmap2,
+		.ordered_events	= true,
  	},
  };
 
#7 lore raw

Em Wed, Aug 19, 2015 at 06:23:51PM +0300, Adrian Hunter escreveu:

Quoted 20 lines · Adrian Hunter
Quoted 19 lines

After recording, 'perf record' post-processes the data to determine which buildids are needed. That processing must process the data in time order, if possible, because otherwise dependent events, like forks and mmaps, will not make sense.

---
 tools/perf/builtin-record.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/perf/builtin-record.c +0 -0
index de165a1b9240..f36b88938499 100644
@@ -500,6 +500,15 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		return -1;
 	}
 
Quoted 14 lines

tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 511

Quoted 9 lines
tools/perf/builtin-record.c +9 -0
+	/*
+	 * Normally perf_session__new would do this, but it doesn't have the
+	 * evlist.
+	 */
+	if (rec->tool.ordered_events && !perf_evlist__sample_id_all(rec->evlist)) {
+		dump_printf("WARNING: No sample_id_all support, falling back to unordered processing\n");
+		rec->tool.ordered_events = false;
+	}
+

In fact this chunk is misplaced, it needs to be below the call to record__open() after which the attributes are correctly configured. And dump_printf() is no good for 'perf record', pr_debug would be better. Don't have time to do that right now, I'll do it tommorow unless Arnaldo fixes it up.

I'll fix it up, thanks!

- Arnaldo

Quoted 16 lines
Quoted 16 lines
tools/perf/builtin-record.c +2 -0
 	fd = perf_data_file__fd(file);
 	rec->session = session;
 
@@ -965,9 +974,11 @@ static struct record record = {
 	.tool = {
 		.sample		= process_sample_event,
 		.fork		= perf_event__process_fork,
+		.exit		= perf_event__process_exit,
 		.comm		= perf_event__process_comm,
 		.mmap		= perf_event__process_mmap,
 		.mmap2		= perf_event__process_mmap2,
+		.ordered_events	= true,
 	},
 };
 
#8 lore raw
In reply to Adrian Hunter
perf tools: Fix buildid processing
Commit-ID:  cca8482c0651cea97aade58cc22109ce9fffbfa2
Gitweb:     http://git.kernel.org/tip/cca8482c0651cea97aade58cc22109ce9fffbfa2
Author:     Adrian Hunter <[email protected]>
AuthorDate: Wed, 19 Aug 2015 17:29:21 +0300
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Wed, 19 Aug 2015 14:15:26 -0300

perf tools: Fix buildid processing

After recording, 'perf record' post-processes the data to determine which buildids are needed.

That processing must process the data in time order, if possible, because otherwise dependent events, like forks and mmaps, will not make sense.

Cc Linus Torvalds <[email protected]>
[ Moved the sample_id_add to after trying to open the events, use pr_warning ]
---
 tools/perf/builtin-record.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/perf/builtin-record.c +11 -0
index de165a1..20b56eb 100644
@@ -521,6 +521,15 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		goto out_child;
 	}
 
+	/*
+	 * Normally perf_session__new would do this, but it doesn't have the
+	 * evlist.
+	 */
+	if (rec->tool.ordered_events && !perf_evlist__sample_id_all(rec->evlist)) {
+		pr_warning("WARNING: No sample_id_all support, falling back to unordered processing\n");
+		rec->tool.ordered_events = false;
+	}
+
 	if (!rec->evlist->nr_groups)
 		perf_header__clear_feat(&session->header, HEADER_GROUP_DESC);
 
@@ -965,9 +974,11 @@ static struct record record = {
 	.tool = {
 		.sample		= process_sample_event,
 		.fork		= perf_event__process_fork,
+		.exit		= perf_event__process_exit,
 		.comm		= perf_event__process_comm,
 		.mmap		= perf_event__process_mmap,
 		.mmap2		= perf_event__process_mmap2,
+		.ordered_events	= true,
 	},
 };
 
#9 lore raw
In reply to Adrian Hunter
Quoted 6 lines · Adrian Hunter

Hi

Here are 3 patches related to the "getting stuck at end" problem.

The first patch dodges the deadlock.

Quoted 7 lines

The second patch stops fork from messing up i.e.
prevents the deadlock condition from arising in
the first place.

The third patch processes the data in order i.e.
prevents fork from having to deal with weird data
anyway.

Tested/Acked-by: Jiri Olsa <[email protected]>

thanks,
jirka