Skip to contentSkip to conversation

perf sched latency: Fix removed thread issue

PATCH In review Jiri Olsa 5 participants 8 messages Nov 2, 2015 – Nov 8, 2015 perf events

lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

PATCH 1/1 perf sched latency: Fix removed thread issue

If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:

thread_atoms_search: Assertion `!(thread != atoms->thread)' failed

Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.

I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.

---
 tools/perf/builtin-sched.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
index 0ee6d900e100..e3d3e32c0a93 100644
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_
 
 static int pid_cmp(struct work_atoms *l, struct work_atoms *r)
 {
+	if (l->thread == r->thread)
+		return 0;
 	if (l->thread->tid < r->thread->tid)
 		return -1;
 	if (l->thread->tid > r->thread->tid)
 		return 1;
-
-	return 0;
+	return (int)(l->thread - r->thread);
 }
 
 static int avg_cmp(struct work_atoms *l, struct work_atoms *r)
Signature · 1 line
2.4.3
maintainer Acked-by on v1 1/1
#1 lore raw
Quoted 1 line · Jiri Olsa

If machine's thread gets excited (EXIT event is received),

Why a thread get *excited* when it received EXIT event? :)

Quoted 10 lines

we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:

thread_atoms_search: Assertion `!(thread != atoms->thread)' failed

Quoted 7 lines

Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.

I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.

Right. I'll work on the v6 soon.

As a work around:

Thanks,
Namhyung

Quoted 31 lines
---
 tools/perf/builtin-sched.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
index 0ee6d900e100..e3d3e32c0a93 100644
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_
 
 static int pid_cmp(struct work_atoms *l, struct work_atoms *r)
 {
+	if (l->thread == r->thread)
+		return 0;
 	if (l->thread->tid < r->thread->tid)
 		return -1;
 	if (l->thread->tid > r->thread->tid)
 		return 1;
-
-	return 0;
+	return (int)(l->thread - r->thread);
 }
 
 static int avg_cmp(struct work_atoms *l, struct work_atoms *r)
Signature · 1 line
2.4.3
#2 lore raw
Quoted 4 lines · Namhyung Kim
Quoted 1 line · Jiri Olsa

If machine's thread gets excited (EXIT event is received),

Why a thread get *excited* when it received EXIT event? :)

wouldn't you? I would.. ;-)

jirka

#3 lore raw
In reply to Jiri Olsa

Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:

Quoted 6 lines

If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid

Humm, wher is the latency subcommand handling the EXIT event?

I see:

   perf_sched__lat
     perf_sched__read_events
       session = perf_session__new(&file, false, &sched->tool);
       perf_session__process_events(session)

And sched->tool->exit() is not set, which will make perf_session__process_events(), when calling perf_tool__fill_defaults() set it to process_event_stub() which will do nothing for PERF_RECORD_EXIT events, no?

Is there some perf.data file somewhere to reproduce this problem?

- Arnaldo

Quoted 42 lines

and tid, the old working atom is found and assert bug condition is hit in search function:

thread_atoms_search: Assertion `!(thread != atoms->thread)' failed

Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.

I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.

---
 tools/perf/builtin-sched.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
index 0ee6d900e100..e3d3e32c0a93 100644
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_
 
 static int pid_cmp(struct work_atoms *l, struct work_atoms *r)
 {
+	if (l->thread == r->thread)
+		return 0;
 	if (l->thread->tid < r->thread->tid)
 		return -1;
 	if (l->thread->tid > r->thread->tid)
 		return 1;
-
-	return 0;
+	return (int)(l->thread - r->thread);
 }
 
 static int avg_cmp(struct work_atoms *l, struct work_atoms *r)
Signature · 1 line
2.4.3
#4 lore raw
Quoted 17 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:

Quoted 6 lines

If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid

Humm, wher is the latency subcommand handling the EXIT event?

I see:

   perf_sched__lat
     perf_sched__read_events
       session = perf_session__new(&file, false, &sched->tool);
       perf_session__process_events(session)
Quoted 4 lines

And sched->tool->exit() is not set, which will make perf_session__process_events(), when calling perf_tool__fill_defaults() set it to process_event_stub() which will do nothing for PERF_RECORD_EXIT events, no?

yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog paragraph might be a little misleading sorry ;-)

could you please change it to:

---
If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---

thanks,
jirka

#5 lore raw

Em Tue, Nov 03, 2015 at 08:41:48AM +0100, Jiri Olsa escreveu:

Quoted 18 lines · Arnaldo Carvalho de Melo
Quoted 17 lines

Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:

Quoted 6 lines

If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid

Humm, wher is the latency subcommand handling the EXIT event?

I see:

   perf_sched__lat
     perf_sched__read_events
       session = perf_session__new(&file, false, &sched->tool);
       perf_session__process_events(session)
Quoted 7 lines
Quoted 4 lines

And sched->tool->exit() is not set, which will make perf_session__process_events(), when calling perf_tool__fill_defaults() set it to process_event_stub() which will do nothing for PERF_RECORD_EXIT events, no?

yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog

So its not related to processing an EXIT event as described in the changelog, ok. And I don't see where is that thread->dead is set, i.e. the sequence is:

machine__process_fork_event()
machine__remove_thread()

The only place where thread->dead is set to true is in thread__exited() and that is only called by machine__process_exit_event(), which is never called by 'perf sched'.

It is not "later removed from machine's tree", it is removed straight away, in __machine__remove_thread().

Anyway, I'm downloading that perf.data file to try to debug this here.

- Arnaldo

Quoted 16 lines

paragraph might be a little misleading sorry ;-)

could you please change it to:

---
If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---

thanks,
jirka

#6 lore raw
Quoted 34 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 03, 2015 at 08:41:48AM +0100, Jiri Olsa escreveu:

Quoted 18 lines · Arnaldo Carvalho de Melo
Quoted 17 lines

Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:

Quoted 6 lines

If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid

Humm, wher is the latency subcommand handling the EXIT event?

I see:

   perf_sched__lat
     perf_sched__read_events
       session = perf_session__new(&file, false, &sched->tool);
       perf_session__process_events(session)
Quoted 7 lines
Quoted 4 lines

And sched->tool->exit() is not set, which will make perf_session__process_events(), when calling perf_tool__fill_defaults() set it to process_event_stub() which will do nothing for PERF_RECORD_EXIT events, no?

yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog

So its not related to processing an EXIT event as described in the changelog, ok. And I don't see where is that thread->dead is set, i.e. the sequence is:

machine__process_fork_event()
machine__remove_thread()

Quoted 6 lines

The only place where thread->dead is set to true is in thread__exited() and that is only called by machine__process_exit_event(), which is never called by 'perf sched'.

It is not "later removed from machine's tree", it is removed straight away, in __machine__remove_thread().

it is removed later via FORK event, as stated in changelog and in updated changelog I sent in previous email:

SNIP

Quoted 11 lines
Quoted 11 lines

could you please change it to:

---
If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.

We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---

thanks,
jirka

#7 lore raw

[tip:perf/urgent] perf sched latency: Fix thread pid reuse issue

In reply to Jiri Olsa
perf sched latency: Fix thread pid reuse issue
Commit-ID:  0014de172d228e450377d1fd079d94e67128d27f
Gitweb:     http://git.kernel.org/tip/0014de172d228e450377d1fd079d94e67128d27f
Author:     Jiri Olsa <[email protected]>
AuthorDate: Mon, 2 Nov 2015 12:10:25 +0100
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Thu, 5 Nov 2015 12:51:00 -0300

perf sched latency: Fix thread pid reuse issue

The latency subcommand holds a tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:

thread_atoms_search: Assertion `!(thread != atoms->thread)' failed

Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.

---
 tools/perf/builtin-sched.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
index 0ee6d90..e3d3e32 100644
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_
 
 static int pid_cmp(struct work_atoms *l, struct work_atoms *r)
 {
+	if (l->thread == r->thread)
+		return 0;
 	if (l->thread->tid < r->thread->tid)
 		return -1;
 	if (l->thread->tid > r->thread->tid)
 		return 1;
-
-	return 0;
+	return (int)(l->thread - r->thread);
 }
 
 static int avg_cmp(struct work_atoms *l, struct work_atoms *r)