perf sched latency: Fix removed thread issue
Jiri Olsa
reviewerPATCH 1/1 perf sched latency: Fix removed thread issue
If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.
The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:
thread_atoms_search: Assertion `!(thread != atoms->thread)' failed
Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.
I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.
--- tools/perf/builtin-sched.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_static int pid_cmp (struct work_atoms *l,struct work_atoms *r) { +if (l->thread == r->thread) +return 0 ;if (l->thread->tid < r->thread->tid)return -1 ;if (l->thread->tid > r->thread->tid)return 1 ; - -return 0 ; +return (int )(l->thread - r->thread); }static int avg_cmp (struct work_atoms *l,struct work_atoms *r)
Signature · 1 line
2.4.3
Quoted 1 line · Jiri Olsa
If machine's thread gets excited (EXIT event is received),
Why a thread get *excited* when it received EXIT event? :)
Quoted 10 lines
we set
thread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:
thread_atoms_search: Assertion `!(thread !=atoms->thread)' failed
Quoted 7 lines
Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.
I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.
Right. I'll work on the v6 soon.
As a work around:
Thanks,
Namhyung
Quoted 31 lines
Reported-by Mohit Agrawal <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/builtin-sched.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)tools/perf/builtin-sched.c +3
-2@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_static int pid_cmp (struct work_atoms *l,struct work_atoms *r) { +if (l->thread == r->thread) +return 0 ;if (l->thread->tid < r->thread->tid)return -1 ;if (l->thread->tid > r->thread->tid)return 1 ; - -return 0 ; +return (int )(l->thread - r->thread); }static int avg_cmp (struct work_atoms *l,struct work_atoms *r)Signature · 1 line
2.4.3
Quoted 4 lines · Namhyung Kim
Quoted 1 line · Jiri Olsa
If machine's thread gets excited (EXIT event is received),
Why a thread get *excited* when it received EXIT event? :)
wouldn't you? I would.. ;-)
jirka
Arnaldo Carvalho de Melo
maintainerEm Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:
Quoted 6 lines
If machine's thread gets excited (EXIT event is received), we set
thread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid
Humm, wher is the latency subcommand handling the EXIT event?
I see:
perf_sched__lat
perf_sched__read_events
session = perf_session__new(&file, false, &sched->tool);
perf_session__process_events(session)And sched->tool->exit() is not set, which will make perf_session__process_events(), when calling perf_tool__fill_defaults() set it to process_event_stub() which will do nothing for PERF_RECORD_EXIT events, no?
Is there some perf.data file somewhere to reproduce this problem?
- Arnaldo
Quoted 42 lines
and tid, the old working atom is found and assert bug condition is hit in search function:
thread_atoms_search: Assertion `!(thread !=atoms->thread)' failedChanging the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.
I think we could change this to the sort based on timestamp of thread creation, once it's added within Namhyung's thread patchset.
Reported-by Mohit Agrawal <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- tools/perf/builtin-sched.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)tools/perf/builtin-sched.c +3
-2@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_static int pid_cmp (struct work_atoms *l,struct work_atoms *r) { +if (l->thread == r->thread) +return 0 ;if (l->thread->tid < r->thread->tid)return -1 ;if (l->thread->tid > r->thread->tid)return 1 ; - -return 0 ; +return (int )(l->thread - r->thread); }static int avg_cmp (struct work_atoms *l,struct work_atoms *r)Signature · 1 line
2.4.3
Quoted 17 lines · Arnaldo Carvalho de Melo
Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:
Quoted 6 lines
If machine's thread gets excited (EXIT event is received), we set
thread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid
Humm, wher is the latency subcommand handling the EXIT event?
I see:
perf_sched__lat perf_sched__read_events session = perf_session__new(&file, false, &sched->tool); perf_session__process_events(session)
Quoted 4 lines
And
sched->tool->exit()is not set, which will makeperf_session__process_events(), when callingperf_tool__fill_defaults()set it toprocess_event_stub()which will do nothing forPERF_RECORD_EXITevents, no?
yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog paragraph might be a little misleading sorry ;-)
could you please change it to:
---
If machine's thread gets excited (EXIT event is received), we set thread->dead = true and it is later on removed from machine's tree if the pid is reused on new thread.
We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---
thanks,
jirka
Arnaldo Carvalho de Melo
maintainerEm Tue, Nov 03, 2015 at 08:41:48AM +0100, Jiri Olsa escreveu:
Quoted 18 lines · Arnaldo Carvalho de Melo
Quoted 17 lines
Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:
Quoted 6 lines
If machine's thread gets excited (EXIT event is received), we set
thread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid
Humm, wher is the latency subcommand handling the EXIT event?
I see:
perf_sched__lat perf_sched__read_events session = perf_session__new(&file, false, &sched->tool); perf_session__process_events(session)
Quoted 7 lines
Quoted 4 lines
And
sched->tool->exit()is not set, which will makeperf_session__process_events(), when callingperf_tool__fill_defaults()set it toprocess_event_stub()which will do nothing forPERF_RECORD_EXITevents, no?yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog
So its not related to processing an EXIT event as described in the changelog, ok. And I don't see where is that thread->dead is set, i.e. the sequence is:
machine__process_fork_event()
machine__remove_thread()
The only place where thread->dead is set to true is in thread__exited() and that is only called by machine__process_exit_event(), which is never called by 'perf sched'.
It is not "later removed from machine's tree", it is removed straight away, in __machine__remove_thread().
Anyway, I'm downloading that perf.data file to try to debug this here.
- Arnaldo
Quoted 16 lines
paragraph might be a little misleading sorry ;-)
could you please change it to:
---
If machine's thread gets excited (EXIT event is received), we setthread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---thanks,
jirka
Quoted 34 lines · Arnaldo Carvalho de Melo
Em Tue, Nov 03, 2015 at 08:41:48AM +0100, Jiri Olsa escreveu:
Quoted 18 lines · Arnaldo Carvalho de Melo
Quoted 17 lines
Em Mon, Nov 02, 2015 at 12:10:25PM +0100, Jiri Olsa escreveu:
Quoted 6 lines
If machine's thread gets excited (EXIT event is received), we set
thread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.The latency subcommand holds tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid
Humm, wher is the latency subcommand handling the EXIT event?
I see:
perf_sched__lat perf_sched__read_events session = perf_session__new(&file, false, &sched->tool); perf_session__process_events(session)Quoted 7 lines
Quoted 4 lines
And
sched->tool->exit()is not set, which will makeperf_session__process_events(), when callingperf_tool__fill_defaults()set it toprocess_event_stub()which will do nothing forPERF_RECORD_EXITevents, no?yep, latency command does not handle EXIT event, but the thread is removed via FORK event.. the first changelog
So its not related to processing an EXIT event as described in the changelog, ok. And I don't see where is that
thread->deadis set, i.e. the sequence is:
machine__process_fork_event()
machine__remove_thread()
Quoted 6 lines
The only place where
thread->deadis set to true is inthread__exited()and that is only called bymachine__process_exit_event(), which is never called by 'perf sched'.It is not "later removed from machine's tree", it is removed straight away, in
__machine__remove_thread().
it is removed later via FORK event, as stated in changelog and in updated changelog I sent in previous email:
SNIP
Quoted 11 lines
Quoted 11 lines
could you please change it to:
---
If machine's thread gets excited (EXIT event is received), we setthread->dead= true and it is later on removed from machine's tree if the pid is reused on new thread.We dont handle EXIT command in 'perf sched latency', however the old thread is removed anyway when FORK event is received for new thrad with same pid/tid.
---
thanks,
jirka
tip-bot for Jiri Olsa
merge bot[tip:perf/urgent] perf sched latency: Fix thread pid reuse issue
In reply to Jiri Olsaperf sched latency: Fix thread pid reuse issue
Commit-ID: 0014de172d228e450377d1fd079d94e67128d27f Gitweb: http://git.kernel.org/tip/0014de172d228e450377d1fd079d94e67128d27f Author: Jiri Olsa <[email protected]> AuthorDate: Mon, 2 Nov 2015 12:10:25 +0100 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Thu, 5 Nov 2015 12:51:00 -0300
perf sched latency: Fix thread pid reuse issue
The latency subcommand holds a tree of working atoms sorted by thread's pid/tid. If there's new thread with same pid and tid, the old working atom is found and assert bug condition is hit in search function:
thread_atoms_search: Assertion `!(thread != atoms->thread)' failed
Changing the sort function to use thread object pointers together with pid and tid check. This way new thread will never find old one with same pid/tid.
--- tools/perf/builtin-sched.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/perf/builtin-sched.c +3 -2
@@ -1203,12 +1203,13 @@ static void output_lat_thread(struct perf_sched *sched, struct work_atoms *work_static int pid_cmp (struct work_atoms *l,struct work_atoms *r) { +if (l->thread == r->thread) +return 0 ;if (l->thread->tid < r->thread->tid)return -1 ;if (l->thread->tid > r->thread->tid)return 1 ; - -return 0 ; +return (int )(l->thread - r->thread); }static int avg_cmp (struct work_atoms *l,struct work_atoms *r)