perf tools: Speed-up perf report by using multi thread (v1)
Namhyung Kim
maintainercover perf tools: Speed-up perf report by using multi thread (v1)
Hello,
This patchset converts perf report to use multiple threads in order to speed up the processing on large data files. I can see a minimum 40% of speedup with this change. The code is still experimental, little bit outdated and contains many rough edges. But I'd like to share and give some feedbacks.
The perf report processes (sample) events like below:
1. preprocess sample to get matching thread/dso/symbol info
2. insert it to hists rbtree (with callchain tree) based on the info
3. optionally collapse hist entries that match given sort key(s)
4. resort hist entries (by overhead) for output
5. display the hist entries
The stage 1 is a preprocessing and mostly act like a read-only operation during the sample processing. Meta events like fork, comm and mmap can change the machine/thread state but symbols can be loaded during the processing (stage 2).
The stage 2 consumes most of the time especially with callchains and
--children option is enabled. And this work can be easily patitioned
as each sample is independent to others. But the resulting hists must be combined/collapsed to a single global hists before going to further steps.
The stage 3 is optional and only needed by certain sort keys - but with stage 2 paralellized, it needs to be done anyway.
The stage 4 and 5 works on whole hists so must be done serially.
So my approach is like this:
Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
For DWARF post-unwinding, dso cache data also needs to be protected by a lock and this causes a huge contention. I just added a front cache that can be accessed without the lock but this should be improved IMHO.
The patch 1-10 are to support multi-file data recording. With
-M/--multi option, perf record will create a directory (named
'perf.data.dir' by default - but maybe renamed 'perf.data' for transparent conversion later) and save meta events to perf.header file and sample events to perf.data.<n> file). It'd be better considering file format change Jiri suggested [1].
The patch 11-20 are to manage machine and thread state using timestamp so that it can be searched when processing samples. The patch 21-35 are to implement parallel report. And finally I implemented 'perf data split' command to convert a single data file into a multi-file format.
This patchset didn't change perf record to use multi-thread. But I think it can be easily done later if needed.
Note that output has a slight difference to original version when compared using splitted data file. But they're mostly unresolved symbols for callchains.
Here is the result:
This is just elapsed (real) time measured by shell 'time' function.
The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.
time perf report --children --no-children + --call-graph none
---------- ------------- -------------------
current 4m43.260s 1m32.779s 0m35.866s patched 4m43.710s 1m29.695s 0m33.995s --multi-thread 2m46.265s 0m45.486s 0m7.570s
This result is with 7.7GB data file using libunwind for callchain.
time perf report --children --no-children + --call-graph none
---------- ------------- -------------------
current 3m51.762s 3m10.451s 0m4.695s patched 2m26.030s 1m49.846s 0m4.105s --multi-thread 0m49.217s 0m35.106s 0m1.457s
Note that the single thread performance improvement in patched version is due to changes in the patch 33-35.
This result is with same file but using libdw for callchain unwind.
time perf report --children --no-children + --call-graph none
---------- ------------- -------------------
current 10m22.472s 11m42.290s 0m4.758s patched 10m10.625s 11m45.480s 0m4.162s --multi-thread 3m47.332s 3m35.235s 0m1.755s
On my archlinux system, callchain unwind using libdw is much slower than libunwind. I'm using elfutils version 0.160. Also I don't know why --children takes less time than --no-children. Anyway we can see the --multi-thread performance is much better for each case.
You can get it from 'perf/threaded-v1' branch on my tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Please take a look and play with it. Any comments are welcome! :)
Thanks,
Namhyung
[1] https://lkml.org/lkml/2013/9/1/20
Jiri Olsa (1):
perf tools: Add new perf data command
Namhyung Kim (36):
perf tools: Set attr.task bit for a tracking event
perf record: Use a software dummy event to track task/mmap events perf tools: Use perf_data_file__fd() consistently
perf tools: Add multi file interface to perf_data_file perf tools: Create separate mmap for dummy tracking event perf tools: Introduce perf_evlist__mmap_multi()
perf tools: Do not use __perf_session__process_events() directly perf tools: Handle multi-file session properly
perf record: Add -M/--multi option for multi file recording perf report: Skip dummy tracking event
perf tools: Introduce thread__comm_time() helpers
perf tools: Add a test case for thread comm handling
perf tools: Use thread__comm_time() when adding hist entries perf tools: Convert dead thread list into rbtree
perf tools: Introduce machine__find*_thread_time()
perf tools: Add a test case for timed thread handling perf tools: Maintain map groups list in a leader thread perf tools: Remove thread when map groups initialization failed perf tools: Introduce thread__find_addr_location_time() and friends perf tools: Add a test case for timed map groups handling perf tools: Protect dso symbol loading using a mutex
perf tools: Protect dso cache tree using dso->lock
perf tools: Protect dso cache fd with a mutex
perf session: Pass struct events stats to event processing functions perf hists: Pass hists struct to hist_entry_iter functions perf tools: Move BUILD_ID_SIZE definition to perf.h
perf report: Parallelize perf report using multi-thread perf tools: Add missing_threads rb tree
perf top: Always creates thread in the current task tree. perf tools: Fix progress ui to support multi thread
perf record: Show total size of multi file data
perf report: Add --multi-thread option and config item perf tools: Add front cache for dso data access
perf tools: Convert lseek + read to pread
perf callchain: Save eh/debug frame offset for dwarf unwind perf data: Implement 'split' subcommand
tools/perf/Documentation/perf-data.txt | 43 ++++ tools/perf/Documentation/perf-record.txt | 5 + tools/perf/Documentation/perf-report.txt | 3 + tools/perf/Makefile.perf | 4 + tools/perf/builtin-annotate.c | 5 +- tools/perf/builtin-data.c | 298 ++++++++++++++++++++++++++ tools/perf/builtin-diff.c | 8 +- tools/perf/builtin-inject.c | 9 +- tools/perf/builtin-record.c | 65 ++++-- tools/perf/builtin-report.c | 107 ++++++++-- tools/perf/builtin-script.c | 5 +- tools/perf/builtin-top.c | 9 +- tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + tools/perf/perf.h | 2 + tools/perf/tests/builtin-test.c | 12 ++ tools/perf/tests/dso-data.c | 5 + tools/perf/tests/dwarf-unwind.c | 10 +- tools/perf/tests/hists_common.c | 3 +- tools/perf/tests/hists_cumulate.c | 4 +- tools/perf/tests/hists_filter.c | 3 +- tools/perf/tests/hists_link.c | 6 +- tools/perf/tests/hists_output.c | 4 +- tools/perf/tests/tests.h | 3 + tools/perf/tests/thread-comm.c | 47 +++++ tools/perf/tests/thread-lookup-time.c | 180 ++++++++++++++++ tools/perf/tests/thread-mg-share.c | 7 +- tools/perf/tests/thread-mg-time.c | 88 ++++++++ tools/perf/ui/browsers/hists.c | 10 +- tools/perf/ui/gtk/hists.c | 3 + tools/perf/util/build-id.c | 9 +- tools/perf/util/build-id.h | 2 - tools/perf/util/data.c | 188 ++++++++++++++++- tools/perf/util/data.h | 17 ++ tools/perf/util/dso.c | 192 ++++++++++++----- tools/perf/util/dso.h | 5 + tools/perf/util/event.c | 85 ++++++-- tools/perf/util/event.h | 6 +- tools/perf/util/evlist.c | 151 ++++++++++++-- tools/perf/util/evlist.h | 22 +- tools/perf/util/evsel.c | 1 + tools/perf/util/evsel.h | 15 ++ tools/perf/util/hist.c | 121 +++++++---- tools/perf/util/hist.h | 12 +- tools/perf/util/machine.c | 251 +++++++++++++++++++--- tools/perf/util/machine.h | 12 +- tools/perf/util/map.c | 1 + tools/perf/util/map.h | 2 + tools/perf/util/ordered-events.c | 4 +- tools/perf/util/session.c | 347 ++++++++++++++++++++++++++----- tools/perf/util/session.h | 8 +- tools/perf/util/symbol.c | 34 ++- tools/perf/util/thread.c | 140 ++++++++++++- tools/perf/util/thread.h | 28 ++- tools/perf/util/tool.h | 17 ++ tools/perf/util/unwind-libdw.c | 11 +- tools/perf/util/unwind-libunwind.c | 49 +++-- tools/perf/util/util.c | 43 ++++ tools/perf/util/util.h | 1 + 60 files changed, 2381 insertions(+), 344 deletions(-) create mode 100644 tools/perf/Documentation/perf-data.txt create mode 100644 tools/perf/builtin-data.c create mode 100644 tools/perf/tests/thread-comm.c create mode 100644 tools/perf/tests/thread-lookup-time.c create mode 100644 tools/perf/tests/thread-mg-time.c
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 1/37 perf tools: Set attr.task bit for a tracking event
The perf_event_attr.task bit is to track task (fork and exit) events but it missed to be set by perf_evsel__config(). While it was not a problem in practice since setting other bits (comm/mmap) ended up being in same result, it'd be good to set it explicitly anyway.
--- tools/perf/util/evsel.c | 1 + 1 file changed, 1 insertion(+)
tools/perf/util/evsel.c +1 -0
@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)if (opts->sample_weight)perf_evsel__set_sample_bit (evsel,WEIGHT ); + attr->task = track; attr->mmap = track; attr->mmap2 = track && !perf_missing_features.mmap2; attr->comm = track;
Signature · 1 line
2.1.3
Re: [PATCH 01/37] perf tools: Set attr.task bit for a tracking event
Quoted 6 lines · Namhyung Kim
The perf_event_attr.task bit is to track task (fork and exit) events but it missed to be set by
perf_evsel__config(). While it was not a problem in practice since setting other bits (comm/mmap) ended up being in same result, it'd be good to set it explicitly anyway.Signed-off-by Namhyung Kim <[email protected]>
Quoted 19 lines
--- tools/perf/util/evsel.c | 1 + 1 file changed, 1 insertion(+)tools/perf/util/evsel.c +1
-0@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)if (opts->sample_weight)perf_evsel__set_sample_bit (evsel,WEIGHT ); + attr->task = track; attr->mmap = track; attr->mmap2 = track && !perf_missing_features.mmap2; attr->comm = track;Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 2/37 perf record: Use a software dummy event to track task/mmap events
In reply to Namhyung KimPrepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
--- tools/perf/builtin-record.c | 3 +++ tools/perf/perf.h | 1 + tools/perf/util/evlist.c | 38 ++++++++++++++++++++++++++++++++++++++ tools/perf/util/evlist.h | 1 + tools/perf/util/evsel.h | 15 +++++++++++++++ 5 files changed, 58 insertions(+)
tools/perf/builtin-record.c +3 -0
@@ -862,6 +862,9 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused) goto out_symbol_exit; } +if (rec->opts.multi_file) +perf_evlist__prepend_dummy (rec->evlist); +if (rec->opts.target.tid && !rec->opts.no_inherit_set) rec->opts.no_inherit = true;
tools/perf/perf.h +1 -0
@@ -53,6 +53,7 @@ struct record_opts {bool sample_time;bool period;bool sample_intr_regs; +bool multi_file;unsigned int freq;unsigned int mmap_pages;unsigned int user_freq;
tools/perf/util/evlist.c +38 -0
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; } +int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY , + }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name) + goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false; + } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true; + +if (!evlist->nr_entries++) +perf_evlist__set_id_pos (evlist); + +return 0 ; +error_free: +perf_evsel__delete (evsel); +error: +return -ENOMEM ; +} +static int perf_evlist__add_attrs (struct perf_evlist *evlist,struct perf_event_attr *attrs,size_t nr_attrs) {
tools/perf/util/evlist.h +1 -0
@@ -67,6 +67,7 @@ void perf_evlist__delete(struct perf_evlist *evlist);void perf_evlist__add (struct perf_evlist *evlist,struct perf_evsel *entry);int perf_evlist__add_default (struct perf_evlist *evlist); +int perf_evlist__prepend_dummy (struct perf_evlist *evlist);int __perf_evlist__add_default_attrs (struct perf_evlist *evlist,struct perf_event_attr *attrs,size_t nr_attrs);
tools/perf/util/evsel.h +15 -0
@@ -331,6 +331,21 @@ static inline bool perf_evsel__is_function_event(struct perf_evsel *evsel)#undef FUNCTION_EVENT } +/** +* perf_evsel__is_dummy_tracking - Return whether given evsel is a dummy +* event for tracking meta events only +* +* @evsel - evsel selector to be tested +* +* Return %true if event is a dummy tracking event +*/ +static inline bool perf_evsel__is_dummy_tracking (struct perf_evsel *evsel) +{ +return evsel->attr.type ==PERF_TYPE_SOFTWARE && + evsel->attr.config ==PERF_COUNT_SW_DUMMY && + evsel->attr.task ==1 && evsel->idx ==0 ; +} +struct perf_attr_details {bool freq;bool verbose;
Signature · 1 line
2.1.3
Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events
Quoted 3 lines · Namhyung Kim
Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?
---8<---
Quoted 25 lines
tools/perf/util/evlist.c +17
-0@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; } +int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY , + }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name)
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222
Quoted 10 lines
tools/perf/util/evlist.c +10
-0+ goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false; + } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true;
perf_evlist__set_tracking_event()?
David
Namhyung Kim
maintainerRe: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events
Hi David,
Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim
Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?
Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.
Quoted 34 lines
---8<---
Quoted 31 lines
tools/perf/util/evlist.c +22
-0@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; } +int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY , + }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name) + goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false;
Quoted 8 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+ } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true;
perf_evlist__set_tracking_event()?
I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the evsel->idx.
Thanks,
Namhyung
Adrian Hunter
reviewerRe: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events
Quoted 32 lines · Namhyung Kim
Hi David,
Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim
Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?
Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.
Quoted 12 lines
---8<---
Quoted 9 lines
tools/perf/util/evlist.c +0
-0@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; }
Quoted 5 lines
Quoted 5 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY ,
Probably need .exclude_kernel = 1, here
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
tools/perf/util/evlist.c +17
-0+ }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name) + goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false;Quoted 8 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+ } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true;
perf_evlist__set_tracking_event()?
Quoted 3 lines
I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the
evsel->idx.
perf_evlist__set_tracking_event() also ensures there is only one tracking event so it is easy to identify. It is the only event with attr->mmap etc set to 1. Then you can use perf_evlist__add().
Namhyung Kim
maintainerRe: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events
Hi Adrian,
Quoted 33 lines · Adrian Hunter
Quoted 32 lines · Namhyung Kim
Hi David,
Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim
Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?
Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.
Quoted 12 lines
---8<---
Quoted 9 lines
tools/perf/util/evlist.c +0
-0@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; }
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY ,Probably need .exclude_kernel = 1, here
Ah, right.
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 13 lines
tools/perf/util/evlist.c +12
-0+ }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name)tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222
Quoted 10 lines
tools/perf/util/evlist.c +10
-0+ goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false; + } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true;
perf_evlist__set_tracking_event()?
Quoted 7 lines
Quoted 3 lines
I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the
evsel->idx.
perf_evlist__set_tracking_event()also ensures there is only one tracking event so it is easy to identify. It is the only event withattr->mmapetc set to 1. Then you can useperf_evlist__add().
Well, yes, I think we can put the dummy tracking event anywhere in the evlist with this, but I still slightly prefer put it at a fixed location for a possible code simplification..
Thanks,
Namhyung
Adrian Hunter
reviewerRe: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events
Quoted 77 lines · Namhyung Kim
Hi Adrian,
Quoted 33 lines · Adrian Hunter
Quoted 32 lines · Namhyung Kim
Hi David,
Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim
Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.
Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?
Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.
Quoted 12 lines
---8<---
Quoted 9 lines
tools/perf/util/evlist.c +0
-0@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)return -ENOMEM ; }Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+int perf_evlist__prepend_dummy (struct perf_evlist *evlist) +{ +struct perf_event_attr attr = { + .type =PERF_TYPE_SOFTWARE , + .config =PERF_COUNT_SW_DUMMY ,Probably need .exclude_kernel = 1, here
Ah, right.
Quoted 31 lines
Quoted 30 lines
Quoted 18 lines
Quoted 18 lines
tools/perf/util/evlist.c +17
-0+ }; +struct perf_evsel *evsel, *pos; + +event_attr_init (&attr); + + evsel =perf_evsel__new (&attr); +if (evsel ==NULL ) + goto error; + +/* use strdup() because free(evsel) assumes name is allocated */ + evsel->name =strdup ("dummy" ); +if (!evsel->name) + goto error_free; + +list_for_each_entry (pos, &evlist->entries, node) { + pos->idx +=1 ; + pos->tracking = false;Quoted 8 lines
tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222
Quoted 5 lines
tools/perf/util/evlist.c +5
-0+ } + +list_add (&evsel->node, &evlist->entries); + evsel->idx =0 ; + evsel->tracking = true;
perf_evlist__set_tracking_event()?I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the
evsel->idx.
Quoted 7 lines
Quoted 3 lines
perf_evlist__set_tracking_event()also ensures there is only one tracking event so it is easy to identify. It is the only event withattr->mmapetc set to 1. Then you can useperf_evlist__add().Well, yes, I think we can put the dummy tracking event anywhere in the evlist with this, but I still slightly prefer put it at a fixed location for a possible code simplification..
Probably you should not be checking for the "tracking event" at all on the processing side and instead just have perf report ignore all dummy events. That generalizes things even more.
What is the code simplification?
Namhyung Kim
maintainerPATCH 3/37 perf tools: Use perf_data_file__fd() consistently
In reply to Namhyung KimDo not reference file->fd directly since we want hide the implementation details from outside to support multi-file store.
--- tools/perf/builtin-inject.c | 9 ++++++--- tools/perf/builtin-record.c | 14 ++++++++------ 2 files changed, 14 insertions(+), 9 deletions(-)
tools/perf/builtin-inject.c +6 -3
@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject) } } -if (!file_out->is_pipe) -lseek (file_out->fd, session->header.data_offset,SEEK_SET ); +if (!file_out->is_pipe) { +lseek (perf_data_file__fd (file_out), session->header.data_offset, +SEEK_SET ); + } ret =perf_session__process_events (session, &inject->tool);
@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)perf_header__set_feat (&session->header,HEADER_BUILD_ID ); session->header.data_size = inject->bytes_written; -perf_session__write_header (session, session->evlist, file_out->fd, true); +perf_session__write_header (session, session->evlist, +perf_data_file__fd (file_out), true); }return ret;
tools/perf/builtin-record.c +8 -6
@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)struct perf_session *session = rec->session;u64 start = session->header.data_offset; -u64 size =lseek (file->fd,0 ,SEEK_CUR ); +u64 size =lseek (perf_data_file__fd (file),0 ,SEEK_CUR );if (size ==0 )return 0 ;
@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)perf_header__clear_feat (&session->header,HEADER_GROUP_DESC );if (file->is_pipe) { - err =perf_header__write_pipe (file->fd); + err =perf_header__write_pipe (perf_data_file__fd (file));if (err <0 ) goto out_child; }else { err =perf_session__write_header (session, rec->evlist, - file->fd, false); +perf_data_file__fd (file), false);if (err <0 ) goto out_child; }
@@ -397,8 +397,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)* return this more properly and also * propagate errors that now are calling die() */ - err =perf_event__synthesize_tracing_data (tool, file->fd, rec->evlist, - process_synthesized_event); + err =perf_event__synthesize_tracing_data (tool, +perf_data_file__fd (file), + rec->evlist, + process_synthesized_event);if (err <=0 ) {pr_err ("Couldn't record tracing data.\n" ); goto out_child;
@@ -541,7 +543,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)if (!rec->no_buildid)process_buildids (rec);perf_session__write_header (rec->session, rec->evlist, - file->fd, true); +perf_data_file__fd (&rec->file), true); } out_delete_session:
Signature · 1 line
2.1.3
Re: [PATCH 03/37] perf tools: Use perf_data_file__fd() consistently
tools/perf/builtin-inject.c static int __cmd_inject(struct perf_inject *inject) · line 377
Quoted 8 lines · Namhyung Kim
tools/perf/builtin-inject.c +0
-0@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject) } }
How about a local variable to make this more readable? fd = perf_data_file__fd(file_out)
Quoted 33 lines
tools/perf/builtin-inject.c +6
-3-if (!file_out->is_pipe) -lseek (file_out->fd, session->header.data_offset,SEEK_SET ); +if (!file_out->is_pipe) { +lseek (perf_data_file__fd (file_out), session->header.data_offset, +SEEK_SET ); + } ret =perf_session__process_events (session, &inject->tool);@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)perf_header__set_feat (&session->header,HEADER_BUILD_ID ); session->header.data_size = inject->bytes_written; -perf_session__write_header (session, session->evlist, file_out->fd, true); +perf_session__write_header (session, session->evlist, +perf_data_file__fd (file_out), true); }return ret;tools/perf/builtin-record.c +1
-1@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)struct perf_session *session = rec->session;u64 start = session->header.data_offset; -u64 size =lseek (file->fd,0 ,SEEK_CUR ); +u64 size =lseek (perf_data_file__fd (file),0 ,SEEK_CUR );if (size ==0 )return 0 ;
tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 360
Quoted 1 line
tools/perf/builtin-record.c +0
-0@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)perf_header__clear_feat (&session->header,HEADER_GROUP_DESC );
Similarly in this function.
Quoted 27 lines
tools/perf/builtin-record.c +6
-4if (file->is_pipe) { - err =perf_header__write_pipe (file->fd); + err =perf_header__write_pipe (perf_data_file__fd (file));if (err <0 ) goto out_child; }else { err =perf_session__write_header (session, rec->evlist, - file->fd, false); +perf_data_file__fd (file), false);if (err <0 ) goto out_child; }@@ -397,8 +397,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)* return this more properly and also * propagate errors that now are calling die() */ - err =perf_event__synthesize_tracing_data (tool, file->fd, rec->evlist, - process_synthesized_event); + err =perf_event__synthesize_tracing_data (tool, +perf_data_file__fd (file), + rec->evlist, + process_synthesized_event);if (err <=0 ) {pr_err ("Couldn't record tracing data.\n" ); goto out_child;
tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 549
Quoted 8 lines
tools/perf/builtin-record.c +1
-1@@ -541,7 +543,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)if (!rec->no_buildid)process_buildids (rec);perf_session__write_header (rec->session, rec->evlist, - file->fd, true); +perf_data_file__fd (&rec->file), true); } out_delete_session:
David
Namhyung Kim
maintainerRe: [PATCH 03/37] perf tools: Use perf_data_file__fd() consistently
Quoted 13 lines · David Ahern
tools/perf/builtin-inject.c static int __cmd_inject(struct perf_inject *inject) · line 377
Quoted 8 lines · Namhyung Kim
tools/perf/builtin-inject.c +0
-0@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject) } }How about a local variable to make this more readable? fd =
perf_data_file__fd(file_out)
Will do.
Quoted 37 lines
Quoted 37 lines
tools/perf/builtin-inject.c +4
-2-if (!file_out->is_pipe) -lseek (file_out->fd, session->header.data_offset, SEEK_SET); +if (!file_out->is_pipe) { +lseek (perf_data_file__fd (file_out), session->header.data_offset, +SEEK_SET ); + }ret = perf_session__process_events(session, &inject->tool);tools/perf/builtin-inject.c +2
-1@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)perf_header__set_feat (&session->header,HEADER_BUILD_ID ); session->header.data_size = inject->bytes_written; -perf_session__write_header (session, session->evlist, file_out->fd, true); +perf_session__write_header (session, session->evlist, +perf_data_file__fd (file_out), true); }return ret;tools/perf/builtin-record.c +1
-1@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)struct perf_session *session = rec->session;u64 start = session->header.data_offset; -u64 size =lseek (file->fd,0 ,SEEK_CUR ); +u64 size =lseek (perf_data_file__fd (file),0 ,SEEK_CUR );if (size ==0 )return 0 ;
Quoted 7 lines
tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int · line 362
Quoted 4 lines
tools/perf/builtin-record.c +0
-0@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)perf_header__clear_feat (&session->header,HEADER_GROUP_DESC );Similarly in this function.
No problem.
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 4/37 perf tools: Add multi file interface to perf_data_file
In reply to Namhyung KimWhen multi file storage is enabled, the perf data files will be saved in a directory (default: perf.data.dir) and it'll have a single header file for metadata (task/comm/mmap events and file header) and multiple data files (sample events) like below:
$ tree perf.data.dir perf.data.dir |-- perf.data.0 |-- perf.data.1 |-- perf.data.2 |-- perf.data.3 `-- perf.header
0 directories, 5 files
Existing data file interface supports multi files internally and add new perf_data_file__prepare_write() and perf_data_file__write_multi() functions in order to support multi-file record. Note that multi read interface is not needed since they're accessed via mmap.
--- tools/perf/util/data.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++-- tools/perf/util/data.h | 14 +++++ tools/perf/util/util.c | 43 ++++++++++++++ tools/perf/util/util.h | 1 + 4 files changed, 201 insertions(+), 6 deletions(-)
tools/perf/util/data.c +143 -6
@@ -4,6 +4,7 @@#include <sys/stat.h> #include <unistd.h> #include <string.h> +#include <dirent.h> #include "data.h" #include "util.h"
@@ -39,26 +40,97 @@ static int check_backup(struct perf_data_file *file)char oldname[PATH_MAX ];snprintf (oldname,sizeof (oldname),"%s.old" , file->path); -unlink (oldname); + +if (S_ISDIR (st.st_mode)) +rm_rf (oldname); +else +unlink (oldname); +rename (file->path, oldname); }return 0 ; } +static int scandir_filter (const struct dirent *d) +{ +return !prefixcmp (d->d_name,"perf.data." ); +} + +static int open_file_read_multi (struct perf_data_file *file) +{ +int i, n; +int ret; +struct dirent **list; + + n =scandir (file->path, &list, scandir_filter, versionsort); +if (n <=0 ) { + ret = -errno; +pr_err ("cannot find multi-data file\n" ); +return ret; + } + + file->multi_fd =malloc (n *sizeof (int )); +if (file->multi_fd ==NULL ) { +free (list); +return -ENOMEM ; + } + +for (i =0 ; i < n; i++) { +char path[PATH_MAX ]; + +scnprintf (path,sizeof (path),"%s/%s" , + file->path, list[i]->d_name); + + ret =open (path,O_RDONLY ); +if (ret <0 ) + goto out_err; + + file->multi_fd[i] = ret; + } + + file->nr_multi = n; + +free (list); +return 0 ; + +out_err: +while (--i >=0 ) +close (file->multi_fd[i]); + +zfree (&file->multi_fd); +free (list); +return ret; +} + +static const char *default_data_path (struct perf_data_file *file) +{ +return file->is_multi ?"perf.data.dir" :"perf.data" ; +} +static int open_file_read (struct perf_data_file *file) {struct stat st; +char path[PATH_MAX ];int fd;char sbuf[STRERR_BUFSIZE ]; - fd =open (file->path,O_RDONLY ); +strcpy (path, file->path); +if (file->is_multi) { +if (open_file_read_multi (file) <0 ) +return -1 ; + +if (path__join (path,sizeof (path), file->path,"perf.header" ) <0 ) +return -1 ; + } + + fd =open (path,O_RDONLY );if (fd <0 ) {int err = errno; -pr_err ("failed to open %s: %s" , file->path, +pr_err ("failed to open %s: %s" , path,strerror_r (err, sbuf,sizeof (sbuf))); -if (err ==ENOENT && !strcmp (file->path,"perf.data" )) +if (err ==ENOENT && !strcmp (path,default_data_path (file)))pr_err (" (try 'perf record' first)" );pr_err ("\n" );return -err;
@@ -90,12 +162,26 @@ static int open_file_read(struct perf_data_file *file)static int open_file_write (struct perf_data_file *file) {int fd; +char path[PATH_MAX ];char sbuf[STRERR_BUFSIZE ];if (check_backup (file))return -1 ; - fd =open (file->path,O_CREAT |O_RDWR |O_TRUNC ,S_IRUSR |S_IWUSR ); +strcpy (path, file->path); + +if (file->is_multi) { +if (mkdir (file->path,S_IRWXU ) <0 ) { +pr_err ("cannot create data directory `%s': %s\n" , + file->path,strerror_r (errno, sbuf,sizeof (sbuf))); +return -1 ; + } + +if (path__join (path,sizeof (path), file->path,"perf.header" ) <0 ) +return -1 ; + } + + fd =open (path,O_CREAT |O_RDWR |O_TRUNC ,S_IRUSR |S_IWUSR );if (fd <0 ) pr_err("failed to open %s : %s\n", file->path,
@@ -121,18 +207,69 @@ int perf_data_file__open(struct perf_data_file *file)return 0 ;if (!file->path) - file->path ="perf.data" ; + file->path =default_data_path (file);return open_file (file); }void perf_data_file__close (struct perf_data_file *file) { +if (file->is_multi) { +int i; + +for (i =0 ; i < file->nr_multi; i++) +close (file->multi_fd[i]); + +zfree (&file->multi_fd); + } +close (file->fd); } +int perf_data_file__prepare_write (struct perf_data_file *file,int nr) +{ +int i; +int ret; +char path[PATH_MAX ]; + +if (!file->is_multi) +return 0 ; + + file->multi_fd =malloc (nr *sizeof (int )); +if (file->multi_fd ==NULL ) +return -ENOMEM ; + +for (i =0 ; i < nr; i++) { +scnprintf (path,sizeof (path),"%s/perf.data.%d" , file->path, i); + ret =open (path,O_RDWR |O_CREAT |O_TRUNC ,S_IRUSR |S_IWUSR ); +if (ret <0 ) + goto out_err; + + file->multi_fd[i] = ret; + } + + file->nr_multi = nr; +return 0 ; + +out_err: +while (--i >=0 ) +close (file->multi_fd[i]); + +zfree (&file->multi_fd); +return ret; +} +ssize_t perf_data_file__write (struct perf_data_file *file,void *buf,size_t size) {return writen (file->fd, buf, size); } + +ssize_t perf_data_file__write_multi (struct perf_data_file *file, +void *buf,size_t size,int idx) +{ +if (!file->is_multi) +return -1 ; + +return writen (file->multi_fd[idx], buf, size); +}
tools/perf/util/data.h +14 -0
@@ -11,6 +11,9 @@ enum perf_data_mode {struct perf_data_file {const char *path;int fd; +int nr_multi; +int *multi_fd; +bool is_multi;bool is_pipe;bool force;unsigned long size;
@@ -37,6 +40,14 @@ static inline int perf_data_file__fd(struct perf_data_file *file)return file->fd; } +static inline int perf_data_file__multi_fd (struct perf_data_file *file,int idx) +{ +if (!file->is_multi || idx >= file->nr_multi) +return -1 ; + +return file->multi_fd[idx]; +} +static inline unsigned long perf_data_file__size (struct perf_data_file *file) {return file->size;
@@ -46,5 +57,8 @@ int perf_data_file__open(struct perf_data_file *file);void perf_data_file__close (struct perf_data_file *file);ssize_t perf_data_file__write (struct perf_data_file *file,void *buf,size_t size); +int perf_data_file__prepare_write (struct perf_data_file *file,int nr); +ssize_t perf_data_file__write_multi (struct perf_data_file *file, +void *buf,size_t size,int idx);#endif /* __PERF_DATA_H */
tools/perf/util/util.c +43 -0
49 lines not on this page · raw message
tools/perf/util/util.h +1 -0
@@ -248,6 +248,7 @@ static inline int sane_case(int x, int high) }int mkdir_p (char *path,mode_t mode); +int rm_rf (char *path);int copyfile (const char *from,const char *to);int copyfile_mode (const char *from,const char *to,mode_t mode);
Signature · 1 line
2.1.3
Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file
On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:
SNIP
Quoted 15 lines
#endif /*
__PERF_DATA_H*/tools/perf/util/util.c +6
-0@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)return (stat (path, &st) &&mkdir (path, mode)) ?-1 :0 ; } +int rm_rf (char *path) +{ +DIR *dir; +int ret =0 ; +struct dirent *d; +char namebuf[PATH_MAX ];
tools/perf/util/util.c int mkdir_p(char *path, mode_t mode) · line 90
Quoted 10 lines
tools/perf/util/util.c +10
-0+ + dir =opendir (path); +if (dir ==NULL ) +return 0 ; + +while ((d =readdir (dir)) !=NULL && !ret) { +struct stat statbuf; + +if (d->d_name[0 ] =='.' ) +continue ;
Could you check for '.' and for '..' to support '.*' removal?
I know tha we will probably not have any '.*' files in perf.data.dir, but this function could be used later like for total cache clean ;-)
also please submit 'rm_rf' in separate patch
thanks,
jirka
Namhyung Kim
maintainerRe: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file
Hi Jiri,
Quoted 32 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:
SNIP
Quoted 15 lines
#endif /*
__PERF_DATA_H*/tools/perf/util/util.c +6
-0@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)return (stat (path, &st) &&mkdir (path, mode)) ?-1 :0 ; } +int rm_rf (char *path) +{ +DIR *dir; +int ret =0 ; +struct dirent *d; +char namebuf[PATH_MAX ];tools/perf/util/util.c int mkdir_p(char *path, mode_t mode) · line 90
Quoted 10 lines
tools/perf/util/util.c +10
-0+ + dir =opendir (path); +if (dir ==NULL ) +return 0 ; + +while ((d =readdir (dir)) !=NULL && !ret) { +struct stat statbuf; + +if (d->d_name[0 ] =='.' ) +continue ;Could you check for '.' and for '..' to support '.*' removal?
Quoted 2 lines
I know tha we will probably not have any '.*' files in perf.data.dir, but this function could be used later like for total cache clean ;-)
Ah, okay.
Quoted 1 line
also please submit '
rm_rf' in separate patch
Will do.
Thanks,
Namhyung
Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file
In reply to Namhyung KimOn Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:
SNIP
tools/perf/util/data.c static int check_backup(struct perf_data_file *file) · line 118
Quoted 10 lines
tools/perf/util/data.c +3
-1+static int open_file_read (struct perf_data_file *file) {struct stat st; +char path[PATH_MAX ];int fd;char sbuf[STRERR_BUFSIZE ]; - fd =open (file->path,O_RDONLY ); +strcpy (path, file->path);
strncpy ?
jirka
Namhyung Kim
maintainerRe: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file
Quoted 12 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:
SNIP
Quoted 8 lines
tools/perf/util/data.c +2
-0+static int open_file_read (struct perf_data_file *file) {struct stat st; +char path[PATH_MAX ];int fd;char sbuf[STRERR_BUFSIZE ];
Quoted 4 lines
tools/perf/util/data.c static int check_backup(struct perf_data_file *file) · line 118
Quoted 2 lines
tools/perf/util/data.c +1
-1- fd =open (file->path,O_RDONLY ); +strcpy (path, file->path);strncpy ?
Will change. :)
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 5/37 perf tools: Create separate mmap for dummy tracking event
In reply to Namhyung KimWhen multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.
--- tools/perf/builtin-record.c | 9 +++- tools/perf/util/evlist.c | 104 +++++++++++++++++++++++++++++++++++--------- tools/perf/util/evlist.h | 11 ++++- 3 files changed, 102 insertions(+), 22 deletions(-)
tools/perf/builtin-record.c +8 -1
@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,static int record__mmap_read (struct record *rec,int idx) { -struct perf_mmap *md = &rec->evlist->mmap[idx]; +struct perf_mmap *md =perf_evlist__mmap_desc (rec->evlist, idx);unsigned int head =perf_mmap__read_head (md);unsigned int old = md->prev;unsigned char *data = md->base + page_size;
@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx) } md->prev = old; +perf_evlist__mmap_consume (rec->evlist, idx); out:return rc;
@@ -263,6 +264,12 @@ static int record__mmap_read_all(struct record *rec) goto out; } } +if (rec->evlist->track_mmap) { +if (record__mmap_read (rec,track_mmap_idx (i)) !=0 ) { + rc =-1 ; + goto out; + } + } }/*
tools/perf/util/evlist.c +84 -20
185 lines not on this page · raw message
tools/perf/util/evlist.h +10 -1
@@ -48,11 +48,14 @@ struct perf_evlist {bool overwrite;struct fdarray pollfd;struct perf_mmap *mmap; +struct perf_mmap *track_mmap;struct thread_map *threads;struct cpu_map *cpus;struct perf_evsel *selected; }; +#define TRACK_MMAP_SIZE (((128 * 1024 / page_size) + 1) * page_size) +struct perf_evsel_str_handler {const char *name;void *handler;
@@ -100,8 +103,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);struct perf_sample_id *perf_evlist__id2sid (struct perf_evlist *evlist,u64 id);union perf_event *perf_evlist__mmap_read (struct perf_evlist *evlist,int idx); -void perf_evlist__mmap_consume (struct perf_evlist *evlist,int idx); +struct perf_mmap *perf_evlist__mmap_desc (struct perf_evlist *evlist,int idx);int perf_evlist__open (struct perf_evlist *evlist);void perf_evlist__close (struct perf_evlist *evlist);
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);void perf_evlist__to_front (struct perf_evlist *evlist,struct perf_evsel *move_evsel); +/* convert from/to negative idx for track mmaps */ +static inline int track_mmap_idx (int idx) +{ +return -idx -1 ; +} +/** * __evlist__for_each - iterate thru all the evsels * @list: list_head instance to iterate
Signature · 1 line
2.1.3
Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
Quoted 8 lines · Namhyung Kim
When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.
maybe this needs to be applied after next patch?
perf tools: Introduce perf_evlist__mmap_multi
I'm getting compile error:
[jolsa@krava perf]$ make JOBS=1
BUILD: Doing 'make -j1' parallel build
CC util/evlist.outil/evlist.c: In function ‘perf_evlist__mmap_per_evsel’:util/evlist.c:937:9: error: ‘struct mmap_params’ has no member named ‘track’
if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
^
util/evlist.c: In function ‘perf_evlist__mmap’:
util/evlist.c:1092:62: error: ‘use_track_mmap’ undeclared (first use in this function)
if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0)
^util/evlist.c:1092:62: note: each undeclared identifier is reported only once for each function it appears in make[1]: *** [util/evlist.o] Error 1
make: *** [all] Error 2
jirka
Namhyung Kim
maintainerRe: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
Quoted 5 lines · Jiri Olsa
Quoted 4 lines · Namhyung Kim
When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Quoted 7 lines
Quoted 4 lines
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.
maybe this needs to be applied after next patch?
perf tools: Introduceperf_evlist__mmap_multi
Oops, sorry. it seems the code was mixed after huge number of rebase.. The commit order is right but it just has a wrong hunk.
Quoted 14 lines
I'm getting compile error:
[jolsa@krava perf]$ make JOBS=1
BUILD: Doing 'make -j1' parallel build
CCutil/evlist.outil/evlist.c: In function ‘perf_evlist__mmap_per_evsel’:util/evlist.c:937:9: error: ‘struct mmap_params’ has no member named ‘track’if (mp->track && perf_evsel__is_dummy_tracking(evsel)) { ^ util/evlist.c: In function ‘perf_evlist__mmap’: util/evlist.c:1092:62: error: ‘use_track_mmap’ undeclared (first use in this function) if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0) ^
Quoted 3 lines
util/evlist.c:1092:62: note: each undeclared identifier is reported only once for each function it appears in make[1]: *** [util/evlist.o] Error 1
make: *** [all] Error 2
Will fix, thanks
Namhyung
Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
In reply to Namhyung KimOn Wed, Dec 24, 2014 at 04:15:01PM +0900, Namhyung Kim wrote:
SNIP
Quoted 9 lines
union
perf_event*perf_evlist__mmap_read(structperf_evlist*evlist, int idx);tools/perf/util/evlist.h +1
-1-void perf_evlist__mmap_consume (struct perf_evlist *evlist,int idx); +struct perf_mmap *perf_evlist__mmap_desc (struct perf_evlist *evlist,int idx);int perf_evlist__open (struct perf_evlist *evlist);void perf_evlist__close (struct perf_evlist *evlist);
tools/perf/util/evlist.h bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str); · line 221
Quoted 8 lines
tools/perf/util/evlist.h +5
-0@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);void perf_evlist__to_front (struct perf_evlist *evlist,struct perf_evsel *move_evsel); +/* convert from/to negative idx for track mmaps */ +static inline int track_mmap_idx (int idx) +{ +return -idx -1 ; +}
hum, whats the logic with negative numbers in here? you still access track_mmap array with this index no?
thanks,
jirka
Namhyung Kim
maintainerRe: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
Quoted 16 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:01PM +0900, Namhyung Kim wrote:
SNIP
Quoted 12 lines
union
perf_event*perf_evlist__mmap_read(structperf_evlist*evlist, int idx);tools/perf/util/evlist.h +1
-1-void perf_evlist__mmap_consume (struct perf_evlist *evlist,int idx); +struct perf_mmap *perf_evlist__mmap_desc (struct perf_evlist *evlist,int idx);int perf_evlist__open (struct perf_evlist *evlist);void perf_evlist__close (struct perf_evlist *evlist);@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);void perf_evlist__to_front (struct perf_evlist *evlist,struct perf_evsel *move_evsel);
Quoted 8 lines
tools/perf/util/evlist.h bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str); · line 221
Quoted 5 lines
tools/perf/util/evlist.h +5
-0+/* convert from/to negative idx for track mmaps */ +static inline int track_mmap_idx (int idx) +{ +return -idx -1 ; +}hum, whats the logic with negative numbers in here? you still access
track_mmaparray with this index no?
For each index (per-cpu or per-task depends on user input), it now has two mmaps - one for normal (sampling) mmap and another for track (metadata) mmap. So I wanted to distinguish them by using positive and negative number. But simply changing the sign did not work due to 0 index. And the subtracting by 1 works nicely for both direction.
0 => -1 => 0
1 => -2 => 1
2 => -3 => 2
3 => -4 => 3
So when it sees a negative index, it'll convert it using track_mmap_idx() to get a matching (track) mmap.
Thanks,
Namhyung
Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
In reply to Namhyung KimQuoted 6 lines · Namhyung Kim
When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I
Have you tried stress tests like a kernel compile with -j Ncpus (Ncpus = 16 or higher)?
David
Namhyung Kim
maintainerRe: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
Quoted 6 lines · David Ahern
Quoted 5 lines · Namhyung Kim
When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Quoted 6 lines
Quoted 2 lines
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I
Have you tried stress tests like a kernel compile with -j Ncpus (Ncpus = 16 or higher)?
Yep, the test results in the cover letter were recorded kernel build with -j 20 on my development machine.
Thanks,
Namhyung
Adrian Hunter
reviewerRe: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
In reply to Namhyung KimQuoted 201 lines · Namhyung Kim
When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.
Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.
Cc Adrian Hunter <[email protected]>Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/builtin-record.c | 9 +++- tools/perf/util/evlist.c | 104 +++++++++++++++++++++++++++++++++++--------- tools/perf/util/evlist.h | 11 ++++- 3 files changed, 102 insertions(+), 22 deletions(-)tools/perf/builtin-record.c +8
-1@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,static int record__mmap_read (struct record *rec,int idx) { -struct perf_mmap *md = &rec->evlist->mmap[idx]; +struct perf_mmap *md =perf_evlist__mmap_desc (rec->evlist, idx);unsigned int head =perf_mmap__read_head (md);unsigned int old = md->prev;unsigned char *data = md->base + page_size;@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx) } md->prev = old; +perf_evlist__mmap_consume (rec->evlist, idx); out:return rc;@@ -263,6 +264,12 @@ static int record__mmap_read_all(struct record *rec) goto out; } } +if (rec->evlist->track_mmap) { +if (record__mmap_read (rec,track_mmap_idx (i)) !=0 ) { + rc =-1 ; + goto out; + } + } }/* tools/perf/util/evlist.c +64
-17@@ -27,6 +27,7 @@static void perf_evlist__mmap_put (struct perf_evlist *evlist,int idx);static void __perf_evlist__munmap (struct perf_evlist *evlist,int idx); +static void __perf_evlist__munmap_track (struct perf_evlist *evlist,int idx);#define FD (e, x, y) (*(int *)xyarray__entry(e->fd, x, y))#define SID (e, x, y) xyarray__entry(e->sample_id, x, y)@@ -735,22 +736,39 @@ static bool perf_mmap__empty(struct perf_mmap *md)return perf_mmap__read_head (md) != md->prev; } +struct perf_mmap *perf_evlist__mmap_desc (struct perf_evlist *evlist,int idx) +{ +if (idx >=0 ) +return &evlist->mmap[idx]; +else +return &evlist->track_mmap[track_mmap_idx (idx)]; +} +static void perf_evlist__mmap_get (struct perf_evlist *evlist,int idx) { - ++evlist->mmap[idx].refcnt; +struct perf_mmap *md =perf_evlist__mmap_desc (evlist, idx); + + ++md->refcnt; }static void perf_evlist__mmap_put (struct perf_evlist *evlist,int idx) { -BUG_ON (evlist->mmap[idx].refcnt ==0 ); +struct perf_mmap *md =perf_evlist__mmap_desc (evlist, idx); + +BUG_ON (md->refcnt ==0 ); + +if (--md->refcnt !=0 ) +return ; -if (--evlist->mmap[idx].refcnt ==0 ) +if (idx >=0 )__perf_evlist__munmap (evlist, idx); +else +__perf_evlist__munmap_track (evlist,track_mmap_idx (idx)); }void perf_evlist__mmap_consume (struct perf_evlist *evlist,int idx) { -struct perf_mmap *md = &evlist->mmap[idx]; +struct perf_mmap *md =perf_evlist__mmap_desc (evlist, idx);if (!evlist->overwrite) {unsigned int old = md->prev;@@ -771,6 +789,15 @@ static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx) } } +static void __perf_evlist__munmap_track (struct perf_evlist *evlist,int idx) +{ +if (evlist->track_mmap[idx].base !=NULL ) { +munmap (evlist->track_mmap[idx].base,TRACK_MMAP_SIZE ); + evlist->track_mmap[idx].base =NULL ; + evlist->track_mmap[idx].refcnt =0 ; + } +} +void perf_evlist__munmap (struct perf_evlist *evlist) {int i;@@ -782,23 +809,43 @@ void perf_evlist__munmap(struct perf_evlist *evlist)__perf_evlist__munmap (evlist, i);zfree (&evlist->mmap); + +if (evlist->track_mmap ==NULL ) +return ; + +for (i =0 ; i < evlist->nr_mmaps; i++) +__perf_evlist__munmap_track (evlist, i); + +zfree (&evlist->track_mmap); } -static int perf_evlist__alloc_mmap (struct perf_evlist *evlist) +static int perf_evlist__alloc_mmap (struct perf_evlist *evlist,bool track_mmap) { evlist->nr_mmaps =cpu_map__nr (evlist->cpus);if (cpu_map__empty (evlist->cpus)) evlist->nr_mmaps =thread_map__nr (evlist->threads); evlist->mmap =zalloc (evlist->nr_mmaps *sizeof (struct perf_mmap )); -return evlist->mmap !=NULL ?0 : -ENOMEM ; +if (evlist->mmap ==NULL ) +return -ENOMEM ; + +if (track_mmap) { + evlist->track_mmap =calloc (evlist->nr_mmaps, +sizeof (struct perf_mmap )); +if (evlist->track_mmap ==NULL ) { +zfree (&evlist->mmap); +return -ENOMEM ; + } + } +return 0 ; }struct mmap_params { -int prot; -int mask; +int prot; +size_t len; }; -static int __perf_evlist__mmap (struct perf_evlist *evlist,int idx, +static int __perf_evlist__mmap (struct perf_evlist *evlist__maybe_unused , +struct perf_mmap *pmmap,struct mmap_params *mp,int fd) {/* @@ -814,15 +861,14 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,* evlist layer can't just drop it when filtering events in * perf_evlist__filter_pollfd(). */ - evlist->mmap[idx].refcnt =2 ; - evlist->mmap[idx].prev =0 ; - evlist->mmap[idx].mask = mp->mask; - evlist->mmap[idx].base =mmap (NULL , evlist->mmap_len, mp->prot, -MAP_SHARED , fd,0 ); -if (evlist->mmap[idx].base ==MAP_FAILED ) { + pmmap->refcnt =2 ; + pmmap->prev =0 ; + pmmap->mask = mp->len - page_size -1 ; + pmmap->base =mmap (NULL , mp->len, mp->prot,MAP_SHARED , fd,0 ); +if (pmmap->base ==MAP_FAILED ) {pr_debug2 ("failed to mmap perf event ring buffer, error %d\n" , errno); - evlist->mmap[idx].base =NULL ; + pmmap->base =NULL ;return -1 ; }@@ -843,9 +889,22 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, fd =FD (evsel, cpu, thread); -if (*output ==-1 ) { +if (perf_evsel__is_dummy_tracking (evsel)) { +struct mmap_params track_mp = {
tools/perf/util/evlist.c static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, · line 903
Quoted 10 lines
tools/perf/util/evlist.c +10
-0+ .prot = mp->prot, + .len = TRACK_MMAP_SIZE, + }; + +if (__perf_evlist__mmap (evlist, &evlist->track_mmap[idx], + &track_mp, fd) <0 ) +return -1 ; + +/* mark idx as track mmap idx (negative) */ + idx =track_mmap_idx (idx);
Do you not still need to do SET_OUTPUT when there are multiple cpus and multiple pids?
Quoted 81 lines
tools/perf/util/evlist.c +10
-3+ }else if (*output ==-1 ) { *output = fd; -if (__perf_evlist__mmap (evlist, idx, mp, *output) <0 ) +if (__perf_evlist__mmap (evlist, &evlist->mmap[idx], + mp, *output) <0 )return -1 ; }else {if (ioctl (fd,PERF_EVENT_IOC_SET_OUTPUT , *output) !=0 )@@ -874,6 +933,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,perf_evlist__set_sid_idx (evlist, evsel, idx, cpu, thread); } + +if (mp->track &&perf_evsel__is_dummy_tracking (evsel)) { +/* restore idx as normal idx (positive) */ + idx =track_mmap_idx (idx); + } }return 0 ;@@ -1025,7 +1089,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages, .prot = PROT_READ | (overwrite ? 0 : PROT_WRITE), }; -if (evlist->mmap ==NULL &&perf_evlist__alloc_mmap (evlist) <0 ) +if (evlist->mmap ==NULL &&perf_evlist__alloc_mmap (evlist, use_track_mmap) <0 )return -ENOMEM ;if (evlist->pollfd.entries ==NULL &&perf_evlist__alloc_pollfd (evlist) <0 )@@ -1034,7 +1098,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages, evlist->overwrite = overwrite; evlist->mmap_len =perf_evlist__mmap_size (pages);pr_debug ("mmap size %zuB\n" , evlist->mmap_len); - mp.mask = evlist->mmap_len - page_size -1 ; + mp.len = evlist->mmap_len;evlist__for_each (evlist, evsel) { if ((evsel->attr.read_format & PERF_FORMAT_ID) &&tools/perf/util/evlist.h +10
-1@@ -48,11 +48,14 @@ struct perf_evlist {bool overwrite;struct fdarray pollfd;struct perf_mmap *mmap; +struct perf_mmap *track_mmap;struct thread_map *threads;struct cpu_map *cpus;struct perf_evsel *selected; }; +#define TRACK_MMAP_SIZE (((128 * 1024 / page_size) + 1) * page_size) +struct perf_evsel_str_handler {const char *name;void *handler;@@ -100,8 +103,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);struct perf_sample_id *perf_evlist__id2sid (struct perf_evlist *evlist,u64 id);union perf_event *perf_evlist__mmap_read (struct perf_evlist *evlist,int idx); -void perf_evlist__mmap_consume (struct perf_evlist *evlist,int idx); +struct perf_mmap *perf_evlist__mmap_desc (struct perf_evlist *evlist,int idx);int perf_evlist__open (struct perf_evlist *evlist);void perf_evlist__close (struct perf_evlist *evlist);@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);void perf_evlist__to_front (struct perf_evlist *evlist,struct perf_evsel *move_evsel); +/* convert from/to negative idx for track mmaps */ +static inline int track_mmap_idx (int idx) +{ +return -idx -1 ; +} +/** * __evlist__for_each - iterate thru all the evsels * @list: list_head instance to iterate
Namhyung Kim
maintainerRe: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event
Quoted 4 lines · Adrian Hunter
Quoted 4 lines
tools/perf/util/evlist.c +0
-0@@ -843,9 +889,22 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, fd =FD (evsel, cpu, thread);
Quoted 16 lines
tools/perf/util/evlist.c static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, · line 903
Quoted 13 lines
tools/perf/util/evlist.c +12
-1-if (*output ==-1 ) { +if (perf_evsel__is_dummy_tracking (evsel)) { +struct mmap_params track_mp = { + .prot = mp->prot, + .len =TRACK_MMAP_SIZE , + }; + +if (__perf_evlist__mmap (evlist, &evlist->track_mmap[idx], + &track_mp, fd) <0 ) +return -1 ; + +/* mark idx as track mmap idx (negative) */ + idx =track_mmap_idx (idx);Do you not still need to do
SET_OUTPUTwhen there are multiple cpus and multiple pids?
You're right. I just considered simple cases, will fix.
Thanks,
Namhyung
Quoted 22 lines
Quoted 22 lines
tools/perf/util/evlist.c +8
-1+ }else if (*output ==-1 ) { *output = fd; -if (__perf_evlist__mmap (evlist, idx, mp, *output) <0 ) +if (__perf_evlist__mmap (evlist, &evlist->mmap[idx], + mp, *output) <0 )return -1 ; }else {if (ioctl (fd,PERF_EVENT_IOC_SET_OUTPUT , *output) !=0 )@@ -874,6 +933,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,perf_evlist__set_sid_idx (evlist, evsel, idx, cpu, thread); } + +if (mp->track &&perf_evsel__is_dummy_tracking (evsel)) { +/* restore idx as normal idx (positive) */ + idx =track_mmap_idx (idx); + } }return 0 ;
Namhyung Kim
maintainerPATCH 6/37 perf tools: Introduce perf_evlist__mmap_multi()
In reply to Namhyung KimThe perf_evlist__mmap_multi function creates data mmaps and optionally tracking mmaps for events. It'll be used for perf record to save tracking events in a separate files. Checking dummy tracking event in perf_evlist__mmap() alone is not enough as users can specify the dummy event at first (like in keep tracking testcase) without the multi-file option.
--- tools/perf/builtin-record.c | 3 ++- tools/perf/util/evlist.c | 11 +++++++---- tools/perf/util/evlist.h | 10 ++++++++-- 3 files changed, 17 insertions(+), 7 deletions(-)
tools/perf/builtin-record.c +2 -1
@@ -169,7 +169,8 @@ static int record__open(struct record *rec) goto out; } -if (perf_evlist__mmap (evlist, opts->mmap_pages, false) <0 ) { +if (perf_evlist__mmap_multi (evlist, opts->mmap_pages, false, + opts->multi_file) <0 ) {if (errno ==EPERM ) {pr_err ("Permission error mapping pages.\n" "Consider increasing "
tools/perf/util/evlist.c +7 -4
@@ -841,6 +841,7 @@ static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)struct mmap_params {int prot; +bool track;size_t len; };
@@ -889,7 +890,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, fd =FD (evsel, cpu, thread); -if (perf_evsel__is_dummy_tracking (evsel)) { +if (mp->track &&perf_evsel__is_dummy_tracking (evsel)) {struct mmap_params track_mp = { .prot = mp->prot, .len =TRACK_MMAP_SIZE ,
@@ -1068,10 +1069,11 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str, }/** -* perf_evlist__mmap - Create mmaps to receive events. +* perf_evlist__mmap_multi - Create mmaps to receive events. * @evlist: list of events * @pages: map length in pages * @overwrite: overwrite older events? +* @use_track_mmap: use another mmaps to track meta events * * If @overwrite is %false the user needs to signal event consumption using * perf_mmap__write_tail(). Using perf_evlist__mmap_read() does this
@@ -1079,14 +1081,15 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,* * Return: %0 on success, negative error code otherwise. */ -int perf_evlist__mmap (struct perf_evlist *evlist,unsigned int pages, -bool overwrite) +int perf_evlist__mmap_multi (struct perf_evlist *evlist,unsigned int pages, +bool overwrite,bool use_track_mmap) {struct perf_evsel *evsel;const struct cpu_map *cpus = evlist->cpus;const struct thread_map *threads = evlist->threads;struct mmap_params mp = { .prot =PROT_READ | (overwrite ?0 :PROT_WRITE ), + .track = use_track_mmap, };if (evlist->mmap ==NULL &&perf_evlist__alloc_mmap (evlist, use_track_mmap) <0 )
tools/perf/util/evlist.h +8 -2
@@ -127,10 +127,16 @@ int perf_evlist__parse_mmap_pages(const struct option *opt,const char *str, int unset); -int perf_evlist__mmap (struct perf_evlist *evlist,unsigned int pages, -bool overwrite); +int perf_evlist__mmap_multi (struct perf_evlist *evlist,unsigned int pages, +bool overwrite,bool use_track_mmap);void perf_evlist__munmap (struct perf_evlist *evlist); +static inline int perf_evlist__mmap (struct perf_evlist *evlist, +unsigned int pages,bool overwrite) +{ +return perf_evlist__mmap_multi (evlist, pages, overwrite, false); +} +void perf_evlist__disable (struct perf_evlist *evlist);void perf_evlist__enable (struct perf_evlist *evlist);
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 7/37 perf tools: Do not use __perf_session__process_events() directly
In reply to Namhyung KimIt's only used for perf record to process build-id because its file size it's not fixed at this time due to remaining header features. However data offset and size is available so that we can use the perf_session__process_events() once we set the file size as the current offset like for now.
It turns out that we can staticize the function again as it's the only user and add multi file support in a single place.
--- tools/perf/builtin-record.c | 7 +++---- tools/perf/util/session.c | 6 +++--- tools/perf/util/session.h | 3 --- 3 files changed, 6 insertions(+), 10 deletions(-)
tools/perf/builtin-record.c +3 -4
@@ -196,12 +196,13 @@ static int process_buildids(struct record *rec) {struct perf_data_file *file = &rec->file;struct perf_session *session = rec->session; -u64 start = session->header.data_offset;u64 size =lseek (perf_data_file__fd (file),0 ,SEEK_CUR );if (size ==0 )return 0 ; + file->size = size; +/* * During this process, it'll load kernel map and replace the * dso->long_name to a real pathname it found. In this case
@@ -213,9 +214,7 @@ static int process_buildids(struct record *rec)*/ symbol_conf.ignore_vmlinux_buildid = true; -return __perf_session__process_events (session, start, - size - start, - size, &build_id__mark_dso_hit_ops); +return perf_session__process_events (session, &build_id__mark_dso_hit_ops); }static void perf_event__synthesize_guest_os (struct machine *machine,void *data)
tools/perf/util/session.c +3 -3
@@ -1252,9 +1252,9 @@ fetch_mmaped_event(struct perf_session *session,#define NUM_MMAPS 128#endif -int __perf_session__process_events (struct perf_session *session, -u64 data_offset,u64 data_size, -u64 file_size,struct perf_tool *tool) +static int __perf_session__process_events (struct perf_session *session, +u64 data_offset,u64 data_size, +u64 file_size,struct perf_tool *tool) {int fd =perf_data_file__fd (session->file);u64 head, page_offset, file_offset, file_pos, size;
tools/perf/util/session.h +0 -3
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,union perf_event **event_ptr, struct perf_sample *sample); -int __perf_session__process_events (struct perf_session *session, -u64 data_offset,u64 data_size,u64 size, -struct perf_tool *tool);int perf_session__process_events (struct perf_session *session,struct perf_tool *tool);
Signature · 1 line
2.1.3
Re: [PATCH 07/37] perf tools: Do not use __perf_session__process_events() directly
Quoted 10 lines · Namhyung Kim
It's only used for perf record to process build-id because its file size it's not fixed at this time due to remaining header features. However data offset and size is available so that we can use the
perf_session__process_events()once we set the file size as the current offset like for now.It turns out that we can staticize the function again as it's the only user and add multi file support in a single place.
Signed-off-by Namhyung Kim <[email protected]>
Quoted 70 lines
--- tools/perf/builtin-record.c | 7 +++---- tools/perf/util/session.c | 6 +++--- tools/perf/util/session.h | 3 --- 3 files changed, 6 insertions(+), 10 deletions(-)tools/perf/builtin-record.c +3
-4@@ -196,12 +196,13 @@ static int process_buildids(struct record *rec) {struct perf_data_file *file = &rec->file;struct perf_session *session = rec->session; -u64 start = session->header.data_offset;u64 size =lseek (perf_data_file__fd (file),0 ,SEEK_CUR );if (size ==0 )return 0 ; + file->size = size; +/* * During this process, it'll load kernel map and replace the * dso->long_name to a real pathname it found. In this case @@ -213,9 +214,7 @@ static int process_buildids(struct record *rec)*/ symbol_conf.ignore_vmlinux_buildid = true; -return __perf_session__process_events (session, start, - size - start, - size, &build_id__mark_dso_hit_ops); +return perf_session__process_events (session, &build_id__mark_dso_hit_ops); }static void perf_event__synthesize_guest_os (struct machine *machine,void *data)tools/perf/util/session.c +3
-3@@ -1252,9 +1252,9 @@ fetch_mmaped_event(struct perf_session *session,#define NUM_MMAPS 128#endif -int __perf_session__process_events (struct perf_session *session, -u64 data_offset,u64 data_size, -u64 file_size,struct perf_tool *tool) +static int __perf_session__process_events (struct perf_session *session, +u64 data_offset,u64 data_size, +u64 file_size,struct perf_tool *tool) {int fd =perf_data_file__fd (session->file);u64 head, page_offset, file_offset, file_pos, size;tools/perf/util/session.h +0
-3@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,union perf_event **event_ptr, struct perf_sample *sample); -int __perf_session__process_events (struct perf_session *session, -u64 data_offset,u64 data_size,u64 size, -struct perf_tool *tool);int perf_session__process_events (struct perf_session *session,struct perf_tool *tool);Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 8/37 perf tools: Handle multi-file session properly
In reply to Namhyung KimWhen perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.
--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/session.c | 41 +++++++++++++++++++++++++++++++---------- 2 files changed, 48 insertions(+), 10 deletions(-)
tools/perf/util/data.c +17 -0
@@ -52,6 +52,21 @@ static int check_backup(struct perf_data_file *file)return 0 ; } +static void check_multi (struct perf_data_file *file) +{ +struct stat st; + +/* +* For write, it'll be determined by user (perf record -M) +* whether to enable multi file data storage. +*/ +if (perf_data_file__is_write (file)) +return ; + +if (!stat (file->path, &st) &&S_ISDIR (st.st_mode)) + file->is_multi = true; +} +static int scandir_filter (const struct dirent *d) {return !prefixcmp (d->d_name,"perf.data." );
@@ -206,6 +221,8 @@ int perf_data_file__open(struct perf_data_file *file)if (check_pipe (file))return 0 ; +check_multi (file); +if (!file->path) file->path =default_data_path (file);
tools/perf/util/session.c +31 -10
60 lines not on this page · raw message
Signature · 1 line
2.1.3
Re: [PATCH 08/37] perf tools: Handle multi-file session properly
Quoted 81 lines · Namhyung Kim
When perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/data.c | 17 +++++++++++++++++ tools/perf/util/session.c | 41 +++++++++++++++++++++++++++++++---------- 2 files changed, 48 insertions(+), 10 deletions(-)tools/perf/util/data.c +17
-0@@ -52,6 +52,21 @@ static int check_backup(struct perf_data_file *file)return 0 ; } +static void check_multi (struct perf_data_file *file) +{ +struct stat st; + +/* +* For write, it'll be determined by user (perf record -M) +* whether to enable multi file data storage. +*/ +if (perf_data_file__is_write (file)) +return ; + +if (!stat (file->path, &st) &&S_ISDIR (st.st_mode)) + file->is_multi = true; +} +static int scandir_filter (const struct dirent *d) {return !prefixcmp (d->d_name,"perf.data." );@@ -206,6 +221,8 @@ int perf_data_file__open(struct perf_data_file *file)if (check_pipe (file))return 0 ; +check_multi (file); +if (!file->path) file->path =default_data_path (file);tools/perf/util/session.c +2
-10@@ -1252,11 +1252,10 @@ fetch_mmaped_event(struct perf_session *session,#define NUM_MMAPS 128#endif -static int __perf_session__process_events (struct perf_session *session, +static int __perf_session__process_events (struct perf_session *session,int fd,u64 data_offset,u64 data_size,u64 file_size,struct perf_tool *tool) { -int fd =perf_data_file__fd (session->file);u64 head, page_offset, file_offset, file_pos, size;int err, mmap_prot, mmap_flags, map_idx =0 ;size_t mmap_size;@@ -1362,18 +1361,40 @@ int perf_session__process_events(struct perf_session *session, struct perf_tool *tool) {u64 size =perf_data_file__size (session->file); -int err; +int err, i;if (perf_session__register_idle_thread (session) ==NULL )return -ENOMEM ; -if (!perf_data_file__is_pipe (session->file)) - err =__perf_session__process_events (session, - session->header.data_offset, - session->header.data_size, - size, tool); -else - err =__perf_session__process_pipe_events (session, tool);
tools/perf/util/session.c int perf_session__process_events(struct perf_session *session, · line 1378
Quoted 10 lines
tools/perf/util/session.c +10
-0+if (perf_data_file__is_pipe (session->file)) +return __perf_session__process_pipe_events (session, tool); + + err =__perf_session__process_events (session, +perf_data_file__fd (session->file), + session->header.data_offset, + session->header.data_size, + size, tool); +if (!session->file->is_multi || err) +return err;
if we have file->is_multi true, the perf_data_file__fd(session->file) is the perf.header file right? So presumably, there's no data in it, and we dont need to call above __perf_session__process_events function?
jirka
Quoted 25 lines
tools/perf/util/session.c +19
-0+ +/* +* For multi-file data storage, events are processed for each +* cpu/thread so it's already ordered. +*/ + tool->ordered_events = false; + +for (i =0 ; i < session->file->nr_multi; i++) { +int fd =perf_data_file__multi_fd (session->file, i); + + size =lseek (fd,0 ,SEEK_END ); +if (size ==0 ) +continue ; + + err =__perf_session__process_events (session, fd, +0 , size, size, tool); +if (err <0 ) +break ; + }return err; }Signature · 1 line
2.1.3
Namhyung Kim
maintainerRe: [PATCH 08/37] perf tools: Handle multi-file session properly
Hi Jiri,
Quoted 6 lines · Jiri Olsa
Quoted 5 lines · Namhyung Kim
When perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.
[SNIP]
Quoted 23 lines
Quoted 23 lines
tools/perf/util/session.c +7
-8@@ -1362,18 +1361,40 @@ int perf_session__process_events(struct perf_session *session, struct perf_tool *tool) {u64 size =perf_data_file__size (session->file); -int err; +int err, i;if (perf_session__register_idle_thread (session) ==NULL )return -ENOMEM ; -if (!perf_data_file__is_pipe (session->file)) - err =__perf_session__process_events (session, - session->header.data_offset, - session->header.data_size, - size, tool); -else - err =__perf_session__process_pipe_events (session, tool); +if (perf_data_file__is_pipe (session->file)) +return __perf_session__process_pipe_events (session, tool); + + err = __perf_session__process_events(session, + perf_data_file__fd(session->file), + session->header.data_offset,
Quoted 8 lines
tools/perf/util/session.c int perf_session__process_events(struct perf_session *session, · line 1378
Quoted 4 lines
tools/perf/util/session.c +4
-0+ session->header.data_size, + size, tool); +if (!session->file->is_multi || err) +return err;if we have
file->is_multitrue, theperf_data_file__fd(session->file)is the perf.header file right? So presumably, there's no data in it, and we dont need to call above__perf_session__process_eventsfunction?
Right, it's for perf.header file and has no samples but it has meta events like fork, comm, mmap so it needs to be processed (before processing samples).
Thanks,
Namhyung
Quoted 26 lines
Quoted 26 lines
tools/perf/util/session.c +19
-1+ +/* +* For multi-file data storage, events are processed for each +* cpu/thread so it's already ordered. +*/ + tool->ordered_events = false; + +for (i =0 ; i < session->file->nr_multi; i++) { +int fd =perf_data_file__multi_fd (session->file, i); + + size =lseek (fd,0 ,SEEK_END ); +if (size ==0 ) +continue ; + + err =__perf_session__process_events (session, fd, +0 , size, size, tool); +if (err <0 ) +break ; + }return err; } --2.1.3
Namhyung Kim
maintainerPATCH 9/37 perf record: Add -M/--multi option for multi file recording
In reply to Namhyung KimThe new -M/--multi option enables multi file storage recording. Now sample data in separate mmap are saved in different files and other events will be recorded in the perf.header file.
--- tools/perf/Documentation/perf-record.txt | 5 +++++ tools/perf/builtin-record.c | 27 ++++++++++++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-record.txt +5 -0
@@ -220,6 +220,11 @@ Capture machine state (registers) at interrupt, i.e., on counter overflows for each sample. List of captured registers depends on the architecture. This option is off by default. +-M:: +--multi:: +Record data in multi-file storage instead of a single data file (perf.data). +This will speed up perf report by parallel processing. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +24 -3
@@ -58,6 +58,19 @@ static int record__write(struct record *rec, void *bf, size_t size)return 0 ; } +static int record__write_multi (struct record *rec,void *bf,size_t size,int idx) +{ +if (rec->file.is_multi && idx >=0 ) { +int ret =perf_data_file__write_multi (rec->session->file, + bf, size, idx); +if (ret <0 ) +pr_err ("failed to write perf data, error: %m\n" ); + +return ret; + } +return record__write (rec, bf, size); +} + static int process_synthesized_event(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample __maybe_unused,
@@ -89,7 +102,7 @@ static int record__mmap_read(struct record *rec, int idx) size = md->mask +1 - (old & md->mask); old += size; -if (record__write (rec, buf, size) <0 ) { +if (record__write_multi (rec, buf, size, idx) <0 ) { rc =-1 ; goto out; }
@@ -99,7 +112,7 @@ static int record__mmap_read(struct record *rec, int idx) size = head - old; old += size; -if (record__write (rec, buf, size) <0 ) { +if (record__write_multi (rec, buf, size, idx) <0 ) { rc =-1 ; goto out; }
@@ -186,6 +199,10 @@ static int record__open(struct record *rec) goto out; } + rc =perf_data_file__prepare_write (session->file, evlist->nr_mmaps); +if (rc <0 ) + goto out; + session->evlist = evlist;perf_session__set_id_hdr_size (session); out:
@@ -822,6 +839,8 @@ struct option __record_options[] = { "use per-thread mmaps"),OPT_BOOLEAN ('I' ,"intr-regs" , &record.opts.sample_intr_regs,"Sample machine registers on interrupt" ), +OPT_BOOLEAN ('M' ,"multi" , &record.opts.multi_file, +"use multi-file storage" ), OPT_END() };
@@ -871,8 +890,10 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused) goto out_symbol_exit; } -if (rec->opts.multi_file) +if (rec->opts.multi_file) { + rec->file.is_multi = true;perf_evlist__prepend_dummy (rec->evlist); + }if (rec->opts.target.tid && !rec->opts.no_inherit_set) rec->opts.no_inherit = true;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 10/37 perf report: Skip dummy tracking event
In reply to Namhyung KimThe dummy tracking event is only for tracking task/comom/mmap events and has no sample data for itself. So no need to report, just skip it.
--- tools/perf/builtin-report.c | 3 +++ tools/perf/ui/browsers/hists.c | 10 ++++++++-- tools/perf/ui/gtk/hists.c | 3 +++ 3 files changed, 14 insertions(+), 2 deletions(-)
tools/perf/builtin-report.c +3 -0
@@ -318,6 +318,9 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,struct hists *hists =evsel__hists (pos);const char *evname =perf_evsel__name (pos); +if (perf_evsel__is_dummy_tracking (pos)) +continue ; +if (symbol_conf.event_group && !perf_evsel__is_group_leader (pos))continue ;
tools/perf/ui/browsers/hists.c +8 -2
@@ -1998,11 +1998,15 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help, struct perf_session_env *env) {int nr_entries = evlist->nr_entries; +struct perf_evsel *first =perf_evlist__first (evlist); + +if (perf_evsel__is_dummy_tracking (first)) { + first =perf_evsel__next (first); + nr_entries--; + } single_entry:if (nr_entries ==1 ) { -struct perf_evsel *first =perf_evlist__first (evlist); -return perf_evsel__hists_browse (first, nr_entries, help, false, hbt, min_pcnt, env);
@@ -2013,6 +2017,8 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help, nr_entries =0 ;evlist__for_each (evlist, pos) { +if (perf_evsel__is_dummy_tracking (pos)) +continue ;if (perf_evsel__is_group_leader (pos)) nr_entries++; }
tools/perf/ui/gtk/hists.c +3 -0
@@ -317,6 +317,9 @@ int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist,char buf[512 ];size_t size =sizeof (buf); +if (perf_evsel__is_dummy_tracking (pos)) +continue ; +if (symbol_conf.event_group) {if (!perf_evsel__is_group_leader (pos))continue ;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 11/37 perf tools: Introduce thread__comm_time() helpers
In reply to Namhyung KimWhen multi-file data storage is enabled, it processes all task, comm and mmap events first and then goes to the sample events. So all it sees is the last comm of a thread although it has information at the time of sample.
Sort thread's comm by time so that it can find appropriate comm at the sample time. The thread__comm_time() will mostly work even if PERF_SAMPLE_TIME bit is off since in that case, sample->time will be -1 so it'll take the last comm anyway.
--- tools/perf/util/thread.c | 34 +++++++++++++++++++++++++++++++++- tools/perf/util/thread.h | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-)
tools/perf/util/thread.c +33 -1
@@ -103,6 +103,22 @@ struct comm *thread__exec_comm(const struct thread *thread)return last; } +struct comm *thread__comm_time (const struct thread *thread,u64 timestamp) +{ +struct comm *comm; + +list_for_each_entry (comm, &thread->comm_list, list) { +if (timestamp >= comm->start) +return comm; + } + +if (list_empty (&thread->comm_list)) +return NULL ; + + return list_last_entry(&thread->comm_list, struct comm, list); +} + +/* CHECKME: time should always be 0 if event aren't ordered */ int __thread__set_comm (struct thread *thread,const char *str,u64 timestamp,bool exec) {
@@ -118,7 +134,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp, new =comm__new (str, timestamp, exec);if (!new)return -ENOMEM ; -list_add (&new->list, &thread->comm_list); + +/* sort by time */ +list_for_each_entry (curr, &thread->comm_list, list) { +if (timestamp >= curr->start) +break ; + } +list_add_tail (&new->list, &curr->list);if (exec)unwind__flush_access (thread);
@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread *thread)return comm__str (comm); } +const char *thread__comm_time_str (const struct thread *thread,u64 timestamp) +{ +const struct comm *comm =thread__comm_time (thread, timestamp); + +if (!comm) +return NULL ; + +return comm__str (comm); +} +/* CHECKME: it should probably better return the max comm len from its comm list */ int thread__comm_len (struct thread *thread) {
tools/perf/util/thread.h +2 -0
@@ -53,7 +53,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,int thread__comm_len (struct thread *thread);struct comm *thread__comm (const struct thread *thread);struct comm *thread__exec_comm (const struct thread *thread); +struct comm *thread__comm_time (const struct thread *thread,u64 timestamp);const char *thread__comm_str (const struct thread *thread); +const char *thread__comm_time_str (const struct thread *thread,u64 timestamp);void thread__insert_map (struct thread *thread,struct map *map);int thread__fork (struct thread *thread,struct thread *parent,u64 timestamp);size_t thread__fprintf (struct thread *thread,FILE *fp);
Signature · 1 line
2.1.3
Re: [PATCH 11/37] perf tools: Introduce thread__comm_time() helpers
Quoted 4 lines · Namhyung Kim
tools/perf/util/thread.c +0
-0@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread *thread)return comm__str (comm); }
tools/perf/util/thread.c const char *thread__comm_str(const struct thread *thread) · line 173
Quoted 10 lines
tools/perf/util/thread.c +10
-0+const char *thread__comm_time_str (const struct thread *thread,u64 timestamp) +{ +const struct comm *comm =thread__comm_time (thread, timestamp); + +if (!comm) +return NULL ; + +return comm__str (comm); +} +
thread__comm_str_time()? time_str suggests a time-based string as opposed to a comm_str based on time.
David
Namhyung Kim
maintainerRe: [PATCH 11/37] perf tools: Introduce thread__comm_time() helpers
Quoted 7 lines · David Ahern
Quoted 6 lines · Namhyung Kim
tools/perf/util/thread.c +0
-0@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread *thread)return comm__str (comm); }
Quoted 15 lines
tools/perf/util/thread.c const char *thread__comm_str(const struct thread *thread) · line 173
Quoted 11 lines
tools/perf/util/thread.c +10
-0+const char *thread__comm_time_str (const struct thread *thread,u64 timestamp) +{ +const struct comm *comm =thread__comm_time (thread, timestamp); + +if (!comm) +return NULL ; + +return comm__str (comm); +} +
thread__comm_str_time()?time_strsuggests a time-based string as opposed to acomm_strbased on time.
Will change - my naming sense is always awful. I'd be happy to hear any naming suggestion. ;-)
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 12/37 perf tools: Add a test case for thread comm handling
In reply to Namhyung KimThe new test case checks various thread comm handling like overridding and time sorting.
--- tools/perf/Makefile.perf | 1 + tools/perf/tests/builtin-test.c | 4 ++++ tools/perf/tests/tests.h | 1 + tools/perf/tests/thread-comm.c | 47 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+) create mode 100644 tools/perf/tests/thread-comm.c
tools/perf/Makefile.perf +1 -0
@@ -447,6 +447,7 @@ endif LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o LIB_OBJS += $(OUTPUT)tests/switch-tracking.o +LIB_OBJS += $(OUTPUT)tests/thread-comm.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
@@ -167,6 +167,10 @@ static struct test { .func = test__fdarray__add, }, { + .desc = "Test thread comm handling", + .func = test__thread_comm, + }, + { .func = NULL, }, };
tools/perf/tests/tests.h +1 -0
@@ -51,6 +51,7 @@ int test__hists_cumulate(void);int test__switch_tracking (void );int test__fdarray__filter (void );int test__fdarray__add (void ); +int test__thread_comm (void );#if defined(__x86_64__) || defined(__i386__) || defined(__arm__)#ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-comm.c new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 13/37 perf tools: Use thread__comm_time() when adding hist entries
In reply to Namhyung KimNow thread->comm can be handled with time properly, use it to find correct comm when adding hist entries.
--- tools/perf/builtin-annotate.c | 5 +++-- tools/perf/builtin-diff.c | 8 ++++---- tools/perf/tests/hists_link.c | 4 ++-- tools/perf/util/hist.c | 19 ++++++++++--------- tools/perf/util/hist.h | 2 +- 5 files changed, 20 insertions(+), 18 deletions(-)
tools/perf/builtin-annotate.c +3 -2
@@ -47,7 +47,7 @@ struct perf_annotate { };static int perf_evsel__add_sample (struct perf_evsel *evsel, -struct perf_sample *sample__maybe_unused , +struct perf_sample *sample,struct addr_location *al,struct perf_annotate *ann) {
@@ -67,7 +67,8 @@ static int perf_evsel__add_sample(struct perf_evsel *evsel,return 0 ; } - he =__hists__add_entry (hists, al,NULL ,NULL ,NULL ,1 ,1 ,0 , true); + he =__hists__add_entry (hists, al,NULL ,NULL ,NULL ,1 ,1 ,0 , + sample->time, true);if (he ==NULL )return -ENOMEM ;
tools/perf/builtin-diff.c +4 -4
@@ -312,10 +312,10 @@ static int formula_fprintf(struct hist_entry *he, struct hist_entry *pair,static int hists__add_entry (struct hists *hists,struct addr_location *al,u64 period, -u64 weight,u64 transaction) +u64 weight,u64 transaction,u64 timestamp) {if (__hists__add_entry (hists, al,NULL ,NULL ,NULL , period, weight, - transaction, true) !=NULL ) + transaction, timestamp, true) !=NULL )return 0 ;return -ENOMEM ; }
@@ -335,8 +335,8 @@ static int diff__process_sample_event(struct perf_tool *tool __maybe_unused,return -1 ; } -if (hists__add_entry (hists, &al, sample->period, - sample->weight, sample->transaction)) { +if (hists__add_entry (hists, &al, sample->period, sample->weight, + sample->transaction, sample->time)) {pr_warning ("problem incrementing symbol period, skipping event\n" );return -1 ; }
tools/perf/tests/hists_link.c +2 -2
@@ -90,7 +90,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine) goto out; he =__hists__add_entry (hists, &al,NULL , -NULL ,NULL ,1 ,1 ,0 , true); +NULL ,NULL ,1 ,1 ,0 ,-1 , true);if (he ==NULL ) goto out;
@@ -114,7 +114,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine) goto out; he =__hists__add_entry (hists, &al,NULL , -NULL ,NULL ,1 ,1 ,0 , true); +NULL ,NULL ,1 ,1 ,0 ,-1 , true);if (he ==NULL ) goto out;
tools/perf/util/hist.c +10 -9
@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists, struct branch_info *bi, struct mem_info *mi, u64 period, u64 weight, u64 transaction, - bool sample_self) + u64 timestamp, bool sample_self) { struct hist_entry entry = { .thread = al->thread, - .comm = thread__comm(al->thread), + .comm = thread__comm_time(al->thread, timestamp), .ms = { .map = al->map, .sym = al->sym,
@@ -513,13 +513,14 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al {u64 cost;struct mem_info *mi = iter->priv; +struct perf_sample *sample = iter->sample;struct hists *hists =evsel__hists (iter->evsel);struct hist_entry *he;if (mi ==NULL )return -EINVAL ; - cost = iter->sample->weight; + cost = sample->weight;if (!cost) cost =1 ;
@@ -531,7 +532,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al* and the he_stat__add_period() function. */ he =__hists__add_entry (hists, al, iter->parent,NULL , mi, - cost, cost,0 , true); + cost, cost,0 , sample->time, true);if (!he)return -ENOMEM ;
@@ -632,7 +633,7 @@ iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *a* and not events sampled. Thus we use a pseudo period of 1. */ he =__hists__add_entry (hists, al, iter->parent, &bi[i],NULL , -1 ,1 ,0 , true); +1 ,1 ,0 , iter->sample->time, true);if (he ==NULL )return -ENOMEM ;
@@ -670,7 +671,7 @@ iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location he =__hists__add_entry (evsel__hists (evsel), al, iter->parent,NULL ,NULL , sample->period, sample->weight, - sample->transaction, true); + sample->transaction, sample->time, true);if (he ==NULL )return -ENOMEM ;
@@ -732,7 +733,7 @@ iter_add_single_cumulative_entry(struct hist_entry_iter *iter, he =__hists__add_entry (hists, al, iter->parent,NULL ,NULL , sample->period, sample->weight, - sample->transaction, true); + sample->transaction, sample->time, true);if (he ==NULL )return -ENOMEM ;
@@ -776,7 +777,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,struct hist_entry he_tmp = { .cpu = al->cpu, .thread = al->thread, - .comm =thread__comm (al->thread), + .comm =thread__comm_time (al->thread, sample->time), .ip = al->addr, .ms = { .map = al->map,
@@ -805,7 +806,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter, he =__hists__add_entry (evsel__hists (evsel), al, iter->parent,NULL ,NULL , sample->period, sample->weight, - sample->transaction, false); + sample->transaction, sample->time, false);if (he ==NULL )return -ENOMEM ;
tools/perf/util/hist.h +1 -1
@@ -109,7 +109,7 @@ struct hist_entry *__hists__add_entry(struct hists *hists,struct branch_info *bi, struct mem_info *mi, u64 period, u64 weight, u64 transaction, - bool sample_self); + u64 timestamp, bool sample_self);int hist_entry_iter__add (struct hist_entry_iter *iter,struct addr_location *al,struct perf_evsel *evsel,struct perf_sample *sample,int max_stack_depth,void *arg);
Signature · 1 line
2.1.3
Re: [PATCH 13/37] perf tools: Use thread__comm_time() when adding hist entries
On Wed, Dec 24, 2014 at 04:15:09PM +0900, Namhyung Kim wrote:
SNIP
Quoted 12 lines
tools/perf/tests/hists_link.c +1
-1he =__hists__add_entry (hists, &al,NULL , -NULL ,NULL ,1 ,1 ,0 , true); +NULL ,NULL ,1 ,1 ,0 ,-1 , true);if (he ==NULL ) goto out;tools/perf/util/hist.c +0
-0
tools/perf/util/hist.c struct hist_entry *__hists__add_entry(struct hists *hists, · line 458
Quoted 10 lines
tools/perf/util/hist.c +2
-2@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists, struct branch_info *bi, struct mem_info *mi, u64 period, u64 weight, u64 transaction, - bool sample_self) + u64 timestamp, bool sample_self) { struct hist_entry entry = { .thread = al->thread, - .comm = thread__comm(al->thread), + .comm = thread__comm_time(al->thread, timestamp),
with thread object having multiple comm entries, could this hurt the single threaded performance?
The thread__comm_time function iterates comm_list each time, maybe you could add some 'last_comm found check' logic in it?
jirka
Namhyung Kim
maintainerRe: [PATCH 13/37] perf tools: Use thread__comm_time() when adding hist entries
Quoted 21 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:09PM +0900, Namhyung Kim wrote:
SNIP
Quoted 17 lines
he = __hists__add_entry(hists, &al, NULL,tools/perf/tests/hists_link.c +1
-1- NULL, NULL, 1, 1, 0, true); + NULL, NULL, 1, 1, 0, -1, true);if (he ==NULL ) goto out;tools/perf/util/hist.c +1
-1@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,struct branch_info *bi, struct mem_info *mi, u64 period, u64 weight, u64 transaction, - bool sample_self) + u64 timestamp, bool sample_self)
Quoted 8 lines
tools/perf/util/hist.c struct hist_entry *__hists__add_entry(struct hists *hists, · line 458
Quoted 5 lines
tools/perf/util/hist.c +1
-1{struct hist_entry entry = { .thread = al->thread, - .comm =thread__comm (al->thread), + .comm =thread__comm_time (al->thread, timestamp),with thread object having multiple comm entries, could this hurt the single threaded performance?
Probably. But in my test, the single threaded performance on a single data file is slightly better or almost same. I didn't investigate it yet where the performance gain comes, but anyway, I think this has almost no effect on the performance since most thread will have just one or two comms I guess.
And JFYI, the single threaded performance on a multi-file data is better (I tested same data using 'perf data split' command and no --multi-thread option to perf report) as IMHO it doesn't need to use the ordered event queue layer.
Quoted 3 lines
The
thread__comm_timefunction iteratescomm_listeach time, maybe you could add some 'last_commfound check' logic in it?
I think it can be easily added later if it really affects the performance.
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 14/37 perf tools: Convert dead thread list into rbtree
In reply to Namhyung KimCurrently perf maintains dead threads in a linked list but this can be a problem if someone needs to search from it. Convert it to a rbtree like normal threads and it'll be used later with multi-file changes.
--- tools/perf/util/machine.c | 59 +++++++++++++++++++++++++++++++++++++++-------- tools/perf/util/machine.h | 2 +- tools/perf/util/thread.c | 1 + tools/perf/util/thread.h | 11 ++++----- 4 files changed, 57 insertions(+), 16 deletions(-)
tools/perf/util/machine.c +50 -9
@@ -28,7 +28,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)dsos__init (&machine->kernel_dsos); machine->threads =RB_ROOT ; -INIT_LIST_HEAD (&machine->dead_threads); + machine->dead_threads =RB_ROOT ; machine->last_match =NULL ; machine->vdso_info =NULL ;
@@ -91,10 +91,21 @@ static void dsos__delete(struct dsos *dsos)void machine__delete_dead_threads (struct machine *machine) { -struct thread *n, *t; +struct rb_node *nd =rb_first (&machine->dead_threads); + +while (nd) { + struct thread *t = rb_entry(nd, struct thread, rb_node); +struct thread *pos; + + nd =rb_next (nd); +rb_erase (&t->rb_node, &machine->dead_threads); + +while (!list_empty (&t->node)) { + pos = list_first_entry(&t->node, struct thread, node); +list_del (&pos->node); +thread__delete (pos); + } -list_for_each_entry_safe (t, n, &machine->dead_threads, node) { -list_del (&t->node);thread__delete (t); } }
@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)while (nd) { struct thread *t = rb_entry(nd, struct thread, rb_node); -rb_erase (&t->rb_node, &machine->threads); nd =rb_next (nd); +rb_erase (&t->rb_node, &machine->threads);thread__delete (t); } }
@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *eventstatic void machine__remove_thread (struct machine *machine,struct thread *th) { +struct rb_node **p = &machine->dead_threads.rb_node; +struct rb_node *parent =NULL ; +struct thread *pos; + machine->last_match =NULL ;rb_erase (&th->rb_node, &machine->threads); + + th->dead = true; +/* * We may have references to this thread, for instance in some hist_entry -* instances, so just move them to a separate list. +* instances, so just move them to a separate list in rbtree. */ -list_add_tail (&th->node, &machine->dead_threads); +while (*p !=NULL ) { + parent = *p; + pos = rb_entry(parent, struct thread, rb_node); + +if (pos->tid == th->tid) { +list_add_tail (&th->node, &pos->node); +return ; + } + +if (th->tid < pos->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } + +rb_link_node (&th->rb_node, parent, p); +rb_insert_color (&th->rb_node, &machine->dead_threads); } int machine__process_fork_event(struct machine *machine, union perf_event *event,
@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine *machine, void *priv) {struct rb_node *nd; -struct thread *thread; +struct thread *thread, *pos;int rc =0 ;for (nd =rb_first (&machine->threads); nd; nd =rb_next (nd)) {
@@ -1659,10 +1693,17 @@ int machine__for_each_thread(struct machine *machine,return rc; } -list_for_each_entry (thread, &machine->dead_threads, node) { +for (nd =rb_first (&machine->dead_threads); nd; nd =rb_next (nd)) { + thread = rb_entry(nd, struct thread, rb_node); rc =fn (thread, priv);if (rc !=0 )return rc; + +list_for_each_entry (pos, &thread->node, node) { + rc =fn (pos, priv); +if (rc !=0 ) +return rc; + } }return rc; }
tools/perf/util/machine.h +1 -1
@@ -30,7 +30,7 @@ struct machine {bool comm_exec;char *root_dir;struct rb_root threads; -struct list_head dead_threads; +struct rb_root dead_threads;struct thread *last_match;struct vdso_info *vdso_info;struct dsos user_dsos;
tools/perf/util/thread.c +1 -0
@@ -38,6 +38,7 @@ struct thread *thread__new(pid_t pid, pid_t tid) thread->ppid =-1 ; thread->cpu =-1 ;INIT_LIST_HEAD (&thread->comm_list); +INIT_LIST_HEAD (&thread->node);if (unwind__prepare_access (thread) <0 ) goto err_thread;
tools/perf/util/thread.h +5 -6
@@ -11,10 +11,8 @@struct thread_stack ;struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;
@@ -22,7 +20,8 @@ struct thread {int cpu;char shortname[3 ];bool comm_set; -bool dead;/* if set thread has exited */ +bool exited;/* if set thread has exited */ +bool dead;/* thread is in dead_threads list */ struct list_head comm_list;int comm_len;u64 db_id;
@@ -39,7 +38,7 @@ int thread__init_map_groups(struct thread *thread, struct machine *machine);void thread__delete (struct thread *thread);static inline void thread__exited (struct thread *thread) { - thread->dead = true; + thread->exited = true; } int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
Signature · 1 line
2.1.3
Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
On Wed, Dec 24, 2014 at 04:15:10PM +0900, Namhyung Kim wrote:
SNIP
Quoted 16 lines
static void
machine__remove_thread(struct machine *machine, struct thread *th){tools/perf/util/machine.c +8
-1+struct rb_node **p = &machine->dead_threads.rb_node; +struct rb_node *parent =NULL ; +struct thread *pos; + machine->last_match =NULL ;rb_erase (&th->rb_node, &machine->threads); + + th->dead = true; +/* * We may have references to this thread, for instance in some hist_entry -* instances, so just move them to a separate list. +* instances, so just move them to a separate list in rbtree.
tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270
Quoted 10 lines
tools/perf/util/machine.c +8
-1*/ -list_add_tail (&th->node, &machine->dead_threads); +while (*p !=NULL ) { + parent = *p; + pos = rb_entry(parent, struct thread, rb_node); + +if (pos->tid == th->tid) { +list_add_tail (&th->node, &pos->node); +return ; + }
hum, why is this 'new list' in thread object necessary? why not to store all in the tree?
Quoted 7 lines
tools/perf/util/machine.c +7
-0+ +if (th->tid < pos->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } +
SNIP
Quoted 18 lines
tools/perf/util/thread.h +2
-4@@ -11,10 +11,8 @@struct thread_stack ;struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;
tools/perf/util/thread.h struct thread { · line 24
Quoted 6 lines
tools/perf/util/thread.h +2
-1@@ -22,7 +20,8 @@ struct thread {int cpu;char shortname[3 ];bool comm_set; -bool dead;/* if set thread has exited */ +bool exited;/* if set thread has exited */ +bool dead;/* thread is in dead_threads list */
looks like this also changes the logic (new exited flag), not just the dead threads storage wheel
jirka
Namhyung Kim
maintainerRe: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
Quoted 25 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:10PM +0900, Namhyung Kim wrote:
SNIP
Quoted 21 lines
static void
machine__remove_thread(struct machine *machine, struct thread *th){tools/perf/util/machine.c +11
-2+struct rb_node **p = &machine->dead_threads.rb_node; +struct rb_node *parent =NULL ; +struct thread *pos; + machine->last_match =NULL ;rb_erase (&th->rb_node, &machine->threads); + + th->dead = true; +/* * We may have references to this thread, for instance in some hist_entry -* instances, so just move them to a separate list. +* instances, so just move them to a separate list in rbtree. */ -list_add_tail (&th->node, &machine->dead_threads); +while (*p !=NULL ) { + parent = *p; + pos = rb_entry(parent, struct thread, rb_node);
Quoted 8 lines
tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270
Quoted 5 lines
tools/perf/util/machine.c +5
-0+ +if (pos->tid == th->tid) { +list_add_tail (&th->node, &pos->node); +return ; + }hum, why is this 'new list' in thread object necessary? why not to store all in the tree?
No absolute reason, but I just wanted to keep them in a single place and to see how many of them exist easily. We could compare pid and then timestamp so that it can be in a single rbtree.
Quoted 31 lines
Quoted 9 lines
tools/perf/util/machine.c +7
-0+ +if (th->tid < pos->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } +SNIP
Quoted 19 lines
tools/perf/util/thread.h +2
-4@@ -11,10 +11,8 @@struct thread_stack ;struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;@@ -22,7 +20,8 @@ struct thread {int cpu;
Quoted 8 lines
tools/perf/util/thread.h struct thread { · line 24
Quoted 5 lines
tools/perf/util/thread.h +2
-1char shortname[3 ];bool comm_set; -bool dead;/* if set thread has exited */ +bool exited;/* if set thread has exited */ +bool dead;/* thread is in dead_threads list */ looks like this also changes the logic (new exited flag), not just the dead threads storage wheel
AFAICS the 'dead' flag is not used other than thread__exited(). And it confused me a dead thread might not be in a dead_threads tree (or list). So I changed the name and no logical change intended.
Thanks,
Namhyung
Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
Quoted 28 lines · Namhyung Kim
Quoted 19 lines
Quoted 19 lines
tools/perf/util/thread.h +2
-4@@ -11,10 +11,8 @@struct thread_stack ;struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;@@ -22,7 +20,8 @@ struct thread {int cpu;Quoted 8 lines
tools/perf/util/thread.h struct thread { · line 24
Quoted 5 lines
tools/perf/util/thread.h +2
-1char shortname[3 ];bool comm_set; -bool dead;/* if set thread has exited */ +bool exited;/* if set thread has exited */ +bool dead;/* thread is in dead_threads list */ looks like this also changes the logic (new exited flag), not just the dead threads storage wheel
Quoted 3 lines
AFAICS the 'dead' flag is not used other than
thread__exited(). And it confused me a dead thread might not be in adead_threadstree (or list). So I changed the name and no logical change intended.
git show 236a3bbd5cb51
David
Namhyung Kim
maintainerRe: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
Quoted 32 lines · David Ahern
Quoted 31 lines · Namhyung Kim
Quoted 29 lines
Quoted 19 lines
tools/perf/util/thread.h +2
-4@@ -11,10 +11,8 @@struct thread_stack ;struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;tools/perf/util/thread.h struct thread { · line 24
Quoted 6 lines
tools/perf/util/thread.h +2
-1@@ -22,7 +20,8 @@ struct thread {int cpu;char shortname[3 ];bool comm_set; -bool dead;/* if set thread has exited */ +bool exited;/* if set thread has exited */ +bool dead;/* thread is in dead_threads list */ looks like this also changes the logic (new exited flag), not just the dead threads storage wheel
Quoted 6 lines
Quoted 3 lines
AFAICS the 'dead' flag is not used other than
thread__exited(). And it confused me a dead thread might not be in adead_threadstree (or list). So I changed the name and no logical change intended.git show
236a3bbd5cb51
Thanks for the pointer. I understand the need of delaying the move to dead_threads list, but anyway it still makes me confused. So I renamed the flag to keep the current behavior and match the name 'dead' to the list/tree management to reduce further confusion.
Thanks,
Namhyung
Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
In reply to Namhyung Kimtools/perf/util/machine.c void machine__delete_threads(struct machine *machine) · line 124
Quoted 10 lines · Namhyung Kim
tools/perf/util/machine.c +1
-1@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)while (nd) { struct thread *t = rb_entry(nd, struct thread, rb_node); -rb_erase (&t->rb_node, &machine->threads); nd =rb_next (nd); +rb_erase (&t->rb_node, &machine->threads);thread__delete (t); } }
unrelated to dead threads. Bug fix? separate patch?
Quoted 17 lines
tools/perf/util/machine.c +8
-1@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *eventstatic void machine__remove_thread (struct machine *machine,struct thread *th) { +struct rb_node **p = &machine->dead_threads.rb_node; +struct rb_node *parent =NULL ; +struct thread *pos; + machine->last_match =NULL ;rb_erase (&th->rb_node, &machine->threads); + + th->dead = true; +/* * We may have references to this thread, for instance in some hist_entry -* instances, so just move them to a separate list. +* instances, so just move them to a separate list in rbtree.
tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270
Quoted 10 lines
tools/perf/util/machine.c +8
-1*/ -list_add_tail (&th->node, &machine->dead_threads); +while (*p !=NULL ) { + parent = *p; + pos = rb_entry(parent, struct thread, rb_node); + +if (pos->tid == th->tid) { +list_add_tail (&th->node, &pos->node); +return ; + }
So you have a linked list for tid collisions (not mentioned in the git log).
tools/perf/util/machine.c int machine__for_each_thread(struct machine *machine,
Quoted 13 lines
tools/perf/util/machine.c +9
-0+ +if (th->tid < pos->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } + +rb_link_node (&th->rb_node, parent, p); +rb_insert_color (&th->rb_node, &machine->dead_threads); } int machine__process_fork_event(struct machine *machine, union perf_event *event,@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine *machine,
---8<---
Quoted 7 lines
tools/perf/util/thread.h +0
-0@@ -11,10 +11,8 @@struct thread_stack ;
tools/perf/util/thread.h line 18
Quoted 10 lines
tools/perf/util/thread.h +2
-4struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;
could use better names for rb_node and node. rb_node is the entry in the dead_threads tree - dead_node?; node is the linked list for tid collisions - tid_node?
David
Namhyung Kim
maintainerRe: [PATCH 14/37] perf tools: Convert dead thread list into rbtree
Hi David,
Quoted 6 lines · David Ahern
Quoted 5 lines · Namhyung Kim
tools/perf/util/machine.c +0
-0@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)while (nd) { struct thread *t = rb_entry(nd, struct thread, rb_node);
Quoted 9 lines
tools/perf/util/machine.c void machine__delete_threads(struct machine *machine) · line 124
Quoted 6 lines
tools/perf/util/machine.c +1
-1-rb_erase (&t->rb_node, &machine->threads); nd =rb_next (nd); +rb_erase (&t->rb_node, &machine->threads);thread__delete (t); } }unrelated to dead threads. Bug fix? separate patch?
Yes, I'll make it a separate bug-fix patch.
Quoted 27 lines
Quoted 27 lines
tools/perf/util/machine.c +7
-0@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *eventstatic void machine__remove_thread (struct machine *machine,struct thread *th) { +struct rb_node **p = &machine->dead_threads.rb_node; +struct rb_node *parent =NULL ; +struct thread *pos; + machine->last_match =NULL ;rb_erase (&th->rb_node, &machine->threads); + + th->dead = true; +/* * We may have references to this thread, for instance in some
hist_entrytools/perf/util/machine.c +4
-2-* instances, so just move them to a separate list. +* instances, so just move them to a separate list in rbtree. */ -list_add_tail (&th->node, &machine->dead_threads); +while (*p !=NULL ) { + parent = *p; + pos = rb_entry(parent, struct thread, rb_node);
Quoted 8 lines
tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270
Quoted 5 lines
tools/perf/util/machine.c +5
-0+ +if (pos->tid == th->tid) { +list_add_tail (&th->node, &pos->node); +return ; + }So you have a linked list for tid collisions (not mentioned in the git log).
Right, will add a description.
Quoted 27 lines
Quoted 15 lines
tools/perf/util/machine.c +9
-0+ +if (th->tid < pos->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + } + +rb_link_node (&th->rb_node, parent, p); +rb_insert_color (&th->rb_node, &machine->dead_threads); } int machine__process_fork_event(struct machine *machine, union
perf_event*event,Review int machine__for_each_thread(struct machine · line 1683
Quoted 1 line
(quoted diff) +0
-0@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine *machine,---8<---
Quoted 7 lines
tools/perf/util/thread.h +0
-0@@ -11,10 +11,8 @@struct thread_stack ;
Quoted 15 lines
tools/perf/util/thread.h line 18
Quoted 10 lines
tools/perf/util/thread.h +2
-4struct thread { -union { -struct rb_node rb_node; -struct list_head node; - }; +struct rb_node rb_node; +struct list_head node;struct map_groups *mg;pid_t pid_;/* Not all tools update this */ pid_t tid;could use better names for
rb_nodeand node.rb_nodeis the entry in thedead_threadstree -dead_node?; node is the linked list for tid collisions -tid_node?
But the rb_node is used for 3 different purpose depends on its state - a thread can be in a (normal) threads tree, dead threads tree or missing threads tree (will be introduced later).
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 15/37 perf tools: Introduce machine__find*_thread_time()
In reply to Namhyung KimWith multi-file data storage is enabled, it needs to search thread based on sample time since sample processing is done after other (task, comm and mmap) events are processed. This can be a problem if a session is very long and pid is recycled - in that case it'll only see the last one.
So keep thread start time in it, and search thread based on the time. This patch introduces machine__find{,new}_thread_time() function for this. It'll first search current thread rbtree and then dead thread tree and list. If it couldn't find anyone, it'll create a new thread.
The sample timestamp of 0 means that this is called from synthesized event so just use current rbtree. The timestamp will be -1 if sample didn't record the timestamp so will see current threads automatically.
Dead threads are managed in a rbtree, and if there's more than one thread has sample tid/pid, it'll be saved in a list.
--- tools/perf/builtin-script.c | 5 +- tools/perf/tests/dwarf-unwind.c | 10 ++-- tools/perf/tests/hists_common.c | 3 +- tools/perf/tests/hists_link.c | 2 +- tools/perf/util/event.c | 4 +- tools/perf/util/machine.c | 109 +++++++++++++++++++++++++++++++++++++++- tools/perf/util/machine.h | 8 ++- tools/perf/util/thread.c | 4 ++ tools/perf/util/thread.h | 1 + 9 files changed, 132 insertions(+), 14 deletions(-)
tools/perf/builtin-script.c +3 -2
@@ -549,8 +549,9 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused, struct machine *machine) {struct addr_location al; -struct thread *thread =machine__findnew_thread (machine, sample->pid, - sample->tid); +struct thread *thread =machine__findnew_thread_time (machine, sample->pid, + sample->tid, + sample->time);if (thread ==NULL ) { pr_debug("problem processing %d event, skipping it.\n",
tools/perf/tests/dwarf-unwind.c +5 -5
@@ -13,10 +13,10 @@static int mmap_handler (struct perf_tool *tool__maybe_unused ,union perf_event *event, -struct perf_sample *sample__maybe_unused , +struct perf_sample *sample,struct machine *machine) { -return machine__process_mmap2_event (machine, event,NULL ); +return machine__process_mmap2_event (machine, event, sample); }static int init_live_machine (struct machine *machine)
@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg) __attribute__ ((noinline))static int unwind_thread (struct thread *thread) { -struct perf_sample sample; +struct perf_sample sample = { + .time =-1ULL , + };unsigned long cnt =0 ;int err =-1 ; -memset (&sample,0 ,sizeof (sample)); -if (test__arch_unwind_sample (&sample, thread)) {pr_debug ("failed to get unwind sample\n" ); goto out;
tools/perf/tests/hists_common.c +2 -1
@@ -80,6 +80,7 @@ static struct {struct machine *setup_fake_machine (struct machines *machines) {struct machine *machine =machines__find (machines,HOST_KERNEL_ID ); +struct perf_sample sample = { .time =-1ULL , };size_t i;if (machine ==NULL ) {
@@ -113,7 +114,7 @@ struct machine *setup_fake_machine(struct machines *machines)strcpy (fake_mmap_event.mmap.filename, fake_mmap_info[i].filename); -machine__process_mmap_event (machine, &fake_mmap_event,NULL ); +machine__process_mmap_event (machine, &fake_mmap_event, &sample); }for (i =0 ; i <ARRAY_SIZE (fake_symbols); i++) {
tools/perf/tests/hists_link.c +1 -1
@@ -64,7 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)struct perf_evsel *evsel;struct addr_location al;struct hist_entry *he; -struct perf_sample sample = { .period =1 , }; +struct perf_sample sample = { .period =1 , .time =-1ULL , };size_t i =0 , k;/*
tools/perf/util/event.c +2 -2
@@ -825,8 +825,8 @@ int perf_event__preprocess_sample(const union perf_event *event, struct perf_sample *sample) {u8 cpumode = event->header.misc &PERF_RECORD_MISC_CPUMODE_MASK ; -struct thread *thread =machine__findnew_thread (machine, sample->pid, - sample->tid); +struct thread *thread =machine__findnew_thread_time (machine, sample->pid, + sample->tid, sample->time);if (thread ==NULL )return -1 ;
tools/perf/util/machine.c +108 -1
127 lines not on this page · raw message
tools/perf/util/machine.h +6 -2
@@ -68,8 +68,6 @@ static inline bool machine__kernel_ip(struct machine *machine, u64 ip)return ip >= kernel_start; } -struct thread *machine__find_thread (struct machine *machine,pid_t pid, -pid_t tid);struct comm *machine__thread_exec_comm (struct machine *machine,struct thread *thread);
@@ -149,6 +147,12 @@ static inline bool machine__is_host(struct machine *machine)struct thread *machine__findnew_thread (struct machine *machine,pid_t pid,pid_t tid); +struct thread *machine__find_thread (struct machine *machine,pid_t pid, +pid_t tid); +struct thread *machine__findnew_thread_time (struct machine *machine,pid_t pid, +pid_t tid,u64 timestamp); +struct thread *machine__find_thread_time (struct machine *machine,pid_t pid, +pid_t tid,u64 timestamp);size_t machine__fprintf (struct machine *machine,FILE *fp);
tools/perf/util/thread.c +4 -0
@@ -128,6 +128,9 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,/* Override the default :tid entry */ if (!thread->comm_set) { +if (!thread->start_time) + thread->start_time = timestamp; + err =comm__override (curr, str, timestamp, exec);if (err)return err;
@@ -229,6 +232,7 @@ int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp) } thread->ppid = parent->tid; + thread->start_time = timestamp;return thread__clone_map_groups (thread, parent); }
tools/perf/util/thread.h +1 -0
@@ -25,6 +25,7 @@ struct thread {struct list_head comm_list;int comm_len;u64 db_id; +u64 start_time;void *priv;struct thread_stack *ts;
Signature · 1 line
2.1.3
Re: [PATCH 15/37] perf tools: Introduce machine__find*_thread_time()
tools/perf/tests/dwarf-unwind.c static int unwind_entry(struct unwind_entry *entry, void *arg) · line 66
Quoted 8 lines · Namhyung Kim
tools/perf/tests/dwarf-unwind.c +3
-1@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg) __attribute__ ((noinline))static int unwind_thread (struct thread *thread) { -struct perf_sample sample; +struct perf_sample sample = { + .time =-1ULL , + };
1-liner like the others?
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 434
Quoted 9 lines
tools/perf/util/machine.c +1
-0@@ -431,6 +431,103 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,return __machine__findnew_thread (machine, pid, tid, false); } +static void machine__remove_thread (struct machine *machine,struct thread *th);
Why is this declaration needed?
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 446
Quoted 12 lines
tools/perf/util/machine.c +12
-0+ +static struct thread *__machine__findnew_thread_time (struct machine *machine, +pid_t pid,pid_t tid, +u64 timestamp,bool create) +{ +struct thread *curr, *pos, *new; +struct thread *th =NULL ; +struct rb_node **p; +struct rb_node *parent =NULL ; +bool initial = timestamp == (u64 )0 ; + + curr =__machine__findnew_thread (machine, pid, tid, initial);
What if create arg is false? Should initial arg also be false?
Quoted 19 lines
tools/perf/util/machine.c +19
-0+if (curr && timestamp >= curr->start_time) +return curr; + + p = &machine->dead_threads.rb_node; +while (*p !=NULL ) { + parent = *p; + th = rb_entry(parent, struct thread, rb_node); + +if (th->tid == tid) { +list_for_each_entry (pos, &th->node, node) { +if (timestamp >= pos->start_time && + pos->start_time > th->start_time) { + th = pos; +break ; + } + } + +if (timestamp >= th->start_time) { +machine__update_thread_pid (machine, th, pid);
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 475
Quoted 10 lines
tools/perf/util/machine.c +10
-0+return th; + } +break ; + } + +if (tid < th->tid) + p = &(*p)->rb_left; +else + p = &(*p)->rb_right; + }
Can the dead_threads search be put in a separate function -- machine__find_dead_thread?
Quoted 16 lines
tools/perf/util/machine.c +16
-0+ +if (!create) +return NULL ; + +if (!curr) +return __machine__findnew_thread (machine, pid, tid, true); + + new =thread__new (pid, tid); +if (new ==NULL ) +return NULL ; + + new->start_time = timestamp; + +if (*p) { +list_for_each_entry (pos, &th->node, node) { +/* sort by time */
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 501
Quoted 10 lines
tools/perf/util/machine.c +10
-0+if (timestamp >= pos->start_time) { + th = pos; +break ; + } + } +list_add_tail (&new->node, &th->node); + }else { +rb_link_node (&new->rb_node, parent, p); +rb_insert_color (&new->rb_node, &machine->dead_threads); + }
Why insert this unknown tid on the dead_threads list?
Quoted 6 lines
tools/perf/util/machine.c +6
-0+ +/* +* We have to initialize map_groups separately +* after rb tree is updated. +* +* The reason is that we call machine__findnew_thread
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 517
Quoted 10 lines
tools/perf/util/machine.c +10
-0+* within thread__init_map_groups to find the thread +* leader and that would screwed the rb tree. +*/ +if (thread__init_map_groups (new, machine)) { +thread__delete (new); +return NULL ; + } + +return new; +}
---8<---
Quoted 4 lines
tools/perf/util/machine.c +0
-0@@ -1265,6 +1362,16 @@ static void machine__remove_thread(struct machine *machine, struct thread *th) pos = rb_entry(parent, struct thread, rb_node);if (pos->tid == th->tid) {
tools/perf/util/machine.c static void machine__remove_thread(struct machine *machine, struct thread *th) · line 1374
Quoted 10 lines
tools/perf/util/machine.c +10
-0+struct thread *old; + +/* sort by time */ +list_for_each_entry (old, &pos->node, node) { +if (th->start_time >= old->start_time) { + pos = old; +break ; + } + } +
this change seems unrelated to the patch subject -- machine__find*_thread_time.
David
Namhyung Kim
maintainerRe: [PATCH 15/37] perf tools: Introduce machine__find*_thread_time()
Quoted 14 lines · David Ahern
tools/perf/tests/dwarf-unwind.c static int unwind_entry(struct unwind_entry *entry, · line 67
Quoted 10 lines · Namhyung Kim
tools/perf/tests/dwarf-unwind.c +3
-1@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg) __attribute__ ((noinline))static int unwind_thread (struct thread *thread) { -struct perf_sample sample; +struct perf_sample sample = { + .time =-1ULL , + };1-liner like the others?
Did you mean this?
struct perf_sample sample = { .time = -1ULL, };Quoted 12 lines
Quoted 12 lines
tools/perf/util/machine.c +0
-0@@ -431,6 +431,103 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid, return __machine__findnew_thread(machine, pid, tid, false); }
Quoted 5 lines
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine · line 436
Quoted 2 lines
tools/perf/util/machine.c +1
-0+static void machine__remove_thread (struct machine *machine,struct thread *th);Why is this declaration needed?
It seems like a leftover from previous changes, will remove.
Quoted 10 lines
Quoted 10 lines
tools/perf/util/machine.c +4
-0+ +static struct thread *__machine__findnew_thread_time (struct machine *machine, +pid_t pid,pid_t tid, + u64 timestamp, boolcreate)
tools/perf/util/machine.c +3
-0+{ +struct thread *curr, *pos, *new; +struct thread *th =NULL ;
Quoted 8 lines
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 446
Quoted 5 lines
tools/perf/util/machine.c +5
-0+struct rb_node **p; +struct rb_node *parent =NULL ; +bool initial = timestamp == (u64 )0 ; + + curr =__machine__findnew_thread (machine, pid, tid, initial);What if create arg is false? Should initial arg also be false?
Nop, the create arg adds the thread to the dead thread tree (or missing thread tree later). And the initial flag adds it to the normal threads tree like in case of synthesized events. This was because I firstly used the *_findnew_thread_time() for the meta event processing, but then I realized that using *_findnew_thread() makes the processing much easier since the meta events are processed sequencially. And this *_findnew_thread_time() is used only for sample processing. Maybe we can use 'false' instead of the 'initial'..
Quoted 27 lines
Quoted 27 lines
tools/perf/util/machine.c +19
-0+if (curr && timestamp >= curr->start_time) +return curr; + + p = &machine->dead_threads.rb_node; +while (*p !=NULL ) { + parent = *p; + th = rb_entry(parent, struct thread, rb_node); + +if (th->tid == tid) { +list_for_each_entry (pos, &th->node, node) { +if (timestamp >= pos->start_time && + pos->start_time > th->start_time) { + th = pos; +break ; + } + } + +if (timestamp >= th->start_time) { + machine__update_thread_pid(machine, th,pid);
tools/perf/util/machine.c +6
-0+return th; + } +break ; + } + +if (tid < th->tid)
Quoted 8 lines
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 475
Quoted 4 lines
tools/perf/util/machine.c +4
-0+ p = &(*p)->rb_left; + else + p = &(*p)->rb_right; + }Can the
dead_threadssearch be put in a separate function --machine__find_dead_thread?
Okay, I can do it.
Quoted 22 lines
Quoted 22 lines
tools/perf/util/machine.c +21
-0+ +if (!create) +return NULL ; + +if (!curr) +return __machine__findnew_thread (machine, pid, tid, true); + + new =thread__new (pid, tid); +if (new ==NULL ) +return NULL ; + + new->start_time = timestamp; + +if (*p) { +list_for_each_entry (pos, &th->node, node) { +/* sort by time */ +if (timestamp >= pos->start_time) { + th = pos; +break ; + } + }
Quoted 8 lines
tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 501
Quoted 5 lines
tools/perf/util/machine.c +5
-0+list_add_tail (&new->node, &th->node); + }else { +rb_link_node (&new->rb_node, parent, p); +rb_insert_color (&new->rb_node, &machine->dead_threads); + }Why insert this unknown tid on the
dead_threadslist?
Well, mostly the search will be succeeded, and if it failed it's either newly created thread so curr = NULL and adds it to the normal tree. Otherwise (rarely) there might be a missing fork event and the sample is for an older thread in that any timestamp didn't match to existing (current and dead) threads. So I added it to the dead thread tree. But with multi-thread support enabled, it'll be added to the new missing threads tree which is protected by a mutex.
Quoted 25 lines
Quoted 7 lines
tools/perf/util/machine.c +6
-0+ +/* +* We have to initialize map_groups separately +* after rb tree is updated. +* +* The reason is that we call machine__findnew_thread tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 517
Quoted 10 lines
tools/perf/util/machine.c +10
-0+* within thread__init_map_groups to find the thread +* leader and that would screwed the rb tree. +*/ +if (thread__init_map_groups (new, machine)) { +thread__delete (new); +return NULL ; + } + +return new; +}---8<---
Quoted 4 lines
tools/perf/util/machine.c +0
-0@@ -1265,6 +1362,16 @@ static void machine__remove_thread(struct machine *machine, struct thread *th) pos = rb_entry(parent, struct thread, rb_node);
Quoted 15 lines
tools/perf/util/machine.c static void machine__remove_thread(struct machine *machine, struct thread *th) · line 1374
Quoted 11 lines
tools/perf/util/machine.c +10
-0if (pos->tid == th->tid) { +struct thread *old; + +/* sort by time */ +list_for_each_entry (old, &pos->node, node) { +if (th->start_time >= old->start_time) { + pos = old; +break ; + } + } +this change seems unrelated to the patch subject -- machine__find*_thread_time.
It searchs a thread in the dead threads tree based on timestamp value. So the list should be sorted by time order. But yes, I agree that this change should be moved to the dead thread conversion patch.
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 16/37 perf tools: Add a test case for timed thread handling
In reply to Namhyung KimA test case for verifying live and dead thread tree management during time change and new machine__find{,new}_thread_time().
--- tools/perf/Makefile.perf | 1 + tools/perf/tests/builtin-test.c | 4 + tools/perf/tests/tests.h | 1 + tools/perf/tests/thread-lookup-time.c | 174 ++++++++++++++++++++++++++++++++++ 4 files changed, 180 insertions(+) create mode 100644 tools/perf/tests/thread-lookup-time.c
tools/perf/Makefile.perf +1 -0
@@ -448,6 +448,7 @@ LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o LIB_OBJS += $(OUTPUT)tests/switch-tracking.o LIB_OBJS += $(OUTPUT)tests/thread-comm.o +LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
@@ -171,6 +171,10 @@ static struct test { .func = test__thread_comm, }, { + .desc = "Test thread lookup with time", + .func = test__thread_lookup_time, + }, + { .func = NULL, }, };
tools/perf/tests/tests.h +1 -0
@@ -52,6 +52,7 @@ int test__switch_tracking(void);int test__fdarray__filter (void );int test__fdarray__add (void );int test__thread_comm (void ); +int test__thread_lookup_time (void );#if defined(__x86_64__) || defined(__i386__) || defined(__arm__)#ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-lookup-time.c new file +174 -0
174 lines not on this page · raw message
Signature · 1 line
2.1.3
Re: [PATCH 16/37] perf tools: Add a test case for timed thread handling
On Wed, Dec 24, 2014 at 04:15:12PM +0900, Namhyung Kim wrote:
SNIP
Quoted 34 lines
tools/perf/tests/builtin-test.c +0
-0.func = NULL, }, };tools/perf/tests/tests.h +1
-0@@ -52,6 +52,7 @@ int test__switch_tracking(void);int test__fdarray__filter (void );int test__fdarray__add (void );int test__thread_comm (void ); +int test__thread_lookup_time (void );#if defined(__x86_64__) || defined(__i386__) || defined(__arm__)#ifdef HAVE_DWARF_UNWIND_SUPPORT tools/perf/tests/thread-lookup-time.c new file +13
-0@@ -0,0 +1,174 @@ +#include "tests.h" +#include "machine.h" +#include "thread.h" +#include "map.h" +#include "debug.h" + +static int thread__print_cb (struct thread *th,void *arg__maybe_unused ) +{ +printf ("thread: %d, start time: %" PRIu64" %s\n" , + th->tid, th->start_time, th->dead ?"(dead)" :"" ); +return 0 ; +} +
tools/perf/tests/thread-lookup-time.c line 23
Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10
-0+static int lookup_with_timestamp (struct machine *machine) +{ +struct thread *t1, *t2, *t3; +union perf_event fork = { + .fork = { + .pid =0 , + .tid =0 , + .ppid =1 , + .ptid =1 , + },
I've got following output from the test:
test child forked, pid 18483
========= after t1 created ==========
thread: 0, start time: 0
========= after t1 set comm ==========
thread: 0, start time: 20000
========= after t2 forked ==========
thread: 0, start time: 50000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 20000 (dead)
========= after t3 forked ==========
thread: 0, start time: 60000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 50000 (dead)
thread: 0, start time: 20000 (dead)
test child finished with 0
'after t2 forked' data shows 'thread 0 with time 50000' and newly added parent 'thread: 1, start time: 0'
this makes me wonder if you wanted switch 0 and 1 for pid and ppid in above sample init and follow with forked pid 1 ... but not sure because you're using the same sample for fork 3 ;-)
my question is if that was intentional, because I've got confused in here
Quoted 24 lines
tools/perf/tests/thread-lookup-time.c new file +24
-0+ }; +struct perf_sample sample = { + .time =50000 , + }; + +/* start_time is set to 0 */ + t1 =machine__findnew_thread (machine,0 ,0 ); + +if (verbose >1 ) { +printf ("========= after t1 created ==========\n" ); +machine__for_each_thread (machine, thread__print_cb,NULL ); + } + +TEST_ASSERT_VAL ("wrong start time of old thread" , t1->start_time ==0 ); + +TEST_ASSERT_VAL ("cannot find current thread" , +machine__find_thread (machine,0 ,0 ) == t1); + +TEST_ASSERT_VAL ("cannot find current thread with time" , +machine__findnew_thread_time (machine,0 ,0 ,10000 ) == t1); + +/* start_time is overwritten to new value */ +thread__set_comm (t1,"/usr/bin/perf" ,20000 ); +
tools/perf/tests/thread-lookup-time.c line 57
Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10
-0+if (verbose >1 ) { +printf ("========= after t1 set comm ==========\n" ); +machine__for_each_thread (machine, thread__print_cb,NULL ); + } + +TEST_ASSERT_VAL ("failed to update start time" , t1->start_time ==20000 ); + +TEST_ASSERT_VAL ("should not find passed thread" , +/* this will create yet another dead thread */ +machine__findnew_thread_time (machine,0 ,0 ,10000 ) != t1);
also this comment say that calling machine__findnew_thread_time will create another dead thread, which actually did not happened based on above test output
thanks,
jirka
Namhyung Kim
maintainerRe: [PATCH 16/37] perf tools: Add a test case for timed thread handling
Quoted 72 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:15:12PM +0900, Namhyung Kim wrote:
SNIP
Quoted 34 lines
.func = NULL,
},
};tools/perf/tests/tests.h +1
-0@@ -52,6 +52,7 @@ int test__switch_tracking(void);int test__fdarray__filter (void );int test__fdarray__add (void );int test__thread_comm (void ); +int test__thread_lookup_time (void );#if defined(__x86_64__) || defined(__i386__) || defined(__arm__)#ifdef HAVE_DWARF_UNWIND_SUPPORT tools/perf/tests/thread-lookup-time.c new file +13
-0@@ -0,0 +1,174 @@ +#include "tests.h" +#include "machine.h" +#include "thread.h" +#include "map.h" +#include "debug.h" + +static int thread__print_cb (struct thread *th,void *arg__maybe_unused ) +{ +printf ("thread: %d, start time: %" PRIu64" %s\n" , + th->tid, th->start_time, th->dead ?"(dead)" :"" ); +return 0 ; +} +tools/perf/tests/thread-lookup-time.c line 23
Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10
-0+static int lookup_with_timestamp (struct machine *machine) +{ +struct thread *t1, *t2, *t3; +union perf_event fork = { + .fork = { + .pid =0 , + .tid =0 , + .ppid =1 , + .ptid =1 , + },I've got following output from the test:
test child forked, pid 18483
========= after t1 created ==========
thread: 0, start time: 0
========= after t1 set comm ==========
thread: 0, start time: 20000
========= after t2 forked ==========
thread: 0, start time: 50000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 20000 (dead)
========= after t3 forked ==========
thread: 0, start time: 60000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 50000 (dead)
thread: 0, start time: 20000 (dead)
test child finished with 0'after t2 forked' data shows 'thread 0 with time 50000' and newly added parent 'thread: 1, start time: 0'
Quoted 6 lines
this makes me wonder if you wanted switch 0 and 1 for pid and ppid in above sample init and follow with forked pid 1 ... but not sure because you're using the same sample for fork 3 ;-)
my question is if that was intentional, because I've got confused in here
Yeah, it's intentional. I'm testing machine__findnew_thread_time() and machine__process_fork_event() can generate threads properly. The former creates a dead thread if the timestamp is before any of existing threads which have a same pid. The latter can create two threads - one for tid and another for ptid (only if it doesn't exist).
Quoted 31 lines
Quoted 31 lines
tools/perf/tests/thread-lookup-time.c new file +30
-0+ }; +struct perf_sample sample = { + .time =50000 , + }; + +/* start_time is set to 0 */ + t1 =machine__findnew_thread (machine,0 ,0 ); + +if (verbose >1 ) { +printf ("========= after t1 created ==========\n" ); +machine__for_each_thread (machine, thread__print_cb,NULL ); + } + +TEST_ASSERT_VAL ("wrong start time of old thread" , t1->start_time ==0 ); + +TEST_ASSERT_VAL ("cannot find current thread" , +machine__find_thread (machine,0 ,0 ) == t1); + +TEST_ASSERT_VAL ("cannot find current thread with time" , +machine__findnew_thread_time (machine,0 ,0 ,10000 ) == t1); + +/* start_time is overwritten to new value */ +thread__set_comm (t1,"/usr/bin/perf" ,20000 ); + +if (verbose >1 ) { +printf ("========= after t1 set comm ==========\n" ); +machine__for_each_thread (machine, thread__print_cb,NULL ); + } + +TEST_ASSERT_VAL ("failed to update start time" , t1->start_time ==20000 );
Quoted 8 lines
tools/perf/tests/thread-lookup-time.c line 57
Quoted 4 lines
tools/perf/tests/thread-lookup-time.c new file +4
-0+ +TEST_ASSERT_VAL ("should not find passed thread" , +/* this will create yet another dead thread */ +machine__findnew_thread_time (machine,0 ,0 ,10000 ) != t1);also this comment say that calling
machine__findnew_thread_timewill create another dead thread, which actually did not happened based on above test output
Oh, it's actually a dead thread - it's in the dead threads tree - but I just missed to set the dead flag. :)
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 17/37 perf tools: Maintain map groups list in a leader thread
In reply to Namhyung KimTo support multi-threaded perf report, we need to maintain time-sorted map groups. Add ->mg_list member to struct thread and sort the list by time. Now leader threads have one more refcnt for map groups in the list so also update the thread-mg-share test case.
Currently only add a new map groups when an exec (comm) event is received.
--- tools/perf/tests/thread-mg-share.c | 7 +++- tools/perf/util/event.c | 2 + tools/perf/util/machine.c | 2 +- tools/perf/util/map.c | 1 + tools/perf/util/map.h | 2 + tools/perf/util/thread.c | 80 +++++++++++++++++++++++++++++++++++++- tools/perf/util/thread.h | 3 ++ 7 files changed, 93 insertions(+), 4 deletions(-)
tools/perf/tests/thread-mg-share.c +5 -2
@@ -23,6 +23,9 @@ int test__thread_mg_share(void)* with several threads and checks they properly share and * maintain map groups info (struct map_groups). * +* Note that a leader thread has one more refcnt for its +* (current) map groups. +* * thread group (pid: 0, tids: 0, 1, 2, 3) * other group (pid: 4, tids: 4, 5) */
@@ -43,7 +46,7 @@ int test__thread_mg_share(void) leader && t1 && t2 && t3 && other); mg = leader->mg; -TEST_ASSERT_VAL ("wrong refcnt" , mg->refcnt ==4 ); +TEST_ASSERT_VAL ("wrong refcnt" , mg->refcnt ==5 );/* test the map groups pointer is shared */ TEST_ASSERT_VAL ("map groups don't match" , mg == t1->mg);
@@ -59,7 +62,7 @@ int test__thread_mg_share(void)TEST_ASSERT_VAL ("failed to find other leader" , other_leader); other_mg = other->mg; -TEST_ASSERT_VAL ("wrong refcnt" , other_mg->refcnt ==2 ); +TEST_ASSERT_VAL ("wrong refcnt" , other_mg->refcnt ==3 );TEST_ASSERT_VAL ("map groups don't match" , other_mg == other_leader->mg);
tools/perf/util/event.c +2 -0
@@ -750,6 +750,8 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,return ; } +BUG_ON (mg ==NULL ); +if (cpumode ==PERF_RECORD_MISC_KERNEL && perf_host) { al->level ='k' ; mg = &machine->kmaps;
tools/perf/util/machine.c +1 -1
@@ -330,7 +330,7 @@ static void machine__update_thread_pid(struct machine *machine, goto out_err;if (!leader->mg) - leader->mg =map_groups__new (machine); +thread__set_map_groups (leader,map_groups__new (machine),0 );if (!leader->mg) goto out_err;
tools/perf/util/map.c +1 -0
@@ -422,6 +422,7 @@ void map_groups__init(struct map_groups *mg, struct machine *machine) } mg->machine = machine; mg->refcnt =1 ; + mg->timestamp =0 ; }static void maps__delete (struct rb_root *maps)
tools/perf/util/map.h +2 -0
@@ -61,7 +61,9 @@ struct map_groups {struct rb_root maps[MAP__NR_TYPES ];struct list_head removed_maps[MAP__NR_TYPES ];struct machine *machine; +u64 timestamp;int refcnt; +struct list_head list; };struct map_groups *map_groups__new (struct machine *machine);
tools/perf/util/thread.c +79 -1
116 lines not on this page · raw message
tools/perf/util/thread.h +3 -0
@@ -14,6 +14,7 @@ struct thread {struct rb_node rb_node;struct list_head node;struct map_groups *mg; +struct list_head mg_list;pid_t pid_;/* Not all tools update this */ pid_t tid;pid_t ppid;
@@ -56,6 +57,8 @@ struct comm *thread__exec_comm(const struct thread *thread);struct comm *thread__comm_time (const struct thread *thread,u64 timestamp);const char *thread__comm_str (const struct thread *thread);const char *thread__comm_time_str (const struct thread *thread,u64 timestamp); +struct map_groups *thread__get_map_groups (struct thread *thread,u64 timestamp); +int thread__set_map_groups (struct thread *thread,struct map_groups *mg,u64 timestamp);void thread__insert_map (struct thread *thread,struct map *map);int thread__fork (struct thread *thread,struct thread *parent,u64 timestamp);size_t thread__fprintf (struct thread *thread,FILE *fp);
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 18/37 perf tools: Remove thread when map groups initialization failed
In reply to Namhyung KimOtherwise it'll break the machine->threads tree.
--- tools/perf/util/machine.c | 1 + 1 file changed, 1 insertion(+)
tools/perf/util/machine.c +1 -0
@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,* leader and that would screwed the rb tree. */ if (thread__init_map_groups (th, machine)) { +rb_erase (&th->rb_node, &machine->threads);thread__delete (th);return NULL ; }
Signature · 1 line
2.1.3
Re: [PATCH 18/37] perf tools: Remove thread when map groups initialization failed
Quoted 13 lines · Namhyung Kim
Otherwise it'll break the
machine->threadstree.Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/machine.c | 1 + 1 file changed, 1 insertion(+)tools/perf/util/machine.c +0
-0
tools/perf/util/machine.c static struct thread *__machine__findnew_thread(struct machine *machine, · line 417
Quoted 7 lines
tools/perf/util/machine.c +1
-0@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,* leader and that would screwed the rb tree. */ if (thread__init_map_groups (th, machine)) { +rb_erase (&th->rb_node, &machine->threads);thread__delete (th);return NULL ; }
Can you move the thread__init_map_groups() before the thread is added to the rbtree? If no, you need to delay setting 'machine->last_match = th' otherwise it references a deleted thread.
David
Namhyung Kim
maintainerRe: [PATCH 18/37] perf tools: Remove thread when map groups initialization failed
Quoted 17 lines · David Ahern
Quoted 16 lines · Namhyung Kim
Otherwise it'll break the
machine->threadstree.Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/util/machine.c | 1 + 1 file changed, 1 insertion(+)tools/perf/util/machine.c +1
-0@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,* leader and that would screwed the rb tree. */ if (thread__init_map_groups (th, machine)) { +rb_erase (&th->rb_node, &machine->threads);
Quoted 8 lines
tools/perf/util/machine.c static struct thread *__machine__findnew_thread(struct machine *machine, · line 417
Quoted 4 lines
tools/perf/util/machine.c +0
-0thread__delete (th);return NULL ; }Can you move the
thread__init_map_groups()before the thread is added to the rbtree? If no, you need to delay setting 'machine->last_match= th' otherwise it references a deleted thread.
You're right - I'll move the setting after the thread__init_map_groups().
Thanks,
Namhyung
Namhyung Kim
maintainerPATCH 19/37 perf tools: Introduce thread__find_addr_location_time() and friends
In reply to Namhyung KimThe *_time() variants are for find appropriate map (and symbol) at the given time. This is based on the fact that map_groups list is sorted by time in the previous patch.
--- tools/perf/util/event.c | 59 ++++++++++++++++++++++++++++++++++---- tools/perf/util/machine.c | 51 ++++++++++++++++++-------------- tools/perf/util/thread.c | 21 ++++++++++++++ tools/perf/util/thread.h | 10 +++++++ tools/perf/util/unwind-libdw.c | 11 +++---- tools/perf/util/unwind-libunwind.c | 18 ++++++------ 6 files changed, 129 insertions(+), 41 deletions(-)
tools/perf/util/event.c +53 -6
@@ -731,16 +731,14 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,return machine__process_event (machine, event, sample); } -void thread__find_addr_map (struct thread *thread,u8 cpumode, -enum map_type type,u64 addr, -struct addr_location *al) +static void map_groups__find_addr_map (struct map_groups *mg,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al) { -struct map_groups *mg = thread->mg;struct machine *machine = mg->machine;bool load_map = false; al->machine = machine; - al->thread = thread; al->addr = addr; al->cpumode = cpumode; al->filtered =0 ;
@@ -809,6 +807,35 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode, } } +void thread__find_addr_map (struct thread *thread,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al) +{ + al->thread = thread; +map_groups__find_addr_map (thread->mg, cpumode, type, addr, al); +} + +void thread__find_addr_map_time (struct thread *thread,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al,u64 timestamp) +{ +struct map_groups *mg; +struct thread *leader; + +if (thread->tid == thread->pid_) + leader = thread; +else + leader =machine__find_thread (thread->mg->machine, + thread->pid_, thread->pid_); + +BUG_ON (leader ==NULL ); + + mg =thread__get_map_groups (leader, timestamp); + + al->thread = thread; +map_groups__find_addr_map (mg, cpumode, type, addr, al); +} +void thread__find_addr_location (struct thread *thread,u8 cpumode,enum map_type type,u64 addr,struct addr_location *al)
@@ -821,6 +848,21 @@ void thread__find_addr_location(struct thread *thread, al->sym =NULL ; } +void thread__find_addr_location_time (struct thread *thread,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al,u64 timestamp) +{ +struct map_groups *mg; + + mg =thread__get_map_groups (thread, timestamp); +map_groups__find_addr_map (mg, cpumode, type, addr, al); +if (al->map !=NULL ) + al->sym =map__find_symbol (al->map, al->addr, + mg->machine->symbol_filter); +else + al->sym =NULL ; +} + int perf_event__preprocess_sample(const union perf_event *event, struct machine *machine, struct addr_location *al,
@@ -845,7 +887,12 @@ int perf_event__preprocess_sample(const union perf_event *event, machine->vmlinux_maps[MAP__FUNCTION] == NULL)machine__create_kernel_maps (machine); -thread__find_addr_map (thread, cpumode,MAP__FUNCTION , sample->ip, al); +/* +* sample->time is -1ULL if !PERF_SAMPLE_TIME which ends up +* with using most recent map_groups (same as default behavior). +*/ +thread__find_addr_map_time (thread, cpumode,MAP__FUNCTION , sample->ip, + al, sample->time);dump_printf (" ...... dso: %s\n" , al->map ? al->map->dso->long_name : al->level =='H' ?"[hypervisor]" :"<not found>" );
tools/perf/util/machine.c +30 -21
131 lines not on this page · raw message
tools/perf/util/thread.c +21 -0
@@ -332,3 +332,24 @@ void thread__find_cpumode_addr_location(struct thread *thread,break ; } } + +void thread__find_cpumode_addr_location_time (struct thread *thread, +enum map_type type,u64 addr, +struct addr_location *al, +u64 timestamp) +{ +size_t i; +const u8 const cpumodes[] = { +PERF_RECORD_MISC_USER , +PERF_RECORD_MISC_KERNEL , +PERF_RECORD_MISC_GUEST_USER , +PERF_RECORD_MISC_GUEST_KERNEL + }; + +for (i =0 ; i <ARRAY_SIZE (cpumodes); i++) { +thread__find_addr_location_time (thread, cpumodes[i], type, + addr, al, timestamp); +if (al->map) +break ; + } +}
tools/perf/util/thread.h +10 -0
@@ -66,14 +66,24 @@ size_t thread__fprintf(struct thread *thread, FILE *fp);void thread__find_addr_map (struct thread *thread,u8 cpumode,enum map_type type,u64 addr,struct addr_location *al); +void thread__find_addr_map_time (struct thread *thread,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al,u64 timestamp);void thread__find_addr_location (struct thread *thread,u8 cpumode,enum map_type type,u64 addr,struct addr_location *al); +void thread__find_addr_location_time (struct thread *thread,u8 cpumode, +enum map_type type,u64 addr, +struct addr_location *al,u64 timestamp);void thread__find_cpumode_addr_location (struct thread *thread,enum map_type type,u64 addr,struct addr_location *al); +void thread__find_cpumode_addr_location_time (struct thread *thread, +enum map_type type,u64 addr, +struct addr_location *al, +u64 timestamp);static inline void *thread__priv (struct thread *thread) {
tools/perf/util/unwind-libdw.c +6 -5
@@ -26,9 +26,10 @@ static int __report_module(struct addr_location *al, u64 ip,Dwfl_Module *mod;struct dso *dso =NULL ; -thread__find_addr_location (ui->thread, -PERF_RECORD_MISC_USER , -MAP__FUNCTION , ip, al); +thread__find_addr_location_time (ui->thread, +PERF_RECORD_MISC_USER , +MAP__FUNCTION , ip, al, + ui->sample->time);if (al->map) dso = al->map->dso;
@@ -89,8 +90,8 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,struct addr_location al;ssize_t size; -thread__find_addr_map (ui->thread,PERF_RECORD_MISC_USER , -MAP__FUNCTION , addr, &al); +thread__find_addr_map_time (ui->thread,PERF_RECORD_MISC_USER , +MAP__FUNCTION , addr, &al, ui->sample->time);if (!al.map) {pr_debug ("unwind: no map for %lx\n" , (unsigned long )addr);return -1 ;
tools/perf/util/unwind-libunwind.c +9 -9
@@ -284,8 +284,8 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui) {struct addr_location al; -thread__find_addr_map (ui->thread,PERF_RECORD_MISC_USER , -MAP__FUNCTION , ip, &al); +thread__find_addr_map_time (ui->thread,PERF_RECORD_MISC_USER , +MAP__FUNCTION , ip, &al, ui->sample->time);return al.map; }
@@ -374,8 +374,8 @@ static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,struct addr_location al;ssize_t size; -thread__find_addr_map (ui->thread,PERF_RECORD_MISC_USER , -MAP__FUNCTION , addr, &al); +thread__find_addr_map_time (ui->thread,PERF_RECORD_MISC_USER , +MAP__FUNCTION , addr, &al, ui->sample->time);if (!al.map) {pr_debug ("unwind: no map for %lx\n" , (unsigned long )addr);return -1 ;
@@ -476,14 +476,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,pr_debug ("unwind: put_unwind_info called\n" ); } -static int entry (u64 ip,struct thread *thread, +static int entry (u64 ip,struct thread *thread,u64 timestamp,unwind_entry_cb_t cb,void *arg) {struct unwind_entry e;struct addr_location al; -thread__find_addr_location (thread,PERF_RECORD_MISC_USER , -MAP__FUNCTION , ip, &al); +thread__find_addr_location_time (thread,PERF_RECORD_MISC_USER , +MAP__FUNCTION , ip, &al, timestamp); e.ip = ip; e.map = al.map;
@@ -585,7 +585,7 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,unw_word_t ip;unw_get_reg (&c,UNW_REG_IP , &ip); - ret = ip ?entry (ip, ui->thread, cb, arg) :0 ; + ret = ip ?entry (ip, ui->thread, ui->sample->time, cb, arg) :0 ; }return ret;
@@ -610,7 +610,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,if (ret)return ret; - ret =entry (ip, thread, cb, arg); + ret =entry (ip, thread, data->time, cb, arg);if (ret)return -ENOMEM ;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 20/37 perf tools: Add a test case for timed map groups handling
In reply to Namhyung KimA test case for verifying thread->mg and ->mg_list handling during time change and new thread__find_addr_map_time() and friends.
--- tools/perf/Makefile.perf | 1 + tools/perf/tests/builtin-test.c | 4 ++ tools/perf/tests/tests.h | 1 + tools/perf/tests/thread-mg-time.c | 88 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 94 insertions(+) create mode 100644 tools/perf/tests/thread-mg-time.c
tools/perf/Makefile.perf +1 -0
@@ -449,6 +449,7 @@ LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o LIB_OBJS += $(OUTPUT)tests/switch-tracking.o LIB_OBJS += $(OUTPUT)tests/thread-comm.o LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o +LIB_OBJS += $(OUTPUT)tests/thread-mg-time.o BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
@@ -175,6 +175,10 @@ static struct test { .func = test__thread_lookup_time, }, { + .desc = "Test thread map group handling with time", + .func = test__thread_mg_time, + }, + { .func = NULL, }, };
tools/perf/tests/tests.h +1 -0
@@ -53,6 +53,7 @@ int test__fdarray__filter(void);int test__fdarray__add (void );int test__thread_comm (void );int test__thread_lookup_time (void ); +int test__thread_mg_time (void );#if defined(__x86_64__) || defined(__i386__) || defined(__arm__)#ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-mg-time.c new file +88 -0
88 lines not on this page · raw message
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 21/37 perf tools: Protect dso symbol loading using a mutex
In reply to Namhyung KimWhen multi-thread support for perf report is enabled, it's possible to access a dso concurrently. Add a new pthread_mutex to protect it from concurrent dso__load().
--- tools/perf/util/dso.c | 2 ++ tools/perf/util/dso.h | 1 + tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++---------- 3 files changed, 27 insertions(+), 10 deletions(-)
tools/perf/util/dso.c +2 -0
@@ -888,6 +888,7 @@ struct dso *dso__new(const char *name)RB_CLEAR_NODE (&dso->rb_node);INIT_LIST_HEAD (&dso->node);INIT_LIST_HEAD (&dso->data.open_entry); +pthread_mutex_init (&dso->lock,NULL ); }return dso;
@@ -917,6 +918,7 @@ void dso__delete(struct dso *dso)dso_cache__free (&dso->data.cache);dso__free_a2l (dso);zfree (&dso->symsrc_filename); +pthread_mutex_destroy (&dso->lock);free (dso); }
tools/perf/util/dso.h +1 -0
@@ -102,6 +102,7 @@ struct dsos { };struct dso { +pthread_mutex_t lock;struct list_head node;struct rb_node rb_node;/* rbtree node sorted by long name */ struct rb_root symbols[MAP__NR_TYPES ];
tools/perf/util/symbol.c +24 -10
@@ -1332,12 +1332,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)struct symsrc *syms_ss =NULL , *runtime_ss =NULL ;bool kmod; -dso__set_loaded (dso, map->type); +pthread_mutex_lock (&dso->lock); + +/* check again under the dso->lock */ +if (dso__loaded (dso, map->type)) { + ret =1 ; + goto out; + } + +if (dso->kernel) { +if (dso->kernel ==DSO_TYPE_KERNEL ) + ret =dso__load_kernel_sym (dso, map, filter); +else if (dso->kernel ==DSO_TYPE_GUEST_KERNEL ) + ret =dso__load_guest_kernel_sym (dso, map, filter); -if (dso->kernel ==DSO_TYPE_KERNEL ) -return dso__load_kernel_sym (dso, map, filter); -else if (dso->kernel ==DSO_TYPE_GUEST_KERNEL ) -return dso__load_guest_kernel_sym (dso, map, filter); + goto out; + }if (map->groups && map->groups->machine) machine = map->groups->machine;
@@ -1350,18 +1360,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)struct stat st;if (lstat (dso->name, &st) <0 ) -return -1 ; + goto out;if (st.st_uid && (st.st_uid !=geteuid ())) {pr_warning ("File %s not owned by current user or root, " "ignoring it.\n" , dso->name); -return -1 ; + goto out; } ret =dso__load_perf_map (dso, map, filter); dso->symtab_type = ret >0 ?DSO_BINARY_TYPE__JAVA_JIT :DSO_BINARY_TYPE__NOT_FOUND ; -return ret; + goto out; }if (machine)
@@ -1369,7 +1379,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter) name =malloc (PATH_MAX );if (!name) -return -1 ; + goto out; kmod = dso->symtab_type ==DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE || dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1450,7 +1460,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter) out_free:free (name);if (ret <0 &&strstr (dso->name," (deleted)" ) !=NULL ) -return 0 ; + ret =0 ; +out: +dso__set_loaded (dso, map->type); +pthread_mutex_unlock (&dso->lock); +return ret; }
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 22/37 perf tools: Protect dso cache tree using dso->lock
In reply to Namhyung KimThe dso cache is accessed during dwarf callchain unwind and it might be processed concurrently when multi-thread report is enabled. Protect it under dso->lock.
--- tools/perf/util/dso.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-)
tools/perf/util/dso.c +32 -9
@@ -443,10 +443,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by) }static void -dso_cache__free (struct rb_root *root) +dso_cache__free (struct dso *dso) { +struct rb_root *root = &dso->data.cache;struct rb_node *next =rb_first (root); +pthread_mutex_lock (&dso->lock);while (next) {struct dso_cache *cache;
@@ -455,14 +457,17 @@ dso_cache__free(struct rb_root *root)rb_erase (&cache->rb_node, root);free (cache); } +pthread_mutex_unlock (&dso->lock); } -static struct dso_cache *dso_cache__find (const struct rb_root *root,u64 offset) +static struct dso_cache *dso_cache__find (struct dso *dso,u64 offset) { +const struct rb_root *root = &dso->data.cache;struct rb_node *const *p = &root->rb_node;const struct rb_node *parent =NULL ;struct dso_cache *cache; +pthread_mutex_lock (&dso->lock);while (*p !=NULL ) {u64 end;
@@ -475,19 +480,24 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)else if (offset >= end) p = &(*p)->rb_right;else -return cache; + goto out; } -return NULL ; + cache =NULL ; +out: +pthread_mutex_unlock (&dso->lock); +return cache; } -static void -dso_cache__insert (struct rb_root *root,struct dso_cache *new) +static struct dso_cache * +dso_cache__insert (struct dso *dso,struct dso_cache *new) { +struct rb_root *root = &dso->data.cache;struct rb_node **p = &root->rb_node;struct rb_node *parent =NULL ;struct dso_cache *cache;u64 offset = new->offset; +pthread_mutex_lock (&dso->lock);while (*p !=NULL ) {u64 end;
@@ -499,10 +509,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new) p = &(*p)->rb_left; else if (offset >= end) p = &(*p)->rb_right; + else + goto out; }rb_link_node (&new->rb_node, parent, p);rb_insert_color (&new->rb_node, root); + + cache =NULL ; +out: +pthread_mutex_unlock (&dso->lock); +return cache; } static ssize_t
@@ -520,6 +537,7 @@ static ssize_t dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size) { struct dso_cache *cache; +struct dso_cache *old;ssize_t ret; do {
@@ -543,7 +561,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size) cache->offset = cache_offset; cache->size = ret; -dso_cache__insert (&dso->data.cache, cache); + old =dso_cache__insert (dso, cache); +if (old) { +/* we lose the race */ +free (cache); + cache = old; + } ret =dso_cache__memcpy (cache, offset, data, size);
@@ -560,7 +583,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset, {struct dso_cache *cache; - cache =dso_cache__find (&dso->data.cache, offset); + cache =dso_cache__find (dso, offset);if (cache)return dso_cache__memcpy (cache, offset, data, size);else
@@ -915,7 +938,7 @@ void dso__delete(struct dso *dso) }dso__data_close (dso); -dso_cache__free (&dso->data.cache); +dso_cache__free (dso);dso__free_a2l (dso);zfree (&dso->symsrc_filename);pthread_mutex_destroy (&dso->lock);
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 23/37 perf tools: Protect dso cache fd with a mutex
In reply to Namhyung KimWhen dso cache is accessed in multi-thread environment, it's possible to close other dso->data.fd during operation due to open file limit. Protect the file descriptors using a separate mutex.
--- tools/perf/tests/dso-data.c | 5 ++ tools/perf/util/dso.c | 136 +++++++++++++++++++++++++++++--------------- 2 files changed, 94 insertions(+), 47 deletions(-)
tools/perf/tests/dso-data.c +5 -0
@@ -111,6 +111,9 @@ int test__dso_data(void)memset (&machine,0 ,sizeof (machine)); dso =dso__new ((const char *)file); +TEST_ASSERT_VAL ("failed to get dso" , dso); + + dso->binary_type =DSO_BINARY_TYPE__SYSTEM_PATH_DSO ;/* Basic 10 bytes tests. */ for (i =0 ; i <ARRAY_SIZE (offsets); i++) {
@@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size) dsos[i] =dso__new (file);TEST_ASSERT_VAL ("failed to get dso" , dsos[i]); + + dsos[i]->binary_type =DSO_BINARY_TYPE__SYSTEM_PATH_DSO ; }return 0 ;
tools/perf/util/dso.c +89 -47
249 lines not on this page · raw message
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 24/37 perf session: Pass struct events stats to event processing functions
In reply to Namhyung KimPass stats structure so that it can point separate object when used in multi-thread environment.
--- tools/perf/util/ordered-events.c | 4 +- tools/perf/util/session.c | 87 +++++++++++++++++++++++----------------- tools/perf/util/session.h | 1 + 3 files changed, 54 insertions(+), 38 deletions(-)
tools/perf/util/ordered-events.c +3 -1
@@ -183,7 +183,9 @@ static int __ordered_events__flush(struct perf_session *s,if (ret)pr_err ("Can't parse sample, err = %d\n" , ret);else { - ret =perf_session__deliver_event (s, iter->event, &sample, tool, + ret =perf_session__deliver_event (s, &s->stats, + iter->event, + &sample, tool, iter->file_offset);if (ret)return ret;
tools/perf/util/session.c +50 -37
263 lines not on this page · raw message
tools/perf/util/session.h +1 -0
@@ -59,6 +59,7 @@ int perf_session_queue_event(struct perf_session *s, union perf_event *event,void perf_tool__fill_defaults (struct perf_tool *tool);int perf_session__deliver_event (struct perf_session *session, +struct events_stats *stats,union perf_event *event,struct perf_sample *sample,struct perf_tool *tool,u64 file_offset);
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 25/37 perf hists: Pass hists struct to hist_entry_iter functions
In reply to Namhyung KimThis is a preparation for perf report multi-thread support. When multi-thread is enable, each thread will have its own hists during the sample processing.
--- tools/perf/builtin-report.c | 4 ++-- tools/perf/builtin-top.c | 4 ++-- tools/perf/tests/hists_cumulate.c | 4 ++-- tools/perf/tests/hists_filter.c | 3 ++- tools/perf/tests/hists_output.c | 4 ++-- tools/perf/util/hist.c | 26 +++++++++++--------------- tools/perf/util/hist.h | 6 ++++-- 7 files changed, 25 insertions(+), 26 deletions(-)
tools/perf/builtin-report.c +2 -2
@@ -166,8 +166,8 @@ static int process_sample_event(struct perf_tool *tool,if (al.map !=NULL ) al.map->dso->hit =1 ; - ret =hist_entry_iter__add (&iter, &al, evsel, sample, rep->max_stack, - rep); + ret =hist_entry_iter__add (&iter,evsel__hists (evsel), evsel, &al, + sample, rep->max_stack, rep);if (ret <0 )pr_debug ("problem adding hist entry, skipping event\n" );
tools/perf/builtin-top.c +2 -2
@@ -784,8 +784,8 @@ static void perf_event__process_sample(struct perf_tool *tool,pthread_mutex_lock (&hists->lock); - err =hist_entry_iter__add (&iter, &al, evsel, sample, - top->max_stack, top); + err =hist_entry_iter__add (&iter,evsel__hists (evsel), evsel, + &al, sample, top->max_stack, top);if (err <0 )pr_err ("Problem incrementing symbol period, skipping event\n" );
tools/perf/tests/hists_cumulate.c +2 -2
@@ -104,8 +104,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine) &sample) < 0) goto out; -if (hist_entry_iter__add (&iter, &al, evsel, &sample, -PERF_MAX_STACK_DEPTH ,NULL ) <0 ) +if (hist_entry_iter__add (&iter,evsel__hists (evsel), evsel, &al, + &sample,PERF_MAX_STACK_DEPTH ,NULL ) <0 ) goto out; fake_samples[i].thread = al.thread;
tools/perf/tests/hists_filter.c +2 -1
@@ -81,7 +81,8 @@ static int add_hist_entries(struct perf_evlist *evlist, &sample) < 0) goto out; -if (hist_entry_iter__add (&iter, &al, evsel, &sample, +if (hist_entry_iter__add (&iter,evsel__hists (evsel), + evsel, &al, &sample,PERF_MAX_STACK_DEPTH ,NULL ) <0 ) goto out;
tools/perf/tests/hists_output.c +2 -2
@@ -70,8 +70,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine) &sample) < 0) goto out; -if (hist_entry_iter__add (&iter, &al, evsel, &sample, -PERF_MAX_STACK_DEPTH ,NULL ) <0 ) +if (hist_entry_iter__add (&iter,evsel__hists (evsel), evsel, &al, + &sample,PERF_MAX_STACK_DEPTH ,NULL ) <0 ) goto out; fake_samples[i].thread = al.thread;
tools/perf/util/hist.c +11 -15
@@ -514,7 +514,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *alu64 cost;struct mem_info *mi = iter->priv;struct perf_sample *sample = iter->sample; -struct hists *hists =evsel__hists (iter->evsel); +struct hists *hists = iter->hists;struct hist_entry *he;if (mi ==NULL )
@@ -544,8 +544,7 @@ static int iter_finish_mem_entry(struct hist_entry_iter *iter, struct addr_location *al __maybe_unused) { - struct perf_evsel *evsel = iter->evsel; -struct hists *hists =evsel__hists (evsel); + struct hists *hists = iter->hists;struct hist_entry *he = iter->he;int err = -EINVAL ;
@@ -617,8 +616,7 @@ static int iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *al) { struct branch_info *bi; -struct perf_evsel *evsel = iter->evsel; -struct hists *hists =evsel__hists (evsel); +struct hists *hists = iter->hists;struct hist_entry *he =NULL ;int i = iter->curr;int err =0 ;
@@ -665,11 +663,10 @@ iter_prepare_normal_entry(struct hist_entry_iter *iter __maybe_unused,static int iter_add_single_normal_entry (struct hist_entry_iter *iter,struct addr_location *al) { -struct perf_evsel *evsel = iter->evsel;struct perf_sample *sample = iter->sample;struct hist_entry *he; - he =__hists__add_entry (evsel__hists (evsel), al, iter->parent,NULL ,NULL , + he =__hists__add_entry (iter->hists, al, iter->parent,NULL ,NULL , sample->period, sample->weight, sample->transaction, sample->time, true);if (he ==NULL )
@@ -684,7 +681,6 @@ iter_finish_normal_entry(struct hist_entry_iter *iter, struct addr_location *al __maybe_unused) {struct hist_entry *he = iter->he; -struct perf_evsel *evsel = iter->evsel;struct perf_sample *sample = iter->sample;if (he ==NULL )
@@ -692,7 +688,7 @@ iter_finish_normal_entry(struct hist_entry_iter *iter, iter->he =NULL ; -hists__inc_nr_samples (evsel__hists (evsel), he->filtered); +hists__inc_nr_samples (iter->hists, he->filtered);return hist_entry__append_callchain (he, sample); }
@@ -724,8 +720,7 @@ static int iter_add_single_cumulative_entry(struct hist_entry_iter *iter, struct addr_location *al) { - struct perf_evsel *evsel = iter->evsel; -struct hists *hists =evsel__hists (evsel); + struct hists *hists = iter->hists;struct perf_sample *sample = iter->sample;struct hist_entry **he_cache = iter->priv;struct hist_entry *he;
@@ -770,7 +765,6 @@ static int iter_add_next_cumulative_entry(struct hist_entry_iter *iter, struct addr_location *al) { - struct perf_evsel *evsel = iter->evsel; struct perf_sample *sample = iter->sample;struct hist_entry **he_cache = iter->priv;struct hist_entry *he;
@@ -804,7 +798,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter, } } - he =__hists__add_entry (evsel__hists (evsel), al, iter->parent,NULL ,NULL , + he =__hists__add_entry (iter->hists, al, iter->parent,NULL ,NULL , sample->period, sample->weight, sample->transaction, sample->time, false);if (he ==NULL )
@@ -860,8 +854,9 @@ const struct hist_iter_ops hist_iter_cumulative = { .finish_entry = iter_finish_cumulative_entry, }; -int hist_entry_iter__add (struct hist_entry_iter *iter,struct addr_location *al, -struct perf_evsel *evsel,struct perf_sample *sample, +int hist_entry_iter__add (struct hist_entry_iter *iter,struct hists *hists, +struct perf_evsel *evsel,struct addr_location *al, +struct perf_sample *sample,int max_stack_depth,void *arg) {int err, err2;
@@ -871,6 +866,7 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,if (err)return err; + iter->hists = hists; iter->evsel = evsel; iter->sample = sample;
tools/perf/util/hist.h +4 -2
@@ -86,6 +86,7 @@ struct hist_entry_iter {bool hide_unresolved; +struct hists *hists;struct perf_evsel *evsel;struct perf_sample *sample;struct hist_entry *he;
@@ -110,8 +111,9 @@ struct hist_entry *__hists__add_entry(struct hists *hists, struct mem_info *mi, u64 period, u64 weight, u64 transaction, u64 timestamp, bool sample_self); -int hist_entry_iter__add (struct hist_entry_iter *iter,struct addr_location *al, -struct perf_evsel *evsel,struct perf_sample *sample, +int hist_entry_iter__add (struct hist_entry_iter *iter,struct hists *hists, +struct perf_evsel *evsel,struct addr_location *al, +struct perf_sample *sample,int max_stack_depth,void *arg);int64_t hist_entry__cmp (struct hist_entry *left,struct hist_entry *right);
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 26/37 perf tools: Move BUILD_ID_SIZE definition to perf.h
In reply to Namhyung KimThe util/event.h includes util/build-id.h only for BUILD_ID_SIZE. This is a problem when I include util/event.h from util/tool.h which is also included by util/build-id.h since it now makes a circular dependency resulting in incomplete type error.
--- tools/perf/perf.h | 1 + tools/perf/util/build-id.h | 2 -- tools/perf/util/dso.h | 1 + tools/perf/util/event.h | 1 - 4 files changed, 2 insertions(+), 3 deletions(-)
tools/perf/perf.h +1 -0
@@ -30,6 +30,7 @@ static inline unsigned long long rdclock(void) }#define MAX_NR_CPUS 256 +#define BUILD_ID_SIZE 20extern const char *input_name;extern bool perf_host, perf_guest;
tools/perf/util/build-id.h +0 -2
@@ -1,8 +1,6 @@#ifndef PERF_BUILD_ID_H_ #define PERF_BUILD_ID_H_ 1 -#define BUILD_ID_SIZE 20 -#include "tool.h" #include <linux/types.h>
tools/perf/util/dso.h +1 -0
@@ -7,6 +7,7 @@#include <linux/types.h> #include <linux/bitops.h> #include "map.h" +#include "perf.h" #include "build-id.h" enum dso_binary_type {
tools/perf/util/event.h +0 -1
@@ -6,7 +6,6 @@#include "../perf.h" #include "map.h" -#include "build-id.h" #include "perf_regs.h" struct mmap_event {
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 27/37 perf report: Parallelize perf report using multi-thread
In reply to Namhyung KimIntroduce perf_session__process_events_mt() to enable multi-thread sample processing. It allocates a struct perf_tool_mt and fills needed info in it. The init and fini callbacks are provided so that we can pass additional data structure if needed.
The session and hists event stats are counted for each thread and summed after finishing the processing. Similarly hist entries are added to per-thread hists first and then move to the original hists using hists__multi_resort(). This function reuses hists__collapse_ resort() code so makes sort__need_collapse force to true and skips the collapsing function.
Note that most of preprocessing stage is already done by processing meta events in dummy tracking evsel first. We can find corresponding thread and map based on the sample time and symbol loading and dso cache access is protected by pthread mutex.
--- tools/perf/builtin-report.c | 90 +++++++++++++++++++++------ tools/perf/util/hist.c | 75 +++++++++++++++++++---- tools/perf/util/hist.h | 3 + tools/perf/util/session.c | 146 ++++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/session.h | 4 ++ tools/perf/util/tool.h | 14 +++++ 6 files changed, 302 insertions(+), 30 deletions(-)
tools/perf/builtin-report.c +73 -17
@@ -128,18 +128,16 @@ static int hist_iter__report_callback(struct hist_entry_iter *iter,return err; } -static int process_sample_event (struct perf_tool *tool, -union perf_event *event, -struct perf_sample *sample, -struct perf_evsel *evsel, -struct machine *machine) +static int __process_sample_event (struct perf_tool *tool__maybe_unused , +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel, +struct machine *machine, +struct hist_entry_iter *iter, +struct hists *hists, +struct report *rep) { - struct report *rep = container_of(tool, struct report, tool);struct addr_location al; -struct hist_entry_iter iter = { - .hide_unresolved = rep->hide_unresolved, - .add_entry_cb = hist_iter__report_callback, - };int ret;if (perf_event__preprocess_sample (event, machine, &al, sample) <0 ) {
@@ -155,18 +153,18 @@ static int process_sample_event(struct perf_tool *tool,return 0 ;if (sort__mode ==SORT_MODE__BRANCH ) - iter.ops = &hist_iter_branch; + iter->ops = &hist_iter_branch;else if (rep->mem_mode) - iter.ops = &hist_iter_mem; + iter->ops = &hist_iter_mem;else if (symbol_conf.cumulate_callchain) - iter.ops = &hist_iter_cumulative; + iter->ops = &hist_iter_cumulative;else - iter.ops = &hist_iter_normal; + iter->ops = &hist_iter_normal;if (al.map !=NULL ) al.map->dso->hit =1 ; - ret =hist_entry_iter__add (&iter,evsel__hists (evsel), evsel, &al, + ret =hist_entry_iter__add (iter, hists, evsel, &al, sample, rep->max_stack, rep);if (ret <0 )pr_debug ("problem adding hist entry, skipping event\n" );
@@ -174,6 +172,52 @@ static int process_sample_event(struct perf_tool *tool,return ret; } +static int process_sample_event (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel, +struct machine *machine) +{ + struct report *rep = container_of(tool, struct report, tool); +struct hist_entry_iter iter = { + .hide_unresolved = rep->hide_unresolved, + .add_entry_cb = hist_iter__report_callback, + }; + +return __process_sample_event (tool, event, sample, evsel, machine, + &iter,evsel__hists (evsel), rep); +} + +static int process_sample_event_multi (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel, +struct machine *machine) +{ + struct perf_tool_mt *mt = container_of(tool, struct perf_tool_mt, tool); +struct report *rep = mt->priv; +struct hist_entry_iter iter = { + .hide_unresolved = rep->hide_unresolved, + }; + +return __process_sample_event (tool, event, sample, evsel, machine, + &iter, &mt->hists[evsel->idx], rep); +} + +static int multi_report_init (struct perf_tool_mt *mt,void *arg) +{ +struct report *rep = arg; + + mt->priv = rep; +return 0 ; +} + +static int multi_report_fini (struct perf_tool_mt *mt,void *arg__maybe_unused ) +{ + mt->priv =NULL ; +return 0 ; +} + static int process_read_event(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample __maybe_unused,
@@ -483,7 +527,14 @@ static int __cmd_report(struct report *rep)if (ret)return ret; - ret =perf_session__process_events (session, &rep->tool); +if (file->is_multi) { + rep->tool.sample = process_sample_event_multi; + ret =perf_session__process_events_mt (session, &rep->tool, + multi_report_init, + multi_report_fini, rep); + }else { + ret =perf_session__process_events (session, &rep->tool); + }if (ret)return ret;
@@ -506,7 +557,12 @@ static int __cmd_report(struct report *rep) } } -report__collapse_hists (rep); +/* +* For multi-file report, it already calls hists__multi_resort() +* so no need to collapse here. +*/ +if (!file->is_multi) +report__collapse_hists (rep);if (session_done ())return 0 ;
tools/perf/util/hist.c +62 -13
122 lines not on this page · raw message
tools/perf/util/hist.h +3 -0
@@ -124,6 +124,7 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,void hist_entry__free (struct hist_entry *);void hists__output_resort (struct hists *hists,struct ui_progress *prog); +void hists__multi_resort (struct hists *dst,struct hists *src);void hists__collapse_resort (struct hists *hists,struct ui_progress *prog);void hists__decay_entries (struct hists *hists,bool zap_user,bool zap_kernel);
@@ -136,6 +137,7 @@ void hists__inc_stats(struct hists *hists, struct hist_entry *h);void hists__inc_nr_events (struct hists *hists,u32 type);void hists__inc_nr_samples (struct hists *hists,bool filtered);void events_stats__inc (struct events_stats *stats,u32 type); +void events_stats__add (struct events_stats *dst,struct events_stats *src);size_t events_stats__fprintf (struct events_stats *stats,FILE *fp); size_t hists__fprintf(struct hists *hists, bool show_header, int max_rows,
@@ -179,6 +181,7 @@ static inline struct hists *evsel__hists(struct perf_evsel *evsel) }int hists__init (void ); +void __hists__init (struct hists *hists);struct perf_hpp {char *buf;
tools/perf/util/session.c +146 -0
152 lines not on this page · raw message
tools/perf/util/session.h +4 -0
@@ -51,6 +51,10 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,int perf_session__process_events (struct perf_session *session,struct perf_tool *tool); +int perf_session__process_events_mt (struct perf_session *session, +struct perf_tool *tool, +mt_tool_callback_t init_cb, +mt_tool_callback_t fini_cb,void *arg); int perf_session_queue_event(struct perf_session *s, union perf_event *event, struct perf_tool *tool, struct perf_sample *sample,
tools/perf/util/tool.h +14 -0
@@ -2,6 +2,7 @@#define __PERF_TOOL_H#include <stdbool.h> +#include "util/event.h" struct perf_session ;union perf_event ;
@@ -10,6 +11,7 @@ struct perf_evsel;struct perf_sample ;struct perf_tool ;struct machine ; +struct hists ; typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event, struct perf_sample *sample,
@@ -45,4 +47,16 @@ struct perf_tool { bool ordering_requires_timestamps; }; +struct perf_tool_mt { +struct perf_tool tool; +struct events_stats stats; +struct hists *hists; +struct perf_session *session; +int idx; + +void *priv; +}; + +typedef int (*mt_tool_callback_t )(struct perf_tool_mt *,void *); +#endif /* __PERF_TOOL_H */
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 28/37 perf tools: Add missing_threads rb tree
In reply to Namhyung KimSometimes it's possible to miss certain meta events like fork/exit and in this case it can fail to find such thread in the machine's rbtree. But adding a thread to the tree is dangerous since it's now executed in multi-thread environment otherwise it'll add an overhead in order to grab a lock for every search. So adds a separate missing_threads tree and protect it with a mutex. It's expected to be accessed only if a thread is not found in a normal tree.
--- tools/perf/tests/thread-lookup-time.c | 8 ++- tools/perf/util/build-id.c | 9 +++- tools/perf/util/machine.c | 91 +++++++++++++++++++++++------------ tools/perf/util/machine.h | 2 + tools/perf/util/session.c | 8 +-- tools/perf/util/thread.h | 1 + 6 files changed, 80 insertions(+), 39 deletions(-)
tools/perf/tests/thread-lookup-time.c +7 -1
@@ -7,7 +7,9 @@static int thread__print_cb (struct thread *th,void *arg__maybe_unused ) {printf ("thread: %d, start time: %" PRIu64" %s\n" , - th->tid, th->start_time, th->dead ?"(dead)" :"" ); + th->tid, th->start_time, + th->dead ?"(dead)" : th->exited ?"(exited)" : + th->missing ?"(missing)" :"" );return 0 ; }
@@ -105,6 +107,8 @@ static int lookup_with_timestamp(struct machine *machine) machine__findnew_thread_time(machine, 0, 0, 70000) == t3);machine__delete_threads (machine); +machine__delete_dead_threads (machine); +machine__delete_missing_threads (machine);return 0 ; }
@@ -146,6 +150,8 @@ static int lookup_without_timestamp(struct machine *machine) machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);machine__delete_threads (machine); +machine__delete_dead_threads (machine); +machine__delete_missing_threads (machine);return 0 ; }
tools/perf/util/build-id.c +8 -1
@@ -60,7 +60,14 @@ static int perf_event__exit_del_thread(struct perf_tool *tool __maybe_unused, event->fork.ppid, event->fork.ptid);if (thread) { -rb_erase (&thread->rb_node, &machine->threads); +if (thread->dead) +rb_erase (&thread->rb_node, &machine->dead_threads); +else if (thread->missing) +rb_erase (&thread->rb_node, &machine->missing_threads); +else +rb_erase (&thread->rb_node, &machine->threads); + +list_del (&thread->node); machine->last_match =NULL ;thread__delete (thread); }
tools/perf/util/machine.c +60 -31
139 lines not on this page · raw message
tools/perf/util/machine.h +2 -0
@@ -31,6 +31,7 @@ struct machine {char *root_dir;struct rb_root threads;struct rb_root dead_threads; +struct rb_root missing_threads;struct thread *last_match;struct vdso_info *vdso_info;struct dsos user_dsos;
@@ -116,6 +117,7 @@ void machines__set_comm_exec(struct machines *machines, bool comm_exec);struct machine *machine__new_host (void );int machine__init (struct machine *machine,const char *root_dir,pid_t pid);void machine__exit (struct machine *machine); +void machine__delete_missing_threads (struct machine *machine);void machine__delete_dead_threads (struct machine *machine);void machine__delete_threads (struct machine *machine);void machine__delete (struct machine *machine);
tools/perf/util/session.c +2 -6
@@ -138,14 +138,11 @@ struct perf_session *perf_session__new(struct perf_data_file *file,return NULL ; } -static void perf_session__delete_dead_threads (struct perf_session *session) -{ -machine__delete_dead_threads (&session->machines.host); -} -static void perf_session__delete_threads (struct perf_session *session) {machine__delete_threads (&session->machines.host); +machine__delete_dead_threads (&session->machines.host); +machine__delete_missing_threads (&session->machines.host); }static void perf_session_env__delete (struct perf_session_env *env)
@@ -167,7 +164,6 @@ static void perf_session_env__delete(struct perf_session_env *env)void perf_session__delete (struct perf_session *session) {perf_session__destroy_kernel_maps (session); -perf_session__delete_dead_threads (session);perf_session__delete_threads (session);perf_session_env__delete (&session->header.env);machines__exit (&session->machines);
tools/perf/util/thread.h +1 -0
@@ -23,6 +23,7 @@ struct thread {bool comm_set;bool exited;/* if set thread has exited */ bool dead;/* thread is in dead_threads list */ +bool missing;/* thread is in missing_threads list */ struct list_head comm_list;int comm_len;u64 db_id;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 29/37 perf top: Always creates thread in the current task tree.
In reply to Namhyung KimWhen machine__findnew_thread_time() creates a new thread, it puts the thread in the missing_threads tree assuming it's rare case that missed related task/mmap events during the recorded header file.
However this is not the case of live profiling - so it should be treated differently. This patch fixes NULL thread->mg reference in case of missing threads in live profiling.
--- tools/perf/builtin-top.c | 5 ++++- tools/perf/util/event.c | 24 ++++++++++++++++++------ tools/perf/util/event.h | 5 +++++ 3 files changed, 27 insertions(+), 7 deletions(-)
tools/perf/builtin-top.c +4 -1
@@ -699,6 +699,7 @@ static void perf_event__process_sample(struct perf_tool *tool, { struct perf_top *top = container_of(tool, struct perf_top, tool);struct addr_location al; +struct thread *thread;int err;if (!machine && perf_guest) {
@@ -724,7 +725,9 @@ static void perf_event__process_sample(struct perf_tool *tool,if (event->header.misc &PERF_RECORD_MISC_EXACT_IP ) top->exact_samples++; -if (perf_event__preprocess_sample (event, machine, &al, sample) <0 ) +/* Always use current thread tree for live profiling */ + thread =machine__findnew_thread (machine, sample->pid, sample->tid); +if (__perf_event__preprocess_sample (event, machine, thread, &al, sample) <0 )return ; if (!top->kptr_restrict_warned &&
tools/perf/util/event.c +18 -6
@@ -863,14 +863,13 @@ void thread__find_addr_location_time(struct thread *thread, u8 cpumode, al->sym =NULL ; } -int perf_event__preprocess_sample (const union perf_event *event, -struct machine *machine, -struct addr_location *al, -struct perf_sample *sample) +int __perf_event__preprocess_sample (const union perf_event *event, +struct machine *machine, +struct thread *thread, +struct addr_location *al, +struct perf_sample *sample) {u8 cpumode = event->header.misc &PERF_RECORD_MISC_CPUMODE_MASK ; -struct thread *thread =machine__findnew_thread_time (machine, sample->pid, - sample->tid, sample->time);if (thread ==NULL )return -1 ;
@@ -928,6 +927,19 @@ int perf_event__preprocess_sample(const union perf_event *event,return 0 ; } +int perf_event__preprocess_sample (const union perf_event *event, +struct machine *machine, +struct addr_location *al, +struct perf_sample *sample) +{ +struct thread *thread; + + thread =machine__findnew_thread_time (machine, sample->pid, + sample->tid, sample->time); +return __perf_event__preprocess_sample (event, machine, thread, + al, sample); +} +bool is_bts_event (struct perf_event_attr *attr) { return attr->type == PERF_TYPE_HARDWARE &&
tools/perf/util/event.h +5 -0
@@ -352,6 +352,11 @@ int perf_event__process(struct perf_tool *tool,struct addr_location ; +int __perf_event__preprocess_sample (const union perf_event *event, +struct machine *machine, +struct thread *thread, +struct addr_location *al, +struct perf_sample *sample); int perf_event__preprocess_sample(const union perf_event *event, struct machine *machine, struct addr_location *al,
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 30/37 perf tools: Fix progress ui to support multi thread
In reply to Namhyung KimSplit ui_progress struct into global and local one. Each thread updates local struct without lock and only updates global one if meaningful progress is done (with lock).
To do that, pass struct ui_progress to __perf_session__process_event() and set it for the total size of multi-file storage.
--- tools/perf/util/data.c | 22 +++++++++ tools/perf/util/data.h | 3 ++ tools/perf/util/hist.c | 5 +- tools/perf/util/hist.h | 3 +- tools/perf/util/session.c | 117 ++++++++++++++++++++++++++++++++++------------ tools/perf/util/tool.h | 3 ++ 6 files changed, 121 insertions(+), 32 deletions(-)
tools/perf/util/data.c +22 -0
@@ -290,3 +290,25 @@ ssize_t perf_data_file__write_multi(struct perf_data_file *file,return writen (file->multi_fd[idx], buf, size); } + +s64 perf_data_file__multi_size (struct perf_data_file *file) +{ +int i; +s64 total_size =perf_data_file__size (file); + +if (!file->is_multi) +return total_size; + +for (i =0 ; i < file->nr_multi; i++) { +int fd =perf_data_file__multi_fd (file, i); +long size; + + size =lseek (fd,0 ,SEEK_END ); +if (size <0 ) +return (s64) -1 ; + + total_size += size; + } + +return total_size; +}
tools/perf/util/data.h +3 -0
@@ -2,6 +2,7 @@#define __PERF_DATA_H#include <stdbool.h> +#include "perf.h" enum perf_data_mode {PERF_DATA_MODE_WRITE ,
@@ -61,4 +62,6 @@ int perf_data_file__prepare_write(struct perf_data_file *file, int nr);ssize_t perf_data_file__write_multi (struct perf_data_file *file,void *buf,size_t size,int idx); +s64 perf_data_file__multi_size (struct perf_data_file *file); +#endif /* __PERF_DATA_H */
tools/perf/util/hist.c +3 -2
@@ -1065,12 +1065,13 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)__hists__collapse_resort (hists, root, prog); } -void hists__multi_resort (struct hists *dst,struct hists *src) +void hists__multi_resort (struct hists *dst,struct hists *src, +struct ui_progress *prog) {struct rb_root *root = src->entries_in; sort__need_collapse =1 ; -__hists__collapse_resort (dst, root,NULL ); +__hists__collapse_resort (dst, root, prog); }static int hist_entry__sort (struct hist_entry *a,struct hist_entry *b)
tools/perf/util/hist.h +2 -1
@@ -124,7 +124,8 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,void hist_entry__free (struct hist_entry *);void hists__output_resort (struct hists *hists,struct ui_progress *prog); -void hists__multi_resort (struct hists *dst,struct hists *src); +void hists__multi_resort (struct hists *dst,struct hists *src, +struct ui_progress *prog);void hists__collapse_resort (struct hists *hists,struct ui_progress *prog);void hists__decay_entries (struct hists *hists,bool zap_user,bool zap_kernel);
tools/perf/util/session.c +88 -29
241 lines not on this page · raw message
tools/perf/util/tool.h +3 -0
@@ -3,6 +3,7 @@#include <stdbool.h> #include "util/event.h" +#include "ui/progress.h" struct perf_session ;union perf_event ;
@@ -52,6 +53,8 @@ struct perf_tool_mt {struct events_stats stats;struct hists *hists;struct perf_session *session; +struct ui_progress prog; +struct ui_progress *global_prog;int idx;void *priv;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 31/37 perf record: Show total size of multi file data
In reply to Namhyung KimCurrently perf record shows header file size only - extends it to show total size of multi-file data storage.
--- tools/perf/builtin-record.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/builtin-record.c +2 -2
@@ -538,9 +538,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)*/ fprintf (stderr,"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64" samples) ]\n" , - (double )rec->bytes_written /1024.0 /1024.0 , + (double )(u64 )perf_data_file__multi_size (file) /1024.0 /1024.0 , file->path, - rec->bytes_written /24 ); +perf_data_file__multi_size (file) /24 ); } out_child:
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 32/37 perf report: Add --multi-thread option and config item
In reply to Namhyung KimThe --multi-thread option is to enable parallel processing so user can force serial processing even for multi-file data. It default to false but users also can changes this by setting "report.multi_thread" config option in ~/.perfconfig file.
--- tools/perf/Documentation/perf-report.txt | 3 +++ tools/perf/builtin-report.c | 18 +++++++++++++++--- 2 files changed, 18 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-report.txt +3 -0
@@ -318,6 +318,9 @@ OPTIONS --header-only:: Show only perf.data header (forces --stdio). +--multi-thread:: + Speed up report by parallelizing sample processing using multi-thread. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-annotate[1]
tools/perf/builtin-report.c +15 -3
@@ -51,6 +51,7 @@ struct report {bool mem_mode;bool header;bool header_only; +bool multi_thread;int max_stack;struct perf_read_values show_threads_values;const char *pretty_printing_style;
@@ -82,6 +83,10 @@ static int report__config(const char *var, const char *value, void *cb) rep->queue_size =perf_config_u64 (var, value);return 0 ; } +if (!strcmp (var,"report.multi-thread" )) { + rep->multi_thread =perf_config_bool (var, value); +return 0 ; + }return perf_default_config (var, value, cb); }
@@ -527,7 +532,7 @@ static int __cmd_report(struct report *rep)if (ret)return ret; -if (file->is_multi) { +if (rep->multi_thread) { rep->tool.sample = process_sample_event_multi; ret = perf_session__process_events_mt(session, &rep->tool, multi_report_init,
@@ -558,10 +563,10 @@ static int __cmd_report(struct report *rep) }/* -* For multi-file report, it already calls hists__multi_resort() +* For multi-thread report, it already calls hists__multi_resort() * so no need to collapse here. */ -if (!file->is_multi) +if (!rep->multi_thread)report__collapse_hists (rep);if (session_done ())
@@ -770,6 +775,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused) "Don't show entries under that percent", parse_percent_limit),OPT_CALLBACK (0 ,"percentage" ,NULL ,"relative|absolute" ,"how to display percentage of filtered entries" , parse_filter_percentage), +OPT_BOOLEAN (0 ,"multi-thread" , &report.multi_thread, +"Speed up sample processing using multi-thead" ),OPT_END () }; struct perf_data_file file = {
@@ -814,6 +821,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused) report.queue_size); } +if (report.multi_thread && !file.is_multi) { +pr_debug ("fallback to single thread for single data file.\n" ); + report.multi_thread = false; + } + report.session = session; has_br_stack = perf_header__has_feat(&session->header,
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 33/37 perf tools: Add front cache for dso data access
In reply to Namhyung KimThere's a high contention in dso_cache__find() due to the dso->lock when dwarf unwinding is done with libunwind. Add last accessed pointers of dso_cache to lockless lookup. It'll fallback to normal tree search when it misses the last cache.
The size 16 is arbitrary and works best for my setting.
--- tools/perf/util/dso.c | 28 ++++++++++++++++++++++++++-- tools/perf/util/dso.h | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-)
tools/perf/util/dso.c +26 -2
@@ -466,12 +466,31 @@ dso_cache__free(struct dso *dso)pthread_mutex_unlock (&dso->lock); } +static void update_last_cache (struct dso *dso,struct dso_cache *cache) +{ +int i; + +for (i =DSO_LAST_CACHE_NR -1 ; i >0 ; i--) + dso->data.last[i] = dso->data.last[i-1 ]; + + dso->data.last[0 ] = cache; +} +static struct dso_cache *dso_cache__find (struct dso *dso,u64 offset) {const struct rb_root *root = &dso->data.cache;struct rb_node *const *p = &root->rb_node;const struct rb_node *parent =NULL ;struct dso_cache *cache; +int i; + +for (i =0 ; i <DSO_LAST_CACHE_NR ; i++) { + cache = dso->data.last[i]; + +if (cache && cache->offset <= offset && + offset < cache->offset +DSO__DATA_CACHE_SIZE ) +return cache; + }pthread_mutex_lock (&dso->lock);while (*p !=NULL ) {
@@ -485,8 +504,10 @@ static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset) p = &(*p)->rb_left; else if (offset >= end) p = &(*p)->rb_right; - else + else { +update_last_cache (dso, cache); goto out; + } } cache =NULL ; out:
@@ -515,13 +536,16 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new) p = &(*p)->rb_left; else if (offset >= end) p = &(*p)->rb_right; - else + else { +update_last_cache (dso, cache); goto out; + } }rb_link_node (&new->rb_node, parent, p);rb_insert_color (&new->rb_node, root); +update_last_cache (dso, new); cache =NULL ; out:pthread_mutex_unlock (&dso->lock);
tools/perf/util/dso.h +2 -0
@@ -136,6 +136,8 @@ struct dso {/* dso data file */ struct {struct rb_root cache; +#define DSO_LAST_CACHE_NR 16 +struct dso_cache *last[DSO_LAST_CACHE_NR ];int fd;int status;u32 status_seen;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 34/37 perf tools: Convert lseek + read to pread
In reply to Namhyung KimWhen dso_cache__read() is called, it reads data from the given offset using lseek + normal read syscall. It can be combined to a single pread syscall.
--- tools/perf/util/dso.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
tools/perf/util/dso.c +1 -4
@@ -593,10 +593,7 @@ dso_cache__read(struct dso *dso, struct machine *machine, } } -if (-1 ==lseek (dso->data.fd, cache_offset,SEEK_SET )) - goto err_unlock; - - ret =read (dso->data.fd, cache->data,DSO__DATA_CACHE_SIZE ); + ret =pread (dso->data.fd, cache->data,DSO__DATA_CACHE_SIZE , cache_offset);if (ret <=0 ) goto err_unlock;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 35/37 perf callchain: Save eh/debug frame offset for dwarf unwind
In reply to Namhyung KimWhen libunwind tries to resolve callchains it needs to know the offset of .eh_frame_hdr or .debug_frame to access the dso. Since it calls dso__data_fd(), it'll try to grab dso->lock everytime for same information. So save it to dso_data struct and reuse it.
Note that there's a window between dso__data_fd() and actual use of the fd. The fd could be closed by other threads to deal with the open file limit in dso cache code. But I think it's ok since in that case elf_section_offset() will return 0 so it'll be tried in next acess.
--- tools/perf/util/dso.h | 1 + tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++----------- 2 files changed, 21 insertions(+), 11 deletions(-)
tools/perf/util/dso.h +1 -0
@@ -143,6 +143,7 @@ struct dso {u32 status_seen;size_t file_size;struct list_head open_entry; +u64 frame_offset; } data;union {/* Tool specific area */
tools/perf/util/unwind-libunwind.c +20 -11
@@ -244,14 +244,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine, u64 *fde_count) {int ret = -EINVAL , fd; -u64 offset; +u64 offset = dso->data.frame_offset; - fd =dso__data_fd (dso, machine); -if (fd <0 ) -return -EINVAL ; +if (offset ==0 ) { + fd =dso__data_fd (dso, machine); +if (fd <0 ) +return -EINVAL ; -/* Check the .eh_frame section for unwinding info */ - offset =elf_section_offset (fd,".eh_frame_hdr" ); +/* Check the .eh_frame section for unwinding info */ + offset =elf_section_offset (fd,".eh_frame_hdr" ); + dso->data.frame_offset = offset; + }if (offset) ret = unwind_spec_ehframe(dso, machine, offset,
@@ -265,14 +268,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,static int read_unwind_spec_debug_frame (struct dso *dso,struct machine *machine,u64 *offset) { -int fd =dso__data_fd (dso, machine); +int fd; +u64 ofs = dso->data.frame_offset; -if (fd <0 ) -return -EINVAL ; +if (ofs ==0 ) { + fd =dso__data_fd (dso, machine); +if (fd <0 ) +return -EINVAL ; -/* Check the .debug_frame section for unwinding info */ - *offset =elf_section_offset (fd,".debug_frame" ); +/* Check the .debug_frame section for unwinding info */ + ofs =elf_section_offset (fd,".debug_frame" ); + dso->data.frame_offset = ofs; + } + *offset = ofs;if (*offset)return 0 ;
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 36/37 perf tools: Add new perf data command
In reply to Namhyung KimFrom: Jiri Olsa <[email protected]>
Adding new 'perf data' command to provide operations over data files.
The 'perf data convert' sub command is coming in following patch, but there's possibility for other useful commands like 'perf data ls' (to display perf data file in directory in ls style).
--- tools/perf/Documentation/perf-data.txt | 15 +++++++ tools/perf/Makefile.perf | 1 + tools/perf/builtin-data.c | 75 ++++++++++++++++++++++++++++++++++ tools/perf/builtin.h | 1 + tools/perf/command-list.txt | 1 + tools/perf/perf.c | 1 + 6 files changed, 94 insertions(+) create mode 100644 tools/perf/Documentation/perf-data.txt create mode 100644 tools/perf/builtin-data.c
tools/perf/Documentation/perf-data.txt new file +15 -0
@@ -0,0 +1,15 @@ +perf-data(1) +============== + +NAME +---- +perf-data - Data file related processing + +SYNOPSIS +-------- +[verse] +'perf data' [<common options>] <command> [<options>]", + +DESCRIPTION +----------- +Data file related processing.
tools/perf/Makefile.perf +1 -0
@@ -485,6 +485,7 @@ BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o BUILTIN_OBJS += $(OUTPUT)builtin-inject.o BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o BUILTIN_OBJS += $(OUTPUT)builtin-mem.o +BUILTIN_OBJS += $(OUTPUT)builtin-data.o PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT)
tools/perf/builtin-data.c new file +75 -0
75 lines not on this page · raw message
tools/perf/builtin.h +1 -0
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);extern int cmd_trace (int argc,const char **argv,const char *prefix);extern int cmd_inject (int argc,const char **argv,const char *prefix);extern int cmd_mem (int argc,const char **argv,const char *prefix); +extern int cmd_data (int argc,const char **argv,const char *prefix);extern int find_scripts (char **scripts_array,char **scripts_path_array);#endif
tools/perf/command-list.txt +1 -0
@@ -7,6 +7,7 @@ perf-archive mainporcelain common perf-bench mainporcelain common perf-buildid-cache mainporcelain common perf-buildid-list mainporcelain common +perf-data mainporcelain common perf-diff mainporcelain common perf-evlist mainporcelain common perf-inject mainporcelain common
tools/perf/perf.c +1 -0
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {#endif { "inject", cmd_inject, 0 }, { "mem", cmd_mem, 0 }, + { "data", cmd_data, 0 }, }; struct pager_config {
Signature · 1 line
2.1.3
Namhyung Kim
maintainerPATCH 37/37 perf data: Implement 'split' subcommand
In reply to Namhyung KimThe perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +28 -0
@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads. + +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path. + +-v:: +--verbose:: + Be more verbose (show counter open errors, etc). + +-f:: +--force:: + Don't complain, do it. + +SEE ALSO +-------- +linkperf:perf[1], linkperf:perf-report[1]
tools/perf/builtin-data.c +223 -0
243 lines not on this page · raw message
Signature · 1 line
2.1.3
Arnaldo Carvalho de Melo
maintainerRe: [PATCH 37/37] perf data: Implement 'split' subcommand
Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:
Quoted 3 lines
The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)
- ARnaldo
Quoted 293 lines
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)tools/perf/Documentation/perf-data.txt +28
-0@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads. + +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path. + +-v:: +--verbose:: + Be more verbose (show counter open errors, etc). + +-f:: +--force:: + Don't complain, do it. + +SEE ALSO +-------- +linkperf:perf[1], linkperf:perf-report[1]tools/perf/builtin-data.c +223
-0@@ -2,10 +2,14 @@#include "builtin.h" #include "perf.h" #include "debug.h" +#include "session.h" +#include "evlist.h" #include "parse-options.h" typedef int (*data_cmd_fn_t )(int argc,const char **argv,const char *prefix); +static const char *output_name; +struct data_cmd {const char *name;const char *summary;@@ -41,10 +45,229 @@ static void print_usage(void)printf ("\n" ); } +static int data_cmd_split (int argc,const char **argv,const char *prefix); +static struct data_cmd data_cmds[] = { + {"split" ,"split single data file into multi-file" , data_cmd_split }, {NULL }, }; +#define FD_HASH_BITS 7 +#define FD_HASH_SIZE (1 << FD_HASH_BITS) +#define FD_HASH_MASK (FD_HASH_SIZE - 1) + +struct data_split { +struct perf_tool tool; +struct perf_session *session; +enum { +PER_CPU , +PER_THREAD , + } mode; +int header_fd; +u64 header_written; +struct hlist_head fd_hash[FD_HASH_SIZE ]; +int fd_hash_nr; +}; + +struct fdhash_node { +int id; +int fd; +struct hlist_node list; +}; + +static struct hlist_head *get_hash (struct data_split *split,int id) +{ +return &split->fd_hash[id %FD_HASH_MASK ]; +} + +static int perf_event__rewrite_header (struct perf_tool *tool, +union perf_event *event) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +ssize_t size; + + size =writen (split->header_fd, event, event->header.size); +if (size <0 ) +return -errno; + + split->header_written += size; +return 0 ; +} + +static int split_other_events (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample__maybe_unused , +struct machine *machine__maybe_unused ) +{ +return perf_event__rewrite_header (tool, event); +} + +static int split_sample_event (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel__maybe_unused , +struct machine *machine__maybe_unused ) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +int id = split->mode ==PER_CPU ? sample->cpu : sample->tid; +int fd =-1 ; +char buf[PATH_MAX ]; +struct hlist_head *head; +struct fdhash_node *node; + + head =get_hash (split, id); +hlist_for_each_entry (node, head, list) { +if (node->id == id) { + fd = node->fd; +break ; + } + } + +if (fd ==-1 ) { +scnprintf (buf,sizeof (buf),"%s/perf.data.%d" , + output_name, split->fd_hash_nr++); + + fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (fd <0 ) { +pr_err ("cannot open data file: %s: %m\n" , buf); +return -1 ; + } + + node =malloc (sizeof (*node)); +if (node ==NULL ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + + node->id = id; + node->fd = fd; + +hlist_add_head (&node->list, head); + } + +return writen (fd, event, event->header.size) >0 ?0 : -errno; +} + +static int __data_cmd_split (struct data_split *split) +{ +struct perf_session *session = split->session; +char *output =NULL ; +char buf[PATH_MAX ]; +u64 sample_type; +int header_fd; +int ret =-1 ; +int i; + +if (!output_name) { +if (asprintf (&output,"%s.dir" , input_name) <0 ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + output_name = output; + } + +mkdir (output_name,0700 ); + +/* +* This is necessary to write (copy) build-id table. After +* processing header, dsos list will contain dso which was on +* the original build-id table. +*/ +dsos__hit_all (session); + +scnprintf (buf,sizeof (buf),"%s/perf.header" , output_name); + header_fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (header_fd <0 ) { +pr_err ("cannot open header file: %s: %m\n" , buf); + goto out; + } + +lseek (header_fd, session->header.data_offset,SEEK_SET ); + + sample_type =perf_evlist__combined_sample_type (session->evlist); +if (sample_type &PERF_SAMPLE_CPU ) + split->mode =PER_CPU ; +else + split->mode =PER_THREAD ; + +pr_debug ("splitting data file for %s\n" , + split->mode ==PER_CPU ?"CPUs" :"threads" ); + + split->header_fd = header_fd; +perf_session__process_events (session, &split->tool); + +for (i =0 ; i <FD_HASH_SIZE ; i++) { +struct fdhash_node *pos; +struct hlist_node *tmp; + +hlist_for_each_entry_safe (pos, tmp, &split->fd_hash[i], list) { +hlist_del (&pos->list); +close (pos->fd); +free (pos); + } + } + + session->header.data_size = split->header_written; +perf_session__write_header (session, session->evlist, header_fd, true); + +close (header_fd); + ret =0 ; +out: +free (output); +return ret; +} + +int data_cmd_split (int argc,const char **argv,const char *prefix__maybe_unused ) +{ +bool force = false; +struct perf_session *session; +struct perf_data_file file = { + .mode =PERF_DATA_MODE_READ , + }; +struct data_split split = { + .tool = { + .sample = split_sample_event, + .fork = split_other_events, + .comm = split_other_events, + .exit = split_other_events, + .mmap = split_other_events, + .mmap2 = split_other_events, + .lost = split_other_events, + .throttle = split_other_events, + .unthrottle = split_other_events, + }, + }; +const char *const split_usage[] = { +"perf data split [<options>]" , +NULL + }; +const struct option split_options[] = { +OPT_STRING ('i' ,"input" , &input_name,"file" ,"input file name" ), +OPT_STRING ('o' ,"output" , &output_name,"file" ,"output directory name" ), +OPT_BOOLEAN ('f' ,"force" , &force,"don't complain, do it" ), +OPT_INCR ('v' ,"verbose" , &verbose,"be more verbose" ), +OPT_END () + }; + + argc =parse_options (argc, argv, split_options, split_usage,0 ); +if (argc) +usage_with_options (split_usage, split_options); + + file.path = input_name; + file.force = force; + session =perf_session__new (&file, false, &split.tool); +if (session ==NULL ) +return -1 ; + + split.session = session; +symbol__init (&session->header.env); + +__data_cmd_split (&split); + +perf_session__delete (session); +return 0 ; +} +int cmd_data (int argc,const char **argv,const char *prefix) {struct data_cmd *cmd;Signature · 1 line
2.1.3
Namhyung Kim
maintainerRe: [PATCH 37/37] perf data: Implement 'split' subcommand
Hi Arnaldo,
Quoted 8 lines · Arnaldo Carvalho de Melo
Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:
Quoted 3 lines
The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)
It depends on the data file - if it's recorded system-wide, the split will be done by cpu, otherwise by thread. It's determined by checking sample type has PERF_SAMPLE_CPU. And metadata is saved in a separate perf.header file like other multi-file data. Will add it to the change log.
Thanks,
Namhyung
Quoted 293 lines
Quoted 293 lines
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)tools/perf/Documentation/perf-data.txt +28
-0@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads. + +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path. + +-v:: +--verbose:: + Be more verbose (show counter open errors, etc). + +-f:: +--force:: + Don't complain, do it. + +SEE ALSO +-------- +linkperf:perf[1], linkperf:perf-report[1]tools/perf/builtin-data.c +223
-1@@ -2,10 +2,14 @@#include "builtin.h" #include "perf.h" #include "debug.h" +#include "session.h" +#include "evlist.h" #include "parse-options.h" typedef int (*data_cmd_fn_t )(int argc,const char **argv,const char *prefix); +static const char *output_name; +struct data_cmd {const char *name;const char *summary;@@ -41,10 +45,229 @@ static void print_usage(void)printf ("\n" ); } +static int data_cmd_split (int argc,const char **argv,const char *prefix); +static struct data_cmd data_cmds[] = { + {"split" ,"split single data file into multi-file" , data_cmd_split }, {NULL }, }; +#define FD_HASH_BITS 7 +#define FD_HASH_SIZE (1 << FD_HASH_BITS) +#define FD_HASH_MASK (FD_HASH_SIZE - 1) + +struct data_split { +struct perf_tool tool; +struct perf_session *session; +enum { +PER_CPU , +PER_THREAD , + } mode; +int header_fd; +u64 header_written; +struct hlist_head fd_hash[FD_HASH_SIZE ]; +int fd_hash_nr; +}; + +struct fdhash_node { +int id; +int fd; +struct hlist_node list; +}; + +static struct hlist_head *get_hash (struct data_split *split,int id) +{ +return &split->fd_hash[id %FD_HASH_MASK ]; +} + +static int perf_event__rewrite_header (struct perf_tool *tool, +union perf_event *event) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +ssize_t size; + + size =writen (split->header_fd, event, event->header.size); +if (size <0 ) +return -errno; + + split->header_written += size; +return 0 ; +} + +static int split_other_events (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample__maybe_unused , +struct machine *machine__maybe_unused ) +{ +return perf_event__rewrite_header (tool, event); +} + +static int split_sample_event (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel__maybe_unused , +struct machine *machine__maybe_unused ) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +int id = split->mode ==PER_CPU ? sample->cpu : sample->tid; +int fd =-1 ; +char buf[PATH_MAX ]; +struct hlist_head *head; +struct fdhash_node *node; + + head =get_hash (split, id); +hlist_for_each_entry (node, head, list) { +if (node->id == id) { + fd = node->fd; +break ; + } + } + +if (fd ==-1 ) { +scnprintf (buf,sizeof (buf),"%s/perf.data.%d" , + output_name, split->fd_hash_nr++); + + fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (fd <0 ) { +pr_err ("cannot open data file: %s: %m\n" , buf); +return -1 ; + } + + node =malloc (sizeof (*node)); +if (node ==NULL ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + + node->id = id; + node->fd = fd; + +hlist_add_head (&node->list, head); + } + +return writen (fd, event, event->header.size) >0 ?0 : -errno; +} + +static int __data_cmd_split (struct data_split *split) +{ +struct perf_session *session = split->session; +char *output =NULL ; +char buf[PATH_MAX ]; +u64 sample_type; +int header_fd; +int ret =-1 ; +int i; + +if (!output_name) { +if (asprintf (&output,"%s.dir" , input_name) <0 ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + output_name = output; + } + +mkdir (output_name,0700 ); + +/* +* This is necessary to write (copy) build-id table. After +* processing header, dsos list will contain dso which was on +* the original build-id table. +*/ +dsos__hit_all (session); + +scnprintf (buf,sizeof (buf),"%s/perf.header" , output_name); + header_fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (header_fd <0 ) { +pr_err ("cannot open header file: %s: %m\n" , buf); + goto out; + } + +lseek (header_fd, session->header.data_offset,SEEK_SET ); + + sample_type =perf_evlist__combined_sample_type (session->evlist); +if (sample_type &PERF_SAMPLE_CPU ) + split->mode =PER_CPU ; +else + split->mode =PER_THREAD ; + +pr_debug ("splitting data file for %s\n" , + split->mode ==PER_CPU ?"CPUs" :"threads" ); + + split->header_fd = header_fd; +perf_session__process_events (session, &split->tool); + +for (i =0 ; i <FD_HASH_SIZE ; i++) { +struct fdhash_node *pos; +struct hlist_node *tmp; + +hlist_for_each_entry_safe (pos, tmp, &split->fd_hash[i], list) { +hlist_del (&pos->list); +close (pos->fd); +free (pos); + } + } + + session->header.data_size = split->header_written; +perf_session__write_header (session, session->evlist, header_fd, true); + +close (header_fd); + ret =0 ; +out: +free (output); +return ret; +} + +int data_cmd_split (int argc,const char **argv,const char *prefix__maybe_unused ) +{ +bool force = false; +struct perf_session *session; +struct perf_data_file file = { + .mode =PERF_DATA_MODE_READ , + }; +struct data_split split = { + .tool = { + .sample = split_sample_event, + .fork = split_other_events, + .comm = split_other_events, + .exit = split_other_events, + .mmap = split_other_events, + .mmap2 = split_other_events, + .lost = split_other_events, + .throttle = split_other_events, + .unthrottle = split_other_events, + }, + }; +const char *const split_usage[] = { +"perf data split [<options>]" , +NULL + }; +const struct option split_options[] = { +OPT_STRING ('i' ,"input" , &input_name,"file" ,"input file name" ), +OPT_STRING ('o' ,"output" , &output_name,"file" ,"output directory name" ), +OPT_BOOLEAN ('f' ,"force" , &force,"don't complain, do it" ), +OPT_INCR ('v' ,"verbose" , &verbose,"be more verbose" ), +OPT_END () + }; + + argc =parse_options (argc, argv, split_options, split_usage,0 ); +if (argc) +usage_with_options (split_usage, split_options); + + file.path = input_name; + file.force = force; + session =perf_session__new (&file, false, &split.tool); +if (session ==NULL ) +return -1 ; + + split.session = session; +symbol__init (&session->header.env); + +__data_cmd_split (&split); + +perf_session__delete (session); +return 0 ; +} +int cmd_data (int argc,const char **argv,const char *prefix) {struct data_cmd *cmd; --2.1.3
Signature · 2 lines
Thanks, Namhyung
Arnaldo Carvalho de Melo
maintainerRe: [PATCH 37/37] perf data: Implement 'split' subcommand
Em Wed, Dec 24, 2014 at 11:14:47PM +0900, Namhyung Kim escreveu:
Quoted 13 lines
Hi Arnaldo,
Quoted 8 lines · Arnaldo Carvalho de Melo
Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:
Quoted 3 lines
The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)
Quoted 5 lines
It depends on the data file - if it's recorded system-wide, the split will be done by cpu, otherwise by thread. It's determined by checking sample type has
PERF_SAMPLE_CPU. And metadata is saved in a separate perf.header file like other multi-file data. Will add it to the change log.
Thanks for the explanation!
Please try to add such more detailed information on the change logs, as the reviewing process starts at the idea/design, and that is first read in the changelog, only after understanding what you want to do is that we should bother looking if you did what you say you would do :-)
- Arnaldo
Quoted 303 lines
Thanks,
NamhyungQuoted 293 lines
Quoted 293 lines
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)tools/perf/Documentation/perf-data.txt +28
-0@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads. + +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path. + +-v:: +--verbose:: + Be more verbose (show counter open errors, etc). + +-f:: +--force:: + Don't complain, do it. + +SEE ALSO +-------- +linkperf:perf[1], linkperf:perf-report[1]tools/perf/builtin-data.c +223
-1@@ -2,10 +2,14 @@#include "builtin.h" #include "perf.h" #include "debug.h" +#include "session.h" +#include "evlist.h" #include "parse-options.h" typedef int (*data_cmd_fn_t )(int argc,const char **argv,const char *prefix); +static const char *output_name; +struct data_cmd {const char *name;const char *summary;@@ -41,10 +45,229 @@ static void print_usage(void)printf ("\n" ); } +static int data_cmd_split (int argc,const char **argv,const char *prefix); +static struct data_cmd data_cmds[] = { + {"split" ,"split single data file into multi-file" , data_cmd_split }, {NULL }, }; +#define FD_HASH_BITS 7 +#define FD_HASH_SIZE (1 << FD_HASH_BITS) +#define FD_HASH_MASK (FD_HASH_SIZE - 1) + +struct data_split { +struct perf_tool tool; +struct perf_session *session; +enum { +PER_CPU , +PER_THREAD , + } mode; +int header_fd; +u64 header_written; +struct hlist_head fd_hash[FD_HASH_SIZE ]; +int fd_hash_nr; +}; + +struct fdhash_node { +int id; +int fd; +struct hlist_node list; +}; + +static struct hlist_head *get_hash (struct data_split *split,int id) +{ +return &split->fd_hash[id %FD_HASH_MASK ]; +} + +static int perf_event__rewrite_header (struct perf_tool *tool, +union perf_event *event) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +ssize_t size; + + size =writen (split->header_fd, event, event->header.size); +if (size <0 ) +return -errno; + + split->header_written += size; +return 0 ; +} + +static int split_other_events (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample__maybe_unused , +struct machine *machine__maybe_unused ) +{ +return perf_event__rewrite_header (tool, event); +} + +static int split_sample_event (struct perf_tool *tool, +union perf_event *event, +struct perf_sample *sample, +struct perf_evsel *evsel__maybe_unused , +struct machine *machine__maybe_unused ) +{ + struct data_split *split = container_of(tool, struct data_split, tool); +int id = split->mode ==PER_CPU ? sample->cpu : sample->tid; +int fd =-1 ; +char buf[PATH_MAX ]; +struct hlist_head *head; +struct fdhash_node *node; + + head =get_hash (split, id); +hlist_for_each_entry (node, head, list) { +if (node->id == id) { + fd = node->fd; +break ; + } + } + +if (fd ==-1 ) { +scnprintf (buf,sizeof (buf),"%s/perf.data.%d" , + output_name, split->fd_hash_nr++); + + fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (fd <0 ) { +pr_err ("cannot open data file: %s: %m\n" , buf); +return -1 ; + } + + node =malloc (sizeof (*node)); +if (node ==NULL ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + + node->id = id; + node->fd = fd; + +hlist_add_head (&node->list, head); + } + +return writen (fd, event, event->header.size) >0 ?0 : -errno; +} + +static int __data_cmd_split (struct data_split *split) +{ +struct perf_session *session = split->session; +char *output =NULL ; +char buf[PATH_MAX ]; +u64 sample_type; +int header_fd; +int ret =-1 ; +int i; + +if (!output_name) { +if (asprintf (&output,"%s.dir" , input_name) <0 ) { +pr_err ("memory allocation failed\n" ); +return -1 ; + } + output_name = output; + } + +mkdir (output_name,0700 ); + +/* +* This is necessary to write (copy) build-id table. After +* processing header, dsos list will contain dso which was on +* the original build-id table. +*/ +dsos__hit_all (session); + +scnprintf (buf,sizeof (buf),"%s/perf.header" , output_name); + header_fd =open (buf,O_RDWR |O_CREAT |O_TRUNC ,0600 ); +if (header_fd <0 ) { +pr_err ("cannot open header file: %s: %m\n" , buf); + goto out; + } + +lseek (header_fd, session->header.data_offset,SEEK_SET ); + + sample_type =perf_evlist__combined_sample_type (session->evlist); +if (sample_type &PERF_SAMPLE_CPU ) + split->mode =PER_CPU ; +else + split->mode =PER_THREAD ; + +pr_debug ("splitting data file for %s\n" , + split->mode ==PER_CPU ?"CPUs" :"threads" ); + + split->header_fd = header_fd; +perf_session__process_events (session, &split->tool); + +for (i =0 ; i <FD_HASH_SIZE ; i++) { +struct fdhash_node *pos; +struct hlist_node *tmp; + +hlist_for_each_entry_safe (pos, tmp, &split->fd_hash[i], list) { +hlist_del (&pos->list); +close (pos->fd); +free (pos); + } + } + + session->header.data_size = split->header_written; +perf_session__write_header (session, session->evlist, header_fd, true); + +close (header_fd); + ret =0 ; +out: +free (output); +return ret; +} + +int data_cmd_split (int argc,const char **argv,const char *prefix__maybe_unused ) +{ +bool force = false; +struct perf_session *session; +struct perf_data_file file = { + .mode =PERF_DATA_MODE_READ , + }; +struct data_split split = { + .tool = { + .sample = split_sample_event, + .fork = split_other_events, + .comm = split_other_events, + .exit = split_other_events, + .mmap = split_other_events, + .mmap2 = split_other_events, + .lost = split_other_events, + .throttle = split_other_events, + .unthrottle = split_other_events, + }, + }; +const char *const split_usage[] = { +"perf data split [<options>]" , +NULL + }; +const struct option split_options[] = { +OPT_STRING ('i' ,"input" , &input_name,"file" ,"input file name" ), +OPT_STRING ('o' ,"output" , &output_name,"file" ,"output directory name" ), +OPT_BOOLEAN ('f' ,"force" , &force,"don't complain, do it" ), +OPT_INCR ('v' ,"verbose" , &verbose,"be more verbose" ), +OPT_END () + }; + + argc =parse_options (argc, argv, split_options, split_usage,0 ); +if (argc) +usage_with_options (split_usage, split_options); + + file.path = input_name; + file.force = force; + session =perf_session__new (&file, false, &split.tool); +if (session ==NULL ) +return -1 ; + + split.session = session; +symbol__init (&session->header.env); + +__data_cmd_split (&split); + +perf_session__delete (session); +return 0 ; +} +int cmd_data (int argc,const char **argv,const char *prefix) {struct data_cmd *cmd; --2.1.3
Signature · 2 lines
Thanks, Namhyung
Re: [PATCH 37/37] perf data: Implement 'split' subcommand
In reply to Namhyung KimQuoted 24 lines · Namhyung Kim
The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)tools/perf/Documentation/perf-data.txt +6
-0@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads.
tools/perf/Documentation/perf-data.txt SYNOPSIS · line 31
Quoted 10 lines
tools/perf/Documentation/perf-data.txt +10
-0+ +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path.
should the -o have 'perf.data.dir' as default?
[jolsa@krava perf]$ ./perf record ls > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB perf.data (~0 samples) ] [jolsa@krava perf]$ ./perf data split
[jolsa@krava perf]$ ll perf.data*
-rw------- 1 jolsa jolsa 16172 Dec 26 14:58 perf.data
jirka
Namhyung Kim
maintainerRe: [PATCH 37/37] perf data: Implement 'split' subcommand
Hi Jiri,
Quoted 38 lines · Jiri Olsa
Quoted 24 lines · Namhyung Kim
The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/Documentation/perf-data.txt | 28 +++++ tools/perf/builtin-data.c | 223 +++++++++++++++++++++++++++++++++ 2 files changed, 251 insertions(+)tools/perf/Documentation/perf-data.txt +6
-0@@ -13,3 +13,31 @@ SYNOPSIS DESCRIPTION ----------- Data file related processing. + +COMMANDS +-------- +split:: + Split single data file (perf.data) into multiple files under a directory + in order to be reported by multiple threads.tools/perf/Documentation/perf-data.txt SYNOPSIS · line 31
Quoted 10 lines
tools/perf/Documentation/perf-data.txt +10
-0+ +OPTIONS for 'split' +--------------------- +-i:: +--input:: + Specify input perf data file path. + +-o:: +--output:: + Specify output perf data directory path.should the -o have 'perf.data.dir' as default?
Quoted 6 lines
[jolsa@krava perf]$ ./perf record ls >
/dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB perf.data (~0 samples) ] [jolsa@krava perf]$ ./perf data split
[jolsa@krava perf]$ ll perf.data*
-rw------- 1 jolsa jolsa 16172 Dec 26 14:58 perf.data
You're right! :)
Thanks,
Namhyung
On Wed, Dec 24, 2014 at 04:14:56PM +0900, Namhyung Kim wrote:
SNIP
Quoted 7 lines
You can get it from 'perf/threaded-v1' branch on my tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitPlease take a look and play with it. Any comments are welcome! :)
very nice at first round check ;-)
I'll do detailed review next week
thanks,
jirka
Namhyung Kim
maintainerQuoted 8 lines · Jiri Olsa
On Wed, Dec 24, 2014 at 04:14:56PM +0900, Namhyung Kim wrote:
SNIP
Quoted 4 lines
You can get it from 'perf/threaded-v1' branch on my tree at:
Quoted 7 lines
Quoted 3 lines
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitPlease take a look and play with it. Any comments are welcome! :)
very nice at first round check ;-)
I'll do detailed review next week
Thank you very much!
Namhyung
Thanks for working on this. Haven't read any code, just some high level comments on the design.
Quoted 8 lines
So my approach is like this:
Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.
Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.
Also I thought Adrian's meta data index already addressed this at least partially.
Quoted 7 lines
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread,
map_groupsand comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.
FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.
It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)
Quoted 5 lines
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.
Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?
BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.
Quoted 8 lines
Here is the result:
This is just elapsed (real) time measured by shell 'time' function.
The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.
Quoted 8 lines
time perf report --children --no-children + --call-graph none
---------- ------------- -------------------current 4m43.260s 1m32.779s 0m35.866s patched 4m43.710s 1m29.695s 0m33.995s --multi-thread 2m46.265s 0m45.486s 0m7.570sThis result is with 7.7GB data file using libunwind for callchain.
Nice results!
-Andi
Quoted 36 lines · Andi Kleen
Thanks for working on this. Haven't read any code, just some high level comments on the design.
Quoted 8 lines
So my approach is like this:
Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.
Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.
Also I thought Adrian's meta data index already addressed this at least partially.
Quoted 7 lines
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread,
map_groupsand comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.
It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.
having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.
We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.
Quoted 29 lines
Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?
BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.
Quoted 8 lines
Here is the result:
This is just elapsed (real) time measured by shell 'time' function.
The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.
Quoted 8 lines
time perf report --children --no-children + --call-graph none
---------- ------------- ------------------- current 4m43.260s 1m32.779s 0m35.866s patched 4m43.710s 1m29.695s 0m33.995s --multi-thread 2m46.265s 0m45.486s 0m7.570sThis result is with 7.7GB data file using libunwind for callchain.
Nice results!
-Andi
Namhyung Kim
maintainerHi Stephane,
Quoted 20 lines · Stephane Eranian
Quoted 19 lines · Andi Kleen
Quoted 6 lines
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread,
map_groupsand comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.
It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)
Quoted 5 lines
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
Quoted 7 lines
Quoted 3 lines
I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.
having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.
I just thought it's easier to handle with multiple thread. Maybe we can concatenate the files after recording.
Quoted 4 lines
We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.
Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?
Thanks,
Namhyung
Quoted 36 lines · Namhyung Kim
Hi Stephane,
Quoted 20 lines · Stephane Eranian
Quoted 19 lines · Andi Kleen
Quoted 6 lines
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread,
map_groupsand comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.
It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)
Quoted 5 lines
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
Quoted 7 lines
Quoted 3 lines
I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.
having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.
I just thought it's easier to handle with multiple thread. Maybe we can concatenate the files after recording.
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.
Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?
There is no seek possible in pipe mode.
The way this is done (as I remember) is by creating pseudo-record types and injecting them in the stream.
Namhyung Kim
maintainerQuoted 8 lines · Stephane Eranian
Quoted 7 lines · Namhyung Kim
Hi Stephane,
Quoted 3 lines · Stephane Eranian
We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.
Quoted 8 lines
Quoted 4 lines
Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?
There is no seek possible in pipe mode.
The way this is done (as I remember) is by creating pseudo-record types and injecting them in the stream.
Right. But I have no idea how I can index a live stream and process it using multiple threads.
Thanks,
Namhyung
Namhyung Kim
maintainerHi Andi,
Quoted 3 lines · Andi Kleen
Thanks for working on this. Haven't read any code, just some high level comments on the design.
Really appreciate it!
Quoted 9 lines
Quoted 8 lines
So my approach is like this:
Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.
Quoted 4 lines
Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.
The files are located in a directory and users only deal with the directory so I don't think it's a big problem. In addition, moving data between different systems requires archiving related debuginfos and I think we can extend perf-archive to put those debuginfo in the data directory so that it can find the symbols more easily.
Quoted 3 lines
Also I thought Adrian's meta data index already addressed this at least partially.
I know Adrian's work might have some common parts but I haven't looked at it deeply, sorry! It'd be great if we can discuss how to coordinate the future direction or something..
Quoted 9 lines
Quoted 8 lines
This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread,
map_groupsand comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.
Quoted 6 lines
FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.
It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)
Right, many mmap events won't be used but we cannot predict which one is used or not.
Quoted 8 lines
Quoted 5 lines
With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.
I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.
Right, I admit that my result ran on a SSD disk.
Quoted 5 lines
Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?
I don't understand it. Could you please elaborate it?
Quoted 3 lines
BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.
I'm not sure how much code I can make simpler with the help of such library. I think most changes in this patchset is preparations to concurrent access in libperf and it's still needed even if the library is used anyway.
Thanks,
Namhyung
Quoted 26 lines
Quoted 9 lines
Here is the result:
This is just elapsed (real) time measured by shell 'time' function.
The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.
Quoted 8 lines
time perf report --children --no-children + --call-graph none
---------- ------------- -------------------current 4m43.260s 1m32.779s 0m35.866s patched 4m43.710s 1m29.695s 0m33.995s --multi-thread 2m46.265s 0m45.486s 0m7.570sThis result is with 7.7GB data file using libunwind for callchain.
Nice results!
-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines
Quoted 4 lines
Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?
I don't understand it. Could you please elaborate it?
Have a thread pool. Then there is a single thread which pre-processes events and puts them into queues, which are then processed in parallel by the threads in the thread pool.
-Andi