Skip to contentSkip to conversation

perf tools: Speed-up perf report by using multi thread (v1)

PATCH In review Namhyung Kim 7 participants 91 messages Dec 24, 2014 – Jan 8, 2015 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

cover perf tools: Speed-up perf report by using multi thread (v1)

Hello,

This patchset converts perf report to use multiple threads in order to speed up the processing on large data files. I can see a minimum 40% of speedup with this change. The code is still experimental, little bit outdated and contains many rough edges. But I'd like to share and give some feedbacks.

The perf report processes (sample) events like below:

1. preprocess sample to get matching thread/dso/symbol info
2. insert it to hists rbtree (with callchain tree) based on the info
3. optionally collapse hist entries that match given sort key(s)
4. resort hist entries (by overhead) for output
5. display the hist entries

The stage 1 is a preprocessing and mostly act like a read-only operation during the sample processing. Meta events like fork, comm and mmap can change the machine/thread state but symbols can be loaded during the processing (stage 2).

The stage 2 consumes most of the time especially with callchains and
--children option is enabled. And this work can be easily patitioned
as each sample is independent to others. But the resulting hists must be combined/collapsed to a single global hists before going to further steps.

The stage 3 is optional and only needed by certain sort keys - but with stage 2 paralellized, it needs to be done anyway.

The stage 4 and 5 works on whole hists so must be done serially.

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

For DWARF post-unwinding, dso cache data also needs to be protected by a lock and this causes a huge contention. I just added a front cache that can be accessed without the lock but this should be improved IMHO.

The patch 1-10 are to support multi-file data recording. With
-M/--multi option, perf record will create a directory (named
'perf.data.dir' by default - but maybe renamed 'perf.data' for transparent conversion later) and save meta events to perf.header file and sample events to perf.data.<n> file). It'd be better considering file format change Jiri suggested [1].

The patch 11-20 are to manage machine and thread state using timestamp so that it can be searched when processing samples. The patch 21-35 are to implement parallel report. And finally I implemented 'perf data split' command to convert a single data file into a multi-file format.

This patchset didn't change perf record to use multi-thread. But I think it can be easily done later if needed.

Note that output has a slight difference to original version when compared using splitted data file. But they're mostly unresolved symbols for callchains.

Here is the result:

This is just elapsed (real) time measured by shell 'time' function.

The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.

time perf report --children --no-children + --call-graph none
---------- ------------- -------------------

 current            4m43.260s      1m32.779s            0m35.866s            
 patched            4m43.710s      1m29.695s            0m33.995s
 --multi-thread     2m46.265s      0m45.486s             0m7.570s

This result is with 7.7GB data file using libunwind for callchain.

time perf report --children --no-children + --call-graph none
---------- ------------- -------------------

 current            3m51.762s      3m10.451s             0m4.695s            
 patched            2m26.030s      1m49.846s             0m4.105s
 --multi-thread     0m49.217s      0m35.106s             0m1.457s

Note that the single thread performance improvement in patched version is due to changes in the patch 33-35.

This result is with same file but using libdw for callchain unwind.

time perf report --children --no-children + --call-graph none
---------- ------------- -------------------

 current           10m22.472s     11m42.290s             0m4.758s            
 patched           10m10.625s     11m45.480s             0m4.162s
 --multi-thread     3m47.332s      3m35.235s             0m1.755s

On my archlinux system, callchain unwind using libdw is much slower than libunwind. I'm using elfutils version 0.160. Also I don't know why --children takes less time than --no-children. Anyway we can see the --multi-thread performance is much better for each case.

You can get it from 'perf/threaded-v1' branch on my tree at:

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it. Any comments are welcome! :)

Thanks,
Namhyung

[1] https://lkml.org/lkml/2013/9/1/20

Jiri Olsa (1):
perf tools: Add new perf data command

Namhyung Kim (36):
perf tools: Set attr.task bit for a tracking event
perf record: Use a software dummy event to track task/mmap events perf tools: Use perf_data_file__fd() consistently
perf tools: Add multi file interface to perf_data_file perf tools: Create separate mmap for dummy tracking event perf tools: Introduce perf_evlist__mmap_multi()
perf tools: Do not use __perf_session__process_events() directly perf tools: Handle multi-file session properly
perf record: Add -M/--multi option for multi file recording perf report: Skip dummy tracking event
perf tools: Introduce thread__comm_time() helpers
perf tools: Add a test case for thread comm handling
perf tools: Use thread__comm_time() when adding hist entries perf tools: Convert dead thread list into rbtree
perf tools: Introduce machine__find*_thread_time()
perf tools: Add a test case for timed thread handling perf tools: Maintain map groups list in a leader thread perf tools: Remove thread when map groups initialization failed perf tools: Introduce thread__find_addr_location_time() and friends perf tools: Add a test case for timed map groups handling perf tools: Protect dso symbol loading using a mutex
perf tools: Protect dso cache tree using dso->lock
perf tools: Protect dso cache fd with a mutex
perf session: Pass struct events stats to event processing functions perf hists: Pass hists struct to hist_entry_iter functions perf tools: Move BUILD_ID_SIZE definition to perf.h
perf report: Parallelize perf report using multi-thread perf tools: Add missing_threads rb tree
perf top: Always creates thread in the current task tree. perf tools: Fix progress ui to support multi thread
perf record: Show total size of multi file data
perf report: Add --multi-thread option and config item perf tools: Add front cache for dso data access
perf tools: Convert lseek + read to pread
perf callchain: Save eh/debug frame offset for dwarf unwind perf data: Implement 'split' subcommand

 tools/perf/Documentation/perf-data.txt   |  43 ++++
 tools/perf/Documentation/perf-record.txt |   5 +
 tools/perf/Documentation/perf-report.txt |   3 +
 tools/perf/Makefile.perf                 |   4 +
 tools/perf/builtin-annotate.c            |   5 +-
 tools/perf/builtin-data.c                | 298 ++++++++++++++++++++++++++
 tools/perf/builtin-diff.c                |   8 +-
 tools/perf/builtin-inject.c              |   9 +-
 tools/perf/builtin-record.c              |  65 ++++--
 tools/perf/builtin-report.c              | 107 ++++++++--
 tools/perf/builtin-script.c              |   5 +-
 tools/perf/builtin-top.c                 |   9 +-
 tools/perf/builtin.h                     |   1 +
 tools/perf/command-list.txt              |   1 +
 tools/perf/perf.c                        |   1 +
 tools/perf/perf.h                        |   2 +
 tools/perf/tests/builtin-test.c          |  12 ++
 tools/perf/tests/dso-data.c              |   5 +
 tools/perf/tests/dwarf-unwind.c          |  10 +-
 tools/perf/tests/hists_common.c          |   3 +-
 tools/perf/tests/hists_cumulate.c        |   4 +-
 tools/perf/tests/hists_filter.c          |   3 +-
 tools/perf/tests/hists_link.c            |   6 +-
 tools/perf/tests/hists_output.c          |   4 +-
 tools/perf/tests/tests.h                 |   3 +
 tools/perf/tests/thread-comm.c           |  47 +++++
 tools/perf/tests/thread-lookup-time.c    | 180 ++++++++++++++++
 tools/perf/tests/thread-mg-share.c       |   7 +-
 tools/perf/tests/thread-mg-time.c        |  88 ++++++++
 tools/perf/ui/browsers/hists.c           |  10 +-
 tools/perf/ui/gtk/hists.c                |   3 +
 tools/perf/util/build-id.c               |   9 +-
 tools/perf/util/build-id.h               |   2 -
 tools/perf/util/data.c                   | 188 ++++++++++++++++-
 tools/perf/util/data.h                   |  17 ++
 tools/perf/util/dso.c                    | 192 ++++++++++++-----
 tools/perf/util/dso.h                    |   5 +
 tools/perf/util/event.c                  |  85 ++++++--
 tools/perf/util/event.h                  |   6 +-
 tools/perf/util/evlist.c                 | 151 ++++++++++++--
 tools/perf/util/evlist.h                 |  22 +-
 tools/perf/util/evsel.c                  |   1 +
 tools/perf/util/evsel.h                  |  15 ++
 tools/perf/util/hist.c                   | 121 +++++++----
 tools/perf/util/hist.h                   |  12 +-
 tools/perf/util/machine.c                | 251 +++++++++++++++++++---
 tools/perf/util/machine.h                |  12 +-
 tools/perf/util/map.c                    |   1 +
 tools/perf/util/map.h                    |   2 +
 tools/perf/util/ordered-events.c         |   4 +-
 tools/perf/util/session.c                | 347 ++++++++++++++++++++++++++-----
 tools/perf/util/session.h                |   8 +-
 tools/perf/util/symbol.c                 |  34 ++-
 tools/perf/util/thread.c                 | 140 ++++++++++++-
 tools/perf/util/thread.h                 |  28 ++-
 tools/perf/util/tool.h                   |  17 ++
 tools/perf/util/unwind-libdw.c           |  11 +-
 tools/perf/util/unwind-libunwind.c       |  49 +++--
 tools/perf/util/util.c                   |  43 ++++
 tools/perf/util/util.h                   |   1 +
 60 files changed, 2381 insertions(+), 344 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
 create mode 100644 tools/perf/tests/thread-comm.c
 create mode 100644 tools/perf/tests/thread-lookup-time.c
 create mode 100644 tools/perf/tests/thread-mg-time.c
Signature · 1 line
2.1.3
author maintainer
#1 lore raw

PATCH 1/37 perf tools: Set attr.task bit for a tracking event

The perf_event_attr.task bit is to track task (fork and exit) events but it missed to be set by perf_evsel__config(). While it was not a problem in practice since setting other bits (comm/mmap) ended up being in same result, it'd be good to set it explicitly anyway.

---
 tools/perf/util/evsel.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/evsel.c +1 -0
index 1e90c8557ede..e17d2b1624bc 100644
@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)
 	if (opts->sample_weight)
 		perf_evsel__set_sample_bit(evsel, WEIGHT);
 
+	attr->task  = track;
 	attr->mmap  = track;
 	attr->mmap2 = track && !perf_missing_features.mmap2;
 	attr->comm  = track;
Signature · 1 line
2.1.3
Acked-by on v1 1/37
#2 lore raw

Re: [PATCH 01/37] perf tools: Set attr.task bit for a tracking event

Quoted 6 lines · Namhyung Kim

The perf_event_attr.task bit is to track task (fork and exit) events but it missed to be set by perf_evsel__config(). While it was not a problem in practice since setting other bits (comm/mmap) ended up being in same result, it'd be good to set it explicitly anyway.

Quoted 19 lines
---
 tools/perf/util/evsel.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/evsel.c +1 -0
index 1e90c8557ede..e17d2b1624bc 100644
@@ -709,6 +709,7 @@ void perf_evsel__config(struct perf_evsel *evsel, struct record_opts *opts)
 	if (opts->sample_weight)
 		perf_evsel__set_sample_bit(evsel, WEIGHT);
 
+	attr->task  = track;
 	attr->mmap  = track;
 	attr->mmap2 = track && !perf_missing_features.mmap2;
 	attr->comm  = track;
Signature · 1 line
2.1.3
author maintainer
#3 lore raw

PATCH 2/37 perf record: Use a software dummy event to track task/mmap events

In reply to Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

---
 tools/perf/builtin-record.c |  3 +++
 tools/perf/perf.h           |  1 +
 tools/perf/util/evlist.c    | 38 ++++++++++++++++++++++++++++++++++++++
 tools/perf/util/evlist.h    |  1 +
 tools/perf/util/evsel.h     | 15 +++++++++++++++
 5 files changed, 58 insertions(+)
tools/perf/builtin-record.c +3 -0
index 8648c6d3003d..aa5fa6aabb31 100644
@@ -862,6 +862,9 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out_symbol_exit;
 	}
 
+	if (rec->opts.multi_file)
+		perf_evlist__prepend_dummy(rec->evlist);
+
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
 
tools/perf/perf.h +1 -0
index 1dabb8553499..37284eb47b56 100644
@@ -53,6 +53,7 @@ struct record_opts {
 	bool	     sample_time;
 	bool	     period;
 	bool	     sample_intr_regs;
+	bool	     multi_file;
 	unsigned int freq;
 	unsigned int mmap_pages;
 	unsigned int user_freq;
tools/perf/util/evlist.c +38 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)
 	return -ENOMEM;
 }
 
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+	struct perf_event_attr attr = {
+		.type = PERF_TYPE_SOFTWARE,
+		.config = PERF_COUNT_SW_DUMMY,
+	};
+	struct perf_evsel *evsel, *pos;
+
+	event_attr_init(&attr);
+
+	evsel = perf_evsel__new(&attr);
+	if (evsel == NULL)
+		goto error;
+
+	/* use strdup() because free(evsel) assumes name is allocated */
+	evsel->name = strdup("dummy");
+	if (!evsel->name)
+		goto error_free;
+
+	list_for_each_entry(pos, &evlist->entries, node) {
+		pos->idx += 1;
+		pos->tracking = false;
+	}
+
+	list_add(&evsel->node, &evlist->entries);
+	evsel->idx = 0;
+	evsel->tracking = true;
+
+	if (!evlist->nr_entries++)
+		perf_evlist__set_id_pos(evlist);
+
+	return 0;
+error_free:
+	perf_evsel__delete(evsel);
+error:
+	return -ENOMEM;
+}
+
 static int perf_evlist__add_attrs(struct perf_evlist *evlist,
 				  struct perf_event_attr *attrs, size_t nr_attrs)
 {
tools/perf/util/evlist.h +1 -0
index 649b0c597283..b974bddf6b8b 100644
@@ -67,6 +67,7 @@ void perf_evlist__delete(struct perf_evlist *evlist);
 
 void perf_evlist__add(struct perf_evlist *evlist, struct perf_evsel *entry);
 int perf_evlist__add_default(struct perf_evlist *evlist);
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist);
 int __perf_evlist__add_default_attrs(struct perf_evlist *evlist,
 				     struct perf_event_attr *attrs, size_t nr_attrs);
 
tools/perf/util/evsel.h +15 -0
index 38622747d130..5b45eea63043 100644
@@ -331,6 +331,21 @@ static inline bool perf_evsel__is_function_event(struct perf_evsel *evsel)
 #undef FUNCTION_EVENT
 }
 
+/**
+ * perf_evsel__is_dummy_tracking - Return whether given evsel is a dummy
+ * event for tracking meta events only
+ *
+ * @evsel - evsel selector to be tested
+ *
+ * Return %true if event is a dummy tracking event
+ */
+static inline bool perf_evsel__is_dummy_tracking(struct perf_evsel *evsel)
+{
+	return evsel->attr.type == PERF_TYPE_SOFTWARE &&
+		evsel->attr.config == PERF_COUNT_SW_DUMMY &&
+		evsel->attr.task == 1 && evsel->idx == 0;
+}
+
 struct perf_attr_details {
 	bool freq;
 	bool verbose;
Signature · 1 line
2.1.3
#4 lore raw

Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events

Quoted 3 lines · Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?

---8<---

Quoted 25 lines
tools/perf/util/evlist.c +17 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist *evlist)
  	return -ENOMEM;
  }
 
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+	struct perf_event_attr attr = {
+		.type = PERF_TYPE_SOFTWARE,
+		.config = PERF_COUNT_SW_DUMMY,
+	};
+	struct perf_evsel *evsel, *pos;
+
+	event_attr_init(&attr);
+
+	evsel = perf_evsel__new(&attr);
+	if (evsel == NULL)
+		goto error;
+
+	/* use strdup() because free(evsel) assumes name is allocated */
+	evsel->name = strdup("dummy");
+	if (!evsel->name)

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222

Quoted 10 lines
tools/perf/util/evlist.c +10 -0
+		goto error_free;
+
+	list_for_each_entry(pos, &evlist->entries, node) {
+		pos->idx += 1;
+		pos->tracking = false;
+	}
+
+	list_add(&evsel->node, &evlist->entries);
+	evsel->idx = 0;
+	evsel->tracking = true;

perf_evlist__set_tracking_event()?

David

author maintainer
#5 lore raw

Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events

Hi David,

Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?

Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.

Quoted 34 lines

---8<---

Quoted 31 lines
tools/perf/util/evlist.c +22 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist
 *evlist)
        return -ENOMEM;
  }
 
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+       struct perf_event_attr attr = {
+               .type = PERF_TYPE_SOFTWARE,
+               .config = PERF_COUNT_SW_DUMMY,
+       };
+       struct perf_evsel *evsel, *pos;
+
+       event_attr_init(&attr);
+
+       evsel = perf_evsel__new(&attr);
+       if (evsel == NULL)
+               goto error;
+
+       /* use strdup() because free(evsel) assumes name is allocated */
+       evsel->name = strdup("dummy");
+       if (!evsel->name)
+               goto error_free;
+
+       list_for_each_entry(pos, &evlist->entries, node) {
+               pos->idx += 1;
+               pos->tracking = false;
Quoted 8 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+       }
+
+       list_add(&evsel->node, &evlist->entries);
+       evsel->idx = 0;
+       evsel->tracking = true;

perf_evlist__set_tracking_event()?

I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the evsel->idx.

Thanks,
Namhyung

reviewer
#6 lore raw

Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events

Quoted 32 lines · Namhyung Kim

Hi David,

Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?

Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.

Quoted 12 lines

---8<---

Quoted 9 lines
tools/perf/util/evlist.c +0 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist
 *evlist)
        return -ENOMEM;
  }
 
Quoted 5 lines
Quoted 5 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+       struct perf_event_attr attr = {
+               .type = PERF_TYPE_SOFTWARE,
+               .config = PERF_COUNT_SW_DUMMY,

Probably need .exclude_kernel = 1, here

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
tools/perf/util/evlist.c +17 -0
+       };
+       struct perf_evsel *evsel, *pos;
+
+       event_attr_init(&attr);
+
+       evsel = perf_evsel__new(&attr);
+       if (evsel == NULL)
+               goto error;
+
+       /* use strdup() because free(evsel) assumes name is allocated */
+       evsel->name = strdup("dummy");
+       if (!evsel->name)
+               goto error_free;
+
+       list_for_each_entry(pos, &evlist->entries, node) {
+               pos->idx += 1;
+               pos->tracking = false;
Quoted 8 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+       }
+
+       list_add(&evsel->node, &evlist->entries);
+       evsel->idx = 0;
+       evsel->tracking = true;

perf_evlist__set_tracking_event()?

Quoted 3 lines

I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the evsel->idx.

perf_evlist__set_tracking_event() also ensures there is only one tracking event so it is easy to identify. It is the only event with attr->mmap etc set to 1. Then you can use perf_evlist__add().

author maintainer
#7 lore raw

Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events

Hi Adrian,

Quoted 33 lines · Adrian Hunter
Quoted 32 lines · Namhyung Kim

Hi David,

Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?

Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.

Quoted 12 lines

---8<---

Quoted 9 lines
tools/perf/util/evlist.c +0 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist
 *evlist)
        return -ENOMEM;
  }
 
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+       struct perf_event_attr attr = {
+               .type = PERF_TYPE_SOFTWARE,
+               .config = PERF_COUNT_SW_DUMMY,

Probably need .exclude_kernel = 1, here

Ah, right.

Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 13 lines
tools/perf/util/evlist.c +12 -0
+       };
+       struct perf_evsel *evsel, *pos;
+
+       event_attr_init(&attr);
+
+       evsel = perf_evsel__new(&attr);
+       if (evsel == NULL)
+               goto error;
+
+       /* use strdup() because free(evsel) assumes name is allocated */
+       evsel->name = strdup("dummy");
+       if (!evsel->name)

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222

Quoted 10 lines
tools/perf/util/evlist.c +10 -0
+               goto error_free;
+
+       list_for_each_entry(pos, &evlist->entries, node) {
+               pos->idx += 1;
+               pos->tracking = false;
+       }
+
+       list_add(&evsel->node, &evlist->entries);
+       evsel->idx = 0;
+       evsel->tracking = true;

perf_evlist__set_tracking_event()?

Quoted 7 lines
Quoted 3 lines

I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the evsel->idx.

perf_evlist__set_tracking_event() also ensures there is only one tracking event so it is easy to identify. It is the only event with attr->mmap etc set to 1. Then you can use perf_evlist__add().

Well, yes, I think we can put the dummy tracking event anywhere in the evlist with this, but I still slightly prefer put it at a fixed location for a possible code simplification..

Thanks,
Namhyung

reviewer
#8 lore raw

Re: [PATCH 02/37] perf record: Use a software dummy event to track task/mmap events

Quoted 77 lines · Namhyung Kim

Hi Adrian,

Quoted 33 lines · Adrian Hunter
Quoted 32 lines · Namhyung Kim

Hi David,

Quoted 10 lines · David Ahern
Quoted 4 lines · Namhyung Kim

Prepend a software dummy event into evlist to track task/comm/mmap events separately. This is a preparation of multi-file/thread support which will come later.

Are you are making this the first event because of how perf internals are coded -- that the first event tracks tasks events? With the tracking bit in evsel you should not need to do that. Is there another reason?

Yeah, I know the tracking bit can be set to any evsel in the evlist. But I'd like to keep it at a fixed index so that it can be easily identified at later stages (like perf report) too. Ideally, it'd be great if we have a way to distinguish this auto-added dummy tracking event from other (user-added) (dummy?) tracking events if any.

Quoted 12 lines

---8<---

Quoted 9 lines
tools/perf/util/evlist.c +0 -0
index cfbe2b99b9aa..72dff295237e 100644
@@ -193,6 +193,44 @@ int perf_evlist__add_default(struct perf_evlist
 *evlist)
        return -ENOMEM;
  }
 
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 200

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+int perf_evlist__prepend_dummy(struct perf_evlist *evlist)
+{
+       struct perf_event_attr attr = {
+               .type = PERF_TYPE_SOFTWARE,
+               .config = PERF_COUNT_SW_DUMMY,

Probably need .exclude_kernel = 1, here

Ah, right.

Quoted 31 lines
Quoted 30 lines
Quoted 18 lines
Quoted 18 lines
tools/perf/util/evlist.c +17 -0
+       };
+       struct perf_evsel *evsel, *pos;
+
+       event_attr_init(&attr);
+
+       evsel = perf_evsel__new(&attr);
+       if (evsel == NULL)
+               goto error;
+
+       /* use strdup() because free(evsel) assumes name is allocated */
+       evsel->name = strdup("dummy");
+       if (!evsel->name)
+               goto error_free;
+
+       list_for_each_entry(pos, &evlist->entries, node) {
+               pos->idx += 1;
+               pos->tracking = false;
Quoted 8 lines

tools/perf/util/evlist.c int perf_evlist__add_default(struct perf_evlist *evlist) · line 222

Quoted 5 lines
tools/perf/util/evlist.c +5 -0
+       }
+
+       list_add(&evsel->node, &evlist->entries);
+       evsel->idx = 0;
+       evsel->tracking = true;

perf_evlist__set_tracking_event()?

I found that after I wrote this, so yes, it can use the function instead of the oped-code. But the loop traversal is needed anyway to fixup the evsel->idx.

Quoted 7 lines
Quoted 3 lines

perf_evlist__set_tracking_event() also ensures there is only one tracking event so it is easy to identify. It is the only event with attr->mmap etc set to 1. Then you can use perf_evlist__add().

Well, yes, I think we can put the dummy tracking event anywhere in the evlist with this, but I still slightly prefer put it at a fixed location for a possible code simplification..

Probably you should not be checking for the "tracking event" at all on the processing side and instead just have perf report ignore all dummy events. That generalizes things even more.

What is the code simplification?

author maintainer
#9 lore raw

PATCH 3/37 perf tools: Use perf_data_file__fd() consistently

In reply to Namhyung Kim

Do not reference file->fd directly since we want hide the implementation details from outside to support multi-file store.

---
 tools/perf/builtin-inject.c |  9 ++++++---
 tools/perf/builtin-record.c | 14 ++++++++------
 2 files changed, 14 insertions(+), 9 deletions(-)
tools/perf/builtin-inject.c +6 -3
index 84df2deed988..d8b13407594d 100644
@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject)
 		}
 	}
 
-	if (!file_out->is_pipe)
-		lseek(file_out->fd, session->header.data_offset, SEEK_SET);
+	if (!file_out->is_pipe) {
+		lseek(perf_data_file__fd(file_out), session->header.data_offset,
+		      SEEK_SET);
+	}
 
 	ret = perf_session__process_events(session, &inject->tool);
 
@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)
 			perf_header__set_feat(&session->header,
 					      HEADER_BUILD_ID);
 		session->header.data_size = inject->bytes_written;
-		perf_session__write_header(session, session->evlist, file_out->fd, true);
+		perf_session__write_header(session, session->evlist,
+					   perf_data_file__fd(file_out), true);
 	}
 
 	return ret;
tools/perf/builtin-record.c +8 -6
index aa5fa6aabb31..054c6e57d3b9 100644
@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)
 	struct perf_session *session = rec->session;
 	u64 start = session->header.data_offset;
 
-	u64 size = lseek(file->fd, 0, SEEK_CUR);
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		perf_header__clear_feat(&session->header, HEADER_GROUP_DESC);
 
 	if (file->is_pipe) {
-		err = perf_header__write_pipe(file->fd);
+		err = perf_header__write_pipe(perf_data_file__fd(file));
 		if (err < 0)
 			goto out_child;
 	} else {
 		err = perf_session__write_header(session, rec->evlist,
-						 file->fd, false);
+						 perf_data_file__fd(file), false);
 		if (err < 0)
 			goto out_child;
 	}
@@ -397,8 +397,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 			 * return this more properly and also
 			 * propagate errors that now are calling die()
 			 */
-			err = perf_event__synthesize_tracing_data(tool, file->fd, rec->evlist,
-								  process_synthesized_event);
+			err = perf_event__synthesize_tracing_data(tool,
+						perf_data_file__fd(file),
+						rec->evlist,
+						process_synthesized_event);
 			if (err <= 0) {
 				pr_err("Couldn't record tracing data.\n");
 				goto out_child;
@@ -541,7 +543,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		if (!rec->no_buildid)
 			process_buildids(rec);
 		perf_session__write_header(rec->session, rec->evlist,
-					   file->fd, true);
+					   perf_data_file__fd(&rec->file), true);
 	}
 
 out_delete_session:
Signature · 1 line
2.1.3
#10 lore raw

Re: [PATCH 03/37] perf tools: Use perf_data_file__fd() consistently

tools/perf/builtin-inject.c static int __cmd_inject(struct perf_inject *inject) · line 377

Quoted 8 lines · Namhyung Kim
tools/perf/builtin-inject.c +0 -0
index 84df2deed988..d8b13407594d 100644
@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject)
  		}
  	}
 

How about a local variable to make this more readable? fd = perf_data_file__fd(file_out)

Quoted 33 lines
tools/perf/builtin-inject.c +6 -3
-	if (!file_out->is_pipe)
-		lseek(file_out->fd, session->header.data_offset, SEEK_SET);
+	if (!file_out->is_pipe) {
+		lseek(perf_data_file__fd(file_out), session->header.data_offset,
+		      SEEK_SET);
+	}
 
  	ret = perf_session__process_events(session, &inject->tool);
 
@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)
  			perf_header__set_feat(&session->header,
  					      HEADER_BUILD_ID);
  		session->header.data_size = inject->bytes_written;
-		perf_session__write_header(session, session->evlist, file_out->fd, true);
+		perf_session__write_header(session, session->evlist,
+					   perf_data_file__fd(file_out), true);
  	}
 
  	return ret;
tools/perf/builtin-record.c +1 -1
index aa5fa6aabb31..054c6e57d3b9 100644
@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)
  	struct perf_session *session = rec->session;
  	u64 start = session->header.data_offset;
 
-	u64 size = lseek(file->fd, 0, SEEK_CUR);
+	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
  	if (size == 0)
  		return 0;
 

tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 360

Quoted 1 line
tools/perf/builtin-record.c +0 -0
@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
  		perf_header__clear_feat(&session->header, HEADER_GROUP_DESC);

Similarly in this function.

Quoted 27 lines
tools/perf/builtin-record.c +6 -4
  	if (file->is_pipe) {
-		err = perf_header__write_pipe(file->fd);
+		err = perf_header__write_pipe(perf_data_file__fd(file));
  		if (err < 0)
  			goto out_child;
  	} else {
  		err = perf_session__write_header(session, rec->evlist,
-						 file->fd, false);
+						 perf_data_file__fd(file), false);
  		if (err < 0)
  			goto out_child;
  	}
@@ -397,8 +397,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
  			 * return this more properly and also
  			 * propagate errors that now are calling die()
  			 */
-			err = perf_event__synthesize_tracing_data(tool, file->fd, rec->evlist,
-								  process_synthesized_event);
+			err = perf_event__synthesize_tracing_data(tool,
+						perf_data_file__fd(file),
+						rec->evlist,
+						process_synthesized_event);
  			if (err <= 0) {
  				pr_err("Couldn't record tracing data.\n");
  				goto out_child;

tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int argc, const char **argv) · line 549

Quoted 8 lines
tools/perf/builtin-record.c +1 -1
@@ -541,7 +543,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
  		if (!rec->no_buildid)
  			process_buildids(rec);
  		perf_session__write_header(rec->session, rec->evlist,
-					   file->fd, true);
+					   perf_data_file__fd(&rec->file), true);
  	}
 
  out_delete_session:

David

author maintainer
#11 lore raw

Re: [PATCH 03/37] perf tools: Use perf_data_file__fd() consistently

Quoted 13 lines · David Ahern

tools/perf/builtin-inject.c static int __cmd_inject(struct perf_inject *inject) · line 377

Quoted 8 lines · Namhyung Kim
tools/perf/builtin-inject.c +0 -0
index 84df2deed988..d8b13407594d 100644
@@ -375,8 +375,10 @@ static int __cmd_inject(struct perf_inject *inject)
                }
        }
 

How about a local variable to make this more readable? fd = perf_data_file__fd(file_out)

Will do.

Quoted 37 lines
Quoted 37 lines
tools/perf/builtin-inject.c +4 -2
-       if (!file_out->is_pipe)
-               lseek(file_out->fd, session->header.data_offset,
 SEEK_SET);
+       if (!file_out->is_pipe) {
+               lseek(perf_data_file__fd(file_out),
 session->header.data_offset,
+                     SEEK_SET);
+       }
 
        ret = perf_session__process_events(session, &inject->tool);
tools/perf/builtin-inject.c +2 -1
@@ -385,7 +387,8 @@ static int __cmd_inject(struct perf_inject *inject)
                        perf_header__set_feat(&session->header,
                                              HEADER_BUILD_ID);
                session->header.data_size = inject->bytes_written;
-               perf_session__write_header(session, session->evlist,
 file_out->fd, true);
+               perf_session__write_header(session, session->evlist,
+                                          perf_data_file__fd(file_out),
 true);
        }
        return ret;
tools/perf/builtin-record.c +1 -1
index aa5fa6aabb31..054c6e57d3b9 100644
@@ -196,7 +196,7 @@ static int process_buildids(struct record *rec)
        struct perf_session *session = rec->session;
        u64 start = session->header.data_offset;
 
-       u64 size = lseek(file->fd, 0, SEEK_CUR);
+       u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
        if (size == 0)
                return 0;
 
Quoted 7 lines

tools/perf/builtin-record.c static int __cmd_record(struct record *rec, int · line 362

Quoted 4 lines
tools/perf/builtin-record.c +0 -0
@@ -360,12 +360,12 @@ static int __cmd_record(struct record *rec, int
 argc, const char **argv)
                perf_header__clear_feat(&session->header,
 HEADER_GROUP_DESC);

Similarly in this function.

No problem.

Thanks,
Namhyung

author maintainer
#12 lore raw

PATCH 4/37 perf tools: Add multi file interface to perf_data_file

In reply to Namhyung Kim

When multi file storage is enabled, the perf data files will be saved in a directory (default: perf.data.dir) and it'll have a single header file for metadata (task/comm/mmap events and file header) and multiple data files (sample events) like below:

  $ tree perf.data.dir
  perf.data.dir
  |-- perf.data.0
  |-- perf.data.1
  |-- perf.data.2
  |-- perf.data.3
  `-- perf.header

0 directories, 5 files

Existing data file interface supports multi files internally and add new perf_data_file__prepare_write() and perf_data_file__write_multi() functions in order to support multi-file record. Note that multi read interface is not needed since they're accessed via mmap.

---
 tools/perf/util/data.c | 149 +++++++++++++++++++++++++++++++++++++++++++++++--
 tools/perf/util/data.h |  14 +++++
 tools/perf/util/util.c |  43 ++++++++++++++
 tools/perf/util/util.h |   1 +
 4 files changed, 201 insertions(+), 6 deletions(-)
tools/perf/util/data.c +143 -6
index 1921942fc2e0..8dacd34659cc 100644
@@ -4,6 +4,7 @@
 #include <sys/stat.h>
 #include <unistd.h>
 #include <string.h>
+#include <dirent.h>
 
 #include "data.h"
 #include "util.h"
@@ -39,26 +40,97 @@ static int check_backup(struct perf_data_file *file)
 		char oldname[PATH_MAX];
 		snprintf(oldname, sizeof(oldname), "%s.old",
 			 file->path);
-		unlink(oldname);
+
+		if (S_ISDIR(st.st_mode))
+			rm_rf(oldname);
+		else
+			unlink(oldname);
+
 		rename(file->path, oldname);
 	}
 
 	return 0;
 }
 
+static int scandir_filter(const struct dirent *d)
+{
+	return !prefixcmp(d->d_name, "perf.data.");
+}
+
+static int open_file_read_multi(struct perf_data_file *file)
+{
+	int i, n;
+	int ret;
+	struct dirent **list;
+
+	n = scandir(file->path, &list, scandir_filter, versionsort);
+	if (n <= 0) {
+		ret = -errno;
+		pr_err("cannot find multi-data file\n");
+		return ret;
+	}
+
+	file->multi_fd = malloc(n * sizeof(int));
+	if (file->multi_fd == NULL) {
+		free(list);
+		return -ENOMEM;
+	}
+
+	for (i = 0; i < n; i++) {
+		char path[PATH_MAX];
+
+		scnprintf(path, sizeof(path), "%s/%s",
+			  file->path, list[i]->d_name);
+
+		ret = open(path, O_RDONLY);
+		if (ret < 0)
+			goto out_err;
+
+		file->multi_fd[i] = ret;
+	}
+
+	file->nr_multi = n;
+
+	free(list);
+	return 0;
+
+out_err:
+	while (--i >= 0)
+		close(file->multi_fd[i]);
+
+	zfree(&file->multi_fd);
+	free(list);
+	return ret;
+}
+
+static const char *default_data_path(struct perf_data_file *file)
+{
+	return file->is_multi ? "perf.data.dir" : "perf.data";
+}
+
 static int open_file_read(struct perf_data_file *file)
 {
 	struct stat st;
+	char path[PATH_MAX];
 	int fd;
 	char sbuf[STRERR_BUFSIZE];
 
-	fd = open(file->path, O_RDONLY);
+	strcpy(path, file->path);
+	if (file->is_multi) {
+		if (open_file_read_multi(file) < 0)
+			return -1;
+
+		if (path__join(path, sizeof(path), file->path, "perf.header") < 0)
+			return -1;
+	}
+
+	fd = open(path, O_RDONLY);
 	if (fd < 0) {
 		int err = errno;
 
-		pr_err("failed to open %s: %s", file->path,
+		pr_err("failed to open %s: %s", path,
 			strerror_r(err, sbuf, sizeof(sbuf)));
-		if (err == ENOENT && !strcmp(file->path, "perf.data"))
+		if (err == ENOENT && !strcmp(path, default_data_path(file)))
 			pr_err("  (try 'perf record' first)");
 		pr_err("\n");
 		return -err;
@@ -90,12 +162,26 @@ static int open_file_read(struct perf_data_file *file)
 static int open_file_write(struct perf_data_file *file)
 {
 	int fd;
+	char path[PATH_MAX];
 	char sbuf[STRERR_BUFSIZE];
 
 	if (check_backup(file))
 		return -1;
 
-	fd = open(file->path, O_CREAT|O_RDWR|O_TRUNC, S_IRUSR|S_IWUSR);
+	strcpy(path, file->path);
+
+	if (file->is_multi) {
+		if (mkdir(file->path, S_IRWXU) < 0) {
+			pr_err("cannot create data directory `%s': %s\n",
+			       file->path, strerror_r(errno, sbuf, sizeof(sbuf)));
+			return -1;
+		}
+
+		if (path__join(path, sizeof(path), file->path, "perf.header") < 0)
+			return -1;
+	}
+
+	fd = open(path, O_CREAT|O_RDWR|O_TRUNC, S_IRUSR|S_IWUSR);
 
 	if (fd < 0)
 		pr_err("failed to open %s : %s\n", file->path,
@@ -121,18 +207,69 @@ int perf_data_file__open(struct perf_data_file *file)
 		return 0;
 
 	if (!file->path)
-		file->path = "perf.data";
+		file->path = default_data_path(file);
 
 	return open_file(file);
 }
 
 void perf_data_file__close(struct perf_data_file *file)
 {
+	if (file->is_multi) {
+		int i;
+
+		for (i = 0; i < file->nr_multi; i++)
+			close(file->multi_fd[i]);
+
+		zfree(&file->multi_fd);
+	}
+
 	close(file->fd);
 }
 
+int perf_data_file__prepare_write(struct perf_data_file *file, int nr)
+{
+	int i;
+	int ret;
+	char path[PATH_MAX];
+
+	if (!file->is_multi)
+		return 0;
+
+	file->multi_fd = malloc(nr * sizeof(int));
+	if (file->multi_fd == NULL)
+		return -ENOMEM;
+
+	for (i = 0; i < nr; i++) {
+		scnprintf(path, sizeof(path), "%s/perf.data.%d", file->path, i);
+		ret = open(path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);
+		if (ret < 0)
+			goto out_err;
+
+		file->multi_fd[i] = ret;
+	}
+
+	file->nr_multi = nr;
+	return 0;
+
+out_err:
+	while (--i >= 0)
+		close(file->multi_fd[i]);
+
+	zfree(&file->multi_fd);
+	return ret;
+}
+
 ssize_t perf_data_file__write(struct perf_data_file *file,
 			      void *buf, size_t size)
 {
 	return writen(file->fd, buf, size);
 }
+
+ssize_t perf_data_file__write_multi(struct perf_data_file *file,
+				    void *buf, size_t size, int idx)
+{
+	if (!file->is_multi)
+		return -1;
+
+	return writen(file->multi_fd[idx], buf, size);
+}
tools/perf/util/data.h +14 -0
index 2b15d0c95c7f..f5c229166614 100644
@@ -11,6 +11,9 @@ enum perf_data_mode {
 struct perf_data_file {
 	const char		*path;
 	int			 fd;
+	int			 nr_multi;
+	int			*multi_fd;
+	bool			 is_multi;
 	bool			 is_pipe;
 	bool			 force;
 	unsigned long		 size;
@@ -37,6 +40,14 @@ static inline int perf_data_file__fd(struct perf_data_file *file)
 	return file->fd;
 }
 
+static inline int perf_data_file__multi_fd(struct perf_data_file *file, int idx)
+{
+	if (!file->is_multi || idx >= file->nr_multi)
+		return -1;
+
+	return file->multi_fd[idx];
+}
+
 static inline unsigned long perf_data_file__size(struct perf_data_file *file)
 {
 	return file->size;
@@ -46,5 +57,8 @@ int perf_data_file__open(struct perf_data_file *file);
 void perf_data_file__close(struct perf_data_file *file);
 ssize_t perf_data_file__write(struct perf_data_file *file,
 			      void *buf, size_t size);
+int perf_data_file__prepare_write(struct perf_data_file *file, int nr);
+ssize_t perf_data_file__write_multi(struct perf_data_file *file,
+				    void *buf, size_t size, int idx);
 
 #endif /* __PERF_DATA_H */
tools/perf/util/util.c +43 -0

49 lines not on this page · raw message

tools/perf/util/util.h +1 -0
index abc445ee4f60..d75975a71d2c 100644
@@ -248,6 +248,7 @@ static inline int sane_case(int x, int high)
 }
 
 int mkdir_p(char *path, mode_t mode);
+int rm_rf(char *path);
 int copyfile(const char *from, const char *to);
 int copyfile_mode(const char *from, const char *to, mode_t mode);
 
Signature · 1 line
2.1.3
#13 lore raw

Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file

On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:

SNIP

Quoted 15 lines

#endif /* __PERF_DATA_H */

tools/perf/util/util.c +6 -0
index d5eab3f3323f..a5046d52e311 100644
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
 	return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+	DIR *dir;
+	int ret = 0;
+	struct dirent *d;
+	char namebuf[PATH_MAX];

tools/perf/util/util.c int mkdir_p(char *path, mode_t mode) · line 90

Quoted 10 lines
tools/perf/util/util.c +10 -0
+
+	dir = opendir(path);
+	if (dir == NULL)
+		return 0;
+
+	while ((d = readdir(dir)) != NULL && !ret) {
+		struct stat statbuf;
+
+		if (d->d_name[0] == '.')
+			continue;

Could you check for '.' and for '..' to support '.*' removal?

I know tha we will probably not have any '.*' files in perf.data.dir, but this function could be used later like for total cache clean ;-)

also please submit 'rm_rf' in separate patch

thanks,
jirka

author maintainer
#14 lore raw

Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file

Hi Jiri,

Quoted 32 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:

SNIP

Quoted 15 lines

#endif /* __PERF_DATA_H */

tools/perf/util/util.c +6 -0
index d5eab3f3323f..a5046d52e311 100644
@@ -72,6 +72,49 @@ int mkdir_p(char *path, mode_t mode)
      return (stat(path, &st) && mkdir(path, mode)) ? -1 : 0;
 }
 
+int rm_rf(char *path)
+{
+     DIR *dir;
+     int ret = 0;
+     struct dirent *d;
+     char namebuf[PATH_MAX];

tools/perf/util/util.c int mkdir_p(char *path, mode_t mode) · line 90

Quoted 10 lines
tools/perf/util/util.c +10 -0
+
+     dir = opendir(path);
+     if (dir == NULL)
+             return 0;
+
+     while ((d = readdir(dir)) != NULL && !ret) {
+             struct stat statbuf;
+
+             if (d->d_name[0] == '.')
+                     continue;

Could you check for '.' and for '..' to support '.*' removal?

Quoted 2 lines

I know tha we will probably not have any '.*' files in perf.data.dir, but this function could be used later like for total cache clean ;-)

Ah, okay.

Quoted 1 line

also please submit 'rm_rf' in separate patch

Will do.

Thanks,
Namhyung

#15 lore raw

Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file

In reply to Namhyung Kim

On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:

SNIP

tools/perf/util/data.c static int check_backup(struct perf_data_file *file) · line 118

Quoted 10 lines
tools/perf/util/data.c +3 -1
+
 static int open_file_read(struct perf_data_file *file)
 {
 	struct stat st;
+	char path[PATH_MAX];
 	int fd;
 	char sbuf[STRERR_BUFSIZE];
 
-	fd = open(file->path, O_RDONLY);
+	strcpy(path, file->path);

strncpy ?

jirka

author maintainer
#16 lore raw

Re: [PATCH 04/37] perf tools: Add multi file interface to perf_data_file

Quoted 12 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:00PM +0900, Namhyung Kim wrote:

SNIP

Quoted 8 lines
tools/perf/util/data.c +2 -0
+
 static int open_file_read(struct perf_data_file *file)
 {
      struct stat st;
+     char path[PATH_MAX];
      int fd;
      char sbuf[STRERR_BUFSIZE];
Quoted 4 lines

tools/perf/util/data.c static int check_backup(struct perf_data_file *file) · line 118

Quoted 2 lines
tools/perf/util/data.c +1 -1
-     fd = open(file->path, O_RDONLY);
+     strcpy(path, file->path);

strncpy ?

Will change. :)

Thanks,
Namhyung

author maintainer
#17 lore raw

PATCH 5/37 perf tools: Create separate mmap for dummy tracking event

In reply to Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.

---
 tools/perf/builtin-record.c |   9 +++-
 tools/perf/util/evlist.c    | 104 +++++++++++++++++++++++++++++++++++---------
 tools/perf/util/evlist.h    |  11 ++++-
 3 files changed, 102 insertions(+), 22 deletions(-)
tools/perf/builtin-record.c +8 -1
index 054c6e57d3b9..129fab35fdc5 100644
@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 
 static int record__mmap_read(struct record *rec, int idx)
 {
-	struct perf_mmap *md = &rec->evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(rec->evlist, idx);
 	unsigned int head = perf_mmap__read_head(md);
 	unsigned int old = md->prev;
 	unsigned char *data = md->base + page_size;
@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	}
 
 	md->prev = old;
+
 	perf_evlist__mmap_consume(rec->evlist, idx);
 out:
 	return rc;
@@ -263,6 +264,12 @@ static int record__mmap_read_all(struct record *rec)
 				goto out;
 			}
 		}
+		if (rec->evlist->track_mmap) {
+			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
+				rc = -1;
+				goto out;
+			}
+		}
 	}
 
 	/*
tools/perf/util/evlist.c +84 -20

185 lines not on this page · raw message

tools/perf/util/evlist.h +10 -1
index b974bddf6b8b..b7f54b8577f7 100644
@@ -48,11 +48,14 @@ struct perf_evlist {
 	bool		 overwrite;
 	struct fdarray	 pollfd;
 	struct perf_mmap *mmap;
+	struct perf_mmap *track_mmap;
 	struct thread_map *threads;
 	struct cpu_map	  *cpus;
 	struct perf_evsel *selected;
 };
 
+#define TRACK_MMAP_SIZE  (((128 * 1024 / page_size) + 1) * page_size)
+
 struct perf_evsel_str_handler {
 	const char *name;
 	void	   *handler;
@@ -100,8 +103,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);
 struct perf_sample_id *perf_evlist__id2sid(struct perf_evlist *evlist, u64 id);
 
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}
+
 /**
  * __evlist__for_each - iterate thru all the evsels
  * @list: list_head instance to iterate
Signature · 1 line
2.1.3
#18 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

Quoted 8 lines · Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.

maybe this needs to be applied after next patch?
perf tools: Introduce perf_evlist__mmap_multi

I'm getting compile error:

[jolsa@krava perf]$ make JOBS=1
BUILD: Doing 'make -j1' parallel build
CC util/evlist.o
util/evlist.c: In function ‘perf_evlist__mmap_per_evsel’:
util/evlist.c:937:9: error: ‘struct mmap_params’ has no member named ‘track’

   if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
         ^
util/evlist.c: In function ‘perf_evlist__mmap’:
util/evlist.c:1092:62: error: ‘use_track_mmap’ undeclared (first use in this function)
  if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0)
                                                              ^

util/evlist.c:1092:62: note: each undeclared identifier is reported only once for each function it appears in make[1]: *** [util/evlist.o] Error 1
make: *** [all] Error 2

jirka

author maintainer
#19 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

Quoted 5 lines · Jiri Olsa
Quoted 4 lines · Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Quoted 7 lines
Quoted 4 lines

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.

maybe this needs to be applied after next patch?
perf tools: Introduce perf_evlist__mmap_multi

Oops, sorry. it seems the code was mixed after huge number of rebase.. The commit order is right but it just has a wrong hunk.

Quoted 14 lines

I'm getting compile error:

[jolsa@krava perf]$ make JOBS=1
BUILD: Doing 'make -j1' parallel build
CC util/evlist.o
util/evlist.c: In function ‘perf_evlist__mmap_per_evsel’:
util/evlist.c:937:9: error: ‘struct mmap_params’ has no member named ‘track’

   if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
         ^
util/evlist.c: In function ‘perf_evlist__mmap’:
util/evlist.c:1092:62: error: ‘use_track_mmap’ undeclared (first use in this function)
  if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0)
                                                              ^
Quoted 3 lines

util/evlist.c:1092:62: note: each undeclared identifier is reported only once for each function it appears in make[1]: *** [util/evlist.o] Error 1
make: *** [all] Error 2

Will fix, thanks
Namhyung

#20 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

In reply to Namhyung Kim

On Wed, Dec 24, 2014 at 04:15:01PM +0900, Namhyung Kim wrote:

SNIP

Quoted 9 lines

union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);

tools/perf/util/evlist.h +1 -1
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);

tools/perf/util/evlist.h bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str); · line 221

Quoted 8 lines
tools/perf/util/evlist.h +5 -0
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}

hum, whats the logic with negative numbers in here? you still access track_mmap array with this index no?

thanks,
jirka

author maintainer
#21 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

Quoted 16 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:01PM +0900, Namhyung Kim wrote:

SNIP

Quoted 12 lines

union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);

tools/perf/util/evlist.h +1 -1
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
                         struct perf_evsel *move_evsel);
 
Quoted 8 lines

tools/perf/util/evlist.h bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str); · line 221

Quoted 5 lines
tools/perf/util/evlist.h +5 -0
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+     return -idx - 1;
+}

hum, whats the logic with negative numbers in here? you still access track_mmap array with this index no?

For each index (per-cpu or per-task depends on user input), it now has two mmaps - one for normal (sampling) mmap and another for track (metadata) mmap. So I wanted to distinguish them by using positive and negative number. But simply changing the sign did not work due to 0 index. And the subtracting by 1 works nicely for both direction.

0 => -1 => 0
1 => -2 => 1
2 => -3 => 2
3 => -4 => 3

So when it sees a negative index, it'll convert it using track_mmap_idx() to get a matching (track) mmap.

Thanks,
Namhyung

#22 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

In reply to Namhyung Kim
Quoted 6 lines · Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I

Have you tried stress tests like a kernel compile with -j Ncpus (Ncpus = 16 or higher)?

David

author maintainer
#23 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

Quoted 6 lines · David Ahern
Quoted 5 lines · Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Quoted 6 lines
Quoted 2 lines

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I

Have you tried stress tests like a kernel compile with -j Ncpus (Ncpus = 16 or higher)?

Yep, the test results in the cover letter were recorded kernel build with -j 20 on my development machine.

Thanks,
Namhyung

reviewer
#24 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

In reply to Namhyung Kim
Quoted 201 lines · Namhyung Kim

When multi file support is enabled, a dummy tracking event will be used to track metadata (like task, comm and mmap events) for a session and actual samples will be recorded in separate files.

Provide separate mmap to the dummy tracking event. The size is fixed to 128KiB (+ 1 page) as the event rate will be lower than samples. I originally wanted to use a single mmap for this but cross-cpu sharing is prohibited so it's per-cpu (or per-task) like normal mmaps.

---
 tools/perf/builtin-record.c |   9 +++-
 tools/perf/util/evlist.c    | 104 +++++++++++++++++++++++++++++++++++---------
 tools/perf/util/evlist.h    |  11 ++++-
 3 files changed, 102 insertions(+), 22 deletions(-)
tools/perf/builtin-record.c +8 -1
index 054c6e57d3b9..129fab35fdc5 100644
@@ -69,7 +69,7 @@ static int process_synthesized_event(struct perf_tool *tool,
 
 static int record__mmap_read(struct record *rec, int idx)
 {
-	struct perf_mmap *md = &rec->evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(rec->evlist, idx);
 	unsigned int head = perf_mmap__read_head(md);
 	unsigned int old = md->prev;
 	unsigned char *data = md->base + page_size;
@@ -105,6 +105,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	}
 
 	md->prev = old;
+
 	perf_evlist__mmap_consume(rec->evlist, idx);
 out:
 	return rc;
@@ -263,6 +264,12 @@ static int record__mmap_read_all(struct record *rec)
 				goto out;
 			}
 		}
+		if (rec->evlist->track_mmap) {
+			if (record__mmap_read(rec, track_mmap_idx(i)) != 0) {
+				rc = -1;
+				goto out;
+			}
+		}
 	}
 
 	/*
tools/perf/util/evlist.c +64 -17
index 72dff295237e..d99343b988fe 100644
@@ -27,6 +27,7 @@
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx);
 static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx);
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx);
 
 #define FD(e, x, y) (*(int *)xyarray__entry(e->fd, x, y))
 #define SID(e, x, y) xyarray__entry(e->sample_id, x, y)
@@ -735,22 +736,39 @@ static bool perf_mmap__empty(struct perf_mmap *md)
 	return perf_mmap__read_head(md) != md->prev;
 }
 
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx)
+{
+	if (idx >= 0)
+		return &evlist->mmap[idx];
+	else
+		return &evlist->track_mmap[track_mmap_idx(idx)];
+}
+
 static void perf_evlist__mmap_get(struct perf_evlist *evlist, int idx)
 {
-	++evlist->mmap[idx].refcnt;
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	++md->refcnt;
 }
 
 static void perf_evlist__mmap_put(struct perf_evlist *evlist, int idx)
 {
-	BUG_ON(evlist->mmap[idx].refcnt == 0);
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
+
+	BUG_ON(md->refcnt == 0);
+
+	if (--md->refcnt != 0)
+		return;
 
-	if (--evlist->mmap[idx].refcnt == 0)
+	if (idx >= 0)
 		__perf_evlist__munmap(evlist, idx);
+	else
+		__perf_evlist__munmap_track(evlist, track_mmap_idx(idx));
 }
 
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx)
 {
-	struct perf_mmap *md = &evlist->mmap[idx];
+	struct perf_mmap *md = perf_evlist__mmap_desc(evlist, idx);
 
 	if (!evlist->overwrite) {
 		unsigned int old = md->prev;
@@ -771,6 +789,15 @@ static void __perf_evlist__munmap(struct perf_evlist *evlist, int idx)
 	}
 }
 
+static void __perf_evlist__munmap_track(struct perf_evlist *evlist, int idx)
+{
+	if (evlist->track_mmap[idx].base != NULL) {
+		munmap(evlist->track_mmap[idx].base, TRACK_MMAP_SIZE);
+		evlist->track_mmap[idx].base = NULL;
+		evlist->track_mmap[idx].refcnt = 0;
+	}
+}
+
 void perf_evlist__munmap(struct perf_evlist *evlist)
 {
 	int i;
@@ -782,23 +809,43 @@ void perf_evlist__munmap(struct perf_evlist *evlist)
 		__perf_evlist__munmap(evlist, i);
 
 	zfree(&evlist->mmap);
+
+	if (evlist->track_mmap == NULL)
+		return;
+
+	for (i = 0; i < evlist->nr_mmaps; i++)
+		__perf_evlist__munmap_track(evlist, i);
+
+	zfree(&evlist->track_mmap);
 }
 
-static int perf_evlist__alloc_mmap(struct perf_evlist *evlist)
+static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 {
 	evlist->nr_mmaps = cpu_map__nr(evlist->cpus);
 	if (cpu_map__empty(evlist->cpus))
 		evlist->nr_mmaps = thread_map__nr(evlist->threads);
 	evlist->mmap = zalloc(evlist->nr_mmaps * sizeof(struct perf_mmap));
-	return evlist->mmap != NULL ? 0 : -ENOMEM;
+	if (evlist->mmap == NULL)
+		return -ENOMEM;
+
+	if (track_mmap) {
+		evlist->track_mmap = calloc(evlist->nr_mmaps,
+					    sizeof(struct perf_mmap));
+		if (evlist->track_mmap == NULL) {
+			zfree(&evlist->mmap);
+			return -ENOMEM;
+		}
+	}
+	return 0;
 }
 
 struct mmap_params {
-	int prot;
-	int mask;
+	int	prot;
+	size_t	len;
 };
 
-static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
+static int __perf_evlist__mmap(struct perf_evlist *evlist __maybe_unused,
+			       struct perf_mmap *pmmap,
 			       struct mmap_params *mp, int fd)
 {
 	/*
@@ -814,15 +861,14 @@ static int __perf_evlist__mmap(struct perf_evlist *evlist, int idx,
 	 * evlist layer can't just drop it when filtering events in
 	 * perf_evlist__filter_pollfd().
 	 */
-	evlist->mmap[idx].refcnt = 2;
-	evlist->mmap[idx].prev = 0;
-	evlist->mmap[idx].mask = mp->mask;
-	evlist->mmap[idx].base = mmap(NULL, evlist->mmap_len, mp->prot,
-				      MAP_SHARED, fd, 0);
-	if (evlist->mmap[idx].base == MAP_FAILED) {
+	pmmap->refcnt = 2;
+	pmmap->prev = 0;
+	pmmap->mask = mp->len - page_size - 1;
+	pmmap->base = mmap(NULL, mp->len, mp->prot, MAP_SHARED, fd, 0);
+	if (pmmap->base == MAP_FAILED) {
 		pr_debug2("failed to mmap perf event ring buffer, error %d\n",
 			  errno);
-		evlist->mmap[idx].base = NULL;
+		pmmap->base = NULL;
 		return -1;
 	}
 
@@ -843,9 +889,22 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (*output == -1) {
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			struct mmap_params track_mp = {

tools/perf/util/evlist.c static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, · line 903

Quoted 10 lines
tools/perf/util/evlist.c +10 -0
+				.prot	= mp->prot,
+				.len	= TRACK_MMAP_SIZE,
+			};
+
+			if (__perf_evlist__mmap(evlist, &evlist->track_mmap[idx],
+						&track_mp, fd) < 0)
+				return -1;
+
+			/* mark idx as track mmap idx (negative) */
+			idx = track_mmap_idx(idx);

Do you not still need to do SET_OUTPUT when there are multiple cpus and multiple pids?

Quoted 81 lines
tools/perf/util/evlist.c +10 -3
+		} else if (*output == -1) {
 			*output = fd;
-			if (__perf_evlist__mmap(evlist, idx, mp, *output) < 0)
+			if (__perf_evlist__mmap(evlist, &evlist->mmap[idx],
+						mp, *output) < 0)
 				return -1;
 		} else {
 			if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, *output) != 0)
@@ -874,6 +933,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 			perf_evlist__set_sid_idx(evlist, evsel, idx, cpu,
 						 thread);
 		}
+
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
+			/* restore idx as normal idx (positive) */
+			idx = track_mmap_idx(idx);
+		}
 	}
 
 	return 0;
@@ -1025,7 +1089,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
 	};
 
-	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist) < 0)
+	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0)
 		return -ENOMEM;
 
 	if (evlist->pollfd.entries == NULL && perf_evlist__alloc_pollfd(evlist) < 0)
@@ -1034,7 +1098,7 @@ int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
 	evlist->overwrite = overwrite;
 	evlist->mmap_len = perf_evlist__mmap_size(pages);
 	pr_debug("mmap size %zuB\n", evlist->mmap_len);
-	mp.mask = evlist->mmap_len - page_size - 1;
+	mp.len = evlist->mmap_len;
 
 	evlist__for_each(evlist, evsel) {
 		if ((evsel->attr.read_format & PERF_FORMAT_ID) &&
tools/perf/util/evlist.h +10 -1
index b974bddf6b8b..b7f54b8577f7 100644
@@ -48,11 +48,14 @@ struct perf_evlist {
 	bool		 overwrite;
 	struct fdarray	 pollfd;
 	struct perf_mmap *mmap;
+	struct perf_mmap *track_mmap;
 	struct thread_map *threads;
 	struct cpu_map	  *cpus;
 	struct perf_evsel *selected;
 };
 
+#define TRACK_MMAP_SIZE  (((128 * 1024 / page_size) + 1) * page_size)
+
 struct perf_evsel_str_handler {
 	const char *name;
 	void	   *handler;
@@ -100,8 +103,8 @@ struct perf_evsel *perf_evlist__id2evsel(struct perf_evlist *evlist, u64 id);
 struct perf_sample_id *perf_evlist__id2sid(struct perf_evlist *evlist, u64 id);
 
 union perf_event *perf_evlist__mmap_read(struct perf_evlist *evlist, int idx);
-
 void perf_evlist__mmap_consume(struct perf_evlist *evlist, int idx);
+struct perf_mmap *perf_evlist__mmap_desc(struct perf_evlist *evlist, int idx);
 
 int perf_evlist__open(struct perf_evlist *evlist);
 void perf_evlist__close(struct perf_evlist *evlist);
@@ -211,6 +214,12 @@ bool perf_evlist__can_select_event(struct perf_evlist *evlist, const char *str);
 void perf_evlist__to_front(struct perf_evlist *evlist,
 			   struct perf_evsel *move_evsel);
 
+/* convert from/to negative idx for track mmaps */
+static inline int track_mmap_idx(int idx)
+{
+	return -idx - 1;
+}
+
 /**
  * __evlist__for_each - iterate thru all the evsels
  * @list: list_head instance to iterate
author maintainer
#25 lore raw

Re: [PATCH 05/37] perf tools: Create separate mmap for dummy tracking event

Quoted 4 lines · Adrian Hunter
Quoted 4 lines
tools/perf/util/evlist.c +0 -0
@@ -843,9 +889,22 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
Quoted 16 lines

tools/perf/util/evlist.c static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx, · line 903

Quoted 13 lines
tools/perf/util/evlist.c +12 -1
-		if (*output == -1) {
+		if (perf_evsel__is_dummy_tracking(evsel)) {
+			struct mmap_params track_mp = {
+				.prot	= mp->prot,
+				.len	= TRACK_MMAP_SIZE,
+			};
+
+			if (__perf_evlist__mmap(evlist, &evlist->track_mmap[idx],
+						&track_mp, fd) < 0)
+				return -1;
+
+			/* mark idx as track mmap idx (negative) */
+			idx = track_mmap_idx(idx);

Do you not still need to do SET_OUTPUT when there are multiple cpus and multiple pids?

You're right. I just considered simple cases, will fix.

Thanks,
Namhyung

Quoted 22 lines
Quoted 22 lines
tools/perf/util/evlist.c +8 -1
+		} else if (*output == -1) {
 			*output = fd;
-			if (__perf_evlist__mmap(evlist, idx, mp, *output) < 0)
+			if (__perf_evlist__mmap(evlist, &evlist->mmap[idx],
+						mp, *output) < 0)
 				return -1;
 		} else {
 			if (ioctl(fd, PERF_EVENT_IOC_SET_OUTPUT, *output) != 0)
@@ -874,6 +933,11 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 			perf_evlist__set_sid_idx(evlist, evsel, idx, cpu,
 						 thread);
 		}
+
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
+			/* restore idx as normal idx (positive) */
+			idx = track_mmap_idx(idx);
+		}
 	}
 
 	return 0;
author maintainer
#26 lore raw

PATCH 6/37 perf tools: Introduce perf_evlist__mmap_multi()

In reply to Namhyung Kim

The perf_evlist__mmap_multi function creates data mmaps and optionally tracking mmaps for events. It'll be used for perf record to save tracking events in a separate files. Checking dummy tracking event in perf_evlist__mmap() alone is not enough as users can specify the dummy event at first (like in keep tracking testcase) without the multi-file option.

---
 tools/perf/builtin-record.c |  3 ++-
 tools/perf/util/evlist.c    | 11 +++++++----
 tools/perf/util/evlist.h    | 10 ++++++++--
 3 files changed, 17 insertions(+), 7 deletions(-)
tools/perf/builtin-record.c +2 -1
index 129fab35fdc5..8c91f25b81f6 100644
@@ -169,7 +169,8 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
-	if (perf_evlist__mmap(evlist, opts->mmap_pages, false) < 0) {
+	if (perf_evlist__mmap_multi(evlist, opts->mmap_pages, false,
+				    opts->multi_file) < 0) {
 		if (errno == EPERM) {
 			pr_err("Permission error mapping pages.\n"
 			       "Consider increasing "
tools/perf/util/evlist.c +7 -4
index d99343b988fe..010188939104 100644
@@ -841,6 +841,7 @@ static int perf_evlist__alloc_mmap(struct perf_evlist *evlist, bool track_mmap)
 
 struct mmap_params {
 	int	prot;
+	bool	track;
 	size_t	len;
 };
 
@@ -889,7 +890,7 @@ static int perf_evlist__mmap_per_evsel(struct perf_evlist *evlist, int idx,
 
 		fd = FD(evsel, cpu, thread);
 
-		if (perf_evsel__is_dummy_tracking(evsel)) {
+		if (mp->track && perf_evsel__is_dummy_tracking(evsel)) {
 			struct mmap_params track_mp = {
 				.prot	= mp->prot,
 				.len	= TRACK_MMAP_SIZE,
@@ -1068,10 +1069,11 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
 }
 
 /**
- * perf_evlist__mmap - Create mmaps to receive events.
+ * perf_evlist__mmap_multi - Create mmaps to receive events.
  * @evlist: list of events
  * @pages: map length in pages
  * @overwrite: overwrite older events?
+ * @use_track_mmap: use another mmaps to track meta events
  *
  * If @overwrite is %false the user needs to signal event consumption using
  * perf_mmap__write_tail().  Using perf_evlist__mmap_read() does this
@@ -1079,14 +1081,15 @@ int perf_evlist__parse_mmap_pages(const struct option *opt, const char *str,
  *
  * Return: %0 on success, negative error code otherwise.
  */
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite)
+int perf_evlist__mmap_multi(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap)
 {
 	struct perf_evsel *evsel;
 	const struct cpu_map *cpus = evlist->cpus;
 	const struct thread_map *threads = evlist->threads;
 	struct mmap_params mp = {
 		.prot = PROT_READ | (overwrite ? 0 : PROT_WRITE),
+		.track = use_track_mmap,
 	};
 
 	if (evlist->mmap == NULL && perf_evlist__alloc_mmap(evlist, use_track_mmap) < 0)
tools/perf/util/evlist.h +8 -2
index b7f54b8577f7..65c1aea6a3a4 100644
@@ -127,10 +127,16 @@ int perf_evlist__parse_mmap_pages(const struct option *opt,
 				  const char *str,
 				  int unset);
 
-int perf_evlist__mmap(struct perf_evlist *evlist, unsigned int pages,
-		      bool overwrite);
+int perf_evlist__mmap_multi(struct perf_evlist *evlist, unsigned int pages,
+			    bool overwrite, bool use_track_mmap);
 void perf_evlist__munmap(struct perf_evlist *evlist);
 
+static inline int perf_evlist__mmap(struct perf_evlist *evlist,
+				    unsigned int pages, bool overwrite)
+{
+	return perf_evlist__mmap_multi(evlist, pages, overwrite, false);
+}
+
 void perf_evlist__disable(struct perf_evlist *evlist);
 void perf_evlist__enable(struct perf_evlist *evlist);
 
Signature · 1 line
2.1.3
author maintainer
#27 lore raw

PATCH 7/37 perf tools: Do not use __perf_session__process_events() directly

In reply to Namhyung Kim

It's only used for perf record to process build-id because its file size it's not fixed at this time due to remaining header features. However data offset and size is available so that we can use the perf_session__process_events() once we set the file size as the current offset like for now.

It turns out that we can staticize the function again as it's the only user and add multi file support in a single place.

---
 tools/perf/builtin-record.c | 7 +++----
 tools/perf/util/session.c   | 6 +++---
 tools/perf/util/session.h   | 3 ---
 3 files changed, 6 insertions(+), 10 deletions(-)
tools/perf/builtin-record.c +3 -4
index 8c91f25b81f6..4f97657f14e7 100644
@@ -196,12 +196,13 @@ static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
-	u64 start = session->header.data_offset;
 
 	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
+	file->size = size;
+
 	/*
 	 * During this process, it'll load kernel map and replace the
 	 * dso->long_name to a real pathname it found.  In this case
@@ -213,9 +214,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return __perf_session__process_events(session, start,
-					      size - start,
-					      size, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
tools/perf/util/session.c +3 -3
index 6ac62ae6b8fa..88aa2f09df93 100644
@@ -1252,9 +1252,9 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size,
-				   u64 file_size, struct perf_tool *tool)
+static int __perf_session__process_events(struct perf_session *session,
+					  u64 data_offset, u64 data_size,
+					  u64 file_size, struct perf_tool *tool)
 {
 	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
tools/perf/util/session.h +0 -3
index dc26ebf60fe4..6d663dc76404 100644
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     union perf_event **event_ptr,
 			     struct perf_sample *sample);
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size, u64 size,
-				   struct perf_tool *tool);
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
 
Signature · 1 line
2.1.3
Acked-by on v1 7/37
#28 lore raw

Re: [PATCH 07/37] perf tools: Do not use __perf_session__process_events() directly

Quoted 10 lines · Namhyung Kim

It's only used for perf record to process build-id because its file size it's not fixed at this time due to remaining header features. However data offset and size is available so that we can use the perf_session__process_events() once we set the file size as the current offset like for now.

It turns out that we can staticize the function again as it's the only user and add multi file support in a single place.

Quoted 70 lines
---
 tools/perf/builtin-record.c | 7 +++----
 tools/perf/util/session.c   | 6 +++---
 tools/perf/util/session.h   | 3 ---
 3 files changed, 6 insertions(+), 10 deletions(-)
tools/perf/builtin-record.c +3 -4
index 8c91f25b81f6..4f97657f14e7 100644
@@ -196,12 +196,13 @@ static int process_buildids(struct record *rec)
 {
 	struct perf_data_file *file  = &rec->file;
 	struct perf_session *session = rec->session;
-	u64 start = session->header.data_offset;
 
 	u64 size = lseek(perf_data_file__fd(file), 0, SEEK_CUR);
 	if (size == 0)
 		return 0;
 
+	file->size = size;
+
 	/*
 	 * During this process, it'll load kernel map and replace the
 	 * dso->long_name to a real pathname it found.  In this case
@@ -213,9 +214,7 @@ static int process_buildids(struct record *rec)
 	 */
 	symbol_conf.ignore_vmlinux_buildid = true;
 
-	return __perf_session__process_events(session, start,
-					      size - start,
-					      size, &build_id__mark_dso_hit_ops);
+	return perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 }
 
 static void perf_event__synthesize_guest_os(struct machine *machine, void *data)
tools/perf/util/session.c +3 -3
index 6ac62ae6b8fa..88aa2f09df93 100644
@@ -1252,9 +1252,9 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size,
-				   u64 file_size, struct perf_tool *tool)
+static int __perf_session__process_events(struct perf_session *session,
+					  u64 data_offset, u64 data_size,
+					  u64 file_size, struct perf_tool *tool)
 {
 	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
tools/perf/util/session.h +0 -3
index dc26ebf60fe4..6d663dc76404 100644
@@ -49,9 +49,6 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 			     union perf_event **event_ptr,
 			     struct perf_sample *sample);
 
-int __perf_session__process_events(struct perf_session *session,
-				   u64 data_offset, u64 data_size, u64 size,
-				   struct perf_tool *tool);
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
 
Signature · 1 line
2.1.3
author maintainer
#29 lore raw

PATCH 8/37 perf tools: Handle multi-file session properly

In reply to Namhyung Kim

When perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.

---
 tools/perf/util/data.c    | 17 +++++++++++++++++
 tools/perf/util/session.c | 41 +++++++++++++++++++++++++++++++----------
 2 files changed, 48 insertions(+), 10 deletions(-)
tools/perf/util/data.c +17 -0
index 8dacd34659cc..b6f7cdc4a39f 100644
@@ -52,6 +52,21 @@ static int check_backup(struct perf_data_file *file)
 	return 0;
 }
 
+static void check_multi(struct perf_data_file *file)
+{
+	struct stat st;
+
+	/*
+	 * For write, it'll be determined by user (perf record -M)
+	 * whether to enable multi file data storage.
+	 */
+	if (perf_data_file__is_write(file))
+		return;
+
+	if (!stat(file->path, &st) && S_ISDIR(st.st_mode))
+		file->is_multi = true;
+}
+
 static int scandir_filter(const struct dirent *d)
 {
 	return !prefixcmp(d->d_name, "perf.data.");
@@ -206,6 +221,8 @@ int perf_data_file__open(struct perf_data_file *file)
 	if (check_pipe(file))
 		return 0;
 
+	check_multi(file);
+
 	if (!file->path)
 		file->path = default_data_path(file);
 
tools/perf/util/session.c +31 -10

60 lines not on this page · raw message

Signature · 1 line
2.1.3
#30 lore raw

Re: [PATCH 08/37] perf tools: Handle multi-file session properly

Quoted 81 lines · Namhyung Kim

When perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.

---
 tools/perf/util/data.c    | 17 +++++++++++++++++
 tools/perf/util/session.c | 41 +++++++++++++++++++++++++++++++----------
 2 files changed, 48 insertions(+), 10 deletions(-)
tools/perf/util/data.c +17 -0
index 8dacd34659cc..b6f7cdc4a39f 100644
@@ -52,6 +52,21 @@ static int check_backup(struct perf_data_file *file)
 	return 0;
 }
 
+static void check_multi(struct perf_data_file *file)
+{
+	struct stat st;
+
+	/*
+	 * For write, it'll be determined by user (perf record -M)
+	 * whether to enable multi file data storage.
+	 */
+	if (perf_data_file__is_write(file))
+		return;
+
+	if (!stat(file->path, &st) && S_ISDIR(st.st_mode))
+		file->is_multi = true;
+}
+
 static int scandir_filter(const struct dirent *d)
 {
 	return !prefixcmp(d->d_name, "perf.data.");
@@ -206,6 +221,8 @@ int perf_data_file__open(struct perf_data_file *file)
 	if (check_pipe(file))
 		return 0;
 
+	check_multi(file);
+
 	if (!file->path)
 		file->path = default_data_path(file);
 
tools/perf/util/session.c +2 -10
index 88aa2f09df93..4f0fcd2d3901 100644
@@ -1252,11 +1252,10 @@ fetch_mmaped_event(struct perf_session *session,
 #define NUM_MMAPS 128
 #endif
 
-static int __perf_session__process_events(struct perf_session *session,
+static int __perf_session__process_events(struct perf_session *session, int fd,
 					  u64 data_offset, u64 data_size,
 					  u64 file_size, struct perf_tool *tool)
 {
-	int fd = perf_data_file__fd(session->file);
 	u64 head, page_offset, file_offset, file_pos, size;
 	int err, mmap_prot, mmap_flags, map_idx = 0;
 	size_t	mmap_size;
@@ -1362,18 +1361,40 @@ int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool)
 {
 	u64 size = perf_data_file__size(session->file);
-	int err;
+	int err, i;
 
 	if (perf_session__register_idle_thread(session) == NULL)
 		return -ENOMEM;
 
-	if (!perf_data_file__is_pipe(session->file))
-		err = __perf_session__process_events(session,
-						     session->header.data_offset,
-						     session->header.data_size,
-						     size, tool);
-	else
-		err = __perf_session__process_pipe_events(session, tool);

tools/perf/util/session.c int perf_session__process_events(struct perf_session *session, · line 1378

Quoted 10 lines
tools/perf/util/session.c +10 -0
+	if (perf_data_file__is_pipe(session->file))
+		return __perf_session__process_pipe_events(session, tool);
+
+	err = __perf_session__process_events(session,
+					     perf_data_file__fd(session->file),
+					     session->header.data_offset,
+					     session->header.data_size,
+					     size, tool);
+	if (!session->file->is_multi || err)
+		return err;

if we have file->is_multi true, the perf_data_file__fd(session->file) is the perf.header file right? So presumably, there's no data in it, and we dont need to call above __perf_session__process_events function?

jirka

Quoted 25 lines
tools/perf/util/session.c +19 -0
+
+	/*
+	 * For multi-file data storage, events are processed for each
+	 * cpu/thread so it's already ordered.
+	 */
+	tool->ordered_events = false;
+
+	for (i = 0; i < session->file->nr_multi; i++) {
+		int fd = perf_data_file__multi_fd(session->file, i);
+
+		size = lseek(fd, 0, SEEK_END);
+		if (size == 0)
+			continue;
+
+		err = __perf_session__process_events(session, fd,
+						     0, size, size, tool);
+		if (err < 0)
+			break;
+	}
 
 	return err;
 }
Signature · 1 line
2.1.3
author maintainer
#31 lore raw

Re: [PATCH 08/37] perf tools: Handle multi-file session properly

Hi Jiri,

Quoted 6 lines · Jiri Olsa
Quoted 5 lines · Namhyung Kim

When perf detects multi-file data directory, process header file first and then rest data files in a row. Note that the multi-file data is recorded for each cpu/thread separately, it's already ordered with respect to themselves so no need to use the ordered event queue interface.

[SNIP]

Quoted 23 lines
Quoted 23 lines
tools/perf/util/session.c +7 -8
@@ -1362,18 +1361,40 @@ int perf_session__process_events(struct perf_session *session,
                               struct perf_tool *tool)
 {
      u64 size = perf_data_file__size(session->file);
-     int err;
+     int err, i;
 
      if (perf_session__register_idle_thread(session) == NULL)
              return -ENOMEM;
 
-     if (!perf_data_file__is_pipe(session->file))
-             err = __perf_session__process_events(session,
-                                                  session->header.data_offset,
-                                                  session->header.data_size,
-                                                  size, tool);
-     else
-             err = __perf_session__process_pipe_events(session, tool);
+     if (perf_data_file__is_pipe(session->file))
+             return __perf_session__process_pipe_events(session, tool);
+
+     err = __perf_session__process_events(session,
+                                          perf_data_file__fd(session->file),
+                                          session->header.data_offset,
Quoted 8 lines

tools/perf/util/session.c int perf_session__process_events(struct perf_session *session, · line 1378

Quoted 4 lines
tools/perf/util/session.c +4 -0
+                                          session->header.data_size,
+                                          size, tool);
+     if (!session->file->is_multi || err)
+             return err;

if we have file->is_multi true, the perf_data_file__fd(session->file) is the perf.header file right? So presumably, there's no data in it, and we dont need to call above __perf_session__process_events function?

Right, it's for perf.header file and has no samples but it has meta events like fork, comm, mmap so it needs to be processed (before processing samples).

Thanks,
Namhyung

Quoted 26 lines
Quoted 26 lines
tools/perf/util/session.c +19 -1
+
+     /*
+      * For multi-file data storage, events are processed for each
+      * cpu/thread so it's already ordered.
+      */
+     tool->ordered_events = false;
+
+     for (i = 0; i < session->file->nr_multi; i++) {
+             int fd = perf_data_file__multi_fd(session->file, i);
+
+             size = lseek(fd, 0, SEEK_END);
+             if (size == 0)
+                     continue;
+
+             err = __perf_session__process_events(session, fd,
+                                                  0, size, size, tool);
+             if (err < 0)
+                     break;
+     }
 
      return err;
 }
--

2.1.3

author maintainer
#32 lore raw

PATCH 9/37 perf record: Add -M/--multi option for multi file recording

In reply to Namhyung Kim

The new -M/--multi option enables multi file storage recording. Now sample data in separate mmap are saved in different files and other events will be recorded in the perf.header file.

---
 tools/perf/Documentation/perf-record.txt |  5 +++++
 tools/perf/builtin-record.c              | 27 ++++++++++++++++++++++++---
 2 files changed, 29 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-record.txt +5 -0
index af9a54ece024..14247ccd7965 100644
@@ -220,6 +220,11 @@ Capture machine state (registers) at interrupt, i.e., on counter overflows for
 each sample. List of captured registers depends on the architecture. This option
 is off by default.
 
+-M::
+--multi::
+Record data in multi-file storage instead of a single data file (perf.data).
+This will speed up perf report by parallel processing.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +24 -3
index 4f97657f14e7..7f7a4725d080 100644
@@ -58,6 +58,19 @@ static int record__write(struct record *rec, void *bf, size_t size)
 	return 0;
 }
 
+static int record__write_multi(struct record *rec, void *bf, size_t size, int idx)
+{
+	if (rec->file.is_multi && idx >= 0) {
+		int ret = perf_data_file__write_multi(rec->session->file,
+						      bf, size, idx);
+		if (ret < 0)
+			pr_err("failed to write perf data, error: %m\n");
+
+		return ret;
+	}
+	return record__write(rec, bf, size);
+}
+
 static int process_synthesized_event(struct perf_tool *tool,
 				     union perf_event *event,
 				     struct perf_sample *sample __maybe_unused,
@@ -89,7 +102,7 @@ static int record__mmap_read(struct record *rec, int idx)
 		size = md->mask + 1 - (old & md->mask);
 		old += size;
 
-		if (record__write(rec, buf, size) < 0) {
+		if (record__write_multi(rec, buf, size, idx) < 0) {
 			rc = -1;
 			goto out;
 		}
@@ -99,7 +112,7 @@ static int record__mmap_read(struct record *rec, int idx)
 	size = head - old;
 	old += size;
 
-	if (record__write(rec, buf, size) < 0) {
+	if (record__write_multi(rec, buf, size, idx) < 0) {
 		rc = -1;
 		goto out;
 	}
@@ -186,6 +199,10 @@ static int record__open(struct record *rec)
 		goto out;
 	}
 
+	rc = perf_data_file__prepare_write(session->file, evlist->nr_mmaps);
+	if (rc < 0)
+		goto out;
+
 	session->evlist = evlist;
 	perf_session__set_id_hdr_size(session);
 out:
@@ -822,6 +839,8 @@ struct option __record_options[] = {
 		    "use per-thread mmaps"),
 	OPT_BOOLEAN('I', "intr-regs", &record.opts.sample_intr_regs,
 		    "Sample machine registers on interrupt"),
+	OPT_BOOLEAN('M', "multi", &record.opts.multi_file,
+		    "use multi-file storage"),
 	OPT_END()
 };
 
@@ -871,8 +890,10 @@ int cmd_record(int argc, const char **argv, const char *prefix __maybe_unused)
 		goto out_symbol_exit;
 	}
 
-	if (rec->opts.multi_file)
+	if (rec->opts.multi_file) {
+		rec->file.is_multi = true;
 		perf_evlist__prepend_dummy(rec->evlist);
+	}
 
 	if (rec->opts.target.tid && !rec->opts.no_inherit_set)
 		rec->opts.no_inherit = true;
Signature · 1 line
2.1.3
author maintainer
#33 lore raw

PATCH 10/37 perf report: Skip dummy tracking event

In reply to Namhyung Kim

The dummy tracking event is only for tracking task/comom/mmap events and has no sample data for itself. So no need to report, just skip it.

---
 tools/perf/builtin-report.c    |  3 +++
 tools/perf/ui/browsers/hists.c | 10 ++++++++--
 tools/perf/ui/gtk/hists.c      |  3 +++
 3 files changed, 14 insertions(+), 2 deletions(-)
tools/perf/builtin-report.c +3 -0
index 2f91094e228b..4cac79ad3085 100644
@@ -318,6 +318,9 @@ static int perf_evlist__tty_browse_hists(struct perf_evlist *evlist,
 		struct hists *hists = evsel__hists(pos);
 		const char *evname = perf_evsel__name(pos);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group &&
 		    !perf_evsel__is_group_leader(pos))
 			continue;
tools/perf/ui/browsers/hists.c +8 -2
index e6bb04b5b09b..be594e1d6a99 100644
@@ -1998,11 +1998,15 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 				  struct perf_session_env *env)
 {
 	int nr_entries = evlist->nr_entries;
+	struct perf_evsel *first = perf_evlist__first(evlist);
+
+	if (perf_evsel__is_dummy_tracking(first)) {
+		first = perf_evsel__next(first);
+		nr_entries--;
+	}
 
 single_entry:
 	if (nr_entries == 1) {
-		struct perf_evsel *first = perf_evlist__first(evlist);
-
 		return perf_evsel__hists_browse(first, nr_entries, help,
 						false, hbt, min_pcnt,
 						env);
@@ -2013,6 +2017,8 @@ int perf_evlist__tui_browse_hists(struct perf_evlist *evlist, const char *help,
 
 		nr_entries = 0;
 		evlist__for_each(evlist, pos) {
+			if (perf_evsel__is_dummy_tracking(pos))
+				continue;
 			if (perf_evsel__is_group_leader(pos))
 				nr_entries++;
 		}
tools/perf/ui/gtk/hists.c +3 -0
index 4b3585eed1e8..83a7ecd5cda8 100644
@@ -317,6 +317,9 @@ int perf_evlist__gtk_browse_hists(struct perf_evlist *evlist,
 		char buf[512];
 		size_t size = sizeof(buf);
 
+		if (perf_evsel__is_dummy_tracking(pos))
+			continue;
+
 		if (symbol_conf.event_group) {
 			if (!perf_evsel__is_group_leader(pos))
 				continue;
Signature · 1 line
2.1.3
author maintainer
#34 lore raw

PATCH 11/37 perf tools: Introduce thread__comm_time() helpers

In reply to Namhyung Kim

When multi-file data storage is enabled, it processes all task, comm and mmap events first and then goes to the sample events. So all it sees is the last comm of a thread although it has information at the time of sample.

Sort thread's comm by time so that it can find appropriate comm at the sample time. The thread__comm_time() will mostly work even if PERF_SAMPLE_TIME bit is off since in that case, sample->time will be -1 so it'll take the last comm anyway.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/util/thread.c | 34 +++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h |  2 ++
 2 files changed, 35 insertions(+), 1 deletion(-)
tools/perf/util/thread.c +33 -1
index 9ebc8b1f9be5..083fa0fcf316 100644
@@ -103,6 +103,22 @@ struct comm *thread__exec_comm(const struct thread *thread)
 	return last;
 }
 
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp)
+{
+	struct comm *comm;
+
+	list_for_each_entry(comm, &thread->comm_list, list) {
+		if (timestamp >= comm->start)
+			return comm;
+	}
+
+	if (list_empty(&thread->comm_list))
+		return NULL;
+
+	return list_last_entry(&thread->comm_list, struct comm, list);
+}
+
+/* CHECKME: time should always be 0 if event aren't ordered */
 int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		       bool exec)
 {
@@ -118,7 +134,13 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 		new = comm__new(str, timestamp, exec);
 		if (!new)
 			return -ENOMEM;
-		list_add(&new->list, &thread->comm_list);
+
+		/* sort by time */
+		list_for_each_entry(curr, &thread->comm_list, list) {
+			if (timestamp >= curr->start)
+				break;
+		}
+		list_add_tail(&new->list, &curr->list);
 
 		if (exec)
 			unwind__flush_access(thread);
@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread *thread)
 	return comm__str(comm);
 }
 
+const char *thread__comm_time_str(const struct thread *thread, u64 timestamp)
+{
+	const struct comm *comm = thread__comm_time(thread, timestamp);
+
+	if (!comm)
+		return NULL;
+
+	return comm__str(comm);
+}
+
 /* CHECKME: it should probably better return the max comm len from its comm list */
 int thread__comm_len(struct thread *thread)
 {
tools/perf/util/thread.h +2 -0
index 160fd066a7d1..0b6dcd70bc8b 100644
@@ -53,7 +53,9 @@ static inline int thread__set_comm(struct thread *thread, const char *comm,
 int thread__comm_len(struct thread *thread);
 struct comm *thread__comm(const struct thread *thread);
 struct comm *thread__exec_comm(const struct thread *thread);
+struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
+const char *thread__comm_time_str(const struct thread *thread, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
Signature · 1 line
2.1.3
#35 lore raw

Re: [PATCH 11/37] perf tools: Introduce thread__comm_time() helpers

Quoted 4 lines · Namhyung Kim
tools/perf/util/thread.c +0 -0
@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread *thread)
  	return comm__str(comm);
  }
 

tools/perf/util/thread.c const char *thread__comm_str(const struct thread *thread) · line 173

Quoted 10 lines
tools/perf/util/thread.c +10 -0
+const char *thread__comm_time_str(const struct thread *thread, u64 timestamp)
+{
+	const struct comm *comm = thread__comm_time(thread, timestamp);
+
+	if (!comm)
+		return NULL;
+
+	return comm__str(comm);
+}
+

thread__comm_str_time()? time_str suggests a time-based string as opposed to a comm_str based on time.

David

author maintainer
#36 lore raw

Re: [PATCH 11/37] perf tools: Introduce thread__comm_time() helpers

Quoted 7 lines · David Ahern
Quoted 6 lines · Namhyung Kim
tools/perf/util/thread.c +0 -0
@@ -139,6 +161,16 @@ const char *thread__comm_str(const struct thread
 *thread)
        return comm__str(comm);
  }
 
Quoted 15 lines

tools/perf/util/thread.c const char *thread__comm_str(const struct thread *thread) · line 173

Quoted 11 lines
tools/perf/util/thread.c +10 -0
+const char *thread__comm_time_str(const struct thread *thread, u64
 timestamp)
+{
+       const struct comm *comm = thread__comm_time(thread, timestamp);
+
+       if (!comm)
+               return NULL;
+
+       return comm__str(comm);
+}
+

thread__comm_str_time()? time_str suggests a time-based string as opposed to a comm_str based on time.

Will change - my naming sense is always awful. I'd be happy to hear any naming suggestion. ;-)

Thanks,
Namhyung

author maintainer
#37 lore raw

PATCH 12/37 perf tools: Add a test case for thread comm handling

In reply to Namhyung Kim

The new test case checks various thread comm handling like overridding and time sorting.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/Makefile.perf        |  1 +
 tools/perf/tests/builtin-test.c |  4 ++++
 tools/perf/tests/tests.h        |  1 +
 tools/perf/tests/thread-comm.c  | 47 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+)
 create mode 100644 tools/perf/tests/thread-comm.c
tools/perf/Makefile.perf +1 -0
index 763e68fb5767..e4528a4a3a8c 100644
@@ -447,6 +447,7 @@ endif
 LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
 LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
+LIB_OBJS += $(OUTPUT)tests/thread-comm.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
index 4b7d9ab0f049..1b463d82a71a 100644
@@ -167,6 +167,10 @@ static struct test {
 		.func = test__fdarray__add,
 	},
 	{
+		.desc = "Test thread comm handling",
+		.func = test__thread_comm,
+	},
+	{
 		.func = NULL,
 	},
 };
tools/perf/tests/tests.h +1 -0
index 00e776a87a9c..43ac17780629 100644
@@ -51,6 +51,7 @@ int test__hists_cumulate(void);
 int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
+int test__thread_comm(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-comm.c new file +47 -0

47 lines not on this page · raw message

Signature · 1 line
2.1.3
author maintainer
#38 lore raw

PATCH 13/37 perf tools: Use thread__comm_time() when adding hist entries

In reply to Namhyung Kim

Now thread->comm can be handled with time properly, use it to find correct comm when adding hist entries.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/builtin-annotate.c |  5 +++--
 tools/perf/builtin-diff.c     |  8 ++++----
 tools/perf/tests/hists_link.c |  4 ++--
 tools/perf/util/hist.c        | 19 ++++++++++---------
 tools/perf/util/hist.h        |  2 +-
 5 files changed, 20 insertions(+), 18 deletions(-)
tools/perf/builtin-annotate.c +3 -2
index 747f86103599..a3b6d9d14925 100644
@@ -47,7 +47,7 @@ struct perf_annotate {
 };
 
 static int perf_evsel__add_sample(struct perf_evsel *evsel,
-				  struct perf_sample *sample __maybe_unused,
+				  struct perf_sample *sample,
 				  struct addr_location *al,
 				  struct perf_annotate *ann)
 {
@@ -67,7 +67,8 @@ static int perf_evsel__add_sample(struct perf_evsel *evsel,
 		return 0;
 	}
 
-	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0, true);
+	he = __hists__add_entry(hists, al, NULL, NULL, NULL, 1, 1, 0,
+				sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
tools/perf/builtin-diff.c +4 -4
index 260b10c19ad6..2b86607a8571 100644
@@ -312,10 +312,10 @@ static int formula_fprintf(struct hist_entry *he, struct hist_entry *pair,
 
 static int hists__add_entry(struct hists *hists,
 			    struct addr_location *al, u64 period,
-			    u64 weight, u64 transaction)
+			    u64 weight, u64 transaction, u64 timestamp)
 {
 	if (__hists__add_entry(hists, al, NULL, NULL, NULL, period, weight,
-			       transaction, true) != NULL)
+			       transaction, timestamp, true) != NULL)
 		return 0;
 	return -ENOMEM;
 }
@@ -335,8 +335,8 @@ static int diff__process_sample_event(struct perf_tool *tool __maybe_unused,
 		return -1;
 	}
 
-	if (hists__add_entry(hists, &al, sample->period,
-			     sample->weight, sample->transaction)) {
+	if (hists__add_entry(hists, &al, sample->period, sample->weight,
+			     sample->transaction, sample->time)) {
 		pr_warning("problem incrementing symbol period, skipping event\n");
 		return -1;
 	}
tools/perf/tests/hists_link.c +2 -2
index 278ba8344c23..fe7cb886c23e 100644
@@ -90,7 +90,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
@@ -114,7 +114,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 				goto out;
 
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
 
tools/perf/util/hist.c +10 -9
index 9314286ed25c..d322264bac22 100644
@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi,
 				      u64 period, u64 weight, u64 transaction,
-				      bool sample_self)
+				      u64 timestamp, bool sample_self)
 {
 	struct hist_entry entry = {
 		.thread	= al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, timestamp),
 		.ms = {
 			.map	= al->map,
 			.sym	= al->sym,
@@ -513,13 +513,14 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 {
 	u64 cost;
 	struct mem_info *mi = iter->priv;
+	struct perf_sample *sample = iter->sample;
 	struct hists *hists = evsel__hists(iter->evsel);
 	struct hist_entry *he;
 
 	if (mi == NULL)
 		return -EINVAL;
 
-	cost = iter->sample->weight;
+	cost = sample->weight;
 	if (!cost)
 		cost = 1;
 
@@ -531,7 +532,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	 * and the he_stat__add_period() function.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, NULL, mi,
-				cost, cost, 0, true);
+				cost, cost, 0, sample->time, true);
 	if (!he)
 		return -ENOMEM;
 
@@ -632,7 +633,7 @@ iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *a
 	 * and not events sampled. Thus we use a pseudo period of 1.
 	 */
 	he = __hists__add_entry(hists, al, iter->parent, &bi[i], NULL,
-				1, 1, 0, true);
+				1, 1, 0, iter->sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -670,7 +671,7 @@ iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -732,7 +733,7 @@ iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, true);
+				sample->transaction, sample->time, true);
 	if (he == NULL)
 		return -ENOMEM;
 
@@ -776,7 +777,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 	struct hist_entry he_tmp = {
 		.cpu = al->cpu,
 		.thread = al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, sample->time),
 		.ip = al->addr,
 		.ms = {
 			.map = al->map,
@@ -805,7 +806,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 
 	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
-				sample->transaction, false);
+				sample->transaction, sample->time, false);
 	if (he == NULL)
 		return -ENOMEM;
 
tools/perf/util/hist.h +1 -1
index 46bd50344f85..b86966206ba8 100644
@@ -109,7 +109,7 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
-				      bool sample_self);
+				      u64 timestamp, bool sample_self);
 int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 			 struct perf_evsel *evsel, struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
Signature · 1 line
2.1.3
#39 lore raw

Re: [PATCH 13/37] perf tools: Use thread__comm_time() when adding hist entries

On Wed, Dec 24, 2014 at 04:15:09PM +0900, Namhyung Kim wrote:

SNIP

Quoted 12 lines
tools/perf/tests/hists_link.c +1 -1
 			he = __hists__add_entry(hists, &al, NULL,
-						NULL, NULL, 1, 1, 0, true);
+						NULL, NULL, 1, 1, 0, -1, true);
 			if (he == NULL)
 				goto out;
tools/perf/util/hist.c +0 -0
index 9314286ed25c..d322264bac22 100644

tools/perf/util/hist.c struct hist_entry *__hists__add_entry(struct hists *hists, · line 458

Quoted 10 lines
tools/perf/util/hist.c +2 -2
@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct branch_info *bi,
 				      struct mem_info *mi,
 				      u64 period, u64 weight, u64 transaction,
-				      bool sample_self)
+				      u64 timestamp, bool sample_self)
 {
 	struct hist_entry entry = {
 		.thread	= al->thread,
-		.comm = thread__comm(al->thread),
+		.comm = thread__comm_time(al->thread, timestamp),

with thread object having multiple comm entries, could this hurt the single threaded performance?

The thread__comm_time function iterates comm_list each time, maybe you could add some 'last_comm found check' logic in it?

jirka

author maintainer
#40 lore raw

Re: [PATCH 13/37] perf tools: Use thread__comm_time() when adding hist entries

Quoted 21 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:09PM +0900, Namhyung Kim wrote:

SNIP

Quoted 17 lines
                      he = __hists__add_entry(hists, &al, NULL,
tools/perf/tests/hists_link.c +1 -1
-                                             NULL, NULL, 1, 1, 0, true);
+                                             NULL, NULL, 1, 1, 0, -1, true);
                      if (he == NULL)
                              goto out;
tools/perf/util/hist.c +1 -1
index 9314286ed25c..d322264bac22 100644
@@ -451,11 +451,11 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
                                    struct branch_info *bi,
                                    struct mem_info *mi,
                                    u64 period, u64 weight, u64 transaction,
-                                   bool sample_self)
+                                   u64 timestamp, bool sample_self)
Quoted 8 lines

tools/perf/util/hist.c struct hist_entry *__hists__add_entry(struct hists *hists, · line 458

Quoted 5 lines
tools/perf/util/hist.c +1 -1
 {
      struct hist_entry entry = {
              .thread = al->thread,
-             .comm = thread__comm(al->thread),
+             .comm = thread__comm_time(al->thread, timestamp),

with thread object having multiple comm entries, could this hurt the single threaded performance?

Probably. But in my test, the single threaded performance on a single data file is slightly better or almost same. I didn't investigate it yet where the performance gain comes, but anyway, I think this has almost no effect on the performance since most thread will have just one or two comms I guess.

And JFYI, the single threaded performance on a multi-file data is better (I tested same data using 'perf data split' command and no --multi-thread option to perf report) as IMHO it doesn't need to use the ordered event queue layer.

Quoted 3 lines

The thread__comm_time function iterates comm_list each time, maybe you could add some 'last_comm found check' logic in it?

I think it can be easily added later if it really affects the performance.

Thanks,
Namhyung

author maintainer
#41 lore raw

PATCH 14/37 perf tools: Convert dead thread list into rbtree

In reply to Namhyung Kim

Currently perf maintains dead threads in a linked list but this can be a problem if someone needs to search from it. Convert it to a rbtree like normal threads and it'll be used later with multi-file changes.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/util/machine.c | 59 +++++++++++++++++++++++++++++++++++++++--------
 tools/perf/util/machine.h |  2 +-
 tools/perf/util/thread.c  |  1 +
 tools/perf/util/thread.h  | 11 ++++-----
 4 files changed, 57 insertions(+), 16 deletions(-)
tools/perf/util/machine.c +50 -9
index 15dd0a9691ce..582e011adc92 100644
@@ -28,7 +28,7 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)
 	dsos__init(&machine->kernel_dsos);
 
 	machine->threads = RB_ROOT;
-	INIT_LIST_HEAD(&machine->dead_threads);
+	machine->dead_threads = RB_ROOT;
 	machine->last_match = NULL;
 
 	machine->vdso_info = NULL;
@@ -91,10 +91,21 @@ static void dsos__delete(struct dsos *dsos)
 
 void machine__delete_dead_threads(struct machine *machine)
 {
-	struct thread *n, *t;
+	struct rb_node *nd = rb_first(&machine->dead_threads);
+
+	while (nd) {
+		struct thread *t = rb_entry(nd, struct thread, rb_node);
+		struct thread *pos;
+
+		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->dead_threads);
+
+		while (!list_empty(&t->node)) {
+			pos = list_first_entry(&t->node, struct thread, node);
+			list_del(&pos->node);
+			thread__delete(pos);
+		}
 
-	list_for_each_entry_safe(t, n, &machine->dead_threads, node) {
-		list_del(&t->node);
 		thread__delete(t);
 	}
 }
@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)
 	while (nd) {
 		struct thread *t = rb_entry(nd, struct thread, rb_node);
 
-		rb_erase(&t->rb_node, &machine->threads);
 		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->threads);
 		thread__delete(t);
 	}
 }
@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 static void machine__remove_thread(struct machine *machine, struct thread *th)
 {
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+
+	th->dead = true;
+
 	/*
 	 * We may have references to this thread, for instance in some hist_entry
-	 * instances, so just move them to a separate list.
+	 * instances, so just move them to a separate list in rbtree.
 	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			list_add_tail(&th->node, &pos->node);
+			return;
+		}
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&th->rb_node, parent, p);
+	rb_insert_color(&th->rb_node, &machine->dead_threads);
 }
 
 int machine__process_fork_event(struct machine *machine, union perf_event *event,
@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine *machine,
 			     void *priv)
 {
 	struct rb_node *nd;
-	struct thread *thread;
+	struct thread *thread, *pos;
 	int rc = 0;
 
 	for (nd = rb_first(&machine->threads); nd; nd = rb_next(nd)) {
@@ -1659,10 +1693,17 @@ int machine__for_each_thread(struct machine *machine,
 			return rc;
 	}
 
-	list_for_each_entry(thread, &machine->dead_threads, node) {
+	for (nd = rb_first(&machine->dead_threads); nd; nd = rb_next(nd)) {
+		thread = rb_entry(nd, struct thread, rb_node);
 		rc = fn(thread, priv);
 		if (rc != 0)
 			return rc;
+
+		list_for_each_entry(pos, &thread->node, node) {
+			rc = fn(pos, priv);
+			if (rc != 0)
+				return rc;
+		}
 	}
 	return rc;
 }
tools/perf/util/machine.h +1 -1
index e8b7779a0a3f..4349946a38ff 100644
@@ -30,7 +30,7 @@ struct machine {
 	bool		  comm_exec;
 	char		  *root_dir;
 	struct rb_root	  threads;
-	struct list_head  dead_threads;
+	struct rb_root	  dead_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
tools/perf/util/thread.c +1 -0
index 083fa0fcf316..b9c5c5d5e718 100644
@@ -38,6 +38,7 @@ struct thread *thread__new(pid_t pid, pid_t tid)
 		thread->ppid = -1;
 		thread->cpu = -1;
 		INIT_LIST_HEAD(&thread->comm_list);
+		INIT_LIST_HEAD(&thread->node);
 
 		if (unwind__prepare_access(thread) < 0)
 			goto err_thread;
tools/perf/util/thread.h +5 -6
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
 struct thread_stack;
 
 struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	node;
 	struct map_groups	*mg;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
@@ -22,7 +20,8 @@ struct thread {
 	int			cpu;
 	char			shortname[3];
 	bool			comm_set;
-	bool			dead; /* if set thread has exited */
+	bool			exited; /* if set thread has exited */
+	bool			dead; /* thread is in dead_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
@@ -39,7 +38,7 @@ int thread__init_map_groups(struct thread *thread, struct machine *machine);
 void thread__delete(struct thread *thread);
 static inline void thread__exited(struct thread *thread)
 {
-	thread->dead = true;
+	thread->exited = true;
 }
 
 int __thread__set_comm(struct thread *thread, const char *comm, u64 timestamp,
Signature · 1 line
2.1.3
#42 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

On Wed, Dec 24, 2014 at 04:15:10PM +0900, Namhyung Kim wrote:

SNIP

Quoted 16 lines

static void machine__remove_thread(struct machine *machine, struct thread *th) {

tools/perf/util/machine.c +8 -1
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
 	machine->last_match = NULL;
 	rb_erase(&th->rb_node, &machine->threads);
+
+	th->dead = true;
+
 	/*
 	 * We may have references to this thread, for instance in some hist_entry
-	 * instances, so just move them to a separate list.
+	 * instances, so just move them to a separate list in rbtree.

tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270

Quoted 10 lines
tools/perf/util/machine.c +8 -1
 	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			list_add_tail(&th->node, &pos->node);
+			return;
+		}

hum, why is this 'new list' in thread object necessary? why not to store all in the tree?

Quoted 7 lines
tools/perf/util/machine.c +7 -0
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+

SNIP

Quoted 18 lines
tools/perf/util/thread.h +2 -4
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
 struct thread_stack;
 
 struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	node;
 	struct map_groups	*mg;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;

tools/perf/util/thread.h struct thread { · line 24

Quoted 6 lines
tools/perf/util/thread.h +2 -1
@@ -22,7 +20,8 @@ struct thread {
 	int			cpu;
 	char			shortname[3];
 	bool			comm_set;
-	bool			dead; /* if set thread has exited */
+	bool			exited; /* if set thread has exited */
+	bool			dead; /* thread is in dead_threads list */

looks like this also changes the logic (new exited flag), not just the dead threads storage wheel

jirka

author maintainer
#43 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

Quoted 25 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:10PM +0900, Namhyung Kim wrote:

SNIP

Quoted 21 lines

static void machine__remove_thread(struct machine *machine, struct thread *th) {

tools/perf/util/machine.c +11 -2
+     struct rb_node **p = &machine->dead_threads.rb_node;
+     struct rb_node *parent = NULL;
+     struct thread *pos;
+
      machine->last_match = NULL;
      rb_erase(&th->rb_node, &machine->threads);
+
+     th->dead = true;
+
      /*
       * We may have references to this thread, for instance in some hist_entry
-      * instances, so just move them to a separate list.
+      * instances, so just move them to a separate list in rbtree.
       */
-     list_add_tail(&th->node, &machine->dead_threads);
+     while (*p != NULL) {
+             parent = *p;
+             pos = rb_entry(parent, struct thread, rb_node);
Quoted 8 lines

tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270

Quoted 5 lines
tools/perf/util/machine.c +5 -0
+
+             if (pos->tid == th->tid) {
+                     list_add_tail(&th->node, &pos->node);
+                     return;
+             }

hum, why is this 'new list' in thread object necessary? why not to store all in the tree?

No absolute reason, but I just wanted to keep them in a single place and to see how many of them exist easily. We could compare pid and then timestamp so that it can be in a single rbtree.

Quoted 31 lines
Quoted 9 lines
tools/perf/util/machine.c +7 -0
+
+             if (th->tid < pos->tid)
+                     p = &(*p)->rb_left;
+             else
+                     p = &(*p)->rb_right;
+     }
+

SNIP

Quoted 19 lines
tools/perf/util/thread.h +2 -4
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
 struct thread_stack;
 
 struct thread {
-     union {
-             struct rb_node   rb_node;
-             struct list_head node;
-     };
+     struct rb_node          rb_node;
+     struct list_head        node;
      struct map_groups       *mg;
      pid_t                   pid_; /* Not all tools update this */
      pid_t                   tid;
@@ -22,7 +20,8 @@ struct thread {
      int                     cpu;
Quoted 8 lines

tools/perf/util/thread.h struct thread { · line 24

Quoted 5 lines
tools/perf/util/thread.h +2 -1
      char                    shortname[3];
      bool                    comm_set;
-     bool                    dead; /* if set thread has exited */
+     bool                    exited; /* if set thread has exited */
+     bool                    dead; /* thread is in dead_threads list */

looks like this also changes the logic (new exited flag), not just the dead threads storage wheel

AFAICS the 'dead' flag is not used other than thread__exited(). And it confused me a dead thread might not be in a dead_threads tree (or list). So I changed the name and no logical change intended.

Thanks,
Namhyung

#44 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

Quoted 28 lines · Namhyung Kim
Quoted 19 lines
Quoted 19 lines
tools/perf/util/thread.h +2 -4
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
  struct thread_stack;
 
  struct thread {
-     union {
-             struct rb_node   rb_node;
-             struct list_head node;
-     };
+     struct rb_node          rb_node;
+     struct list_head        node;
       struct map_groups       *mg;
       pid_t                   pid_; /* Not all tools update this */
       pid_t                   tid;
@@ -22,7 +20,8 @@ struct thread {
       int                     cpu;
Quoted 8 lines

tools/perf/util/thread.h struct thread { · line 24

Quoted 5 lines
tools/perf/util/thread.h +2 -1
       char                    shortname[3];
       bool                    comm_set;
-     bool                    dead; /* if set thread has exited */
+     bool                    exited; /* if set thread has exited */
+     bool                    dead; /* thread is in dead_threads list */

looks like this also changes the logic (new exited flag), not just the dead threads storage wheel

Quoted 3 lines

AFAICS the 'dead' flag is not used other than thread__exited(). And it confused me a dead thread might not be in a dead_threads tree (or list). So I changed the name and no logical change intended.

git show 236a3bbd5cb51

David

author maintainer
#45 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

Quoted 32 lines · David Ahern
Quoted 31 lines · Namhyung Kim
Quoted 29 lines
Quoted 19 lines
tools/perf/util/thread.h +2 -4
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
  struct thread_stack;
 
  struct thread {
-     union {
-             struct rb_node   rb_node;
-             struct list_head node;
-     };
+     struct rb_node          rb_node;
+     struct list_head        node;
       struct map_groups       *mg;
       pid_t                   pid_; /* Not all tools update this */
       pid_t                   tid;

tools/perf/util/thread.h struct thread { · line 24

Quoted 6 lines
tools/perf/util/thread.h +2 -1
@@ -22,7 +20,8 @@ struct thread {
       int                     cpu;
       char                    shortname[3];
       bool                    comm_set;
-     bool                    dead; /* if set thread has exited */
+     bool                    exited; /* if set thread has exited */
+     bool                    dead; /* thread is in dead_threads list */

looks like this also changes the logic (new exited flag), not just the dead threads storage wheel

Quoted 6 lines
Quoted 3 lines

AFAICS the 'dead' flag is not used other than thread__exited(). And it confused me a dead thread might not be in a dead_threads tree (or list). So I changed the name and no logical change intended.

git show 236a3bbd5cb51

Thanks for the pointer. I understand the need of delaying the move to dead_threads list, but anyway it still makes me confused. So I renamed the flag to keep the current behavior and match the name 'dead' to the list/tree management to reduce further confusion.

Thanks,
Namhyung

#46 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

In reply to Namhyung Kim

tools/perf/util/machine.c void machine__delete_threads(struct machine *machine) · line 124

Quoted 10 lines · Namhyung Kim
tools/perf/util/machine.c +1 -1
@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)
  	while (nd) {
  		struct thread *t = rb_entry(nd, struct thread, rb_node);
 
-		rb_erase(&t->rb_node, &machine->threads);
  		nd = rb_next(nd);
+		rb_erase(&t->rb_node, &machine->threads);
  		thread__delete(t);
  	}
  }

unrelated to dead threads. Bug fix? separate patch?

Quoted 17 lines
tools/perf/util/machine.c +8 -1
@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
  static void machine__remove_thread(struct machine *machine, struct thread *th)
  {
+	struct rb_node **p = &machine->dead_threads.rb_node;
+	struct rb_node *parent = NULL;
+	struct thread *pos;
+
  	machine->last_match = NULL;
  	rb_erase(&th->rb_node, &machine->threads);
+
+	th->dead = true;
+
  	/*
  	 * We may have references to this thread, for instance in some hist_entry
-	 * instances, so just move them to a separate list.
+	 * instances, so just move them to a separate list in rbtree.

tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270

Quoted 10 lines
tools/perf/util/machine.c +8 -1
  	 */
-	list_add_tail(&th->node, &machine->dead_threads);
+	while (*p != NULL) {
+		parent = *p;
+		pos = rb_entry(parent, struct thread, rb_node);
+
+		if (pos->tid == th->tid) {
+			list_add_tail(&th->node, &pos->node);
+			return;
+		}

So you have a linked list for tid collisions (not mentioned in the git log).

tools/perf/util/machine.c int machine__for_each_thread(struct machine *machine,

Quoted 13 lines
tools/perf/util/machine.c +9 -0
+
+		if (th->tid < pos->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}
+
+	rb_link_node(&th->rb_node, parent, p);
+	rb_insert_color(&th->rb_node, &machine->dead_threads);
  }
 
  int machine__process_fork_event(struct machine *machine, union perf_event *event,
@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine *machine,

---8<---

Quoted 7 lines
tools/perf/util/thread.h +0 -0
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
  struct thread_stack;
 

tools/perf/util/thread.h line 18

Quoted 10 lines
tools/perf/util/thread.h +2 -4
  struct thread {
-	union {
-		struct rb_node	 rb_node;
-		struct list_head node;
-	};
+	struct rb_node	 	rb_node;
+	struct list_head 	node;
  	struct map_groups	*mg;
  	pid_t			pid_; /* Not all tools update this */
  	pid_t			tid;

could use better names for rb_node and node. rb_node is the entry in the dead_threads tree - dead_node?; node is the linked list for tid collisions - tid_node?

David

author maintainer
#47 lore raw

Re: [PATCH 14/37] perf tools: Convert dead thread list into rbtree

Hi David,

Quoted 6 lines · David Ahern
Quoted 5 lines · Namhyung Kim
tools/perf/util/machine.c +0 -0
@@ -106,8 +117,8 @@ void machine__delete_threads(struct machine *machine)
        while (nd) {
                struct thread *t = rb_entry(nd, struct thread, rb_node);
 
Quoted 9 lines

tools/perf/util/machine.c void machine__delete_threads(struct machine *machine) · line 124

Quoted 6 lines
tools/perf/util/machine.c +1 -1
-               rb_erase(&t->rb_node, &machine->threads);
                nd = rb_next(nd);
+               rb_erase(&t->rb_node, &machine->threads);
                thread__delete(t);
        }
  }

unrelated to dead threads. Bug fix? separate patch?

Yes, I'll make it a separate bug-fix patch.

Quoted 27 lines
Quoted 27 lines
tools/perf/util/machine.c +7 -0
@@ -1236,13 +1247,36 @@ int machine__process_mmap_event(struct machine
 *machine, union perf_event *event
 
  static void machine__remove_thread(struct machine *machine, struct
 thread *th)
  {
+       struct rb_node **p = &machine->dead_threads.rb_node;
+       struct rb_node *parent = NULL;
+       struct thread *pos;
+
        machine->last_match = NULL;
        rb_erase(&th->rb_node, &machine->threads);
+
+       th->dead = true;
+
        /*
         * We may have references to this thread, for instance in some

hist_entry

tools/perf/util/machine.c +4 -2
-        * instances, so just move them to a separate list.
+        * instances, so just move them to a separate list in rbtree.
         */
-       list_add_tail(&th->node, &machine->dead_threads);
+       while (*p != NULL) {
+               parent = *p;
+               pos = rb_entry(parent, struct thread, rb_node);
Quoted 8 lines

tools/perf/util/machine.c int machine__process_mmap_event(struct machine *machine, union perf_event *event · line 1270

Quoted 5 lines
tools/perf/util/machine.c +5 -0
+
+               if (pos->tid == th->tid) {
+                       list_add_tail(&th->node, &pos->node);
+                       return;
+               }

So you have a linked list for tid collisions (not mentioned in the git log).

Right, will add a description.

Quoted 27 lines
Quoted 15 lines
tools/perf/util/machine.c +9 -0
+
+               if (th->tid < pos->tid)
+                       p = &(*p)->rb_left;
+               else
+                       p = &(*p)->rb_right;
+       }
+
+       rb_link_node(&th->rb_node, parent, p);
+       rb_insert_color(&th->rb_node, &machine->dead_threads);
  }
 
  int machine__process_fork_event(struct machine *machine, union

perf_event *event,

Review int machine__for_each_thread(struct machine · line 1683

Quoted 1 line
(quoted diff) +0 -0
@@ -1649,7 +1683,7 @@ int machine__for_each_thread(struct machine
 *machine,

---8<---

Quoted 7 lines
tools/perf/util/thread.h +0 -0
index 0b6dcd70bc8b..413f28cf689b 100644
@@ -11,10 +11,8 @@
  struct thread_stack;
 
Quoted 15 lines

tools/perf/util/thread.h line 18

Quoted 10 lines
tools/perf/util/thread.h +2 -4
  struct thread {
-       union {
-               struct rb_node   rb_node;
-               struct list_head node;
-       };
+       struct rb_node          rb_node;
+       struct list_head        node;
        struct map_groups       *mg;
        pid_t                   pid_; /* Not all tools update this */
        pid_t                   tid;

could use better names for rb_node and node. rb_node is the entry in the dead_threads tree - dead_node?; node is the linked list for tid collisions - tid_node?

But the rb_node is used for 3 different purpose depends on its state - a thread can be in a (normal) threads tree, dead threads tree or missing threads tree (will be introduced later).

Thanks,
Namhyung

author maintainer
#48 lore raw

PATCH 15/37 perf tools: Introduce machine__find*_thread_time()

In reply to Namhyung Kim

With multi-file data storage is enabled, it needs to search thread based on sample time since sample processing is done after other (task, comm and mmap) events are processed. This can be a problem if a session is very long and pid is recycled - in that case it'll only see the last one.

So keep thread start time in it, and search thread based on the time. This patch introduces machine__find{,new}_thread_time() function for this. It'll first search current thread rbtree and then dead thread tree and list. If it couldn't find anyone, it'll create a new thread.

The sample timestamp of 0 means that this is called from synthesized event so just use current rbtree. The timestamp will be -1 if sample didn't record the timestamp so will see current threads automatically.

Dead threads are managed in a rbtree, and if there's more than one thread has sample tid/pid, it'll be saved in a list.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/builtin-script.c     |   5 +-
 tools/perf/tests/dwarf-unwind.c |  10 ++--
 tools/perf/tests/hists_common.c |   3 +-
 tools/perf/tests/hists_link.c   |   2 +-
 tools/perf/util/event.c         |   4 +-
 tools/perf/util/machine.c       | 109 +++++++++++++++++++++++++++++++++++++++-
 tools/perf/util/machine.h       |   8 ++-
 tools/perf/util/thread.c        |   4 ++
 tools/perf/util/thread.h        |   1 +
 9 files changed, 132 insertions(+), 14 deletions(-)
tools/perf/builtin-script.c +3 -2
index ce304dfd962a..85122b388d8e 100644
@@ -549,8 +549,9 @@ static int process_sample_event(struct perf_tool *tool __maybe_unused,
 				struct machine *machine)
 {
 	struct addr_location al;
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread = machine__findnew_thread_time(machine, sample->pid,
+							     sample->tid,
+							     sample->time);
 
 	if (thread == NULL) {
 		pr_debug("problem processing %d event, skipping it.\n",
tools/perf/tests/dwarf-unwind.c +5 -5
index ab28cca2cb97..dfecd49fef90 100644
@@ -13,10 +13,10 @@
 
 static int mmap_handler(struct perf_tool *tool __maybe_unused,
 			union perf_event *event,
-			struct perf_sample *sample __maybe_unused,
+			struct perf_sample *sample,
 			struct machine *machine)
 {
-	return machine__process_mmap2_event(machine, event, NULL);
+	return machine__process_mmap2_event(machine, event, sample);
 }
 
 static int init_live_machine(struct machine *machine)
@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
 __attribute__ ((noinline))
 static int unwind_thread(struct thread *thread)
 {
-	struct perf_sample sample;
+	struct perf_sample sample = {
+		.time = -1ULL,
+	};
 	unsigned long cnt = 0;
 	int err = -1;
 
-	memset(&sample, 0, sizeof(sample));
-
 	if (test__arch_unwind_sample(&sample, thread)) {
 		pr_debug("failed to get unwind sample\n");
 		goto out;
tools/perf/tests/hists_common.c +2 -1
index a62c09134516..86a8fdb41804 100644
@@ -80,6 +80,7 @@ static struct {
 struct machine *setup_fake_machine(struct machines *machines)
 {
 	struct machine *machine = machines__find(machines, HOST_KERNEL_ID);
+	struct perf_sample sample = { .time = -1ULL, };
 	size_t i;
 
 	if (machine == NULL) {
@@ -113,7 +114,7 @@ struct machine *setup_fake_machine(struct machines *machines)
 		strcpy(fake_mmap_event.mmap.filename,
 		       fake_mmap_info[i].filename);
 
-		machine__process_mmap_event(machine, &fake_mmap_event, NULL);
+		machine__process_mmap_event(machine, &fake_mmap_event, &sample);
 	}
 
 	for (i = 0; i < ARRAY_SIZE(fake_symbols); i++) {
tools/perf/tests/hists_link.c +1 -1
index fe7cb886c23e..07f1d19b88b5 100644
@@ -64,7 +64,7 @@ static int add_hist_entries(struct perf_evlist *evlist, struct machine *machine)
 	struct perf_evsel *evsel;
 	struct addr_location al;
 	struct hist_entry *he;
-	struct perf_sample sample = { .period = 1, };
+	struct perf_sample sample = { .period = 1, .time = -1ULL, };
 	size_t i = 0, k;
 
 	/*
tools/perf/util/event.c +2 -2
index 6c6d044e959a..ff7594a27c73 100644
@@ -825,8 +825,8 @@ int perf_event__preprocess_sample(const union perf_event *event,
 				  struct perf_sample *sample)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
-	struct thread *thread = machine__findnew_thread(machine, sample->pid,
-							sample->tid);
+	struct thread *thread = machine__findnew_thread_time(machine, sample->pid,
+							     sample->tid, sample->time);
 
 	if (thread == NULL)
 		return -1;
tools/perf/util/machine.c +108 -1

127 lines not on this page · raw message

tools/perf/util/machine.h +6 -2
index 4349946a38ff..9571b6b1c5b5 100644
@@ -68,8 +68,6 @@ static inline bool machine__kernel_ip(struct machine *machine, u64 ip)
 	return ip >= kernel_start;
 }
 
-struct thread *machine__find_thread(struct machine *machine, pid_t pid,
-				    pid_t tid);
 struct comm *machine__thread_exec_comm(struct machine *machine,
 				       struct thread *thread);
 
@@ -149,6 +147,12 @@ static inline bool machine__is_host(struct machine *machine)
 
 struct thread *machine__findnew_thread(struct machine *machine, pid_t pid,
 				       pid_t tid);
+struct thread *machine__find_thread(struct machine *machine, pid_t pid,
+				    pid_t tid);
+struct thread *machine__findnew_thread_time(struct machine *machine, pid_t pid,
+					    pid_t tid, u64 timestamp);
+struct thread *machine__find_thread_time(struct machine *machine, pid_t pid,
+					 pid_t tid, u64 timestamp);
 
 size_t machine__fprintf(struct machine *machine, FILE *fp);
 
tools/perf/util/thread.c +4 -0
index b9c5c5d5e718..f2465f17cf16 100644
@@ -128,6 +128,9 @@ int __thread__set_comm(struct thread *thread, const char *str, u64 timestamp,
 
 	/* Override the default :tid entry */
 	if (!thread->comm_set) {
+		if (!thread->start_time)
+			thread->start_time = timestamp;
+
 		err = comm__override(curr, str, timestamp, exec);
 		if (err)
 			return err;
@@ -229,6 +232,7 @@ int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp)
 	}
 
 	thread->ppid = parent->tid;
+	thread->start_time = timestamp;
 	return thread__clone_map_groups(thread, parent);
 }
 
tools/perf/util/thread.h +1 -0
index 413f28cf689b..5c07cee3b64e 100644
@@ -25,6 +25,7 @@ struct thread {
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
+	u64			start_time;
 
 	void			*priv;
 	struct thread_stack	*ts;
Signature · 1 line
2.1.3
#49 lore raw

Re: [PATCH 15/37] perf tools: Introduce machine__find*_thread_time()

tools/perf/tests/dwarf-unwind.c static int unwind_entry(struct unwind_entry *entry, void *arg) · line 66

Quoted 8 lines · Namhyung Kim
tools/perf/tests/dwarf-unwind.c +3 -1
@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry, void *arg)
  __attribute__ ((noinline))
  static int unwind_thread(struct thread *thread)
  {
-	struct perf_sample sample;
+	struct perf_sample sample = {
+		.time = -1ULL,
+	};

1-liner like the others?

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 434

Quoted 9 lines
tools/perf/util/machine.c +1 -0
index 582e011adc92..2cc088d71922 100644
@@ -431,6 +431,103 @@ struct thread *machine__find_thread(struct machine *machine, pid_t pid,
  	return __machine__findnew_thread(machine, pid, tid, false);
  }
 
+static void machine__remove_thread(struct machine *machine, struct thread *th);

Why is this declaration needed?

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 446

Quoted 12 lines
tools/perf/util/machine.c +12 -0
+
+static struct thread *__machine__findnew_thread_time(struct machine *machine,
+						     pid_t pid, pid_t tid,
+						     u64 timestamp, bool create)
+{
+	struct thread *curr, *pos, *new;
+	struct thread *th = NULL;
+	struct rb_node **p;
+	struct rb_node *parent = NULL;
+	bool initial = timestamp == (u64)0;
+
+	curr = __machine__findnew_thread(machine, pid, tid, initial);

What if create arg is false? Should initial arg also be false?

Quoted 19 lines
tools/perf/util/machine.c +19 -0
+	if (curr && timestamp >= curr->start_time)
+		return curr;
+
+	p = &machine->dead_threads.rb_node;
+	while (*p != NULL) {
+		parent = *p;
+		th = rb_entry(parent, struct thread, rb_node);
+
+		if (th->tid == tid) {
+			list_for_each_entry(pos, &th->node, node) {
+				if (timestamp >= pos->start_time &&
+				    pos->start_time > th->start_time) {
+					th = pos;
+					break;
+				}
+			}
+
+			if (timestamp >= th->start_time) {
+				machine__update_thread_pid(machine, th, pid);

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 475

Quoted 10 lines
tools/perf/util/machine.c +10 -0
+				return th;
+			}
+			break;
+		}
+
+		if (tid < th->tid)
+			p = &(*p)->rb_left;
+		else
+			p = &(*p)->rb_right;
+	}

Can the dead_threads search be put in a separate function -- machine__find_dead_thread?

Quoted 16 lines
tools/perf/util/machine.c +16 -0
+
+	if (!create)
+		return NULL;
+
+	if (!curr)
+		return __machine__findnew_thread(machine, pid, tid, true);
+
+	new = thread__new(pid, tid);
+	if (new == NULL)
+		return NULL;
+
+	new->start_time = timestamp;
+
+	if (*p) {
+		list_for_each_entry(pos, &th->node, node) {
+			/* sort by time */

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 501

Quoted 10 lines
tools/perf/util/machine.c +10 -0
+			if (timestamp >= pos->start_time) {
+				th = pos;
+				break;
+			}
+		}
+		list_add_tail(&new->node, &th->node);
+	} else {
+		rb_link_node(&new->rb_node, parent, p);
+		rb_insert_color(&new->rb_node, &machine->dead_threads);
+	}

Why insert this unknown tid on the dead_threads list?

Quoted 6 lines
tools/perf/util/machine.c +6 -0
+
+	/*
+	 * We have to initialize map_groups separately
+	 * after rb tree is updated.
+	 *
+	 * The reason is that we call machine__findnew_thread

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 517

Quoted 10 lines
tools/perf/util/machine.c +10 -0
+	 * within thread__init_map_groups to find the thread
+	 * leader and that would screwed the rb tree.
+	 */
+	if (thread__init_map_groups(new, machine)) {
+		thread__delete(new);
+		return NULL;
+	}
+
+	return new;
+}

---8<---

Quoted 4 lines
tools/perf/util/machine.c +0 -0
@@ -1265,6 +1362,16 @@ static void machine__remove_thread(struct machine *machine, struct thread *th)
  		pos = rb_entry(parent, struct thread, rb_node);
 
  		if (pos->tid == th->tid) {

tools/perf/util/machine.c static void machine__remove_thread(struct machine *machine, struct thread *th) · line 1374

Quoted 10 lines
tools/perf/util/machine.c +10 -0
+			struct thread *old;
+
+			/* sort by time */
+			list_for_each_entry(old, &pos->node, node) {
+				if (th->start_time >= old->start_time) {
+					pos = old;
+					break;
+				}
+			}
+

this change seems unrelated to the patch subject -- machine__find*_thread_time.

David

author maintainer
#50 lore raw

Re: [PATCH 15/37] perf tools: Introduce machine__find*_thread_time()

Quoted 14 lines · David Ahern

tools/perf/tests/dwarf-unwind.c static int unwind_entry(struct unwind_entry *entry, · line 67

Quoted 10 lines · Namhyung Kim
tools/perf/tests/dwarf-unwind.c +3 -1
@@ -61,12 +61,12 @@ static int unwind_entry(struct unwind_entry *entry,
 void *arg)
  __attribute__ ((noinline))
  static int unwind_thread(struct thread *thread)
  {
-       struct perf_sample sample;
+       struct perf_sample sample = {
+               .time = -1ULL,
+       };

1-liner like the others?

Did you mean this?

        struct perf_sample sample = { .time = -1ULL, };
Quoted 12 lines
Quoted 12 lines
tools/perf/util/machine.c +0 -0
index 582e011adc92..2cc088d71922 100644
@@ -431,6 +431,103 @@ struct thread *machine__find_thread(struct machine
 *machine, pid_t pid,
        return __machine__findnew_thread(machine, pid, tid, false);
  }
 
Quoted 5 lines

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine · line 436

Quoted 2 lines
tools/perf/util/machine.c +1 -0
+static void machine__remove_thread(struct machine *machine, struct thread
 *th);

Why is this declaration needed?

It seems like a leftover from previous changes, will remove.

Quoted 10 lines
Quoted 10 lines
tools/perf/util/machine.c +4 -0
+
+static struct thread *__machine__findnew_thread_time(struct machine
 *machine,
+                                                    pid_t pid, pid_t tid,
+                                                    u64 timestamp, bool

create)

tools/perf/util/machine.c +3 -0
+{
+       struct thread *curr, *pos, *new;
+       struct thread *th = NULL;
Quoted 8 lines

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 446

Quoted 5 lines
tools/perf/util/machine.c +5 -0
+       struct rb_node **p;
+       struct rb_node *parent = NULL;
+       bool initial = timestamp == (u64)0;
+
+       curr = __machine__findnew_thread(machine, pid, tid, initial);

What if create arg is false? Should initial arg also be false?

Nop, the create arg adds the thread to the dead thread tree (or missing thread tree later). And the initial flag adds it to the normal threads tree like in case of synthesized events. This was because I firstly used the *_findnew_thread_time() for the meta event processing, but then I realized that using *_findnew_thread() makes the processing much easier since the meta events are processed sequencially. And this *_findnew_thread_time() is used only for sample processing. Maybe we can use 'false' instead of the 'initial'..

Quoted 27 lines
Quoted 27 lines
tools/perf/util/machine.c +19 -0
+       if (curr && timestamp >= curr->start_time)
+               return curr;
+
+       p = &machine->dead_threads.rb_node;
+       while (*p != NULL) {
+               parent = *p;
+               th = rb_entry(parent, struct thread, rb_node);
+
+               if (th->tid == tid) {
+                       list_for_each_entry(pos, &th->node, node) {
+                               if (timestamp >= pos->start_time &&
+                                   pos->start_time > th->start_time) {
+                                       th = pos;
+                                       break;
+                               }
+                       }
+
+                       if (timestamp >= th->start_time) {
+                               machine__update_thread_pid(machine, th,

pid);

tools/perf/util/machine.c +6 -0
+                               return th;
+                       }
+                       break;
+               }
+
+               if (tid < th->tid)
Quoted 8 lines

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 475

Quoted 4 lines
tools/perf/util/machine.c +4 -0
+                       p = &(*p)->rb_left;
+               else
+                       p = &(*p)->rb_right;
+       }

Can the dead_threads search be put in a separate function -- machine__find_dead_thread?

Okay, I can do it.

Quoted 22 lines
Quoted 22 lines
tools/perf/util/machine.c +21 -0
+
+       if (!create)
+               return NULL;
+
+       if (!curr)
+               return __machine__findnew_thread(machine, pid, tid, true);
+
+       new = thread__new(pid, tid);
+       if (new == NULL)
+               return NULL;
+
+       new->start_time = timestamp;
+
+       if (*p) {
+               list_for_each_entry(pos, &th->node, node) {
+                       /* sort by time */
+                       if (timestamp >= pos->start_time) {
+                               th = pos;
+                               break;
+                       }
+               }
Quoted 8 lines

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 501

Quoted 5 lines
tools/perf/util/machine.c +5 -0
+               list_add_tail(&new->node, &th->node);
+       } else {
+               rb_link_node(&new->rb_node, parent, p);
+               rb_insert_color(&new->rb_node, &machine->dead_threads);
+       }

Why insert this unknown tid on the dead_threads list?

Well, mostly the search will be succeeded, and if it failed it's either newly created thread so curr = NULL and adds it to the normal tree. Otherwise (rarely) there might be a missing fork event and the sample is for an older thread in that any timestamp didn't match to existing (current and dead) threads. So I added it to the dead thread tree. But with multi-thread support enabled, it'll be added to the new missing threads tree which is protected by a mutex.

Quoted 25 lines
Quoted 7 lines
tools/perf/util/machine.c +6 -0
+
+       /*
+        * We have to initialize map_groups separately
+        * after rb tree is updated.
+        *
+        * The reason is that we call machine__findnew_thread

tools/perf/util/machine.c struct thread *machine__find_thread(struct machine *machine, pid_t pid, · line 517

Quoted 10 lines
tools/perf/util/machine.c +10 -0
+        * within thread__init_map_groups to find the thread
+        * leader and that would screwed the rb tree.
+        */
+       if (thread__init_map_groups(new, machine)) {
+               thread__delete(new);
+               return NULL;
+       }
+
+       return new;
+}

---8<---

Quoted 4 lines
tools/perf/util/machine.c +0 -0
@@ -1265,6 +1362,16 @@ static void machine__remove_thread(struct machine
 *machine, struct thread *th)
                pos = rb_entry(parent, struct thread, rb_node);
 
Quoted 15 lines

tools/perf/util/machine.c static void machine__remove_thread(struct machine *machine, struct thread *th) · line 1374

Quoted 11 lines
tools/perf/util/machine.c +10 -0
                if (pos->tid == th->tid) {
+                       struct thread *old;
+
+                       /* sort by time */
+                       list_for_each_entry(old, &pos->node, node) {
+                               if (th->start_time >= old->start_time) {
+                                       pos = old;
+                                       break;
+                               }
+                       }
+

this change seems unrelated to the patch subject -- machine__find*_thread_time.

It searchs a thread in the dead threads tree based on timestamp value. So the list should be sorted by time order. But yes, I agree that this change should be moved to the dead thread conversion patch.

Thanks,
Namhyung

author maintainer
#51 lore raw

PATCH 16/37 perf tools: Add a test case for timed thread handling

In reply to Namhyung Kim

A test case for verifying live and dead thread tree management during time change and new machine__find{,new}_thread_time().

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/Makefile.perf              |   1 +
 tools/perf/tests/builtin-test.c       |   4 +
 tools/perf/tests/tests.h              |   1 +
 tools/perf/tests/thread-lookup-time.c | 174 ++++++++++++++++++++++++++++++++++
 4 files changed, 180 insertions(+)
 create mode 100644 tools/perf/tests/thread-lookup-time.c
tools/perf/Makefile.perf +1 -0
index e4528a4a3a8c..6094f0a10d8b 100644
@@ -448,6 +448,7 @@ LIB_OBJS += $(OUTPUT)tests/mmap-thread-lookup.o
 LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
 LIB_OBJS += $(OUTPUT)tests/thread-comm.o
+LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
index 1b463d82a71a..e4d335de19ea 100644
@@ -171,6 +171,10 @@ static struct test {
 		.func = test__thread_comm,
 	},
 	{
+		.desc = "Test thread lookup with time",
+		.func = test__thread_lookup_time,
+	},
+	{
 		.func = NULL,
 	},
 };
tools/perf/tests/tests.h +1 -0
index 43ac17780629..1090337f63e5 100644
@@ -52,6 +52,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-lookup-time.c new file +174 -0

174 lines not on this page · raw message

Signature · 1 line
2.1.3
#52 lore raw

Re: [PATCH 16/37] perf tools: Add a test case for timed thread handling

On Wed, Dec 24, 2014 at 04:15:12PM +0900, Namhyung Kim wrote:

SNIP

Quoted 34 lines
tools/perf/tests/builtin-test.c +0 -0
 		.func = NULL,
 	},
 };
tools/perf/tests/tests.h +1 -0
index 43ac17780629..1090337f63e5 100644
@@ -52,6 +52,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-lookup-time.c new file +13 -0
new file mode 100644
index 000000000000..6237ecf8caae
@@ -0,0 +1,174 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
+{
+	printf("thread: %d, start time: %"PRIu64" %s\n",
+	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	return 0;
+}
+

tools/perf/tests/thread-lookup-time.c line 23

Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10 -0
+static int lookup_with_timestamp(struct machine *machine)
+{
+	struct thread *t1, *t2, *t3;
+	union perf_event fork = {
+		.fork = {
+			.pid = 0,
+			.tid = 0,
+			.ppid = 1,
+			.ptid = 1,
+		},

I've got following output from the test:

test child forked, pid 18483
========= after t1 created ==========
thread: 0, start time: 0
========= after t1 set comm ==========
thread: 0, start time: 20000
========= after t2 forked ==========
thread: 0, start time: 50000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 20000 (dead)
========= after t3 forked ==========
thread: 0, start time: 60000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 50000 (dead)
thread: 0, start time: 20000 (dead)
test child finished with 0

'after t2 forked' data shows 'thread 0 with time 50000' and newly added parent 'thread: 1, start time: 0'

this makes me wonder if you wanted switch 0 and 1 for pid and ppid in above sample init and follow with forked pid 1 ... but not sure because you're using the same sample for fork 3 ;-)

my question is if that was intentional, because I've got confused in here

Quoted 24 lines
tools/perf/tests/thread-lookup-time.c new file +24 -0
+	};
+	struct perf_sample sample = {
+		.time = 50000,
+	};
+
+	/* start_time is set to 0 */
+	t1 = machine__findnew_thread(machine, 0, 0);
+
+	if (verbose > 1) {
+		printf("========= after t1 created ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("wrong start time of old thread", t1->start_time == 0);
+
+	TEST_ASSERT_VAL("cannot find current thread",
+			machine__find_thread(machine, 0, 0) == t1);
+
+	TEST_ASSERT_VAL("cannot find current thread with time",
+			machine__findnew_thread_time(machine, 0, 0, 10000) == t1);
+
+	/* start_time is overwritten to new value */
+	thread__set_comm(t1, "/usr/bin/perf", 20000);
+

tools/perf/tests/thread-lookup-time.c line 57

Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10 -0
+	if (verbose > 1) {
+		printf("========= after t1 set comm ==========\n");
+		machine__for_each_thread(machine, thread__print_cb, NULL);
+	}
+
+	TEST_ASSERT_VAL("failed to update start time", t1->start_time == 20000);
+
+	TEST_ASSERT_VAL("should not find passed thread",
+			/* this will create yet another dead thread */
+			machine__findnew_thread_time(machine, 0, 0, 10000) != t1);

also this comment say that calling machine__findnew_thread_time will create another dead thread, which actually did not happened based on above test output

thanks,
jirka

author maintainer
#53 lore raw

Re: [PATCH 16/37] perf tools: Add a test case for timed thread handling

Quoted 72 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:15:12PM +0900, Namhyung Kim wrote:

SNIP

Quoted 34 lines

.func = NULL,
},
};

tools/perf/tests/tests.h +1 -0
index 43ac17780629..1090337f63e5 100644
@@ -52,6 +52,7 @@ int test__switch_tracking(void);
 int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
+int test__thread_lookup_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-lookup-time.c new file +13 -0
new file mode 100644
index 000000000000..6237ecf8caae
@@ -0,0 +1,174 @@
+#include "tests.h"
+#include "machine.h"
+#include "thread.h"
+#include "map.h"
+#include "debug.h"
+
+static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
+{
+     printf("thread: %d, start time: %"PRIu64" %s\n",
+            th->tid, th->start_time, th->dead ? "(dead)" : "");
+     return 0;
+}
+

tools/perf/tests/thread-lookup-time.c line 23

Quoted 10 lines
tools/perf/tests/thread-lookup-time.c new file +10 -0
+static int lookup_with_timestamp(struct machine *machine)
+{
+     struct thread *t1, *t2, *t3;
+     union perf_event fork = {
+             .fork = {
+                     .pid = 0,
+                     .tid = 0,
+                     .ppid = 1,
+                     .ptid = 1,
+             },

I've got following output from the test:

test child forked, pid 18483
========= after t1 created ==========
thread: 0, start time: 0
========= after t1 set comm ==========
thread: 0, start time: 20000
========= after t2 forked ==========
thread: 0, start time: 50000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 20000 (dead)
========= after t3 forked ==========
thread: 0, start time: 60000
thread: 1, start time: 0
thread: 0, start time: 10000
thread: 0, start time: 50000 (dead)
thread: 0, start time: 20000 (dead)
test child finished with 0

'after t2 forked' data shows 'thread 0 with time 50000' and newly added parent 'thread: 1, start time: 0'

Quoted 6 lines

this makes me wonder if you wanted switch 0 and 1 for pid and ppid in above sample init and follow with forked pid 1 ... but not sure because you're using the same sample for fork 3 ;-)

my question is if that was intentional, because I've got confused in here

Yeah, it's intentional. I'm testing machine__findnew_thread_time() and machine__process_fork_event() can generate threads properly. The former creates a dead thread if the timestamp is before any of existing threads which have a same pid. The latter can create two threads - one for tid and another for ptid (only if it doesn't exist).

Quoted 31 lines
Quoted 31 lines
tools/perf/tests/thread-lookup-time.c new file +30 -0
+     };
+     struct perf_sample sample = {
+             .time = 50000,
+     };
+
+     /* start_time is set to 0 */
+     t1 = machine__findnew_thread(machine, 0, 0);
+
+     if (verbose > 1) {
+             printf("========= after t1 created ==========\n");
+             machine__for_each_thread(machine, thread__print_cb, NULL);
+     }
+
+     TEST_ASSERT_VAL("wrong start time of old thread", t1->start_time == 0);
+
+     TEST_ASSERT_VAL("cannot find current thread",
+                     machine__find_thread(machine, 0, 0) == t1);
+
+     TEST_ASSERT_VAL("cannot find current thread with time",
+                     machine__findnew_thread_time(machine, 0, 0, 10000) == t1);
+
+     /* start_time is overwritten to new value */
+     thread__set_comm(t1, "/usr/bin/perf", 20000);
+
+     if (verbose > 1) {
+             printf("========= after t1 set comm ==========\n");
+             machine__for_each_thread(machine, thread__print_cb, NULL);
+     }
+
+     TEST_ASSERT_VAL("failed to update start time", t1->start_time == 20000);
Quoted 8 lines

tools/perf/tests/thread-lookup-time.c line 57

Quoted 4 lines
tools/perf/tests/thread-lookup-time.c new file +4 -0
+
+     TEST_ASSERT_VAL("should not find passed thread",
+                     /* this will create yet another dead thread */
+                     machine__findnew_thread_time(machine, 0, 0, 10000) != t1);

also this comment say that calling machine__findnew_thread_time will create another dead thread, which actually did not happened based on above test output

Oh, it's actually a dead thread - it's in the dead threads tree - but I just missed to set the dead flag. :)

Thanks,
Namhyung

author maintainer
#54 lore raw

PATCH 17/37 perf tools: Maintain map groups list in a leader thread

In reply to Namhyung Kim

To support multi-threaded perf report, we need to maintain time-sorted map groups. Add ->mg_list member to struct thread and sort the list by time. Now leader threads have one more refcnt for map groups in the list so also update the thread-mg-share test case.

Currently only add a new map groups when an exec (comm) event is received.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/tests/thread-mg-share.c |  7 +++-
 tools/perf/util/event.c            |  2 +
 tools/perf/util/machine.c          |  2 +-
 tools/perf/util/map.c              |  1 +
 tools/perf/util/map.h              |  2 +
 tools/perf/util/thread.c           | 80 +++++++++++++++++++++++++++++++++++++-
 tools/perf/util/thread.h           |  3 ++
 7 files changed, 93 insertions(+), 4 deletions(-)
tools/perf/tests/thread-mg-share.c +5 -2
index b028499dd3cf..8933e01d0549 100644
@@ -23,6 +23,9 @@ int test__thread_mg_share(void)
 	 * with several threads and checks they properly share and
 	 * maintain map groups info (struct map_groups).
 	 *
+	 * Note that a leader thread has one more refcnt for its
+	 * (current) map groups.
+	 *
 	 * thread group (pid: 0, tids: 0, 1, 2, 3)
 	 * other  group (pid: 4, tids: 4, 5)
 	*/
@@ -43,7 +46,7 @@ int test__thread_mg_share(void)
 			leader && t1 && t2 && t3 && other);
 
 	mg = leader->mg;
-	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 4);
+	TEST_ASSERT_VAL("wrong refcnt", mg->refcnt == 5);
 
 	/* test the map groups pointer is shared */
 	TEST_ASSERT_VAL("map groups don't match", mg == t1->mg);
@@ -59,7 +62,7 @@ int test__thread_mg_share(void)
 	TEST_ASSERT_VAL("failed to find other leader", other_leader);
 
 	other_mg = other->mg;
-	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 2);
+	TEST_ASSERT_VAL("wrong refcnt", other_mg->refcnt == 3);
 
 	TEST_ASSERT_VAL("map groups don't match", other_mg == other_leader->mg);
 
tools/perf/util/event.c +2 -0
index ff7594a27c73..2d04949bdc7d 100644
@@ -750,6 +750,8 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 		return;
 	}
 
+	BUG_ON(mg == NULL);
+
 	if (cpumode == PERF_RECORD_MISC_KERNEL && perf_host) {
 		al->level = 'k';
 		mg = &machine->kmaps;
tools/perf/util/machine.c +1 -1
index 2cc088d71922..031bace39fdc 100644
@@ -330,7 +330,7 @@ static void machine__update_thread_pid(struct machine *machine,
 		goto out_err;
 
 	if (!leader->mg)
-		leader->mg = map_groups__new(machine);
+		thread__set_map_groups(leader, map_groups__new(machine), 0);
 
 	if (!leader->mg)
 		goto out_err;
tools/perf/util/map.c +1 -0
index 62ca9f2607d5..f0c1e2a24fee 100644
@@ -422,6 +422,7 @@ void map_groups__init(struct map_groups *mg, struct machine *machine)
 	}
 	mg->machine = machine;
 	mg->refcnt = 1;
+	mg->timestamp = 0;
 }
 
 static void maps__delete(struct rb_root *maps)
tools/perf/util/map.h +2 -0
index 6951a9d42339..dd92510b03d1 100644
@@ -61,7 +61,9 @@ struct map_groups {
 	struct rb_root	 maps[MAP__NR_TYPES];
 	struct list_head removed_maps[MAP__NR_TYPES];
 	struct machine	 *machine;
+	u64		 timestamp;
 	int		 refcnt;
+	struct list_head list;
 };
 
 struct map_groups *map_groups__new(struct machine *machine);
tools/perf/util/thread.c +79 -1

116 lines not on this page · raw message

tools/perf/util/thread.h +3 -0
index 5c07cee3b64e..8b9a67764613 100644
@@ -14,6 +14,7 @@ struct thread {
 	struct rb_node	 	rb_node;
 	struct list_head 	node;
 	struct map_groups	*mg;
+	struct list_head	mg_list;
 	pid_t			pid_; /* Not all tools update this */
 	pid_t			tid;
 	pid_t			ppid;
@@ -56,6 +57,8 @@ struct comm *thread__exec_comm(const struct thread *thread);
 struct comm *thread__comm_time(const struct thread *thread, u64 timestamp);
 const char *thread__comm_str(const struct thread *thread);
 const char *thread__comm_time_str(const struct thread *thread, u64 timestamp);
+struct map_groups *thread__get_map_groups(struct thread *thread, u64 timestamp);
+int thread__set_map_groups(struct thread *thread, struct map_groups *mg, u64 timestamp);
 void thread__insert_map(struct thread *thread, struct map *map);
 int thread__fork(struct thread *thread, struct thread *parent, u64 timestamp);
 size_t thread__fprintf(struct thread *thread, FILE *fp);
Signature · 1 line
2.1.3
author maintainer
#55 lore raw

PATCH 18/37 perf tools: Remove thread when map groups initialization failed

In reply to Namhyung Kim

Otherwise it'll break the machine->threads tree.

---
 tools/perf/util/machine.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/machine.c +1 -0
index 031bace39fdc..beae6e8fe789 100644
@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,
 		 * leader and that would screwed the rb tree.
 		 */
 		if (thread__init_map_groups(th, machine)) {
+			rb_erase(&th->rb_node, &machine->threads);
 			thread__delete(th);
 			return NULL;
 		}
Signature · 1 line
2.1.3
#56 lore raw

Re: [PATCH 18/37] perf tools: Remove thread when map groups initialization failed

Quoted 13 lines · Namhyung Kim

Otherwise it'll break the machine->threads tree.

---
  tools/perf/util/machine.c | 1 +
  1 file changed, 1 insertion(+)
tools/perf/util/machine.c +0 -0
index 031bace39fdc..beae6e8fe789 100644

tools/perf/util/machine.c static struct thread *__machine__findnew_thread(struct machine *machine, · line 417

Quoted 7 lines
tools/perf/util/machine.c +1 -0
@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,
  		 * leader and that would screwed the rb tree.
  		 */
  		if (thread__init_map_groups(th, machine)) {
+			rb_erase(&th->rb_node, &machine->threads);
  			thread__delete(th);
  			return NULL;
  		}

Can you move the thread__init_map_groups() before the thread is added to the rbtree? If no, you need to delay setting 'machine->last_match = th' otherwise it references a deleted thread.

David

author maintainer
#57 lore raw

Re: [PATCH 18/37] perf tools: Remove thread when map groups initialization failed

Quoted 17 lines · David Ahern
Quoted 16 lines · Namhyung Kim

Otherwise it'll break the machine->threads tree.

---
 tools/perf/util/machine.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/util/machine.c +1 -0
index 031bace39fdc..beae6e8fe789 100644
@@ -411,6 +411,7 @@ static struct thread *__machine__findnew_thread(struct machine *machine,
 		 * leader and that would screwed the rb tree.
 		 */
 		if (thread__init_map_groups(th, machine)) {
+			rb_erase(&th->rb_node, &machine->threads);
Quoted 8 lines

tools/perf/util/machine.c static struct thread *__machine__findnew_thread(struct machine *machine, · line 417

Quoted 4 lines
tools/perf/util/machine.c +0 -0
 			thread__delete(th);
 			return NULL;
 		}

Can you move the thread__init_map_groups() before the thread is added to the rbtree? If no, you need to delay setting 'machine->last_match = th' otherwise it references a deleted thread.

You're right - I'll move the setting after the thread__init_map_groups().

Thanks,
Namhyung

author maintainer
#58 lore raw

PATCH 19/37 perf tools: Introduce thread__find_addr_location_time() and friends

In reply to Namhyung Kim

The *_time() variants are for find appropriate map (and symbol) at the given time. This is based on the fact that map_groups list is sorted by time in the previous patch.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/util/event.c            | 59 ++++++++++++++++++++++++++++++++++----
 tools/perf/util/machine.c          | 51 ++++++++++++++++++--------------
 tools/perf/util/thread.c           | 21 ++++++++++++++
 tools/perf/util/thread.h           | 10 +++++++
 tools/perf/util/unwind-libdw.c     | 11 +++----
 tools/perf/util/unwind-libunwind.c | 18 ++++++------
 6 files changed, 129 insertions(+), 41 deletions(-)
tools/perf/util/event.c +53 -6
index 2d04949bdc7d..3bb186a26314 100644
@@ -731,16 +731,14 @@ int perf_event__process(struct perf_tool *tool __maybe_unused,
 	return machine__process_event(machine, event, sample);
 }
 
-void thread__find_addr_map(struct thread *thread, u8 cpumode,
-			   enum map_type type, u64 addr,
-			   struct addr_location *al)
+static void map_groups__find_addr_map(struct map_groups *mg, u8 cpumode,
+				      enum map_type type, u64 addr,
+				      struct addr_location *al)
 {
-	struct map_groups *mg = thread->mg;
 	struct machine *machine = mg->machine;
 	bool load_map = false;
 
 	al->machine = machine;
-	al->thread = thread;
 	al->addr = addr;
 	al->cpumode = cpumode;
 	al->filtered = 0;
@@ -809,6 +807,35 @@ void thread__find_addr_map(struct thread *thread, u8 cpumode,
 	}
 }
 
+void thread__find_addr_map(struct thread *thread, u8 cpumode,
+			   enum map_type type, u64 addr,
+			   struct addr_location *al)
+{
+	al->thread = thread;
+	map_groups__find_addr_map(thread->mg, cpumode, type, addr, al);
+}
+
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+	struct thread *leader;
+
+	if (thread->tid == thread->pid_)
+		leader = thread;
+	else
+		leader = machine__find_thread(thread->mg->machine,
+					      thread->pid_, thread->pid_);
+
+	BUG_ON(leader == NULL);
+
+	mg = thread__get_map_groups(leader, timestamp);
+
+	al->thread = thread;
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+}
+
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al)
@@ -821,6 +848,21 @@ void thread__find_addr_location(struct thread *thread,
 		al->sym = NULL;
 }
 
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp)
+{
+	struct map_groups *mg;
+
+	mg = thread__get_map_groups(thread, timestamp);
+	map_groups__find_addr_map(mg, cpumode, type, addr, al);
+	if (al->map != NULL)
+		al->sym = map__find_symbol(al->map, al->addr,
+					   mg->machine->symbol_filter);
+	else
+		al->sym = NULL;
+}
+
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
@@ -845,7 +887,12 @@ int perf_event__preprocess_sample(const union perf_event *event,
 	    machine->vmlinux_maps[MAP__FUNCTION] == NULL)
 		machine__create_kernel_maps(machine);
 
-	thread__find_addr_map(thread, cpumode, MAP__FUNCTION, sample->ip, al);
+	/*
+	 * sample->time is -1ULL if !PERF_SAMPLE_TIME which ends up
+	 * with using most recent map_groups (same as default behavior).
+	 */
+	thread__find_addr_map_time(thread, cpumode, MAP__FUNCTION, sample->ip,
+				   al, sample->time);
 	dump_printf(" ...... dso: %s\n",
 		    al->map ? al->map->dso->long_name :
 			al->level == 'H' ? "[hypervisor]" : "<not found>");
tools/perf/util/machine.c +30 -21

131 lines not on this page · raw message

tools/perf/util/thread.c +21 -0
index 109ceb5e2a85..87c20308b91f 100644
@@ -332,3 +332,24 @@ void thread__find_cpumode_addr_location(struct thread *thread,
 			break;
 	}
 }
+
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp)
+{
+	size_t i;
+	const u8 const cpumodes[] = {
+		PERF_RECORD_MISC_USER,
+		PERF_RECORD_MISC_KERNEL,
+		PERF_RECORD_MISC_GUEST_USER,
+		PERF_RECORD_MISC_GUEST_KERNEL
+	};
+
+	for (i = 0; i < ARRAY_SIZE(cpumodes); i++) {
+		thread__find_addr_location_time(thread, cpumodes[i], type,
+						addr, al, timestamp);
+		if (al->map)
+			break;
+	}
+}
tools/perf/util/thread.h +10 -0
index 8b9a67764613..0b88ca22bc3d 100644
@@ -66,14 +66,24 @@ size_t thread__fprintf(struct thread *thread, FILE *fp);
 void thread__find_addr_map(struct thread *thread,
 			   u8 cpumode, enum map_type type, u64 addr,
 			   struct addr_location *al);
+void thread__find_addr_map_time(struct thread *thread, u8 cpumode,
+				enum map_type type, u64 addr,
+				struct addr_location *al, u64 timestamp);
 
 void thread__find_addr_location(struct thread *thread,
 				u8 cpumode, enum map_type type, u64 addr,
 				struct addr_location *al);
+void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
+				     enum map_type type, u64 addr,
+				     struct addr_location *al, u64 timestamp);
 
 void thread__find_cpumode_addr_location(struct thread *thread,
 					enum map_type type, u64 addr,
 					struct addr_location *al);
+void thread__find_cpumode_addr_location_time(struct thread *thread,
+					     enum map_type type, u64 addr,
+					     struct addr_location *al,
+					     u64 timestamp);
 
 static inline void *thread__priv(struct thread *thread)
 {
tools/perf/util/unwind-libdw.c +6 -5
index 2dcfe9a7c8d0..ba8d8e41d680 100644
@@ -26,9 +26,10 @@ static int __report_module(struct addr_location *al, u64 ip,
 	Dwfl_Module *mod;
 	struct dso *dso = NULL;
 
-	thread__find_addr_location(ui->thread,
-				   PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, al);
+	thread__find_addr_location_time(ui->thread,
+					PERF_RECORD_MISC_USER,
+					MAP__FUNCTION, ip, al,
+					ui->sample->time);
 
 	if (al->map)
 		dso = al->map->dso;
@@ -89,8 +90,8 @@ static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, addr, &al, ui->sample->time);
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
tools/perf/util/unwind-libunwind.c +9 -9
index 371219a6daf1..94929ec491f3 100644
@@ -284,8 +284,8 @@ static struct map *find_map(unw_word_t ip, struct unwind_info *ui)
 {
 	struct addr_location al;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, ip, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, ip, &al, ui->sample->time);
 	return al.map;
 }
 
@@ -374,8 +374,8 @@ static int access_dso_mem(struct unwind_info *ui, unw_word_t addr,
 	struct addr_location al;
 	ssize_t size;
 
-	thread__find_addr_map(ui->thread, PERF_RECORD_MISC_USER,
-			      MAP__FUNCTION, addr, &al);
+	thread__find_addr_map_time(ui->thread, PERF_RECORD_MISC_USER,
+				   MAP__FUNCTION, addr, &al, ui->sample->time);
 	if (!al.map) {
 		pr_debug("unwind: no map for %lx\n", (unsigned long)addr);
 		return -1;
@@ -476,14 +476,14 @@ static void put_unwind_info(unw_addr_space_t __maybe_unused as,
 	pr_debug("unwind: put_unwind_info called\n");
 }
 
-static int entry(u64 ip, struct thread *thread,
+static int entry(u64 ip, struct thread *thread, u64 timestamp,
 		 unwind_entry_cb_t cb, void *arg)
 {
 	struct unwind_entry e;
 	struct addr_location al;
 
-	thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
-				   MAP__FUNCTION, ip, &al);
+	thread__find_addr_location_time(thread, PERF_RECORD_MISC_USER,
+					MAP__FUNCTION, ip, &al, timestamp);
 
 	e.ip = ip;
 	e.map = al.map;
@@ -585,7 +585,7 @@ static int get_entries(struct unwind_info *ui, unwind_entry_cb_t cb,
 		unw_word_t ip;
 
 		unw_get_reg(&c, UNW_REG_IP, &ip);
-		ret = ip ? entry(ip, ui->thread, cb, arg) : 0;
+		ret = ip ? entry(ip, ui->thread, ui->sample->time, cb, arg) : 0;
 	}
 
 	return ret;
@@ -610,7 +610,7 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (ret)
 		return ret;
 
-	ret = entry(ip, thread, cb, arg);
+	ret = entry(ip, thread, data->time, cb, arg);
 	if (ret)
 		return -ENOMEM;
 
Signature · 1 line
2.1.3
author maintainer
#59 lore raw

PATCH 20/37 perf tools: Add a test case for timed map groups handling

In reply to Namhyung Kim

A test case for verifying thread->mg and ->mg_list handling during time change and new thread__find_addr_map_time() and friends.

Cc Frederic Weisbecker <[email protected]>
---
 tools/perf/Makefile.perf          |  1 +
 tools/perf/tests/builtin-test.c   |  4 ++
 tools/perf/tests/tests.h          |  1 +
 tools/perf/tests/thread-mg-time.c | 88 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 94 insertions(+)
 create mode 100644 tools/perf/tests/thread-mg-time.c
tools/perf/Makefile.perf +1 -0
index 6094f0a10d8b..47d933454492 100644
@@ -449,6 +449,7 @@ LIB_OBJS += $(OUTPUT)tests/thread-mg-share.o
 LIB_OBJS += $(OUTPUT)tests/switch-tracking.o
 LIB_OBJS += $(OUTPUT)tests/thread-comm.o
 LIB_OBJS += $(OUTPUT)tests/thread-lookup-time.o
+LIB_OBJS += $(OUTPUT)tests/thread-mg-time.o
 
 BUILTIN_OBJS += $(OUTPUT)builtin-annotate.o
 BUILTIN_OBJS += $(OUTPUT)builtin-bench.o
tools/perf/tests/builtin-test.c +4 -0
index e4d335de19ea..8f61a7e291ee 100644
@@ -175,6 +175,10 @@ static struct test {
 		.func = test__thread_lookup_time,
 	},
 	{
+		.desc = "Test thread map group handling with time",
+		.func = test__thread_mg_time,
+	},
+	{
 		.func = NULL,
 	},
 };
tools/perf/tests/tests.h +1 -0
index 1090337f63e5..03557563f31d 100644
@@ -53,6 +53,7 @@ int test__fdarray__filter(void);
 int test__fdarray__add(void);
 int test__thread_comm(void);
 int test__thread_lookup_time(void);
+int test__thread_mg_time(void);
 
 #if defined(__x86_64__) || defined(__i386__) || defined(__arm__)
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
tools/perf/tests/thread-mg-time.c new file +88 -0

88 lines not on this page · raw message

Signature · 1 line
2.1.3
author maintainer
#60 lore raw

PATCH 21/37 perf tools: Protect dso symbol loading using a mutex

In reply to Namhyung Kim

When multi-thread support for perf report is enabled, it's possible to access a dso concurrently. Add a new pthread_mutex to protect it from concurrent dso__load().

---
 tools/perf/util/dso.c    |  2 ++
 tools/perf/util/dso.h    |  1 +
 tools/perf/util/symbol.c | 34 ++++++++++++++++++++++++----------
 3 files changed, 27 insertions(+), 10 deletions(-)
tools/perf/util/dso.c +2 -0
index 45be944d450a..3da75816b8f8 100644
@@ -888,6 +888,7 @@ struct dso *dso__new(const char *name)
 		RB_CLEAR_NODE(&dso->rb_node);
 		INIT_LIST_HEAD(&dso->node);
 		INIT_LIST_HEAD(&dso->data.open_entry);
+		pthread_mutex_init(&dso->lock, NULL);
 	}
 
 	return dso;
@@ -917,6 +918,7 @@ void dso__delete(struct dso *dso)
 	dso_cache__free(&dso->data.cache);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
+	pthread_mutex_destroy(&dso->lock);
 	free(dso);
 }
 
tools/perf/util/dso.h +1 -0
index 3782c82c6e44..ac753594a469 100644
@@ -102,6 +102,7 @@ struct dsos {
 };
 
 struct dso {
+	pthread_mutex_t	 lock;
 	struct list_head node;
 	struct rb_node	 rb_node;	/* rbtree node sorted by long name */
 	struct rb_root	 symbols[MAP__NR_TYPES];
tools/perf/util/symbol.c +24 -10
index c24c5b83156c..6c764a5b3464 100644
@@ -1332,12 +1332,22 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 	struct symsrc *syms_ss = NULL, *runtime_ss = NULL;
 	bool kmod;
 
-	dso__set_loaded(dso, map->type);
+	pthread_mutex_lock(&dso->lock);
+
+	/* check again under the dso->lock */
+	if (dso__loaded(dso, map->type)) {
+		ret = 1;
+		goto out;
+	}
+
+	if (dso->kernel) {
+		if (dso->kernel == DSO_TYPE_KERNEL)
+			ret = dso__load_kernel_sym(dso, map, filter);
+		else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
+			ret = dso__load_guest_kernel_sym(dso, map, filter);
 
-	if (dso->kernel == DSO_TYPE_KERNEL)
-		return dso__load_kernel_sym(dso, map, filter);
-	else if (dso->kernel == DSO_TYPE_GUEST_KERNEL)
-		return dso__load_guest_kernel_sym(dso, map, filter);
+		goto out;
+	}
 
 	if (map->groups && map->groups->machine)
 		machine = map->groups->machine;
@@ -1350,18 +1360,18 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 		struct stat st;
 
 		if (lstat(dso->name, &st) < 0)
-			return -1;
+			goto out;
 
 		if (st.st_uid && (st.st_uid != geteuid())) {
 			pr_warning("File %s not owned by current user or root, "
 				"ignoring it.\n", dso->name);
-			return -1;
+			goto out;
 		}
 
 		ret = dso__load_perf_map(dso, map, filter);
 		dso->symtab_type = ret > 0 ? DSO_BINARY_TYPE__JAVA_JIT :
 					     DSO_BINARY_TYPE__NOT_FOUND;
-		return ret;
+		goto out;
 	}
 
 	if (machine)
@@ -1369,7 +1379,7 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 
 	name = malloc(PATH_MAX);
 	if (!name)
-		return -1;
+		goto out;
 
 	kmod = dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE ||
 		dso->symtab_type == DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE_COMP ||
@@ -1450,7 +1460,11 @@ int dso__load(struct dso *dso, struct map *map, symbol_filter_t filter)
 out_free:
 	free(name);
 	if (ret < 0 && strstr(dso->name, " (deleted)") != NULL)
-		return 0;
+		ret = 0;
+out:
+	dso__set_loaded(dso, map->type);
+	pthread_mutex_unlock(&dso->lock);
+
 	return ret;
 }
 
Signature · 1 line
2.1.3
author maintainer
#61 lore raw

PATCH 22/37 perf tools: Protect dso cache tree using dso->lock

In reply to Namhyung Kim

The dso cache is accessed during dwarf callchain unwind and it might be processed concurrently when multi-thread report is enabled. Protect it under dso->lock.

---
 tools/perf/util/dso.c | 41 ++++++++++++++++++++++++++++++++---------
 1 file changed, 32 insertions(+), 9 deletions(-)
tools/perf/util/dso.c +32 -9
index 3da75816b8f8..9555b1772fb5 100644
@@ -443,10 +443,12 @@ bool dso__data_status_seen(struct dso *dso, enum dso_data_status_seen by)
 }
 
 static void
-dso_cache__free(struct rb_root *root)
+dso_cache__free(struct dso *dso)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node *next = rb_first(root);
 
+	pthread_mutex_lock(&dso->lock);
 	while (next) {
 		struct dso_cache *cache;
 
@@ -455,14 +457,17 @@ dso_cache__free(struct rb_root *root)
 		rb_erase(&cache->rb_node, root);
 		free(cache);
 	}
+	pthread_mutex_unlock(&dso->lock);
 }
 
-static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
+static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
+	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -475,19 +480,24 @@ static struct dso_cache *dso_cache__find(const struct rb_root *root, u64 offset)
 		else if (offset >= end)
 			p = &(*p)->rb_right;
 		else
-			return cache;
+			goto out;
 	}
-	return NULL;
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
-static void
-dso_cache__insert(struct rb_root *root, struct dso_cache *new)
+static struct dso_cache *
+dso_cache__insert(struct dso *dso, struct dso_cache *new)
 {
+	struct rb_root *root = &dso->data.cache;
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
 	struct dso_cache *cache;
 	u64 offset = new->offset;
 
+	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
 		u64 end;
 
@@ -499,10 +509,17 @@ dso_cache__insert(struct rb_root *root, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
+		else
+			goto out;
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
+
+	cache = NULL;
+out:
+	pthread_mutex_unlock(&dso->lock);
+	return cache;
 }
 
 static ssize_t
@@ -520,6 +537,7 @@ static ssize_t
 dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 {
 	struct dso_cache *cache;
+	struct dso_cache *old;
 	ssize_t ret;
 
 	do {
@@ -543,7 +561,12 @@ dso_cache__read(struct dso *dso, u64 offset, u8 *data, ssize_t size)
 
 		cache->offset = cache_offset;
 		cache->size   = ret;
-		dso_cache__insert(&dso->data.cache, cache);
+		old = dso_cache__insert(dso, cache);
+		if (old) {
+			/* we lose the race */
+			free(cache);
+			cache = old;
+		}
 
 		ret = dso_cache__memcpy(cache, offset, data, size);
 
@@ -560,7 +583,7 @@ static ssize_t dso_cache_read(struct dso *dso, u64 offset,
 {
 	struct dso_cache *cache;
 
-	cache = dso_cache__find(&dso->data.cache, offset);
+	cache = dso_cache__find(dso, offset);
 	if (cache)
 		return dso_cache__memcpy(cache, offset, data, size);
 	else
@@ -915,7 +938,7 @@ void dso__delete(struct dso *dso)
 	}
 
 	dso__data_close(dso);
-	dso_cache__free(&dso->data.cache);
+	dso_cache__free(dso);
 	dso__free_a2l(dso);
 	zfree(&dso->symsrc_filename);
 	pthread_mutex_destroy(&dso->lock);
Signature · 1 line
2.1.3
author maintainer
#62 lore raw

PATCH 23/37 perf tools: Protect dso cache fd with a mutex

In reply to Namhyung Kim

When dso cache is accessed in multi-thread environment, it's possible to close other dso->data.fd during operation due to open file limit. Protect the file descriptors using a separate mutex.

---
 tools/perf/tests/dso-data.c |   5 ++
 tools/perf/util/dso.c       | 136 +++++++++++++++++++++++++++++---------------
 2 files changed, 94 insertions(+), 47 deletions(-)
tools/perf/tests/dso-data.c +5 -0
index caaf37f079b1..0276e7d2d41b 100644
@@ -111,6 +111,9 @@ int test__dso_data(void)
 	memset(&machine, 0, sizeof(machine));
 
 	dso = dso__new((const char *)file);
+	TEST_ASSERT_VAL("failed to get dso", dso);
+
+	dso->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
 
 	/* Basic 10 bytes tests. */
 	for (i = 0; i < ARRAY_SIZE(offsets); i++) {
@@ -199,6 +202,8 @@ static int dsos__create(int cnt, int size)
 
 		dsos[i] = dso__new(file);
 		TEST_ASSERT_VAL("failed to get dso", dsos[i]);
+
+		dsos[i]->binary_type = DSO_BINARY_TYPE__SYSTEM_PATH_DSO;
 	}
 
 	return 0;
tools/perf/util/dso.c +89 -47

249 lines not on this page · raw message

Signature · 1 line
2.1.3
author maintainer
#63 lore raw

PATCH 24/37 perf session: Pass struct events stats to event processing functions

In reply to Namhyung Kim

Pass stats structure so that it can point separate object when used in multi-thread environment.

---
 tools/perf/util/ordered-events.c |  4 +-
 tools/perf/util/session.c        | 87 +++++++++++++++++++++++-----------------
 tools/perf/util/session.h        |  1 +
 3 files changed, 54 insertions(+), 38 deletions(-)
tools/perf/util/ordered-events.c +3 -1
index fd4be94125fb..e933c51d7090 100644
@@ -183,7 +183,9 @@ static int __ordered_events__flush(struct perf_session *s,
 		if (ret)
 			pr_err("Can't parse sample, err = %d\n", ret);
 		else {
-			ret = perf_session__deliver_event(s, iter->event, &sample, tool,
+			ret = perf_session__deliver_event(s, &s->stats,
+							  iter->event,
+							  &sample, tool,
 							  iter->file_offset);
 			if (ret)
 				return ret;
tools/perf/util/session.c +50 -37

263 lines not on this page · raw message

tools/perf/util/session.h +1 -0
index 6d663dc76404..8fc067d931cd 100644
@@ -59,6 +59,7 @@ int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 void perf_tool__fill_defaults(struct perf_tool *tool);
 
 int perf_session__deliver_event(struct perf_session *session,
+				struct events_stats *stats,
 				union perf_event *event,
 				struct perf_sample *sample,
 				struct perf_tool *tool, u64 file_offset);
Signature · 1 line
2.1.3
author maintainer
#64 lore raw

PATCH 25/37 perf hists: Pass hists struct to hist_entry_iter functions

In reply to Namhyung Kim

This is a preparation for perf report multi-thread support. When multi-thread is enable, each thread will have its own hists during the sample processing.

---
 tools/perf/builtin-report.c       |  4 ++--
 tools/perf/builtin-top.c          |  4 ++--
 tools/perf/tests/hists_cumulate.c |  4 ++--
 tools/perf/tests/hists_filter.c   |  3 ++-
 tools/perf/tests/hists_output.c   |  4 ++--
 tools/perf/util/hist.c            | 26 +++++++++++---------------
 tools/perf/util/hist.h            |  6 ++++--
 7 files changed, 25 insertions(+), 26 deletions(-)
tools/perf/builtin-report.c +2 -2
index 4cac79ad3085..aabcfc24afd1 100644
@@ -166,8 +166,8 @@ static int process_sample_event(struct perf_tool *tool,
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, &al, evsel, sample, rep->max_stack,
-				   rep);
+	ret = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+				   sample, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
 
tools/perf/builtin-top.c +2 -2
index 961cea183a83..818ae35cbd7b 100644
@@ -784,8 +784,8 @@ static void perf_event__process_sample(struct perf_tool *tool,
 
 		pthread_mutex_lock(&hists->lock);
 
-		err = hist_entry_iter__add(&iter, &al, evsel, sample,
-					   top->max_stack, top);
+		err = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel,
+					   &al, sample, top->max_stack, top);
 		if (err < 0)
 			pr_err("Problem incrementing symbol period, skipping event\n");
 
tools/perf/tests/hists_cumulate.c +2 -2
index 4b8226e19a91..b1a2bed721a8 100644
@@ -104,8 +104,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+					 &sample, PERF_MAX_STACK_DEPTH, NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
tools/perf/tests/hists_filter.c +2 -1
index 59e53db7914c..3c54264ec58d 100644
@@ -81,7 +81,8 @@ static int add_hist_entries(struct perf_evlist *evlist,
 							  &sample) < 0)
 				goto out;
 
-			if (hist_entry_iter__add(&iter, &al, evsel, &sample,
+			if (hist_entry_iter__add(&iter, evsel__hists(evsel),
+						 evsel, &al, &sample,
 						 PERF_MAX_STACK_DEPTH, NULL) < 0)
 				goto out;
 
tools/perf/tests/hists_output.c +2 -2
index f5547610da02..c705bc7a5e78 100644
@@ -70,8 +70,8 @@ static int add_hist_entries(struct hists *hists, struct machine *machine)
 						  &sample) < 0)
 			goto out;
 
-		if (hist_entry_iter__add(&iter, &al, evsel, &sample,
-					 PERF_MAX_STACK_DEPTH, NULL) < 0)
+		if (hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+					 &sample, PERF_MAX_STACK_DEPTH, NULL) < 0)
 			goto out;
 
 		fake_samples[i].thread = al.thread;
tools/perf/util/hist.c +11 -15
index d322264bac22..d7cee7165bcd 100644
@@ -514,7 +514,7 @@ iter_add_single_mem_entry(struct hist_entry_iter *iter, struct addr_location *al
 	u64 cost;
 	struct mem_info *mi = iter->priv;
 	struct perf_sample *sample = iter->sample;
-	struct hists *hists = evsel__hists(iter->evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he;
 
 	if (mi == NULL)
@@ -544,8 +544,7 @@ static int
 iter_finish_mem_entry(struct hist_entry_iter *iter,
 		      struct addr_location *al __maybe_unused)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = iter->he;
 	int err = -EINVAL;
 
@@ -617,8 +616,7 @@ static int
 iter_add_next_branch_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
 	struct branch_info *bi;
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct hist_entry *he = NULL;
 	int i = iter->curr;
 	int err = 0;
@@ -665,11 +663,10 @@ iter_prepare_normal_entry(struct hist_entry_iter *iter __maybe_unused,
 static int
 iter_add_single_normal_entry(struct hist_entry_iter *iter, struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry *he;
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, true);
 	if (he == NULL)
@@ -684,7 +681,6 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 			 struct addr_location *al __maybe_unused)
 {
 	struct hist_entry *he = iter->he;
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 
 	if (he == NULL)
@@ -692,7 +688,7 @@ iter_finish_normal_entry(struct hist_entry_iter *iter,
 
 	iter->he = NULL;
 
-	hists__inc_nr_samples(evsel__hists(evsel), he->filtered);
+	hists__inc_nr_samples(iter->hists, he->filtered);
 
 	return hist_entry__append_callchain(he, sample);
 }
@@ -724,8 +720,7 @@ static int
 iter_add_single_cumulative_entry(struct hist_entry_iter *iter,
 				 struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
-	struct hists *hists = evsel__hists(evsel);
+	struct hists *hists = iter->hists;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -770,7 +765,6 @@ static int
 iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 			       struct addr_location *al)
 {
-	struct perf_evsel *evsel = iter->evsel;
 	struct perf_sample *sample = iter->sample;
 	struct hist_entry **he_cache = iter->priv;
 	struct hist_entry *he;
@@ -804,7 +798,7 @@ iter_add_next_cumulative_entry(struct hist_entry_iter *iter,
 		}
 	}
 
-	he = __hists__add_entry(evsel__hists(evsel), al, iter->parent, NULL, NULL,
+	he = __hists__add_entry(iter->hists, al, iter->parent, NULL, NULL,
 				sample->period, sample->weight,
 				sample->transaction, sample->time, false);
 	if (he == NULL)
@@ -860,8 +854,9 @@ const struct hist_iter_ops hist_iter_cumulative = {
 	.finish_entry 		= iter_finish_cumulative_entry,
 };
 
-int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
+int hist_entry_iter__add(struct hist_entry_iter *iter, struct hists *hists,
+			 struct perf_evsel *evsel, struct addr_location *al,
+			 struct perf_sample *sample,
 			 int max_stack_depth, void *arg)
 {
 	int err, err2;
@@ -871,6 +866,7 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
 	if (err)
 		return err;
 
+	iter->hists = hists;
 	iter->evsel = evsel;
 	iter->sample = sample;
 
tools/perf/util/hist.h +4 -2
index b86966206ba8..2ee0e40cf44c 100644
@@ -86,6 +86,7 @@ struct hist_entry_iter {
 
 	bool hide_unresolved;
 
+	struct hists *hists;
 	struct perf_evsel *evsel;
 	struct perf_sample *sample;
 	struct hist_entry *he;
@@ -110,8 +111,9 @@ struct hist_entry *__hists__add_entry(struct hists *hists,
 				      struct mem_info *mi, u64 period,
 				      u64 weight, u64 transaction,
 				      u64 timestamp, bool sample_self);
-int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
-			 struct perf_evsel *evsel, struct perf_sample *sample,
+int hist_entry_iter__add(struct hist_entry_iter *iter, struct hists *hists,
+			 struct perf_evsel *evsel, struct addr_location *al,
+			 struct perf_sample *sample,
 			 int max_stack_depth, void *arg);
 
 int64_t hist_entry__cmp(struct hist_entry *left, struct hist_entry *right);
Signature · 1 line
2.1.3
author maintainer
#65 lore raw

PATCH 26/37 perf tools: Move BUILD_ID_SIZE definition to perf.h

In reply to Namhyung Kim

The util/event.h includes util/build-id.h only for BUILD_ID_SIZE. This is a problem when I include util/event.h from util/tool.h which is also included by util/build-id.h since it now makes a circular dependency resulting in incomplete type error.

---
 tools/perf/perf.h          | 1 +
 tools/perf/util/build-id.h | 2 --
 tools/perf/util/dso.h      | 1 +
 tools/perf/util/event.h    | 1 -
 4 files changed, 2 insertions(+), 3 deletions(-)
tools/perf/perf.h +1 -0
index 37284eb47b56..56ce258314fc 100644
@@ -30,6 +30,7 @@ static inline unsigned long long rdclock(void)
 }
 
 #define MAX_NR_CPUS			256
+#define BUILD_ID_SIZE			20
 
 extern const char *input_name;
 extern bool perf_host, perf_guest;
tools/perf/util/build-id.h +0 -2
index 8236319514d5..8f31545edc5b 100644
@@ -1,8 +1,6 @@
 #ifndef PERF_BUILD_ID_H_
 #define PERF_BUILD_ID_H_ 1
 
-#define BUILD_ID_SIZE 20
-
 #include "tool.h"
 #include <linux/types.h>
 
tools/perf/util/dso.h +1 -0
index ac753594a469..c18fcc0e8081 100644
@@ -7,6 +7,7 @@
 #include <linux/types.h>
 #include <linux/bitops.h>
 #include "map.h"
+#include "perf.h"
 #include "build-id.h"
 
 enum dso_binary_type {
tools/perf/util/event.h +0 -1
index 09b9e8d3fcf7..5f66abfa61ca 100644
@@ -6,7 +6,6 @@
 
 #include "../perf.h"
 #include "map.h"
-#include "build-id.h"
 #include "perf_regs.h"
 
 struct mmap_event {
Signature · 1 line
2.1.3
author maintainer
#66 lore raw

PATCH 27/37 perf report: Parallelize perf report using multi-thread

In reply to Namhyung Kim

Introduce perf_session__process_events_mt() to enable multi-thread sample processing. It allocates a struct perf_tool_mt and fills needed info in it. The init and fini callbacks are provided so that we can pass additional data structure if needed.

The session and hists event stats are counted for each thread and summed after finishing the processing. Similarly hist entries are added to per-thread hists first and then move to the original hists using hists__multi_resort(). This function reuses hists__collapse_ resort() code so makes sort__need_collapse force to true and skips the collapsing function.

Note that most of preprocessing stage is already done by processing meta events in dummy tracking evsel first. We can find corresponding thread and map based on the sample time and symbol loading and dso cache access is protected by pthread mutex.

---
 tools/perf/builtin-report.c |  90 +++++++++++++++++++++------
 tools/perf/util/hist.c      |  75 +++++++++++++++++++----
 tools/perf/util/hist.h      |   3 +
 tools/perf/util/session.c   | 146 ++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/session.h   |   4 ++
 tools/perf/util/tool.h      |  14 +++++
 6 files changed, 302 insertions(+), 30 deletions(-)
tools/perf/builtin-report.c +73 -17
index aabcfc24afd1..796db514db31 100644
@@ -128,18 +128,16 @@ static int hist_iter__report_callback(struct hist_entry_iter *iter,
 	return err;
 }
 
-static int process_sample_event(struct perf_tool *tool,
-				union perf_event *event,
-				struct perf_sample *sample,
-				struct perf_evsel *evsel,
-				struct machine *machine)
+static int __process_sample_event(struct perf_tool *tool __maybe_unused,
+				  union perf_event *event,
+				  struct perf_sample *sample,
+				  struct perf_evsel *evsel,
+				  struct machine *machine,
+				  struct hist_entry_iter *iter,
+				  struct hists *hists,
+				  struct report *rep)
 {
-	struct report *rep = container_of(tool, struct report, tool);
 	struct addr_location al;
-	struct hist_entry_iter iter = {
-		.hide_unresolved = rep->hide_unresolved,
-		.add_entry_cb = hist_iter__report_callback,
-	};
 	int ret;
 
 	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0) {
@@ -155,18 +153,18 @@ static int process_sample_event(struct perf_tool *tool,
 		return 0;
 
 	if (sort__mode == SORT_MODE__BRANCH)
-		iter.ops = &hist_iter_branch;
+		iter->ops = &hist_iter_branch;
 	else if (rep->mem_mode)
-		iter.ops = &hist_iter_mem;
+		iter->ops = &hist_iter_mem;
 	else if (symbol_conf.cumulate_callchain)
-		iter.ops = &hist_iter_cumulative;
+		iter->ops = &hist_iter_cumulative;
 	else
-		iter.ops = &hist_iter_normal;
+		iter->ops = &hist_iter_normal;
 
 	if (al.map != NULL)
 		al.map->dso->hit = 1;
 
-	ret = hist_entry_iter__add(&iter, evsel__hists(evsel), evsel, &al,
+	ret = hist_entry_iter__add(iter, hists, evsel, &al,
 				   sample, rep->max_stack, rep);
 	if (ret < 0)
 		pr_debug("problem adding hist entry, skipping event\n");
@@ -174,6 +172,52 @@ static int process_sample_event(struct perf_tool *tool,
 	return ret;
 }
 
+static int process_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel,
+				struct machine *machine)
+{
+	struct report *rep = container_of(tool, struct report, tool);
+	struct hist_entry_iter iter = {
+		.hide_unresolved = rep->hide_unresolved,
+		.add_entry_cb = hist_iter__report_callback,
+	};
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      &iter, evsel__hists(evsel), rep);
+}
+
+static int process_sample_event_multi(struct perf_tool *tool,
+				      union perf_event *event,
+				      struct perf_sample *sample,
+				      struct perf_evsel *evsel,
+				      struct machine *machine)
+{
+	struct perf_tool_mt *mt = container_of(tool, struct perf_tool_mt, tool);
+	struct report *rep = mt->priv;
+	struct hist_entry_iter iter = {
+		.hide_unresolved = rep->hide_unresolved,
+	};
+
+	return __process_sample_event(tool, event, sample, evsel, machine,
+				      &iter, &mt->hists[evsel->idx], rep);
+}
+
+static int multi_report_init(struct perf_tool_mt *mt, void *arg)
+{
+	struct report *rep = arg;
+
+	mt->priv = rep;
+	return 0;
+}
+
+static int multi_report_fini(struct perf_tool_mt *mt, void *arg __maybe_unused)
+{
+	mt->priv = NULL;
+	return 0;
+}
+
 static int process_read_event(struct perf_tool *tool,
 			      union perf_event *event,
 			      struct perf_sample *sample __maybe_unused,
@@ -483,7 +527,14 @@ static int __cmd_report(struct report *rep)
 	if (ret)
 		return ret;
 
-	ret = perf_session__process_events(session, &rep->tool);
+	if (file->is_multi) {
+		rep->tool.sample = process_sample_event_multi;
+		ret = perf_session__process_events_mt(session, &rep->tool,
+						      multi_report_init,
+						      multi_report_fini, rep);
+	} else {
+		ret = perf_session__process_events(session, &rep->tool);
+	}
 	if (ret)
 		return ret;
 
@@ -506,7 +557,12 @@ static int __cmd_report(struct report *rep)
 		}
 	}
 
-	report__collapse_hists(rep);
+	/*
+	 * For multi-file report, it already calls hists__multi_resort()
+	 * so no need to collapse here.
+	 */
+	if (!file->is_multi)
+		report__collapse_hists(rep);
 
 	if (session_done())
 		return 0;
tools/perf/util/hist.c +62 -13

122 lines not on this page · raw message

tools/perf/util/hist.h +3 -0
index 2ee0e40cf44c..4d975f5501ed 100644
@@ -124,6 +124,7 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__free(struct hist_entry *);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
+void hists__multi_resort(struct hists *dst, struct hists *src);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
@@ -136,6 +137,7 @@ void hists__inc_stats(struct hists *hists, struct hist_entry *h);
 void hists__inc_nr_events(struct hists *hists, u32 type);
 void hists__inc_nr_samples(struct hists *hists, bool filtered);
 void events_stats__inc(struct events_stats *stats, u32 type);
+void events_stats__add(struct events_stats *dst, struct events_stats *src);
 size_t events_stats__fprintf(struct events_stats *stats, FILE *fp);
 
 size_t hists__fprintf(struct hists *hists, bool show_header, int max_rows,
@@ -179,6 +181,7 @@ static inline struct hists *evsel__hists(struct perf_evsel *evsel)
 }
 
 int hists__init(void);
+void __hists__init(struct hists *hists);
 
 struct perf_hpp {
 	char *buf;
tools/perf/util/session.c +146 -0

152 lines not on this page · raw message

tools/perf/util/session.h +4 -0
index 8fc067d931cd..1d0750d891ba 100644
@@ -51,6 +51,10 @@ int perf_session__peek_event(struct perf_session *session, off_t file_offset,
 
 int perf_session__process_events(struct perf_session *session,
 				 struct perf_tool *tool);
+int perf_session__process_events_mt(struct perf_session *session,
+				    struct perf_tool *tool,
+				    mt_tool_callback_t init_cb,
+				    mt_tool_callback_t fini_cb, void *arg);
 
 int perf_session_queue_event(struct perf_session *s, union perf_event *event,
 			     struct perf_tool *tool, struct perf_sample *sample,
tools/perf/util/tool.h +14 -0
index bb2708bbfaca..222ebd4df6c1 100644
@@ -2,6 +2,7 @@
 #define __PERF_TOOL_H
 
 #include <stdbool.h>
+#include "util/event.h"
 
 struct perf_session;
 union perf_event;
@@ -10,6 +11,7 @@ struct perf_evsel;
 struct perf_sample;
 struct perf_tool;
 struct machine;
+struct hists;
 
 typedef int (*event_sample)(struct perf_tool *tool, union perf_event *event,
 			    struct perf_sample *sample,
@@ -45,4 +47,16 @@ struct perf_tool {
 	bool		ordering_requires_timestamps;
 };
 
+struct perf_tool_mt {
+	struct perf_tool	tool;
+	struct events_stats	stats;
+	struct hists		*hists;
+	struct perf_session	*session;
+	int			idx;
+
+	void			*priv;
+};
+
+typedef int (*mt_tool_callback_t)(struct perf_tool_mt *, void *);
+
 #endif /* __PERF_TOOL_H */
Signature · 1 line
2.1.3
author maintainer
#67 lore raw

PATCH 28/37 perf tools: Add missing_threads rb tree

In reply to Namhyung Kim

Sometimes it's possible to miss certain meta events like fork/exit and in this case it can fail to find such thread in the machine's rbtree. But adding a thread to the tree is dangerous since it's now executed in multi-thread environment otherwise it'll add an overhead in order to grab a lock for every search. So adds a separate missing_threads tree and protect it with a mutex. It's expected to be accessed only if a thread is not found in a normal tree.

---
 tools/perf/tests/thread-lookup-time.c |  8 ++-
 tools/perf/util/build-id.c            |  9 +++-
 tools/perf/util/machine.c             | 91 +++++++++++++++++++++++------------
 tools/perf/util/machine.h             |  2 +
 tools/perf/util/session.c             |  8 +--
 tools/perf/util/thread.h              |  1 +
 6 files changed, 80 insertions(+), 39 deletions(-)
tools/perf/tests/thread-lookup-time.c +7 -1
index 6237ecf8caae..04cdde9329d6 100644
@@ -7,7 +7,9 @@
 static int thread__print_cb(struct thread *th, void *arg __maybe_unused)
 {
 	printf("thread: %d, start time: %"PRIu64" %s\n",
-	       th->tid, th->start_time, th->dead ? "(dead)" : "");
+	       th->tid, th->start_time,
+	       th->dead ? "(dead)" : th->exited ? "(exited)" :
+	       th->missing ? "(missing)" : "");
 	return 0;
 }
 
@@ -105,6 +107,8 @@ static int lookup_with_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, 70000) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
@@ -146,6 +150,8 @@ static int lookup_without_timestamp(struct machine *machine)
 			machine__findnew_thread_time(machine, 0, 0, -1ULL) == t3);
 
 	machine__delete_threads(machine);
+	machine__delete_dead_threads(machine);
+	machine__delete_missing_threads(machine);
 	return 0;
 }
 
tools/perf/util/build-id.c +8 -1
index 0c72680a977f..98446d089b08 100644
@@ -60,7 +60,14 @@ static int perf_event__exit_del_thread(struct perf_tool *tool __maybe_unused,
 		    event->fork.ppid, event->fork.ptid);
 
 	if (thread) {
-		rb_erase(&thread->rb_node, &machine->threads);
+		if (thread->dead)
+			rb_erase(&thread->rb_node, &machine->dead_threads);
+		else if (thread->missing)
+			rb_erase(&thread->rb_node, &machine->missing_threads);
+		else
+			rb_erase(&thread->rb_node, &machine->threads);
+
+		list_del(&thread->node);
 		machine->last_match = NULL;
 		thread__delete(thread);
 	}
tools/perf/util/machine.c +60 -31

139 lines not on this page · raw message

tools/perf/util/machine.h +2 -0
index 9571b6b1c5b5..40af1f59e360 100644
@@ -31,6 +31,7 @@ struct machine {
 	char		  *root_dir;
 	struct rb_root	  threads;
 	struct rb_root	  dead_threads;
+	struct rb_root	  missing_threads;
 	struct thread	  *last_match;
 	struct vdso_info  *vdso_info;
 	struct dsos	  user_dsos;
@@ -116,6 +117,7 @@ void machines__set_comm_exec(struct machines *machines, bool comm_exec);
 struct machine *machine__new_host(void);
 int machine__init(struct machine *machine, const char *root_dir, pid_t pid);
 void machine__exit(struct machine *machine);
+void machine__delete_missing_threads(struct machine *machine);
 void machine__delete_dead_threads(struct machine *machine);
 void machine__delete_threads(struct machine *machine);
 void machine__delete(struct machine *machine);
tools/perf/util/session.c +2 -6
index c1a17110ec6a..34956983ae8e 100644
@@ -138,14 +138,11 @@ struct perf_session *perf_session__new(struct perf_data_file *file,
 	return NULL;
 }
 
-static void perf_session__delete_dead_threads(struct perf_session *session)
-{
-	machine__delete_dead_threads(&session->machines.host);
-}
-
 static void perf_session__delete_threads(struct perf_session *session)
 {
 	machine__delete_threads(&session->machines.host);
+	machine__delete_dead_threads(&session->machines.host);
+	machine__delete_missing_threads(&session->machines.host);
 }
 
 static void perf_session_env__delete(struct perf_session_env *env)
@@ -167,7 +164,6 @@ static void perf_session_env__delete(struct perf_session_env *env)
 void perf_session__delete(struct perf_session *session)
 {
 	perf_session__destroy_kernel_maps(session);
-	perf_session__delete_dead_threads(session);
 	perf_session__delete_threads(session);
 	perf_session_env__delete(&session->header.env);
 	machines__exit(&session->machines);
tools/perf/util/thread.h +1 -0
index 0b88ca22bc3d..87188ba9465b 100644
@@ -23,6 +23,7 @@ struct thread {
 	bool			comm_set;
 	bool			exited; /* if set thread has exited */
 	bool			dead; /* thread is in dead_threads list */
+	bool			missing; /* thread is in missing_threads list */
 	struct list_head	comm_list;
 	int			comm_len;
 	u64			db_id;
Signature · 1 line
2.1.3
author maintainer
#68 lore raw

PATCH 29/37 perf top: Always creates thread in the current task tree.

In reply to Namhyung Kim

When machine__findnew_thread_time() creates a new thread, it puts the thread in the missing_threads tree assuming it's rare case that missed related task/mmap events during the recorded header file.

However this is not the case of live profiling - so it should be treated differently. This patch fixes NULL thread->mg reference in case of missing threads in live profiling.

---
 tools/perf/builtin-top.c |  5 ++++-
 tools/perf/util/event.c  | 24 ++++++++++++++++++------
 tools/perf/util/event.h  |  5 +++++
 3 files changed, 27 insertions(+), 7 deletions(-)
tools/perf/builtin-top.c +4 -1
index 818ae35cbd7b..2d75a7fab470 100644
@@ -699,6 +699,7 @@ static void perf_event__process_sample(struct perf_tool *tool,
 {
 	struct perf_top *top = container_of(tool, struct perf_top, tool);
 	struct addr_location al;
+	struct thread *thread;
 	int err;
 
 	if (!machine && perf_guest) {
@@ -724,7 +725,9 @@ static void perf_event__process_sample(struct perf_tool *tool,
 	if (event->header.misc & PERF_RECORD_MISC_EXACT_IP)
 		top->exact_samples++;
 
-	if (perf_event__preprocess_sample(event, machine, &al, sample) < 0)
+	/* Always use current thread tree for live profiling */
+	thread = machine__findnew_thread(machine, sample->pid, sample->tid);
+	if (__perf_event__preprocess_sample(event, machine, thread, &al, sample) < 0)
 		return;
 
 	if (!top->kptr_restrict_warned &&
tools/perf/util/event.c +18 -6
index 3bb186a26314..452af8f4b0f3 100644
@@ -863,14 +863,13 @@ void thread__find_addr_location_time(struct thread *thread, u8 cpumode,
 		al->sym = NULL;
 }
 
-int perf_event__preprocess_sample(const union perf_event *event,
-				  struct machine *machine,
-				  struct addr_location *al,
-				  struct perf_sample *sample)
+int __perf_event__preprocess_sample(const union perf_event *event,
+				    struct machine *machine,
+				    struct thread *thread,
+				    struct addr_location *al,
+				    struct perf_sample *sample)
 {
 	u8 cpumode = event->header.misc & PERF_RECORD_MISC_CPUMODE_MASK;
-	struct thread *thread = machine__findnew_thread_time(machine, sample->pid,
-							     sample->tid, sample->time);
 
 	if (thread == NULL)
 		return -1;
@@ -928,6 +927,19 @@ int perf_event__preprocess_sample(const union perf_event *event,
 	return 0;
 }
 
+int perf_event__preprocess_sample(const union perf_event *event,
+				  struct machine *machine,
+				  struct addr_location *al,
+				  struct perf_sample *sample)
+{
+	struct thread *thread;
+
+	thread = machine__findnew_thread_time(machine, sample->pid,
+					      sample->tid, sample->time);
+	return __perf_event__preprocess_sample(event, machine, thread,
+					       al, sample);
+}
+
 bool is_bts_event(struct perf_event_attr *attr)
 {
 	return attr->type == PERF_TYPE_HARDWARE &&
tools/perf/util/event.h +5 -0
index 5f66abfa61ca..1fbd37864241 100644
@@ -352,6 +352,11 @@ int perf_event__process(struct perf_tool *tool,
 
 struct addr_location;
 
+int __perf_event__preprocess_sample(const union perf_event *event,
+				    struct machine *machine,
+				    struct thread *thread,
+				    struct addr_location *al,
+				    struct perf_sample *sample);
 int perf_event__preprocess_sample(const union perf_event *event,
 				  struct machine *machine,
 				  struct addr_location *al,
Signature · 1 line
2.1.3
author maintainer
#69 lore raw

PATCH 30/37 perf tools: Fix progress ui to support multi thread

In reply to Namhyung Kim

Split ui_progress struct into global and local one. Each thread updates local struct without lock and only updates global one if meaningful progress is done (with lock).

To do that, pass struct ui_progress to __perf_session__process_event() and set it for the total size of multi-file storage.

---
 tools/perf/util/data.c    |  22 +++++++++
 tools/perf/util/data.h    |   3 ++
 tools/perf/util/hist.c    |   5 +-
 tools/perf/util/hist.h    |   3 +-
 tools/perf/util/session.c | 117 ++++++++++++++++++++++++++++++++++------------
 tools/perf/util/tool.h    |   3 ++
 6 files changed, 121 insertions(+), 32 deletions(-)
tools/perf/util/data.c +22 -0
index b6f7cdc4a39f..37f75f231c4d 100644
@@ -290,3 +290,25 @@ ssize_t perf_data_file__write_multi(struct perf_data_file *file,
 
 	return writen(file->multi_fd[idx], buf, size);
 }
+
+s64 perf_data_file__multi_size(struct perf_data_file *file)
+{
+	int i;
+	s64 total_size = perf_data_file__size(file);
+
+	if (!file->is_multi)
+		return total_size;
+
+	for (i = 0; i < file->nr_multi; i++) {
+		int fd = perf_data_file__multi_fd(file, i);
+		long size;
+
+		size = lseek(fd, 0, SEEK_END);
+		if (size < 0)
+			return (s64) -1;
+
+		total_size += size;
+	}
+
+	return total_size;
+}
tools/perf/util/data.h +3 -0
index f5c229166614..0f0013ac9e30 100644
@@ -2,6 +2,7 @@
 #define __PERF_DATA_H
 
 #include <stdbool.h>
+#include "perf.h"
 
 enum perf_data_mode {
 	PERF_DATA_MODE_WRITE,
@@ -61,4 +62,6 @@ int perf_data_file__prepare_write(struct perf_data_file *file, int nr);
 ssize_t perf_data_file__write_multi(struct perf_data_file *file,
 				    void *buf, size_t size, int idx);
 
+s64 perf_data_file__multi_size(struct perf_data_file *file);
+
 #endif /* __PERF_DATA_H */
tools/perf/util/hist.c +3 -2
index f3b39b45f2ec..60b55a92f23e 100644
@@ -1065,12 +1065,13 @@ void hists__collapse_resort(struct hists *hists, struct ui_progress *prog)
 	__hists__collapse_resort(hists, root, prog);
 }
 
-void hists__multi_resort(struct hists *dst, struct hists *src)
+void hists__multi_resort(struct hists *dst, struct hists *src,
+			 struct ui_progress *prog)
 {
 	struct rb_root *root = src->entries_in;
 
 	sort__need_collapse = 1;
-	__hists__collapse_resort(dst, root, NULL);
+	__hists__collapse_resort(dst, root, prog);
 }
 
 static int hist_entry__sort(struct hist_entry *a, struct hist_entry *b)
tools/perf/util/hist.h +2 -1
index 4d975f5501ed..e2abc4c75158 100644
@@ -124,7 +124,8 @@ int hist_entry__sort_snprintf(struct hist_entry *he, char *bf, size_t size,
 void hist_entry__free(struct hist_entry *);
 
 void hists__output_resort(struct hists *hists, struct ui_progress *prog);
-void hists__multi_resort(struct hists *dst, struct hists *src);
+void hists__multi_resort(struct hists *dst, struct hists *src,
+			 struct ui_progress *prog);
 void hists__collapse_resort(struct hists *hists, struct ui_progress *prog);
 
 void hists__decay_entries(struct hists *hists, bool zap_user, bool zap_kernel);
tools/perf/util/session.c +88 -29

241 lines not on this page · raw message

tools/perf/util/tool.h +3 -0
index 222ebd4df6c1..bd273c85e3cb 100644
@@ -3,6 +3,7 @@
 
 #include <stdbool.h>
 #include "util/event.h"
+#include "ui/progress.h"
 
 struct perf_session;
 union perf_event;
@@ -52,6 +53,8 @@ struct perf_tool_mt {
 	struct events_stats	stats;
 	struct hists		*hists;
 	struct perf_session	*session;
+	struct ui_progress	prog;
+	struct ui_progress	*global_prog;
 	int			idx;
 
 	void			*priv;
Signature · 1 line
2.1.3
author maintainer
#70 lore raw

PATCH 31/37 perf record: Show total size of multi file data

In reply to Namhyung Kim

Currently perf record shows header file size only - extends it to show total size of multi-file data storage.

---
 tools/perf/builtin-record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/builtin-record.c +2 -2
index 7f7a4725d080..eb485f1ee66e 100644
@@ -538,9 +538,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 		 */
 		fprintf(stderr,
 			"[ perf record: Captured and wrote %.3f MB %s (~%" PRIu64 " samples) ]\n",
-			(double)rec->bytes_written / 1024.0 / 1024.0,
+			(double)(u64)perf_data_file__multi_size(file) / 1024.0 / 1024.0,
 			file->path,
-			rec->bytes_written / 24);
+			perf_data_file__multi_size(file) / 24);
 	}
 
 out_child:
Signature · 1 line
2.1.3
author maintainer
#71 lore raw

PATCH 32/37 perf report: Add --multi-thread option and config item

In reply to Namhyung Kim

The --multi-thread option is to enable parallel processing so user can force serial processing even for multi-file data. It default to false but users also can changes this by setting "report.multi_thread" config option in ~/.perfconfig file.

---
 tools/perf/Documentation/perf-report.txt |  3 +++
 tools/perf/builtin-report.c              | 18 +++++++++++++++---
 2 files changed, 18 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-report.txt +3 -0
index dd7cccdde498..e00077a658c1 100644
@@ -318,6 +318,9 @@ OPTIONS
 --header-only::
 	Show only perf.data header (forces --stdio).
 
+--multi-thread::
+	Speed up report by parallelizing sample processing using multi-thread.
+
 SEE ALSO
 --------
 linkperf:perf-stat[1], linkperf:perf-annotate[1]
tools/perf/builtin-report.c +15 -3
index 796db514db31..6e260eaf3b1a 100644
@@ -51,6 +51,7 @@ struct report {
 	bool			mem_mode;
 	bool			header;
 	bool			header_only;
+	bool			multi_thread;
 	int			max_stack;
 	struct perf_read_values	show_threads_values;
 	const char		*pretty_printing_style;
@@ -82,6 +83,10 @@ static int report__config(const char *var, const char *value, void *cb)
 		rep->queue_size = perf_config_u64(var, value);
 		return 0;
 	}
+	if (!strcmp(var, "report.multi-thread")) {
+		rep->multi_thread = perf_config_bool(var, value);
+		return 0;
+	}
 
 	return perf_default_config(var, value, cb);
 }
@@ -527,7 +532,7 @@ static int __cmd_report(struct report *rep)
 	if (ret)
 		return ret;
 
-	if (file->is_multi) {
+	if (rep->multi_thread) {
 		rep->tool.sample = process_sample_event_multi;
 		ret = perf_session__process_events_mt(session, &rep->tool,
 						      multi_report_init,
@@ -558,10 +563,10 @@ static int __cmd_report(struct report *rep)
 	}
 
 	/*
-	 * For multi-file report, it already calls hists__multi_resort()
+	 * For multi-thread report, it already calls hists__multi_resort()
 	 * so no need to collapse here.
 	 */
-	if (!file->is_multi)
+	if (!rep->multi_thread)
 		report__collapse_hists(rep);
 
 	if (session_done())
@@ -770,6 +775,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 		     "Don't show entries under that percent", parse_percent_limit),
 	OPT_CALLBACK(0, "percentage", NULL, "relative|absolute",
 		     "how to display percentage of filtered entries", parse_filter_percentage),
+	OPT_BOOLEAN(0, "multi-thread", &report.multi_thread,
+		    "Speed up sample processing using multi-thead"),
 	OPT_END()
 	};
 	struct perf_data_file file = {
@@ -814,6 +821,11 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)
 					       report.queue_size);
 	}
 
+	if (report.multi_thread && !file.is_multi) {
+		pr_debug("fallback to single thread for single data file.\n");
+		report.multi_thread = false;
+	}
+
 	report.session = session;
 
 	has_br_stack = perf_header__has_feat(&session->header,
Signature · 1 line
2.1.3
author maintainer
#72 lore raw

PATCH 33/37 perf tools: Add front cache for dso data access

In reply to Namhyung Kim

There's a high contention in dso_cache__find() due to the dso->lock when dwarf unwinding is done with libunwind. Add last accessed pointers of dso_cache to lockless lookup. It'll fallback to normal tree search when it misses the last cache.

The size 16 is arbitrary and works best for my setting.

---
 tools/perf/util/dso.c | 28 ++++++++++++++++++++++++++--
 tools/perf/util/dso.h |  2 ++
 2 files changed, 28 insertions(+), 2 deletions(-)
tools/perf/util/dso.c +26 -2
index 6c1f5619f423..d8ee1fd826e7 100644
@@ -466,12 +466,31 @@ dso_cache__free(struct dso *dso)
 	pthread_mutex_unlock(&dso->lock);
 }
 
+static void update_last_cache(struct dso *dso, struct dso_cache *cache)
+{
+	int i;
+
+	for (i = DSO_LAST_CACHE_NR - 1; i > 0; i--)
+		dso->data.last[i] = dso->data.last[i-1];
+
+	dso->data.last[0] = cache;
+}
+
 static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 {
 	const struct rb_root *root = &dso->data.cache;
 	struct rb_node * const *p = &root->rb_node;
 	const struct rb_node *parent = NULL;
 	struct dso_cache *cache;
+	int i;
+
+	for (i = 0; i < DSO_LAST_CACHE_NR; i++) {
+		cache = dso->data.last[i];
+
+		if (cache && cache->offset <= offset &&
+		    offset < cache->offset + DSO__DATA_CACHE_SIZE)
+			return cache;
+	}
 
 	pthread_mutex_lock(&dso->lock);
 	while (*p != NULL) {
@@ -485,8 +504,10 @@ static struct dso_cache *dso_cache__find(struct dso *dso, u64 offset)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
-		else
+		else {
+			update_last_cache(dso, cache);
 			goto out;
+		}
 	}
 	cache = NULL;
 out:
@@ -515,13 +536,16 @@ dso_cache__insert(struct dso *dso, struct dso_cache *new)
 			p = &(*p)->rb_left;
 		else if (offset >= end)
 			p = &(*p)->rb_right;
-		else
+		else {
+			update_last_cache(dso, cache);
 			goto out;
+		}
 	}
 
 	rb_link_node(&new->rb_node, parent, p);
 	rb_insert_color(&new->rb_node, root);
 
+	update_last_cache(dso, new);
 	cache = NULL;
 out:
 	pthread_mutex_unlock(&dso->lock);
tools/perf/util/dso.h +2 -0
index c18fcc0e8081..28e8bb320495 100644
@@ -136,6 +136,8 @@ struct dso {
 	/* dso data file */
 	struct {
 		struct rb_root	 cache;
+#define DSO_LAST_CACHE_NR  16
+		struct dso_cache *last[DSO_LAST_CACHE_NR];
 		int		 fd;
 		int		 status;
 		u32		 status_seen;
Signature · 1 line
2.1.3
author maintainer
#73 lore raw

PATCH 34/37 perf tools: Convert lseek + read to pread

In reply to Namhyung Kim

When dso_cache__read() is called, it reads data from the given offset using lseek + normal read syscall. It can be combined to a single pread syscall.

---
 tools/perf/util/dso.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
tools/perf/util/dso.c +1 -4
index d8ee1fd826e7..95c8d5a2b934 100644
@@ -593,10 +593,7 @@ dso_cache__read(struct dso *dso, struct machine *machine,
 		}
 	}
 
-	if (-1 == lseek(dso->data.fd, cache_offset, SEEK_SET))
-		goto err_unlock;
-
-	ret = read(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE);
+	ret = pread(dso->data.fd, cache->data, DSO__DATA_CACHE_SIZE, cache_offset);
 	if (ret <= 0)
 		goto err_unlock;
 
Signature · 1 line
2.1.3
author maintainer
#74 lore raw

PATCH 35/37 perf callchain: Save eh/debug frame offset for dwarf unwind

In reply to Namhyung Kim

When libunwind tries to resolve callchains it needs to know the offset of .eh_frame_hdr or .debug_frame to access the dso. Since it calls dso__data_fd(), it'll try to grab dso->lock everytime for same information. So save it to dso_data struct and reuse it.

Note that there's a window between dso__data_fd() and actual use of the fd. The fd could be closed by other threads to deal with the open file limit in dso cache code. But I think it's ok since in that case elf_section_offset() will return 0 so it'll be tried in next acess.

---
 tools/perf/util/dso.h              |  1 +
 tools/perf/util/unwind-libunwind.c | 31 ++++++++++++++++++++-----------
 2 files changed, 21 insertions(+), 11 deletions(-)
tools/perf/util/dso.h +1 -0
index 28e8bb320495..5cf8dfe04eac 100644
@@ -143,6 +143,7 @@ struct dso {
 		u32		 status_seen;
 		size_t		 file_size;
 		struct list_head open_entry;
+		u64		 frame_offset;
 	} data;
 
 	union { /* Tool specific area */
tools/perf/util/unwind-libunwind.c +20 -11
index 94929ec491f3..3cb78c36faad 100644
@@ -244,14 +244,17 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 				     u64 *fde_count)
 {
 	int ret = -EINVAL, fd;
-	u64 offset;
+	u64 offset = dso->data.frame_offset;
 
-	fd = dso__data_fd(dso, machine);
-	if (fd < 0)
-		return -EINVAL;
+	if (offset == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .eh_frame section for unwinding info */
-	offset = elf_section_offset(fd, ".eh_frame_hdr");
+		/* Check the .eh_frame section for unwinding info */
+		offset = elf_section_offset(fd, ".eh_frame_hdr");
+		dso->data.frame_offset = offset;
+	}
 
 	if (offset)
 		ret = unwind_spec_ehframe(dso, machine, offset,
@@ -265,14 +268,20 @@ static int read_unwind_spec_eh_frame(struct dso *dso, struct machine *machine,
 static int read_unwind_spec_debug_frame(struct dso *dso,
 					struct machine *machine, u64 *offset)
 {
-	int fd = dso__data_fd(dso, machine);
+	int fd;
+	u64 ofs = dso->data.frame_offset;
 
-	if (fd < 0)
-		return -EINVAL;
+	if (ofs == 0) {
+		fd = dso__data_fd(dso, machine);
+		if (fd < 0)
+			return -EINVAL;
 
-	/* Check the .debug_frame section for unwinding info */
-	*offset = elf_section_offset(fd, ".debug_frame");
+		/* Check the .debug_frame section for unwinding info */
+		ofs = elf_section_offset(fd, ".debug_frame");
+		dso->data.frame_offset = ofs;
+	}
 
+	*offset = ofs;
 	if (*offset)
 		return 0;
 
Signature · 1 line
2.1.3
author maintainer
#75 lore raw

PATCH 36/37 perf tools: Add new perf data command

In reply to Namhyung Kim

From: Jiri Olsa <[email protected]>

Adding new 'perf data' command to provide operations over data files.

The 'perf data convert' sub command is coming in following patch, but there's possibility for other useful commands like 'perf data ls' (to display perf data file in directory in ls style).

---
 tools/perf/Documentation/perf-data.txt | 15 +++++++
 tools/perf/Makefile.perf               |  1 +
 tools/perf/builtin-data.c              | 75 ++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                   |  1 +
 tools/perf/command-list.txt            |  1 +
 tools/perf/perf.c                      |  1 +
 6 files changed, 94 insertions(+)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
tools/perf/Documentation/perf-data.txt new file +15 -0
new file mode 100644
index 000000000000..b8c83947715c
@@ -0,0 +1,15 @@
+perf-data(1)
+==============
+
+NAME
+----
+perf-data - Data file related processing
+
+SYNOPSIS
+--------
+[verse]
+'perf data' [<common options>] <command> [<options>]",
+
+DESCRIPTION
+-----------
+Data file related processing.
tools/perf/Makefile.perf +1 -0
index 47d933454492..f22085cb0b24 100644
@@ -485,6 +485,7 @@ BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o
 BUILTIN_OBJS += $(OUTPUT)builtin-inject.o
 BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o
 BUILTIN_OBJS += $(OUTPUT)builtin-mem.o
+BUILTIN_OBJS += $(OUTPUT)builtin-data.o
 
 PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT)
 
tools/perf/builtin-data.c new file +75 -0

75 lines not on this page · raw message

tools/perf/builtin.h +1 -0
index b210d62907e4..3688ad29085f 100644
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_inject(int argc, const char **argv, const char *prefix);
 extern int cmd_mem(int argc, const char **argv, const char *prefix);
+extern int cmd_data(int argc, const char **argv, const char *prefix);
 
 extern int find_scripts(char **scripts_array, char **scripts_path_array);
 #endif
tools/perf/command-list.txt +1 -0
index 0906fc401c52..00fcaf8a5b8d 100644
@@ -7,6 +7,7 @@ perf-archive			mainporcelain common
 perf-bench			mainporcelain common
 perf-buildid-cache		mainporcelain common
 perf-buildid-list		mainporcelain common
+perf-data			mainporcelain common
 perf-diff			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
tools/perf/perf.c +1 -0
index 3700a7faca6c..f3c66b81c6be 100644
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {
 #endif
 	{ "inject",	cmd_inject,	0 },
 	{ "mem",	cmd_mem,	0 },
+	{ "data",	cmd_data,	0 },
 };
 
 struct pager_config {
Signature · 1 line
2.1.3
author maintainer
#76 lore raw

PATCH 37/37 perf data: Implement 'split' subcommand

In reply to Namhyung Kim

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +28 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+	Split single data file (perf.data) into multiple files under a directory
+	in order to be reported by multiple threads.
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
+SEE ALSO
+--------
+linkperf:perf[1], linkperf:perf-report[1]
tools/perf/builtin-data.c +223 -0

243 lines not on this page · raw message

Signature · 1 line
2.1.3
#77 lore raw

Re: [PATCH 37/37] perf data: Implement 'split' subcommand

Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:

Quoted 3 lines

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)

- ARnaldo

Quoted 293 lines
---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +28 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+	Split single data file (perf.data) into multiple files under a directory
+	in order to be reported by multiple threads.
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
+SEE ALSO
+--------
+linkperf:perf[1], linkperf:perf-report[1]
tools/perf/builtin-data.c +223 -0
index 1eee97d020fa..5f3173826850 100644
@@ -2,10 +2,14 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
 	const char	*name;
 	const char	*summary;
@@ -41,10 +45,229 @@ static void print_usage(void)
 	printf("\n");
 }
 
+static int data_cmd_split(int argc, const char **argv, const char *prefix);
+
 static struct data_cmd data_cmds[] = {
+	{ "split", "split single data file into multi-file", data_cmd_split },
 	{ NULL },
 };
 
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_split {
+	struct perf_tool	tool;
+	struct perf_session	*session;
+	enum {
+		PER_CPU,
+		PER_THREAD,
+	} mode;
+	int 			header_fd;
+	u64			header_written;
+	struct hlist_head	fd_hash[FD_HASH_SIZE];
+	int			fd_hash_nr;
+};
+
+struct fdhash_node {
+	int			id;
+	int			fd;
+	struct hlist_node	list;
+};
+
+static struct hlist_head *get_hash(struct data_split *split, int id)
+{
+	return &split->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+				      union perf_event *event)
+{
+	struct data_split *split = container_of(tool, struct data_split, tool);
+	ssize_t size;
+
+	size = writen(split->header_fd, event, event->header.size);
+	if (size < 0)
+		return -errno;
+
+	split->header_written += size;
+	return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+				union perf_event *event,
+				struct perf_sample *sample,
+				struct perf_evsel *evsel __maybe_unused,
+				struct machine *machine __maybe_unused)
+{
+	struct data_split *split = container_of(tool, struct data_split, tool);
+	int id = split->mode == PER_CPU ? sample->cpu : sample->tid;
+	int fd = -1;
+	char buf[PATH_MAX];
+	struct hlist_head *head;
+	struct fdhash_node *node;
+
+	head = get_hash(split, id);
+	hlist_for_each_entry(node, head, list) {
+		if (node->id == id) {
+			fd = node->fd;
+			break;
+		}
+	}
+
+	if (fd == -1) {
+		scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+			  output_name, split->fd_hash_nr++);
+
+		fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+		if (fd < 0) {
+			pr_err("cannot open data file: %s: %m\n", buf);
+			return -1;
+		}
+
+		node = malloc(sizeof(*node));
+		if (node == NULL) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+
+		node->id = id;
+		node->fd = fd;
+
+		hlist_add_head(&node->list, head);
+	}
+
+	return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int __data_cmd_split(struct data_split *split)
+{
+	struct perf_session *session = split->session;
+	char *output = NULL;
+	char buf[PATH_MAX];
+	u64 sample_type;
+	int header_fd;
+	int ret = -1;
+	int i;
+
+	if (!output_name) {
+		if (asprintf(&output, "%s.dir", input_name) < 0) {
+			pr_err("memory allocation failed\n");
+			return -1;
+		}
+		output_name = output;
+	}
+
+	mkdir(output_name, 0700);
+
+	/*
+	 * This is necessary to write (copy) build-id table.  After
+	 * processing header, dsos list will contain dso which was on
+	 * the original build-id table.
+	 */
+	dsos__hit_all(session);
+
+	scnprintf(buf, sizeof(buf), "%s/perf.header", output_name);
+	header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+	if (header_fd < 0) {
+		pr_err("cannot open header file: %s: %m\n", buf);
+		goto out;
+	}
+
+	lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+	sample_type = perf_evlist__combined_sample_type(session->evlist);
+	if (sample_type & PERF_SAMPLE_CPU)
+		split->mode = PER_CPU;
+	else
+		split->mode = PER_THREAD;
+
+	pr_debug("splitting data file for %s\n",
+		 split->mode == PER_CPU ? "CPUs" : "threads");
+
+	split->header_fd = header_fd;
+	perf_session__process_events(session, &split->tool);
+
+	for (i = 0; i < FD_HASH_SIZE; i++) {
+		struct fdhash_node *pos;
+		struct hlist_node *tmp;
+
+		hlist_for_each_entry_safe(pos, tmp, &split->fd_hash[i], list) {
+			hlist_del(&pos->list);
+			close(pos->fd);
+			free(pos);
+		}
+	}
+
+	session->header.data_size = split->header_written;
+	perf_session__write_header(session, session->evlist, header_fd, true);
+
+	close(header_fd);
+	ret = 0;
+out:
+	free(output);
+	return ret;
+}
+
+int data_cmd_split(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+	bool force = false;
+	struct perf_session *session;
+	struct perf_data_file file = {
+		.mode  = PERF_DATA_MODE_READ,
+	};
+	struct data_split split = {
+		.tool = {
+			.sample		= split_sample_event,
+			.fork		= split_other_events,
+			.comm		= split_other_events,
+			.exit		= split_other_events,
+			.mmap		= split_other_events,
+			.mmap2		= split_other_events,
+			.lost		= split_other_events,
+			.throttle	= split_other_events,
+			.unthrottle	= split_other_events,
+		},
+	};
+	const char * const split_usage[] = {
+		"perf data split [<options>]",
+		NULL
+	};
+	const struct option split_options[] = {
+	OPT_STRING('i', "input", &input_name, "file", "input file name"),
+	OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+	OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+	OPT_END()
+	};
+
+	argc = parse_options(argc, argv, split_options, split_usage, 0);
+	if (argc)
+		usage_with_options(split_usage, split_options);
+
+	file.path = input_name;
+	file.force = force;
+	session = perf_session__new(&file, false, &split.tool);
+	if (session == NULL)
+		return -1;
+
+	split.session = session;
+	symbol__init(&session->header.env);
+
+	__data_cmd_split(&split);
+
+	perf_session__delete(session);
+	return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
 	struct data_cmd *cmd;
Signature · 1 line
2.1.3
author maintainer
#78 lore raw

Re: [PATCH 37/37] perf data: Implement 'split' subcommand

Hi Arnaldo,

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:

Quoted 3 lines

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)

It depends on the data file - if it's recorded system-wide, the split will be done by cpu, otherwise by thread. It's determined by checking sample type has PERF_SAMPLE_CPU. And metadata is saved in a separate perf.header file like other multi-file data. Will add it to the change log.

Thanks,
Namhyung

Quoted 293 lines
Quoted 293 lines
---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +28 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+     Split single data file (perf.data) into multiple files under a directory
+     in order to be reported by multiple threads.
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+     Specify input perf data file path.
+
+-o::
+--output::
+     Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
+SEE ALSO
+--------
+linkperf:perf[1], linkperf:perf-report[1]
tools/perf/builtin-data.c +223 -1
index 1eee97d020fa..5f3173826850 100644
@@ -2,10 +2,14 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
      const char      *name;
      const char      *summary;
@@ -41,10 +45,229 @@ static void print_usage(void)
      printf("\n");
 }
 
+static int data_cmd_split(int argc, const char **argv, const char *prefix);
+
 static struct data_cmd data_cmds[] = {
+     { "split", "split single data file into multi-file", data_cmd_split },
      { NULL },
 };
 
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_split {
+     struct perf_tool        tool;
+     struct perf_session     *session;
+     enum {
+             PER_CPU,
+             PER_THREAD,
+     } mode;
+     int                     header_fd;
+     u64                     header_written;
+     struct hlist_head       fd_hash[FD_HASH_SIZE];
+     int                     fd_hash_nr;
+};
+
+struct fdhash_node {
+     int                     id;
+     int                     fd;
+     struct hlist_node       list;
+};
+
+static struct hlist_head *get_hash(struct data_split *split, int id)
+{
+     return &split->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+                                   union perf_event *event)
+{
+     struct data_split *split = container_of(tool, struct data_split, tool);
+     ssize_t size;
+
+     size = writen(split->header_fd, event, event->header.size);
+     if (size < 0)
+             return -errno;
+
+     split->header_written += size;
+     return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+                             union perf_event *event,
+                             struct perf_sample *sample __maybe_unused,
+                             struct machine *machine __maybe_unused)
+{
+     return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+                             union perf_event *event,
+                             struct perf_sample *sample,
+                             struct perf_evsel *evsel __maybe_unused,
+                             struct machine *machine __maybe_unused)
+{
+     struct data_split *split = container_of(tool, struct data_split, tool);
+     int id = split->mode == PER_CPU ? sample->cpu : sample->tid;
+     int fd = -1;
+     char buf[PATH_MAX];
+     struct hlist_head *head;
+     struct fdhash_node *node;
+
+     head = get_hash(split, id);
+     hlist_for_each_entry(node, head, list) {
+             if (node->id == id) {
+                     fd = node->fd;
+                     break;
+             }
+     }
+
+     if (fd == -1) {
+             scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+                       output_name, split->fd_hash_nr++);
+
+             fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+             if (fd < 0) {
+                     pr_err("cannot open data file: %s: %m\n", buf);
+                     return -1;
+             }
+
+             node = malloc(sizeof(*node));
+             if (node == NULL) {
+                     pr_err("memory allocation failed\n");
+                     return -1;
+             }
+
+             node->id = id;
+             node->fd = fd;
+
+             hlist_add_head(&node->list, head);
+     }
+
+     return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int __data_cmd_split(struct data_split *split)
+{
+     struct perf_session *session = split->session;
+     char *output = NULL;
+     char buf[PATH_MAX];
+     u64 sample_type;
+     int header_fd;
+     int ret = -1;
+     int i;
+
+     if (!output_name) {
+             if (asprintf(&output, "%s.dir", input_name) < 0) {
+                     pr_err("memory allocation failed\n");
+                     return -1;
+             }
+             output_name = output;
+     }
+
+     mkdir(output_name, 0700);
+
+     /*
+      * This is necessary to write (copy) build-id table.  After
+      * processing header, dsos list will contain dso which was on
+      * the original build-id table.
+      */
+     dsos__hit_all(session);
+
+     scnprintf(buf, sizeof(buf), "%s/perf.header", output_name);
+     header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+     if (header_fd < 0) {
+             pr_err("cannot open header file: %s: %m\n", buf);
+             goto out;
+     }
+
+     lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+     sample_type = perf_evlist__combined_sample_type(session->evlist);
+     if (sample_type & PERF_SAMPLE_CPU)
+             split->mode = PER_CPU;
+     else
+             split->mode = PER_THREAD;
+
+     pr_debug("splitting data file for %s\n",
+              split->mode == PER_CPU ? "CPUs" : "threads");
+
+     split->header_fd = header_fd;
+     perf_session__process_events(session, &split->tool);
+
+     for (i = 0; i < FD_HASH_SIZE; i++) {
+             struct fdhash_node *pos;
+             struct hlist_node *tmp;
+
+             hlist_for_each_entry_safe(pos, tmp, &split->fd_hash[i], list) {
+                     hlist_del(&pos->list);
+                     close(pos->fd);
+                     free(pos);
+             }
+     }
+
+     session->header.data_size = split->header_written;
+     perf_session__write_header(session, session->evlist, header_fd, true);
+
+     close(header_fd);
+     ret = 0;
+out:
+     free(output);
+     return ret;
+}
+
+int data_cmd_split(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+     bool force = false;
+     struct perf_session *session;
+     struct perf_data_file file = {
+             .mode  = PERF_DATA_MODE_READ,
+     };
+     struct data_split split = {
+             .tool = {
+                     .sample         = split_sample_event,
+                     .fork           = split_other_events,
+                     .comm           = split_other_events,
+                     .exit           = split_other_events,
+                     .mmap           = split_other_events,
+                     .mmap2          = split_other_events,
+                     .lost           = split_other_events,
+                     .throttle       = split_other_events,
+                     .unthrottle     = split_other_events,
+             },
+     };
+     const char * const split_usage[] = {
+             "perf data split [<options>]",
+             NULL
+     };
+     const struct option split_options[] = {
+     OPT_STRING('i', "input", &input_name, "file", "input file name"),
+     OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+     OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+     OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+     OPT_END()
+     };
+
+     argc = parse_options(argc, argv, split_options, split_usage, 0);
+     if (argc)
+             usage_with_options(split_usage, split_options);
+
+     file.path = input_name;
+     file.force = force;
+     session = perf_session__new(&file, false, &split.tool);
+     if (session == NULL)
+             return -1;
+
+     split.session = session;
+     symbol__init(&session->header.env);
+
+     __data_cmd_split(&split);
+
+     perf_session__delete(session);
+     return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
      struct data_cmd *cmd;
--

2.1.3

Signature · 2 lines
Thanks,
Namhyung
#79 lore raw

Re: [PATCH 37/37] perf data: Implement 'split' subcommand

Em Wed, Dec 24, 2014 at 11:14:47PM +0900, Namhyung Kim escreveu:

Quoted 13 lines

Hi Arnaldo,

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Wed, Dec 24, 2014 at 04:15:33PM +0900, Namhyung Kim escreveu:

Quoted 3 lines

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

How is it split? By CPU?
Will the metadata stay in a different file? Please be as verbose on the description as possible :-)

Quoted 5 lines

It depends on the data file - if it's recorded system-wide, the split will be done by cpu, otherwise by thread. It's determined by checking sample type has PERF_SAMPLE_CPU. And metadata is saved in a separate perf.header file like other multi-file data. Will add it to the change log.

Thanks for the explanation!

Please try to add such more detailed information on the change logs, as the reviewing process starts at the idea/design, and that is first read in the changelog, only after understanding what you want to do is that we should bother looking if you did what you say you would do :-)

- Arnaldo

Quoted 303 lines

Thanks,
Namhyung

Quoted 293 lines
Quoted 293 lines
---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +28 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+     Split single data file (perf.data) into multiple files under a directory
+     in order to be reported by multiple threads.
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+     Specify input perf data file path.
+
+-o::
+--output::
+     Specify output perf data directory path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+-f::
+--force::
+        Don't complain, do it.
+
+SEE ALSO
+--------
+linkperf:perf[1], linkperf:perf-report[1]
tools/perf/builtin-data.c +223 -1
index 1eee97d020fa..5f3173826850 100644
@@ -2,10 +2,14 @@
 #include "builtin.h"
 #include "perf.h"
 #include "debug.h"
+#include "session.h"
+#include "evlist.h"
 #include "parse-options.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
+static const char *output_name;
+
 struct data_cmd {
      const char      *name;
      const char      *summary;
@@ -41,10 +45,229 @@ static void print_usage(void)
      printf("\n");
 }
 
+static int data_cmd_split(int argc, const char **argv, const char *prefix);
+
 static struct data_cmd data_cmds[] = {
+     { "split", "split single data file into multi-file", data_cmd_split },
      { NULL },
 };
 
+#define FD_HASH_BITS  7
+#define FD_HASH_SIZE  (1 << FD_HASH_BITS)
+#define FD_HASH_MASK  (FD_HASH_SIZE - 1)
+
+struct data_split {
+     struct perf_tool        tool;
+     struct perf_session     *session;
+     enum {
+             PER_CPU,
+             PER_THREAD,
+     } mode;
+     int                     header_fd;
+     u64                     header_written;
+     struct hlist_head       fd_hash[FD_HASH_SIZE];
+     int                     fd_hash_nr;
+};
+
+struct fdhash_node {
+     int                     id;
+     int                     fd;
+     struct hlist_node       list;
+};
+
+static struct hlist_head *get_hash(struct data_split *split, int id)
+{
+     return &split->fd_hash[id % FD_HASH_MASK];
+}
+
+static int perf_event__rewrite_header(struct perf_tool *tool,
+                                   union perf_event *event)
+{
+     struct data_split *split = container_of(tool, struct data_split, tool);
+     ssize_t size;
+
+     size = writen(split->header_fd, event, event->header.size);
+     if (size < 0)
+             return -errno;
+
+     split->header_written += size;
+     return 0;
+}
+
+static int split_other_events(struct perf_tool *tool,
+                             union perf_event *event,
+                             struct perf_sample *sample __maybe_unused,
+                             struct machine *machine __maybe_unused)
+{
+     return perf_event__rewrite_header(tool, event);
+}
+
+static int split_sample_event(struct perf_tool *tool,
+                             union perf_event *event,
+                             struct perf_sample *sample,
+                             struct perf_evsel *evsel __maybe_unused,
+                             struct machine *machine __maybe_unused)
+{
+     struct data_split *split = container_of(tool, struct data_split, tool);
+     int id = split->mode == PER_CPU ? sample->cpu : sample->tid;
+     int fd = -1;
+     char buf[PATH_MAX];
+     struct hlist_head *head;
+     struct fdhash_node *node;
+
+     head = get_hash(split, id);
+     hlist_for_each_entry(node, head, list) {
+             if (node->id == id) {
+                     fd = node->fd;
+                     break;
+             }
+     }
+
+     if (fd == -1) {
+             scnprintf(buf, sizeof(buf), "%s/perf.data.%d",
+                       output_name, split->fd_hash_nr++);
+
+             fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+             if (fd < 0) {
+                     pr_err("cannot open data file: %s: %m\n", buf);
+                     return -1;
+             }
+
+             node = malloc(sizeof(*node));
+             if (node == NULL) {
+                     pr_err("memory allocation failed\n");
+                     return -1;
+             }
+
+             node->id = id;
+             node->fd = fd;
+
+             hlist_add_head(&node->list, head);
+     }
+
+     return writen(fd, event, event->header.size) > 0 ? 0 : -errno;
+}
+
+static int __data_cmd_split(struct data_split *split)
+{
+     struct perf_session *session = split->session;
+     char *output = NULL;
+     char buf[PATH_MAX];
+     u64 sample_type;
+     int header_fd;
+     int ret = -1;
+     int i;
+
+     if (!output_name) {
+             if (asprintf(&output, "%s.dir", input_name) < 0) {
+                     pr_err("memory allocation failed\n");
+                     return -1;
+             }
+             output_name = output;
+     }
+
+     mkdir(output_name, 0700);
+
+     /*
+      * This is necessary to write (copy) build-id table.  After
+      * processing header, dsos list will contain dso which was on
+      * the original build-id table.
+      */
+     dsos__hit_all(session);
+
+     scnprintf(buf, sizeof(buf), "%s/perf.header", output_name);
+     header_fd = open(buf, O_RDWR|O_CREAT|O_TRUNC, 0600);
+     if (header_fd < 0) {
+             pr_err("cannot open header file: %s: %m\n", buf);
+             goto out;
+     }
+
+     lseek(header_fd, session->header.data_offset, SEEK_SET);
+
+     sample_type = perf_evlist__combined_sample_type(session->evlist);
+     if (sample_type & PERF_SAMPLE_CPU)
+             split->mode = PER_CPU;
+     else
+             split->mode = PER_THREAD;
+
+     pr_debug("splitting data file for %s\n",
+              split->mode == PER_CPU ? "CPUs" : "threads");
+
+     split->header_fd = header_fd;
+     perf_session__process_events(session, &split->tool);
+
+     for (i = 0; i < FD_HASH_SIZE; i++) {
+             struct fdhash_node *pos;
+             struct hlist_node *tmp;
+
+             hlist_for_each_entry_safe(pos, tmp, &split->fd_hash[i], list) {
+                     hlist_del(&pos->list);
+                     close(pos->fd);
+                     free(pos);
+             }
+     }
+
+     session->header.data_size = split->header_written;
+     perf_session__write_header(session, session->evlist, header_fd, true);
+
+     close(header_fd);
+     ret = 0;
+out:
+     free(output);
+     return ret;
+}
+
+int data_cmd_split(int argc, const char **argv, const char *prefix __maybe_unused)
+{
+     bool force = false;
+     struct perf_session *session;
+     struct perf_data_file file = {
+             .mode  = PERF_DATA_MODE_READ,
+     };
+     struct data_split split = {
+             .tool = {
+                     .sample         = split_sample_event,
+                     .fork           = split_other_events,
+                     .comm           = split_other_events,
+                     .exit           = split_other_events,
+                     .mmap           = split_other_events,
+                     .mmap2          = split_other_events,
+                     .lost           = split_other_events,
+                     .throttle       = split_other_events,
+                     .unthrottle     = split_other_events,
+             },
+     };
+     const char * const split_usage[] = {
+             "perf data split [<options>]",
+             NULL
+     };
+     const struct option split_options[] = {
+     OPT_STRING('i', "input", &input_name, "file", "input file name"),
+     OPT_STRING('o', "output", &output_name, "file", "output directory name"),
+     OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
+     OPT_INCR('v', "verbose", &verbose, "be more verbose"),
+     OPT_END()
+     };
+
+     argc = parse_options(argc, argv, split_options, split_usage, 0);
+     if (argc)
+             usage_with_options(split_usage, split_options);
+
+     file.path = input_name;
+     file.force = force;
+     session = perf_session__new(&file, false, &split.tool);
+     if (session == NULL)
+             return -1;
+
+     split.session = session;
+     symbol__init(&session->header.env);
+
+     __data_cmd_split(&split);
+
+     perf_session__delete(session);
+     return 0;
+}
+
 int cmd_data(int argc, const char **argv, const char *prefix)
 {
      struct data_cmd *cmd;
--

2.1.3

Signature · 2 lines
Thanks,
Namhyung
#80 lore raw

Re: [PATCH 37/37] perf data: Implement 'split' subcommand

In reply to Namhyung Kim
Quoted 24 lines · Namhyung Kim

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +6 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+	Split single data file (perf.data) into multiple files under a directory
+	in order to be reported by multiple threads.

tools/perf/Documentation/perf-data.txt SYNOPSIS · line 31

Quoted 10 lines
tools/perf/Documentation/perf-data.txt +10 -0
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+	Specify input perf data file path.
+
+-o::
+--output::
+	Specify output perf data directory path.

should the -o have 'perf.data.dir' as default?

[jolsa@krava perf]$ ./perf record ls > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB perf.data (~0 samples) ] [jolsa@krava perf]$ ./perf data split
[jolsa@krava perf]$ ll perf.data*
-rw------- 1 jolsa jolsa 16172 Dec 26 14:58 perf.data

jirka

author maintainer
#81 lore raw

Re: [PATCH 37/37] perf data: Implement 'split' subcommand

Hi Jiri,

Quoted 38 lines · Jiri Olsa
Quoted 24 lines · Namhyung Kim

The perf data split command is for splitting a (large) single data file into multiple files under a directory (perf.data.dir by default) so that it can be processed and reported using multiple threads.

---
 tools/perf/Documentation/perf-data.txt |  28 +++++
 tools/perf/builtin-data.c              | 223 +++++++++++++++++++++++++++++++++
 2 files changed, 251 insertions(+)
tools/perf/Documentation/perf-data.txt +6 -0
index b8c83947715c..42708702f10c 100644
@@ -13,3 +13,31 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+split::
+     Split single data file (perf.data) into multiple files under a directory
+     in order to be reported by multiple threads.

tools/perf/Documentation/perf-data.txt SYNOPSIS · line 31

Quoted 10 lines
tools/perf/Documentation/perf-data.txt +10 -0
+
+OPTIONS for 'split'
+---------------------
+-i::
+--input::
+     Specify input perf data file path.
+
+-o::
+--output::
+     Specify output perf data directory path.

should the -o have 'perf.data.dir' as default?

Quoted 6 lines

[jolsa@krava perf]$ ./perf record ls > /dev/null
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.000 MB perf.data (~0 samples) ] [jolsa@krava perf]$ ./perf data split
[jolsa@krava perf]$ ll perf.data*
-rw------- 1 jolsa jolsa 16172 Dec 26 14:58 perf.data

You're right! :)

Thanks,
Namhyung

#82 lore raw
In reply to Namhyung Kim

On Wed, Dec 24, 2014 at 04:14:56PM +0900, Namhyung Kim wrote:

SNIP

Quoted 7 lines

You can get it from 'perf/threaded-v1' branch on my tree at:

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it. Any comments are welcome! :)

very nice at first round check ;-)

I'll do detailed review next week

thanks,
jirka

author maintainer
#83 lore raw
Quoted 8 lines · Jiri Olsa

On Wed, Dec 24, 2014 at 04:14:56PM +0900, Namhyung Kim wrote:

SNIP

Quoted 4 lines

You can get it from 'perf/threaded-v1' branch on my tree at:

Quoted 7 lines
Quoted 3 lines

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Please take a look and play with it. Any comments are welcome! :)

very nice at first round check ;-)

I'll do detailed review next week

Thank you very much!
Namhyung

#84 lore raw
In reply to Namhyung Kim

Thanks for working on this. Haven't read any code, just some high level comments on the design.

Quoted 8 lines

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.

Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.

Also I thought Adrian's meta data index already addressed this at least partially.

Quoted 7 lines

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.

It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)

Quoted 5 lines

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.

Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?

BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.

Quoted 8 lines

Here is the result:

This is just elapsed (real) time measured by shell 'time' function.

The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.

Quoted 8 lines

time perf report --children --no-children + --call-graph none
---------- ------------- -------------------

 current            4m43.260s      1m32.779s            0m35.866s            
 patched            4m43.710s      1m29.695s            0m33.995s
 --multi-thread     2m46.265s      0m45.486s             0m7.570s

This result is with 7.7GB data file using libunwind for callchain.

Nice results!

-Andi

#85 lore raw
Quoted 36 lines · Andi Kleen

Thanks for working on this. Haven't read any code, just some high level comments on the design.

Quoted 8 lines

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.

Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.

Also I thought Adrian's meta data index already addressed this at least partially.

Quoted 7 lines

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.

It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)

Quoted 1 line
Quoted 8 lines
Quoted 4 lines

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.

having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.

We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.

Quoted 29 lines

Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?

BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.

Quoted 8 lines

Here is the result:

This is just elapsed (real) time measured by shell 'time' function.

The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.

Quoted 8 lines

time perf report --children --no-children + --call-graph none

                 ----------  -------------  -------------------
 current            4m43.260s      1m32.779s            0m35.866s
 patched            4m43.710s      1m29.695s            0m33.995s
 --multi-thread     2m46.265s      0m45.486s             0m7.570s

This result is with 7.7GB data file using libunwind for callchain.

Nice results!

-Andi

author maintainer
#86 lore raw

Hi Stephane,

Quoted 20 lines · Stephane Eranian
Quoted 19 lines · Andi Kleen
Quoted 6 lines

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.

It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)

Quoted 5 lines

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

Quoted 7 lines
Quoted 3 lines

I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.

having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.

I just thought it's easier to handle with multiple thread. Maybe we can concatenate the files after recording.

Quoted 4 lines

We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.

Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?

Thanks,
Namhyung

#87 lore raw
Quoted 36 lines · Namhyung Kim

Hi Stephane,

Quoted 20 lines · Stephane Eranian
Quoted 19 lines · Andi Kleen
Quoted 6 lines

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.

It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)

Quoted 5 lines

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

Quoted 7 lines
Quoted 3 lines

I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.

having to manage two separate files is a major change which I don't particularly like. It will cause problems. I don't see why this cannot be appended to the perf.data file with a index at the beginning. There is already an index for sections in file mode.

I just thought it's easier to handle with multiple thread. Maybe we can concatenate the files after recording.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.

Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?

There is no seek possible in pipe mode.

The way this is done (as I remember) is by creating pseudo-record types and injecting them in the stream.

author maintainer
#88 lore raw
Quoted 8 lines · Stephane Eranian
Quoted 7 lines · Namhyung Kim

Hi Stephane,

Quoted 3 lines · Stephane Eranian

We use the pipe mode a lot and this would not work there. So no, I don't like the 2 files solution. But I like the idea of using multiple threads to speed up processing.

Quoted 8 lines
Quoted 4 lines

Actually it's not 2 files, it's 1 + N files. :) Anyway, I think the single file + index approach requires seeking to process them, is it ok for pipe-mode?

There is no seek possible in pipe mode.

The way this is done (as I remember) is by creating pseudo-record types and injecting them in the stream.

Right. But I have no idea how I can index a live stream and process it using multiple threads.

Thanks,
Namhyung

author maintainer
#89 lore raw
In reply to Andi Kleen

Hi Andi,

Quoted 3 lines · Andi Kleen

Thanks for working on this. Haven't read any code, just some high level comments on the design.

Really appreciate it!

Quoted 9 lines
Quoted 8 lines

So my approach is like this:

Partially do stage 1 first - but only for meta events that changes machine state. To do this I add a dummy tracking event to perf record and make it collect such meta events only. They are saved in a separate file (perf.header) and processed before sample events at perf report time.

Quoted 4 lines

Can't you just use seek to put the offset into the perf.data header like it's already done for other sections? Managing another file would be a big change for users and especially is a problem if the data is moved between different systems.

The files are located in a directory and users only deal with the directory so I don't think it's a big problem. In addition, moving data between different systems requires archiving related debuginfos and I think we can extend perf-archive to put those debuginfo in the data directory so that it can find the symbols more easily.

Quoted 3 lines

Also I thought Adrian's meta data index already addressed this at least partially.

I know Adrian's work might have some common parts but I haven't looked at it deeply, sorry! It'd be great if we can discuss how to coordinate the future direction or something..

Quoted 9 lines
Quoted 8 lines

This also requires to handle multiple files and to find a corresponding machine state when processing samples. On a large profiling session, many tasks were created and exited so pid might be recycled (even more than once!). To deal with it, I managed to have thread, map_groups and comm in time sorted. The only remaining thing is symbol loading as it's done lazily when sample requires it.

Quoted 6 lines

FWIW there's often a lot of unnecessary information in this (e.g. mmaps that are not used). The Quipper page claims large saving in data files by avoided redundancies.

It would be probably better if perf record avoided writing redundant information better (I realize that's not easy)

Right, many mmap events won't be used but we cannot predict which one is used or not.

Quoted 8 lines
Quoted 5 lines

With that being done, the stage 2 can be done by multiple threads. I also save each sample data (per-cpu or per-thread) in separate files during record. On perf report time, each file will be processed by each thread. And symbol loading is protected by a mutex lock.

I really don't like the multiple files. See above. Also it could easily cause additional seeking on spinning disks.

Right, I admit that my result ran on a SSD disk.

Quoted 5 lines

Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?

I don't understand it. Could you please elaborate it?

Quoted 3 lines

BTW I suspect if you used cilk plus or a similar library that would make the code much simpler.

I'm not sure how much code I can make simpler with the help of such library. I think most changes in this patchset is preparations to concurrent access in libperf and it's still needed even if the library is used anyway.

Thanks,
Namhyung

Quoted 26 lines
Quoted 9 lines

Here is the result:

This is just elapsed (real) time measured by shell 'time' function.

The data file was recorded during kernel build with fp callchain and size is 2.1GB. The machine has 6 core with hyper-threading enabled and I got a similar result on my laptop too.

Quoted 8 lines

time perf report --children --no-children + --call-graph none
---------- ------------- -------------------

 current            4m43.260s      1m32.779s            0m35.866s            
 patched            4m43.710s      1m29.695s            0m33.995s
 --multi-thread     2m46.265s      0m45.486s             0m7.570s

This result is with 7.7GB data file using libunwind for callchain.

Nice results!

-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#90 lore raw
Quoted 6 lines
Quoted 4 lines

Isn't it fast enough to have a single thread that pre scans the events (perhaps with some single-thread optimizations like vectorization), and then load balances the work to a thread pool?

I don't understand it. Could you please elaborate it?

Have a thread pool. Then there is a single thread which pre-processes events and puts them into queues, which are then processed in parallel by the threads in the thread pool.

-Andi