Skip to content

Introduce threaded trace streaming for basic perf record operation

Alexey Budankov lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest12 patches 12 repliesIn review
  2. v215 patches 48 repliesSuperseded
  3. v115 patches 29 repliesSuperseded

Changes in v3 from the cover letter

- avoided skipped redundant patch 3/15
- applied "data file" and "data directory" terms allover the patch set
- captured Acked-by: tags by Namhyung Kim
- avoided braces where don't needed
- employed thread local variable for serial trace streaming
- added specs for --thread option - core, socket, numa and user defined
- added parallel loading of data directory files similar to the prototype [1]

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12perf record: introduce thread affinity and mmap maskstools/perf/builtin-record.c+116 -01 reply
2/12perf record: introduce thread specific data arraytools/lib/api/fd/array.c tools/lib/api/fd/array.h tools/perf/builtin-record.c+206 -31 reply
3/12perf record: introduce thread local variabletools/perf/builtin-record.c tools/perf/util/evlist.c tools/perf/util/evlist.h+109 -451 reply
4/12perf record: stop threads in the end of trace streamingtools/perf/builtin-record.c+30 -00 replies
5/12perf record: start threads in the beginning of trace streamingtools/perf/builtin-record.c+102 -10 replies
6/12perf record: introduce data file at mmap buffer objecttools/perf/builtin-record.c tools/perf/util/mmap.c tools/perf/util/mmap.h+63 -131 reply
7/12perf record: init data file at mmap buffer objecttools/perf/builtin-record.c tools/perf/util/record.h+35 -75 replies
8/12perf record: introduce --threads=<spec> command line optiontools/include/linux/bitmap.h tools/lib/bitmap.c tools/perf/builtin-record.c +1 more+332 -21 reply
9/12perf record: document parallel data streaming modetools/perf/Documentation/perf-record.txt tools/perf/builtin-record.c+58 -30 replies
10/12perf report: output data file name in raw trace dumptools/perf/builtin-inject.c tools/perf/util/ordered-events.h tools/perf/util/session.c +2 more+52 -330 replies
11/12perf session: load data directory files for analysistools/perf/util/session.c tools/perf/util/session.h+315 -390 replies
12/12perf session: use reader functions to load perf data filetools/perf/util/session.c+66 -1490 replies

Reviews trailers per patch

1 of 12 patches has a Reviewed-by or Acked-by.

PatchNamhyung Kim
1/12
2/12
3/12
4/12
5/12
6/12
7/12
8/12
9/12
10/12A-b
11/12
12/12

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [PATCH v3 00/12] Introduce threaded trace streaming for basic perf record operation

Changes in v3:
- avoided skipped redundant patch 3/15
- applied "data file" and "data directory" terms allover the patch set
- captured Acked-by: tags by Namhyung Kim
- avoided braces where don't needed
- employed thread local variable for serial trace streaming
- added specs for --thread option - core, socket, numa and user defined
- added parallel loading of data directory files similar to the prototype [1]

v2: https://lore.kernel.org/lkml/[email protected]/

Changes in v2:
- explicitly added credit tags to patches 6/15 and 15/15, additionally to cites [1], [2]
- updated description of 3/15 to explicitly mention the reason to open data directories in read access mode (e.g. for perf report)
- implemented fix for compilation error of 2/15
- explicitly elaborated on found issues to be resolved for threaded AUX trace capture

v1: https://lore.kernel.org/lkml/[email protected]/

Patch set provides parallel threaded trace streaming mode for basic perf record operation. Provided mode mitigates profiling data losses and resolves scalability issues of serial and asynchronous (--aio) trace streaming modes on multicore server systems. The design and implementation are based on the prototype [1], [2].

Parallel threaded mode executes trace streaming threads that read kernel data buffers and write captured data into several data files located at data directory. Layout of trace streaming threads and their mapping to data buffers to read can be configured using a value of --thread command line option. Specification value provides masks separated by colon so the masks define cpus to be monitored by one thread and thread affinity mask is separated by slash. <cpus mask 1>/<affinity mask 1>:<cpu mask 2>/<affinity mask 2> specifies parallel threads layout that consists of two threads with corresponding assigned cpus to be monitored. Specification value can be a string e.g. "cpu", "core" or "socket" meaning creation of data streaming thread for monitoring every cpu, whole core or socket. The option provided with no or empty value defaults to "cpu" layout creating data streaming thread for every cpu being monitored. Specification masks are filtered by the mask provided via -C option.

Parallel streaming mode is compatible with Zstd compression/decompression (--compression-level) and external control commands (--control). The mode is not enabled for pipe mode. The mode is not enabled for AUX area tracing, related and derived modes like --snapshot or --aux-sample. --switch-output-* and --timestamp-filename options are not enabled for parallel streaming. Initial intent to enable AUX area tracing faced the need to define some optimal way to store index data in data directory. --switch-output-* and --timestamp-filename use cases are not clear for data directories. Asynchronous(--aio) trace streaming and affinity (--affinity) modes are mutually exclusive to parallel streaming mode.

Basic analysis of data directories is provided in perf report mode. Raw dump and aggregated reports are available for data directories, still with no memory consumption optimizations.

Tested:

tools/perf/perf record -o prof.data --threads -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads= -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads=cpu -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads=core -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads=socket -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads=numa -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data --threads=0-3/3:4-7/4 -- matrix.gcc.g.O3
tools/perf/perf record -o prof.data -C 2,5 --threads=0-3/3:4-7/4 -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data -C 3,4 --threads=0-3/3:4-7/4 -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data -C 0,4,2,6 --threads=core -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data -C 0,4,2,6 --threads=numa -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data --threads -g --call-graph dwarf,4096 -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data --threads -g --call-graph dwarf,4096 --compression-level=3 -- matrix.gcc.g.O3 tools/perf/perf record -o prof.data --threads -a
tools/perf/perf record -D -1 -e cpu-cycles -a --control fd:10,11 -- sleep 30
tools/perf/perf record --threads -D -1 -e cpu-cycles -a --control fd:10,11 -- sleep 30

tools/perf/perf report -i prof.data tools/perf/perf report -i prof.data --call-graph=callee tools/perf/perf report -i prof.data --stdio --header tools/perf/perf report -i prof.data -D --header

[1] git clone https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git -b perf/record_threads
[2] https://lore.kernel.org/lkml/[email protected]/

---
Alexey Budankov (12):
perf record: introduce thread affinity and mmap masks perf record: introduce thread specific data array perf record: introduce thread local variable
perf record: stop threads in the end of trace streaming perf record: start threads in the beginning of trace streaming perf record: introduce data file at mmap buffer object perf record: init data file at mmap buffer object perf record: introduce --threads=<spec> command line option perf record: document parallel data streaming mode perf report: output data file name in raw trace dump perf session: load data directory files for analysis perf session: use reader functions to load perf data file

 tools/include/linux/bitmap.h             |   11 +
 tools/lib/api/fd/array.c                 |   17 +
 tools/lib/api/fd/array.h                 |    1 +
 tools/lib/bitmap.c                       |   14 +
 tools/perf/Documentation/perf-record.txt |   18 +
 tools/perf/builtin-inject.c              |    3 +-
 tools/perf/builtin-record.c              | 1019 ++++++++++++++++++++--
 tools/perf/util/evlist.c                 |   16 +
 tools/perf/util/evlist.h                 |    1 +
 tools/perf/util/mmap.c                   |    6 +
 tools/perf/util/mmap.h                   |    6 +
 tools/perf/util/ordered-events.h         |    1 +
 tools/perf/util/record.h                 |    2 +
 tools/perf/util/session.c                |  484 +++++++---
 tools/perf/util/session.h                |    5 +
 tools/perf/util/tool.h                   |    3 +-
 16 files changed, 1398 insertions(+), 209 deletions(-)
Signature · 1 line
2.24.1