Skip to content

RFC Add perf.data tracepoint events to trace.dat conversion

Tanushree Shah linux-perf-users linuxppc-dev 4 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 RFC Latest5 patches 0 repliesNew
  2. v3 RFC5 patches 3 repliesSuperseded
  3. v2 RFC5 patches 4 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v1 RFC4 patches6 repliesSuperseded

Changes in v3 from the cover letter

- Rebase on latest perf-tools-next and resolve merge conflicts.
- Drop evsel parameter from process_sample_event() following upstream commit "perf tool: Remove evsel from tool APIs that pass the sample".

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5perf/trace-dat: Add trace.dat export infrastructuretools/perf/util/Build tools/perf/util/trace-dat.c tools/perf/util/trace-dat.h+909 -00 replies
2/5perf/trace-event: Write trace.dat metadata sections during parsingtools/perf/util/trace-dat.c tools/perf/util/trace-dat.h tools/perf/util/trace-event-read.c+465 -940 replies
3/5perf data-convert: Add perf.data to trace.dat conversion backendtools/perf/util/Build tools/perf/util/data-convert-trace.c tools/perf/util/data-convert.h+248 -00 replies
4/5perf data: Add --to-trace-dat option for converting perf.data tracepoint events into trace.dat formattools/perf/Documentation/perf-data.txt tools/perf/builtin-data.c tools/perf/util/data-convert-trace.c +2 more+87 -50 replies
5/5perf test: Add test validating trace.dat generated by 'perf data convert --to-trace-dat'tools/perf/tests/shell/test_perf_data_converter_tracepoints_trace_dat.sh+173 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [RFC PATCH v4 0/5] Add perf.data tracepoint events to trace.dat conversion

This RFC patch series introduces support for converting perf.data files containing tracepoint events into trace.dat format, enabling seamless visualization and analysis using KernelShark.

======================
Background and Motivation
======================

Currently, perf and trace-cmd operate as separate tracing ecosystems with incompatible data formats. Users who collect tracepoint data with 'perf record' cannot easily visualize it in KernelShark's graphical timeline view or leverage trace-cmd's analysis capabilities.

This creates workflow friction when users need to:

- Visualize perf tracepoint data in KernelShark's interactive graphical timeline
- Share trace data between perf and trace-cmd workflows and toolchains
- Perform architecture-independent conversion and analysis of traces

This conversion bridge eliminates these barriers by enabling seamless data exchange between perf and trace-cmd ecosystems, allowing users to choose the best tool for each analysis phase.

======================
Implementation Overview
======================

The series implements the trace.dat file format specification (version 7) within perf's data conversion framework.

**Patch 1/5: Core trace.dat Export Infrastructure** Introduces util/trace-dat.c and util/trace-dat.h implementing:
- Per-CPU raw event buffer management (init, collect, free)
- Ftrace ring buffer page construction
- trace.dat section writers (strings, options, flyrecord sections)

**Patch 2/5: Metadata Integration**
Extends util/trace-event-read.c to write trace.dat metadata during perf.data
parsing:
- Initial format header (magic, version, endian, page size, compression)
- Section 16: HEADER INFO (header_page + header_event)
- Section 17: FTRACE EVENT FORMATS
- Section 18: EVENT FORMATS (per system/event format files)
- Section 19: KALLSYMS
- Section 21: CMDLINES
- Section 15: STRINGS (written last after all sections)

**Patch 3/5: Conversion Backend**
Implements util/data-convert-trace.c with trace_convert__perf2dat() function:
- Processes PERF_TYPE_TRACEPOINT samples via process_sample_event()
- Collects raw event data per-CPU using trace_dat__collect_cpu_event()
- Writes OPTIONS sections (CPUCOUNT, TRACECLOCK, metadata offsets)
- Writes FLYRECORD section with per-CPU ring buffer pages

**Patch 4/5: User Interface**
Extends tools/perf/builtin-data.c with --to-trace-dat option:
- Adds command-line option for trace.dat output
- Mutually exclusive with --to-ctf and --to-json
- Calls trace_convert__perf2dat() to perform conversion

**Patch 5/5: Shell Test**
Adds a shell test (tools/perf/tests/shell/) covering normal tracepoint recordings, pipe mode and mixed tracepoint/non-tracepoint recording conversions and --force flag behaviour.

======================
Current Implementation Details
======================

**trace.dat Format Version:**
The implementation currently targets trace.dat format version 7, which is the stable version supported by current trace-cmd releases (v3.x). This version is hardcoded to ensure compatibility with existing trace-cmd and KernelShark installations. Future enhancements could add version negotiation or support for newer format versions as they become standardized.

**Compression Strategy:**
Compression is explicitly disabled (set to NONE) in the generated trace.dat files.
This design choice:
- Simplifies the initial implementation and testing
- Ensures maximum compatibility across trace-cmd versions
- Avoids external compression library dependencies

Future work could add support for various compression algorithms (zlib, zstd, lz4) with runtime selection via command-line options, significantly reducing file sizes for large traces.

======================
Usage Example
======================

```bash
*Record tracepoint events with perf*
perf record -e sched:sched_switch -e sched:sched_wakeup -a sleep 10

*Convert to trace.dat format*
perf data convert --to-trace-dat=output.dat

*Verify trace.dat structure*
trace-cmd dump --summary output.dat

*Analyze with trace-cmd*
trace-cmd report output.dat

*Visualize in KernelShark*
kernelshark output.dat
```

**Conversion Output:**

```
[ perf data convert: Converted 'perf.data' into trace.dat format
'output.dat' ]
[ perf data convert: Converted 2684 events ]
```
**trace-cmd dump --summary Output:**
```
 Tracing meta data in file output.dat:
	[Initial format]
		7	[Version]
		0	[Little endian]
		8	[Bytes in a long]
		65536	[Page size, bytes]
		none	[Compression algorithm]
			[Compression version]
	[buffer "", "local" clock, 65536 page size, 16 cpus, 1048576 bytes
    flyrecord data]
	[10 options]
	[Saved command lines, 0 bytes]
	[Kallsyms, 0 bytes]
	[Ftrace format, 0 events]
	[Header page, 206 bytes]
	[Header event, 205 bytes]
	[Events format, 1 systems]
	[9 sections]
```	

======================
Testing and Verification
======================

The series has been extensively tested with:
- Various tracepoint events (sched, irq, syscalls, block I/O)
- Mixed recordings containing both tracepoint and non-tracepoint events (only tracepoints converted)
- Verification with trace-cmd report and KernelShark visualization
- Memory leak testing with Valgrind (0 bytes leaked)
- Cross-architecture testing: v1 tested x86_64 and ppc64le. v2 adds s390 (big-endian) perf.data converted on both ppc64le (little-endian) and x86_64 (little-endian) hosts, in addition to same-arch x86_64 (LE->LE) and ppc64le (BE->BE) conversion.
- Pipe mode support has been tested end-to-end. (in v2)
- Time filtering: verified --time option correctly limits converted events to the requested range, consistent with --to-json behaviour. (in v4)

All generated trace.dat files successfully open in:
- trace-cmd report (v3.1+)
- KernelShark (v2.0+)

======================
Next Steps
======================

We would highly appreciate reviews, comments, and feedback on:
- The overall architectural approach and integration points
- Compatibility considerations with trace-cmd ecosystem
- Performance characteristics for large-scale traces
- Additional use cases or workflow scenarios
- Future enhancement priorities

---
Changes in v4

Addressing Sashiko AI review findings on v3:

Timestamp correctness (trace-dat.c):
- Fix TIME_EXTEND delta_upper shift: >> 5 (TRACE_DAT_RECORD_TIME_SHIFT) should be >> 27, causing wildly inflated timestamps in trace-cmd.
- Advance base_ts after every event so time_delta is relative to the preceding record, not a stale page base.
- Introduce page_base_ts updated only at page boundaries, keeping the page header timestamp correct independently of per-event base_ts.

Memory safety (trace-dat.c):
- Fix leak on TIME_EXTEND calloc() failure: use goto out_free instead of return -ENOMEM so page_records/page_rec_sizes are freed.

I/O correctness (trace-event-read.c):
- Check fseek() return values when patching the event formats section size; set trace_dat_write_failed on any failure to prevent silent corruption of subsequent writes.

Resource management (data-convert-trace.c):
- Fix orphaned 0-byte output file when fdopen() fails in !opts->force
path: use goto out_close so unlink() is called on error.
- Honor --time filtering using perf_time__parse_for_ranges() and perf_time__ranges_skip_sample(), consistent with JSON/CTF converters.
- Add explicit <stdio.h> include for fdopen()/fopen()/fclose().

Header (trace-dat.h):
- Add explicit <stdint.h> for uint16_t/uint32_t/uint64_t on musl libc.

Shell test (patch 5):
- Replace 'cycles' with 'cpu-clock -a' for portability on s390, KVM guests and unprivileged containers without hardware PMU support.
- Add -a to all perf record/sleep invocations per Ian's suggestion.
- Fix EXIT trap to pass through exit code 2 (skip) unchanged.
- Move second mktemp after trap registration to avoid temp file leak.
- Drop rm -f "$result" before converter in test_trace_converter_command; --force handles overwriting and deleting first opens a symlink race.
- Move rm -f "$result" to before perf record in pipe and mixed-events tests to prevent stale output being validated on early return.

Changes in v3

- Rebase on latest perf-tools-next and resolve merge conflicts.
- Drop evsel parameter from process_sample_event() following upstream commit "perf tool: Remove evsel from tool APIs that pass the sample".

Changes in v2

Addressing the Sashiko AI review findings on v1:

Cross-arch correctness:
- Introduce to_file_u16/u32/u64 helpers (wrapping tep_read_number()) to write all multi-byte fields in the recorded machine's byte order; apply throughout metadata sections and flyrecord page/record headers (ts, commit, TIME_EXTEND, large-event data_len).
- Fix flyrecord record header bit layout for big-endian files. The record header word bit layout differs by file endianness, matching kbuffer-parse.c type_len4host()/ts4host():
LE: type_len in bits [4:0], time_delta in bits [31:5]
BE: type_len in bits [31:27], time_delta in bits [26:0]

Pipe mode:
- Add process_attr(), process_feature(), and process_tracing_data() callbacks required for pipe mode operation.
- Defer CPU buffer initialisation until the first tracepoint sample, after process_feature()/process_tracing_data() have populated the session header. This ensures the recorded machine's CPU count is used rather than the host's - critical for cross-platform analysis.

Format compliance:
- Implement TIME_EXTEND records for timestamp deltas >27 bits to prevent silent truncation and maintain chronological ordering.
- Fix large event encoding (>=29 words): use type_len=0 with a separate 32-bit length word, avoiding collision with reserved types (PADDING=29, TIME_EXTEND=30, TIME_STAMP=31).
- Add bounds check rejecting records larger than a page payload before batching, preventing heap overflow in trace_dat__write_page().
- Fix flyrecord section_size to exclude the 16-byte section header, matching trace.dat specification and trace-cmd behaviour.

CLI behavior:
- Fix --force flag: open with O_CREAT|O_EXCL when force is not set, failing with -EEXIST instead of silently overwriting existing files.

Memory safety:
- Fix realloc overwrite of cpu_events->events and page_records on
failure: use temporary pointers, only commit on success.
- Fix use-after-free/double-free in sequential page_records realloc
failure: replace with malloc+memcpy+free pattern.
- Fix section_size computed from before section header position.
- Add NULL checks for get_tracing_file(), calloc() padding, and trace_dat_options_offset assignment on write failure.
- Use goto out_free on record allocation failure to avoid leaking accumulated page_records entries.
- Replace direct read() with do_read() in read_proc_kallsyms() to handle short reads correctly.
- On fwrite failure, set trace_dat_write_failed and continue parsing so that perf.data processing completes normally.

Testing (new in v2):
- Add shell test covering conversion, trace-cmd dump validation, sched_switch event verification, and --force flag behaviour.

Documentation (new in v2):
- Add documentation for 'perf data convert --to-trace-dat', covering usage and supported options.

v1: https://lore.kernel.org/linux-perf-users/[email protected]/

Tanushree Shah (5):
perf/trace-dat: Add trace.dat export infrastructure perf/trace-event: Write trace.dat metadata sections during parsing perf data-convert: Add perf.data to trace.dat conversion backend perf data: Add --to-trace-dat option for converting perf.data
tracepoint events into trace.dat format
perf test: Add test validating trace.dat generated by 'perf data

    convert --to-trace-dat'
 tools/perf/Documentation/perf-data.txt        |   7 +
 tools/perf/builtin-data.c                     |  43 +
 ...rf_data_converter_tracepoints_trace_dat.sh | 173 ++++
 tools/perf/util/Build                         |   2 +
 tools/perf/util/data-convert-trace.c          | 264 ++++++
 tools/perf/util/data-convert.h                |   4 +
 tools/perf/util/trace-dat.c                   | 883 ++++++++++++++++++
 tools/perf/util/trace-dat.h                   | 114 +++
 tools/perf/util/trace-event-read.c            | 307 +++++-
 9 files changed, 1790 insertions(+), 7 deletions(-)
 create mode 100755 tools/perf/tests/shell/test_perf_data_converter_tracepoints_trace_dat.sh
 create mode 100644 tools/perf/util/data-convert-trace.c
 create mode 100644 tools/perf/util/trace-dat.c
 create mode 100644 tools/perf/util/trace-dat.h
Signature · 1 line
2.47.1