perf to ctf converter
RFC 1/1 perf to ctf converter
I've been playing with python bindings of perf and babeltrace and came up with a way to covert the perf trace into the CTF format. It supports both ftrace events (perf record -e raw_syscalls:* w) and perf counters (perf record -e cache-misses w).
The recorded trace is first read via the "perf script" interface and saved as python pickle. In a second step the pickled-data is converted into a CTF file format.
The perf part requires
"perf script: move the number processing into its own function" "perf script: handle the num array type in python properly"
https://lkml.org/lkml/2014/5/27/434
for array support and
"perf script: pass more arguments to the python event handler"
https://lkml.org/lkml/2014/5/30/392
for more data while reading the "events" traces. The latter will be probably replaced by https://lkml.org/lkml/2014/4/3/217.
Babeltrace needs only
"ctf-writer: Add support for the cpu_id field"
https://www.mail-archive.com/[email protected]/msg06057.html
for the assignment of the CPU number.
The pickle step is nice because I see all type of events before I start writing the CTF trace and can create the necessary objects. On the other hand it eats a lot of memory for huge traces so I will try to replace it with something that saves the data in a streaming like fashion.
The other limitation is that babeltrace doesn't seem to work with python2 while perf doesn't compile against python3.
What I haven't figured out yet is how to pass to the meta environment informations that is displayed by "perf script --header-only -I" and if that information is really important. Probably an optional python callback will do it.
The required steps:
| perf record -e raw_syscalls:* w | perf script -s ./to-pickle.py | ./ctf_writer
ctf_writer.py +132 -0
@@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +# ctf_writer.py +# + +from babeltrace import * +import pickle + +trace_file = "ctf-data.pickle" +trace_path = "ctf-out" + +print("Writing trace at %s from %s" %(trace_path, trace_file)) + +trace = pickle.load(open(trace_file, "rb")) + +writer = CTFWriter.Writer(trace_path) + +clock = CTFWriter.Clock("monotonic") +clock.description = "Monotonic Clock" +clock.offset = 0 # XXX + +writer.add_clock(clock) +writer.add_environment_field("hostname", "bach") +writer.add_environment_field("domain", "kernel") +writer.add_environment_field("sysname", "Linux") +writer.add_environment_field("kernel_release", "3.6.0") # XXX +writer.add_environment_field("kernel_version", "#8 SMP Fri May 23 15:29:41 CEST 2014") # XXX +writer.add_environment_field("tracer_name", "perf") +writer.add_environment_field("tracer_major", "2") +writer.add_environment_field("tracer_minor", "4") +writer.add_environment_field("tracer_patchlevel", "0") + +stream_class = CTFWriter.StreamClass("stream") +stream_class.clock = clock + +# certain file types may be 32bit or 64bit. Even the first event we find and +# build our type might pass NULL which would mean 32bit. The second event +# might pass a 64bit. +# For now we default hex to u64 for array, have a list of hex u64 and everything +# else is s32. +list_type_h_uint64 = [ "addr" ] + +int32_type = CTFWriter.IntegerFieldDeclaration(32) +int32_type.signed = True + +uint64_type = CTFWriter.IntegerFieldDeclaration(64) +uint64_type.signed = False + +hex_uint64_type = CTFWriter.IntegerFieldDeclaration(64) +hex_uint64_type.signed = False +hex_uint64_type.base = 16 + +string_type = CTFWriter.StringFieldDeclaration() + +events = {} +last_cpu = -1 + +list_ev_entry_ignore = [ "common_s", "common_ns", "common_cpu" ] + +# First create all possible event class-es +for entry in trace: + event_name = entry[0] + event_record = entry[1] + + try: + event_class = events[event_name] + except: + event_class = CTFWriter.EventClass(event_name); + for ev_entry in sorted(event_record): + if ev_entry in list_ev_entry_ignore: + continue + val = event_record[ev_entry] + if isinstance(val, int): + if ev_entry in list_type_h_uint64: + event_class.add_field(hex_uint64_type, ev_entry) + else: + event_class.add_field(int32_type, ev_entry) + elif isinstance(val, str): + event_class.add_field(string_type, ev_entry) + elif isinstance(val, list): + array_type = CTFWriter.ArrayFieldDeclaration(hex_uint64_type, len(val)) + event_class.add_field(array_type, ev_entry) + else: + print("Unknown type in trace: %s" %(type(event_record[ev_entry]))) + raise Exception("Unknown type in trace.") + + # Add complete class with all event members. + print("New event type: %s" %(event_name)) + stream_class.add_event_class(event_class) + events[event_name] = event_class + +print("Event types complete") +stream = writer.create_stream(stream_class) + +# Step two, fill it with data +for entry in trace: + event_name = entry[0] + event_record = entry[1] + + ts = int((int(event_record["common_s"]) * 1e9 + int(event_record["common_ns"]))) + + event_class = events[event_name] + event = CTFWriter.Event(event_class) + + clock.time = ts + + for ev_entry in event_record: + if ev_entry in list_ev_entry_ignore: + continue + + field = event.payload(ev_entry) + val = event_record[ev_entry] + if isinstance(val, int): + field.value = int(val) + elif isinstance(val, str): + field.value = val + elif isinstance(val, list): + for i in range(len(val)): + a_idx = field.field(i) + a_idx.value = int(val[i]) + else: + print("Unexpected entry type: %s" %(type(val))) + raise Exception("Unexpected type in trace.") + + stream.append_event(event) + cur_cpu = int(event_record["common_cpu"]) + if cur_cpu != last_cpu: + stream.append_cpu_id(cur_cpu) + last_cpu = cur_cpu + stream.flush() + +stream.flush() +print("Done.")
to-pickle.py +57 -0
57 lines not on this page · raw message
Quoted 5 lines · Sebastian Andrzej Siewior
I've been playing with python bindings of perf and babeltrace and came up with a way to covert the perf trace into the CTF format. It supports both ftrace events (perf record -e raw_syscalls:* w) and perf counters (perf record -e cache-misses w).
Quoted 8 lines
The recorded trace is first read via the "perf script" interface and saved as python pickle. In a second step the pickled-data is converted into a CTF file format.
The perf part requires
"perf script: move the number processing into its own function" "perf script: handle the num array type in python properly"
https://lkml.org/lkml/2014/5/27/434
I saw those 2 already in Arnaldo's tree
Quoted 4 lines
for array support and
"perf script: pass more arguments to the python event handler"
https://lkml.org/lkml/2014/5/30/392
and there's some other replacement for this one comming in soon IIUC
Quoted 6 lines
for more data while reading the "events" traces. The latter will be probably replaced by https://lkml.org/lkml/2014/4/3/217.
Babeltrace needs only
"ctf-writer: Add support for thecpu_idfield"
https://www.mail-archive.com/[email protected]/msg06057.html
any idea when this one will land in babeltrace git tree?
Quoted 16 lines
for the assignment of the CPU number.
The pickle step is nice because I see all type of events before I start writing the CTF trace and can create the necessary objects. On the other hand it eats a lot of memory for huge traces so I will try to replace it with something that saves the data in a streaming like fashion.
The other limitation is that babeltrace doesn't seem to work with python2 while perf doesn't compile against python3.What I haven't figured out yet is how to pass to the meta environment informations that is displayed by "perf script --header-only -I" and if that information is really important. Probably an optional python callback will do it.
Quoted 4 lines
The required steps:
| perf record -e raw_syscalls:* w | perf script -s ./to-pickle.py | ./ctf_writer
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2
It's able to convert single event (HW type) perf.data file into CTF data, by adding just one integer field "period" and single stream, like:
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
...
[jolsa@krava babeltrace]$ /opt/libbabeltrace/bin/babeltrace /home/jolsa/kernel.org/linux-perf/tools/perf/ctf-data [08:14:45.814456098] (+?.?????????) cycles: { }, { period = 1 }
[08:14:45.814459237] (+0.000003139) cycles: { }, { period = 1 }
[08:14:45.814460684] (+0.000001447) cycles: { }, { period = 9 }
[08:14:45.814462073] (+0.000001389) cycles: { }, { period = 182 }
[08:14:45.814463491] (+0.000001418) cycles: { }, { period = 4263 }
[08:14:45.814465874] (+0.000002383) cycles: { }, { period = 97878 }
[08:14:45.814506385] (+0.000040511) cycles: { }, { period = 1365965 }
[08:14:45.815056528] (+0.000550143) cycles: { }, { period = 2250012 }
---
the goals for me is to have a convert tool, like in above example perf data command and support in perf record/report to directl write/read ctf data
Using python for this seems nice.. I'm not experienced python coder, so just small comments/questions
SNIP
Quoted 10 lines
ctf_writer.py +10
-0+list_type_h_uint64 = [ "addr" ] + +int32_type = CTFWriter.IntegerFieldDeclaration(32) +int32_type.signed = True + +uint64_type = CTFWriter.IntegerFieldDeclaration(64) +uint64_type.signed = False + +hex_uint64_type = CTFWriter.IntegerFieldDeclaration(64) +hex_uint64_type.signed = False
ctf_writer.py line 59
Quoted 10 lines
ctf_writer.py +10
-0+hex_uint64_type.base = 16 + +string_type = CTFWriter.StringFieldDeclaration() + +events = {} +last_cpu = -1 + +list_ev_entry_ignore = [ "common_s", "common_ns", "common_cpu" ] + +# First create all possible event class-es
this first iteration could be handled in the to-pickle step, which could gather events description and store/pickle it before the trace data
Quoted 19 lines
ctf_writer.py +19
-0+for entry in trace: + event_name = entry[0] + event_record = entry[1] + + try: + event_class = events[event_name] + except: + event_class = CTFWriter.EventClass(event_name); + for ev_entry in sorted(event_record): + if ev_entry in list_ev_entry_ignore: + continue + val = event_record[ev_entry] + if isinstance(val, int): + if ev_entry in list_type_h_uint64: + event_class.add_field(hex_uint64_type, ev_entry) + else: + event_class.add_field(int32_type, ev_entry) + elif isinstance(val, str): + event_class.add_field(string_type, ev_entry)
SNIP
Quoted 11 lines
to-pickle.py +11
-0+ +def process_event(event_fields_dict): + entry = [] + entry.append(str(event_fields_dict["ev_name"])) + fields = {} + fields["common_s"] = event_fields_dict["s"] + fields["common_ns"] = event_fields_dict["ns"] + fields["common_comm"] = event_fields_dict["comm"] + fields["common_pid"] = event_fields_dict["pid"] + fields["addr"] = event_fields_dict["addr"] +
to-pickle.py line 48
Quoted 10 lines
to-pickle.py +10
-0+ dso = "" + symbol = "" + try: + dso = event_fields_dict["dso"] + except: + pass + try: + symbol = event_fields_dict["symbol"] + except: + pass
I understand this is just a early stage, but we want here detection of the all event arguments, right?
I wonder we could add separated python callback for that
thanks,
jirka
Quoted 7 lines · Jiri Olsa
Quoted 5 lines
for more data while reading the "events" traces. The latter will be probably replaced by https://lkml.org/lkml/2014/4/3/217.
Babeltrace needs only
"ctf-writer: Add support for thecpu_idfield"
https://www.mail-archive.com/[email protected]/msg06057.htmlany idea when this one will land in babeltrace git tree?
Need to re-do them the way they asked. Could take some time. However I wanted first to make sure it make sense to continue that approach.
Quoted 21 lines
Quoted 16 lines
for the assignment of the CPU number.
The pickle step is nice because I see all type of events before I start writing the CTF trace and can create the necessary objects. On the other hand it eats a lot of memory for huge traces so I will try to replace it with something that saves the data in a streaming like fashion.
The other limitation is that babeltrace doesn't seem to work with python2 while perf doesn't compile against python3.What I haven't figured out yet is how to pass to the meta environment informations that is displayed by "perf script --header-only -I" and if that information is really important. Probably an optional python callback will do it.
Quoted 4 lines
The required steps:
| perf record -e raw_syscalls:* w | perf script -s ./to-pickle.py | ./ctf_writer
Quoted 6 lines
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2
Let me try it, maybe I can migrate my effort into one code basis.
Quoted 16 lines
It's able to convert single event (HW type) perf.data file into CTF data, by adding just one integer field "period" and single stream, like:
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
...
[jolsa@krava babeltrace]$/opt/libbabeltrace/bin/babeltrace/home/jolsa/kernel.org/linux-perf/tools/perf/ctf-data[08:14:45.814456098] (+?.?????????) cycles: { }, { period = 1 }
[08:14:45.814459237] (+0.000003139) cycles: { }, { period = 1 }
[08:14:45.814460684] (+0.000001447) cycles: { }, { period = 9 }
[08:14:45.814462073] (+0.000001389) cycles: { }, { period = 182 }
[08:14:45.814463491] (+0.000001418) cycles: { }, { period = 4263 }
[08:14:45.814465874] (+0.000002383) cycles: { }, { period = 97878 }
[08:14:45.814506385] (+0.000040511) cycles: { }, { period = 1365965 }
[08:14:45.815056528] (+0.000550143) cycles: { }, { period = 2250012 }
---
Quoted 6 lines
the goals for me is to have a convert tool, like in above example perf data command and support in perf record/report to directl write/read ctf data
Using python for this seems nice.. I'm not experienced python coder, so just small comments/questions
python looked nice because I saw libraries / interfaces on both sides.
Quoted 18 lines
SNIP
Quoted 16 lines
ctf_writer.py +16
-0+list_type_h_uint64 = [ "addr" ] + +int32_type = CTFWriter.IntegerFieldDeclaration(32) +int32_type.signed = True + +uint64_type = CTFWriter.IntegerFieldDeclaration(64) +uint64_type.signed = False + +hex_uint64_type = CTFWriter.IntegerFieldDeclaration(64) +hex_uint64_type.signed = False +hex_uint64_type.base = 16 + +string_type = CTFWriter.StringFieldDeclaration() + +events = {} +last_cpu = -1
Quoted 8 lines
ctf_writer.py line 59
Quoted 4 lines
ctf_writer.py +4
-0+ +list_ev_entry_ignore = [ "common_s", "common_ns", "common_cpu" ] + +# First create all possible event class-esthis first iteration could be handled in the to-pickle step, which could gather events description and store/pickle it before the trace data
yes.
Quoted 39 lines
Quoted 19 lines
ctf_writer.py +19
-0+for entry in trace: + event_name = entry[0] + event_record = entry[1] + + try: + event_class = events[event_name] + except: + event_class = CTFWriter.EventClass(event_name); + for ev_entry in sorted(event_record): + if ev_entry in list_ev_entry_ignore: + continue + val = event_record[ev_entry] + if isinstance(val, int): + if ev_entry in list_type_h_uint64: + event_class.add_field(hex_uint64_type, ev_entry) + else: + event_class.add_field(int32_type, ev_entry) + elif isinstance(val, str): + event_class.add_field(string_type, ev_entry)SNIP
Quoted 16 lines
to-pickle.py +16
-0+ +def process_event(event_fields_dict): + entry = [] + entry.append(str(event_fields_dict["ev_name"])) + fields = {} + fields["common_s"] = event_fields_dict["s"] + fields["common_ns"] = event_fields_dict["ns"] + fields["common_comm"] = event_fields_dict["comm"] + fields["common_pid"] = event_fields_dict["pid"] + fields["addr"] = event_fields_dict["addr"] + + dso = "" + symbol = "" + try: + dso = event_fields_dict["dso"] + except:
Quoted 8 lines
to-pickle.py line 48
Quoted 5 lines
to-pickle.py +5
-0+ pass + try: + symbol = event_fields_dict["symbol"] + except: + passI understand this is just a early stage, but we want here detection of the all event arguments, right?
Yes. The CTF writer is stupid and takes all arguments as-is and passes it over the babeltrace part of CTF writer. This works well for the ftrace events (handled by trace_unhandled()).
Quoted 1 line
I wonder we could add separated python callback for that
This (the to pickle part) tries come up with the common basis for the CPU events. Therefore it renames the first few arguments (like s to common_s) to make it consistent with the ftrace events.
The dso and symbol members look optional depending whether or not this data was available at trace time. I *think* those may change within a stream say if one library has debug symbols available and the other does not. So I have no idea how you plan specific callbacks for those.
Quoted 2 lines
thanks,
jirka
Sebastian
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2Let me try it, maybe I can migrate my effort into one code basis.
FYI: I rebased your code antop of current tip-tree. The C-API looks better. Especially since I can evlist__for_each() and create a list of all events. So the C version might get cuter…
Sebastian
----- Original Message -----
Quoted 14 lines
From: "Sebastian Andrzej Siewior" <[email protected]>
To: "Jiri Olsa" <[email protected]>
Cc: [email protected], [email protected], "Mathieu Desnoyers" <[email protected]>, [email protected], "namhyung kim" <[email protected]>, [email protected]
Sent: Friday, July 18, 2014 8:34:04 AM
Subject: Re: [RFC] perf to ctf converterQuoted 6 lines · Jiri Olsa
Quoted 5 lines
for more data while reading the "events" traces. The latter will be probably replaced by https://lkml.org/lkml/2014/4/3/217.
Babeltrace needs only
"ctf-writer: Add support for thecpu_idfield"
https://www.mail-archive.com/[email protected]/msg06057.html
Quoted 4 lines
Quoted 1 line
any idea when this one will land in babeltrace git tree?
Need to re-do them the way they asked. Could take some time. However I wanted first to make sure it make sense to continue that approach.
CCing Jérémie, co-maintainer of Babeltrace. He will be able to answer your questions and help out.
Thanks!
Mathieu
Quoted 155 lines
Quoted 22 lines
Quoted 17 lines
for the assignment of the CPU number.
The pickle step is nice because I see all type of events before I start writing the CTF trace and can create the necessary objects. On the other hand it eats a lot of memory for huge traces so I will try to replace it with something that saves the data in a streaming like fashion.
The other limitation is that babeltrace doesn't seem to work with python2 while perf doesn't compile against python3.What I haven't figured out yet is how to pass to the meta environment informations that is displayed by "perf script --header-only -I" and if that information is really important. Probably an optional python callback will do it.
Quoted 4 lines
The required steps:
| perf record -e raw_syscalls:* w | perf script -s ./to-pickle.py | ./ctf_writerQuoted 6 lines
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2Let me try it, maybe I can migrate my effort into one code basis.
Quoted 18 lines
It's able to convert single event (HW type) perf.data file into CTF data, by adding just one integer field "period" and single stream, like:
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
...
[jolsa@krava babeltrace]$/opt/libbabeltrace/bin/babeltrace/home/jolsa/kernel.org/linux-perf/tools/perf/ctf-data[08:14:45.814456098] (+?.?????????) cycles: { }, { period = 1 } [08:14:45.814459237] (+0.000003139) cycles: { }, { period = 1 } [08:14:45.814460684] (+0.000001447) cycles: { }, { period = 9 } [08:14:45.814462073] (+0.000001389) cycles: { }, { period = 182 } [08:14:45.814463491] (+0.000001418) cycles: { }, { period = 4263 } [08:14:45.814465874] (+0.000002383) cycles: { }, { period = 97878 } [08:14:45.814506385] (+0.000040511) cycles: { }, { period = 1365965 } [08:14:45.815056528] (+0.000550143) cycles: { }, { period = 2250012 }
---Quoted 6 lines
the goals for me is to have a convert tool, like in above example perf data command and support in perf record/report to directl write/read ctf data
Using python for this seems nice.. I'm not experienced python coder, so just small comments/questions
python looked nice because I saw libraries / interfaces on both sides.
Quoted 18 lines
SNIP
Quoted 16 lines
ctf_writer.py +16
-0+list_type_h_uint64 = [ "addr" ] + +int32_type = CTFWriter.IntegerFieldDeclaration(32) +int32_type.signed = True + +uint64_type = CTFWriter.IntegerFieldDeclaration(64) +uint64_type.signed = False + +hex_uint64_type = CTFWriter.IntegerFieldDeclaration(64) +hex_uint64_type.signed = False +hex_uint64_type.base = 16 + +string_type = CTFWriter.StringFieldDeclaration() + +events = {} +last_cpu = -1Quoted 8 lines
ctf_writer.py line 59
Quoted 4 lines
ctf_writer.py +4
-0+ +list_ev_entry_ignore = [ "common_s", "common_ns", "common_cpu" ] + +# First create all possible event class-esthis first iteration could be handled in the to-pickle step, which could gather events description and store/pickle it before the trace data
yes.
Quoted 39 lines
Quoted 19 lines
ctf_writer.py +19
-0+for entry in trace: + event_name = entry[0] + event_record = entry[1] + + try: + event_class = events[event_name] + except: + event_class = CTFWriter.EventClass(event_name); + for ev_entry in sorted(event_record): + if ev_entry in list_ev_entry_ignore: + continue + val = event_record[ev_entry] + if isinstance(val, int): + if ev_entry in list_type_h_uint64: + event_class.add_field(hex_uint64_type, ev_entry) + else: + event_class.add_field(int32_type, ev_entry) + elif isinstance(val, str): + event_class.add_field(string_type, ev_entry)SNIP
Quoted 16 lines
to-pickle.py +16
-0+ +def process_event(event_fields_dict): + entry = [] + entry.append(str(event_fields_dict["ev_name"])) + fields = {} + fields["common_s"] = event_fields_dict["s"] + fields["common_ns"] = event_fields_dict["ns"] + fields["common_comm"] = event_fields_dict["comm"] + fields["common_pid"] = event_fields_dict["pid"] + fields["addr"] = event_fields_dict["addr"] + + dso = "" + symbol = "" + try: + dso = event_fields_dict["dso"] + except:Quoted 8 lines
to-pickle.py line 48
Quoted 5 lines
to-pickle.py +5
-0+ pass + try: + symbol = event_fields_dict["symbol"] + except: + passI understand this is just a early stage, but we want here detection of the all event arguments, right?
Yes. The CTF writer is stupid and takes all arguments as-is and passes it over the babeltrace part of CTF writer. This works well for the ftrace events (handled by
trace_unhandled()).Quoted 1 line
I wonder we could add separated python callback for that
This (the to pickle part) tries come up with the common basis for the CPU events. Therefore it renames the first few arguments (like s to
common_s) to make it consistent with the ftrace events.
The dso and symbol members look optional depending whether or not this data was available at trace time. I *think* those may change within a stream say if one library has debug symbols available and the other does not. So I have no idea how you plan specific callbacks for those.Quoted 2 lines
thanks,
jirkaSebastian
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 7 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Jiri Olsa
Quoted 5 lines
for more data while reading the "events" traces. The latter will be probably replaced by https://lkml.org/lkml/2014/4/3/217.
Babeltrace needs only
"ctf-writer: Add support for thecpu_idfield"
https://www.mail-archive.com/[email protected]/msg06057.html
Quoted 5 lines
Quoted 1 line
any idea when this one will land in babeltrace git tree?
Need to re-do them the way they asked. Could take some time. However I wanted first to make sure it make sense to continue that approach.
FYI, I have made the changes and they are now upstream in Babeltrace as of
commit 12c8a1a3121ed7125e8758065c44658d8eda1333
Author: Jérémie Galarneau <[email protected]> Date: Tue Jul 29 16:51:51 2014 -0400
Add stream packet header accessors
Stream packet contexts may now be modified to contain custom fields. The events_discarded field is now handled like a generic packet context field.
Regards,
Jérémie
Quoted 156 lines
Quoted 21 lines
Quoted 16 lines
for the assignment of the CPU number.
The pickle step is nice because I see all type of events before I start writing the CTF trace and can create the necessary objects. On the other hand it eats a lot of memory for huge traces so I will try to replace it with something that saves the data in a streaming like fashion.
The other limitation is that babeltrace doesn't seem to work with python2 while perf doesn't compile against python3.What I haven't figured out yet is how to pass to the meta environment informations that is displayed by "perf script --header-only -I" and if that information is really important. Probably an optional python callback will do it.
Quoted 4 lines
The required steps:
| perf record -e raw_syscalls:* w | perf script -s ./to-pickle.py | ./ctf_writerQuoted 6 lines
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2Let me try it, maybe I can migrate my effort into one code basis.
Quoted 16 lines
It's able to convert single event (HW type) perf.data file into CTF data, by adding just one integer field "period" and single stream, like:
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
...
[jolsa@krava babeltrace]$/opt/libbabeltrace/bin/babeltrace/home/jolsa/kernel.org/linux-perf/tools/perf/ctf-data[08:14:45.814456098] (+?.?????????) cycles: { }, { period = 1 }
[08:14:45.814459237] (+0.000003139) cycles: { }, { period = 1 }
[08:14:45.814460684] (+0.000001447) cycles: { }, { period = 9 }
[08:14:45.814462073] (+0.000001389) cycles: { }, { period = 182 }
[08:14:45.814463491] (+0.000001418) cycles: { }, { period = 4263 }
[08:14:45.814465874] (+0.000002383) cycles: { }, { period = 97878 }
[08:14:45.814506385] (+0.000040511) cycles: { }, { period = 1365965 }
[08:14:45.815056528] (+0.000550143) cycles: { }, { period = 2250012 }
---Quoted 6 lines
the goals for me is to have a convert tool, like in above example perf data command and support in perf record/report to directl write/read ctf data
Using python for this seems nice.. I'm not experienced python coder, so just small comments/questions
python looked nice because I saw libraries / interfaces on both sides.
Quoted 18 lines
SNIP
Quoted 16 lines
ctf_writer.py +16
-0+list_type_h_uint64 = [ "addr" ] + +int32_type = CTFWriter.IntegerFieldDeclaration(32) +int32_type.signed = True + +uint64_type = CTFWriter.IntegerFieldDeclaration(64) +uint64_type.signed = False + +hex_uint64_type = CTFWriter.IntegerFieldDeclaration(64) +hex_uint64_type.signed = False +hex_uint64_type.base = 16 + +string_type = CTFWriter.StringFieldDeclaration() + +events = {} +last_cpu = -1Quoted 8 lines
ctf_writer.py line 59
Quoted 4 lines
ctf_writer.py +4
-0+ +list_ev_entry_ignore = [ "common_s", "common_ns", "common_cpu" ] + +# First create all possible event class-esthis first iteration could be handled in the to-pickle step, which could gather events description and store/pickle it before the trace data
yes.
Quoted 39 lines
Quoted 19 lines
ctf_writer.py +19
-0+for entry in trace: + event_name = entry[0] + event_record = entry[1] + + try: + event_class = events[event_name] + except: + event_class = CTFWriter.EventClass(event_name); + for ev_entry in sorted(event_record): + if ev_entry in list_ev_entry_ignore: + continue + val = event_record[ev_entry] + if isinstance(val, int): + if ev_entry in list_type_h_uint64: + event_class.add_field(hex_uint64_type, ev_entry) + else: + event_class.add_field(int32_type, ev_entry) + elif isinstance(val, str): + event_class.add_field(string_type, ev_entry)SNIP
Quoted 16 lines
to-pickle.py +16
-0+ +def process_event(event_fields_dict): + entry = [] + entry.append(str(event_fields_dict["ev_name"])) + fields = {} + fields["common_s"] = event_fields_dict["s"] + fields["common_ns"] = event_fields_dict["ns"] + fields["common_comm"] = event_fields_dict["comm"] + fields["common_pid"] = event_fields_dict["pid"] + fields["addr"] = event_fields_dict["addr"] + + dso = "" + symbol = "" + try: + dso = event_fields_dict["dso"] + except:Quoted 8 lines
to-pickle.py line 48
Quoted 5 lines
to-pickle.py +5
-0+ pass + try: + symbol = event_fields_dict["symbol"] + except: + passI understand this is just a early stage, but we want here detection of the all event arguments, right?
Yes. The CTF writer is stupid and takes all arguments as-is and passes it over the babeltrace part of CTF writer. This works well for the ftrace events (handled by
trace_unhandled()).Quoted 1 line
I wonder we could add separated python callback for that
This (the to pickle part) tries come up with the common basis for the CPU events. Therefore it renames the first few arguments (like s to
common_s) to make it consistent with the ftrace events.
The dso and symbol members look optional depending whether or not this data was available at trace time. I *think* those may change within a stream say if one library has debug symbols available and the other does not. So I have no idea how you plan specific callbacks for those.Quoted 2 lines
thanks,
jirkaSebastian
_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Signature · 3 lines
Jérémie Galarneau EfficiOS Inc. http://www.efficios.com
Quoted 1 line · Jérémie Galarneau
FYI, I have made the changes and they are now upstream in Babeltrace as of
Thanks you. Let me wire that up…
Quoted 3 lines
Regards,
Jérémie
Sebastian
Quoted 23 lines · * Jiri Olsa | 2014-07-14 16:15:33 [+0200]:
I made similar effort in C:
---
I made some *VERY* early perf convert example, mostly to try the ctf-writer interface.. you can check in here:
https://git.kernel.org/cgit/linux/kernel/git/jolsa/perf.git/log/?h=perf/ctf_2It's able to convert single event (HW type) perf.data file into CTF data, by adding just one integer field "period" and single stream, like:
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib/ ./perf data convert --to-ctf=./ctf-data
...
[jolsa@krava babeltrace]$/opt/libbabeltrace/bin/babeltrace/home/jolsa/kernel.org/linux-perf/tools/perf/ctf-data[08:14:45.814456098] (+?.?????????) cycles: { }, { period = 1 }
[08:14:45.814459237] (+0.000003139) cycles: { }, { period = 1 }
[08:14:45.814460684] (+0.000001447) cycles: { }, { period = 9 }
[08:14:45.814462073] (+0.000001389) cycles: { }, { period = 182 }
[08:14:45.814463491] (+0.000001418) cycles: { }, { period = 4263 }
[08:14:45.814465874] (+0.000002383) cycles: { }, { period = 97878 }
[08:14:45.814506385] (+0.000040511) cycles: { }, { period = 1365965 }
[08:14:45.815056528] (+0.000550143) cycles: { }, { period = 2250012 }
---
Quoted 3 lines
the goals for me is to have a convert tool, like in above example perf data command and support in perf record/report to directl write/read ctf data
I have the following now:
|$ ../perf data convert -i perf.data.backup --to-ctf ctf-out-backup && babeltrace ctf-out-backup
|[11:01:45.468071953] (+?.?????????) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x0, [1] = 0x7FFF18EB71F0, [2] = 0x7FFF18EB7170, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468074246] (+0.000002293) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x2, [1] = 0x7FFF18EB7170, [2] = 0x0, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468076200] (+0.000001954) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x0, args = [ [0] = 0xE, [1] = 0x7FFF18EB3140, [2] = 0x4000, [3] = 0x0, [4] = 0x8CF9, [5] = 0x0 ] }
|[11:01:45.468097941] (+0.000021741) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x17, args = [ [0] = 0x18, [1] = 0x7F9E804FBBB0, [2] = 0x7F9E804FBB90, [3] = 0x0, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468100727] (+0.000002786) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x0, [1] = 0x7FFF18EB71F0, [2] = 0x7FFF18EB7170, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468101797] (+0.000001070) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x2, [1] = 0x7FFF18EB7170, [2] = 0x0, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468103615] (+0.000001818) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x1, args = [ [0] = 0x3, [1] = 0x7F9E805472E0, [2] = 0x30, [3] = 0x0, [4] = 0x8CF9, [5] = 0x0 ] }
|[11:01:45.468126271] (+0.000022656) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x17, args = [ [0] = 0x18, [1] = 0x7F9E804FBBB0, [2] = 0x7F9E804FBB90, [3] = 0x0, [4] = 0x0, [5] = 0x0 ] }
|[11:01:45.468140058] (+0.000013787) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 7121, comm = "perf_3.14", id = 0x10, args = [ [0] = 0x10, [1] = 0x2400, [2] = 0x0, [3] = 0x7FFFCCDED220, [4] = 0x2505110, [5] = 0x7FB588E31780 ] }
|[11:01:45.468141518] (+0.000001460) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 7121, comm = "perf_3.14", id = 0x10, args = [ [0] = 0x18, [1] = 0x2400, [2] = 0x0, [3] = 0x7FFFCCDED220, [4] = 0x2505110, [5] = 0x7FB588E31780 ] }…
|[11:01:45.468208465] (+0.000000062) irq:softirq_raise: { cpu_id = 0 }, { pid = 0, comm = "swapper", vec = 3 } |[11:01:45.468209788] (+0.000001323) irq:softirq_entry: { cpu_id = 0 }, { pid = 0, comm = "swapper", vec = 3 } …
In brief:
- added support for PERF_TYPE_TRACEPOINT, broked everything else. Fixing this on todo :)
- added support for multiple type of arguments (pid & comm is "generic", id, args, vec is based on tp_format which I don't fully understand but it seems to work (the python script writes NR instead id or adds "[action=NET_RX]" behind vec=3 and I haven't figured out what kind of magic that is)).
I pushed my current state to:
http://git.breakpoint.cc/cgit/bigeasy/linux.git/log/?h=perf_ctf_3
git://git.breakpoint.cc/bigeasy/linux.git perf_ctf_3
It is based on TIP tree from last friday and I fixed up some of your cows :)
Some of your patches lack a sign-off by line. If I am allowed to add them then I would post the complete thing for a public review. Otherwiese I am open to suggestions how we could proceed here.
Quoted 2 lines
thanks,
jirka
Sebastian
On Mon, Jul 21, 2014 at 07:11:51PM +0200, Sebastian Andrzej Siewior wrote:
SNIP
Quoted 13 lines
I have the following now:
|$ ../perf data convert -i perf.data.backup --to-ctf ctf-out-backup && babeltrace ctf-out-backup |[11:01:45.468071953] (+?.?????????) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x0, [1] = 0x7FFF18EB71F0, [2] = 0x7FFF18EB7170, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468074246] (+0.000002293) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x2, [1] = 0x7FFF18EB7170, [2] = 0x0, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468076200] (+0.000001954) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x0, args = [ [0] = 0xE, [1] = 0x7FFF18EB3140, [2] = 0x4000, [3] = 0x0, [4] = 0x8CF9, [5] = 0x0 ] } |[11:01:45.468097941] (+0.000021741) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x17, args = [ [0] = 0x18, [1] = 0x7F9E804FBBB0, [2] = 0x7F9E804FBB90, [3] = 0x0, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468100727] (+0.000002786) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x0, [1] = 0x7FFF18EB71F0, [2] = 0x7FFF18EB7170, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468101797] (+0.000001070) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0xE, args = [ [0] = 0x2, [1] = 0x7FFF18EB7170, [2] = 0x0, [3] = 0x8, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468103615] (+0.000001818) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x1, args = [ [0] = 0x3, [1] = 0x7F9E805472E0, [2] = 0x30, [3] = 0x0, [4] = 0x8CF9, [5] = 0x0 ] } |[11:01:45.468126271] (+0.000022656) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 2906, comm = "sshd", id = 0x17, args = [ [0] = 0x18, [1] = 0x7F9E804FBBB0, [2] = 0x7F9E804FBB90, [3] = 0x0, [4] = 0x0, [5] = 0x0 ] } |[11:01:45.468140058] (+0.000013787) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 7121, comm = "perf_3.14", id = 0x10, args = [ [0] = 0x10, [1] = 0x2400, [2] = 0x0, [3] = 0x7FFFCCDED220, [4] = 0x2505110, [5] = 0x7FB588E31780 ] } |[11:01:45.468141518] (+0.000001460) raw_syscalls:sys_enter: { cpu_id = 0 }, { pid = 7121, comm = "perf_3.14", id = 0x10, args = [ [0] = 0x18, [1] = 0x2400, [2] = 0x0, [3] = 0x7FFFCCDED220, [4] = 0x2505110, [5] = 0x7FB588E31780 ] }
Quoted 3 lines
…
|[11:01:45.468208465] (+0.000000062) irq:softirq_raise: {cpu_id= 0 }, { pid = 0, comm = "swapper", vec = 3 } |[11:01:45.468209788] (+0.000001323) irq:softirq_entry: {cpu_id= 0 }, { pid = 0, comm = "swapper", vec = 3 }
heya,
I've got following build error:
CC util/data-bt.outil/data-bt.c: In function ‘add_event_tracepoint_value’:util/data-bt.c:293:3: error: implicit declaration of function ‘bt_ctf_event_class_get_field_by_name’ [-Werror=implicit-function-declaration]
type = bt_ctf_event_class_get_field_by_name(
^util/data-bt.c:293:3: error: nested extern declaration of ‘bt_ctf_event_class_get_field_by_name’ [-Werror=nested-externs] util/data-bt.c:293:8: error: assignment makes pointer from integer without a cast [-Werror]
type = bt_ctf_event_class_get_field_by_name(
^
cc1: all warnings being treated as errors
make[1]: *** [util/data-bt.o] Error 1
make: *** [all] Error 2
but I might be missing some of the babeltrace changes, my branch:
5805251d8079 Fix: mmap trace read the stream_id from the first packet
Quoted 15 lines
…
In brief:
- added support forPERF_TYPE_TRACEPOINT, broked everything else. Fixing this on todo :)
- added support for multiple type of arguments (pid & comm is "generic", id, args, vec is based ontp_formatwhich I don't fully understand but it seems to work (the python script writes NR instead id or adds "[action=NET_RX]" behind vec=3 and I haven't figured out what kind of magic that is)).I pushed my current state to:
http://git.breakpoint.cc/cgit/bigeasy/linux.git/log/?h=perf_ctf_3
git://git.breakpoint.cc/bigeasy/linux.gitperf_ctf_3
Quoted 6 lines
It is based on TIP tree from last friday and I fixed up some of your cows :)
Some of your patches lack a sign-off by line. If I am allowed to add them then I would post the complete thing for a public review. Otherwiese I am open to suggestions how we could proceed here.
feel free to use/change my commits as you wish ;-)
I think the best would be to merge your changes with mine into some meaningful patchset, before it goes to review
I'll check your changes
thanks,
jirka
Quoted 13 lines · Jiri Olsa
On Mon, Jul 21, 2014 at 07:11:51PM +0200, Sebastian Andrzej Siewior wrote:
heya,
I've got following build error:CC
util/data-bt.outil/data-bt.c: In function ‘add_event_tracepoint_value’:util/data-bt.c:293:3: error: implicit declaration of function ‘bt_ctf_event_class_get_field_by_name’ [-Werror=implicit-function-declaration]
type = bt_ctf_event_class_get_field_by_name(
^util/data-bt.c:293:3: error: nested extern declaration of ‘bt_ctf_event_class_get_field_by_name’ [-Werror=nested-externs]util/data-bt.c:293:8: error: assignment makes pointer from integer without a cast [-Werror]
type = bt_ctf_event_class_get_field_by_name(
Quoted 8 lines
^cc1: all warnings being treated as errors
make[1]: *** [util/data-bt.o] Error 1
make: *** [all] Error 2but I might be missing some of the babeltrace changes, my branch:
5805251d8079Fix: mmap trace read thestream_idfrom the first packet
Hmm. This function should be there as of commit 2f100782231 in the master branch. However since my first post and now the babeltrace source got moved around a little and as a result you have some more header files.
The function in question is defined in "babeltrace/ctf-ir/event.h" and included by "babeltrace/ctf-writer/event.h".
Quoted 1 line
feel free to use/change my commits as you wish ;-)
Thank you.
Quoted 3 lines
I think the best would be to merge your changes with mine into some meaningful patchset, before it goes to review
Yeah. I merged some of the fixups back and for the data-bt.c changes, dunno. Either all my changes as a single patch or merge it back into yours as it doesn't make sense to post each incremental change.
Quoted 5 lines
I'll check your changes
thanks,
jirka
Sebastian
Quoted 23 lines · Sebastian Andrzej Siewior
Quoted 13 lines · Jiri Olsa
On Mon, Jul 21, 2014 at 07:11:51PM +0200, Sebastian Andrzej Siewior wrote:
heya,
I've got following build error:CC
util/data-bt.outil/data-bt.c: In function ‘add_event_tracepoint_value’:util/data-bt.c:293:3: error: implicit declaration of function ‘bt_ctf_event_class_get_field_by_name’ [-Werror=implicit-function-declaration]
type = bt_ctf_event_class_get_field_by_name(
^util/data-bt.c:293:3: error: nested extern declaration of ‘bt_ctf_event_class_get_field_by_name’ [-Werror=nested-externs]util/data-bt.c:293:8: error: assignment makes pointer from integer without a cast [-Werror]
type = bt_ctf_event_class_get_field_by_name(Quoted 8 lines
^cc1: all warnings being treated as errors
make[1]: *** [util/data-bt.o] Error 1
make: *** [all] Error 2but I might be missing some of the babeltrace changes, my branch:
5805251d8079Fix: mmap trace read thestream_idfrom the first packet
Quoted 6 lines
Hmm. This function should be there as of commit
2f100782231in the master branch. However since my first post and now the babeltrace source got moved around a little and as a result you have some more header files.
The function in question is defined in "babeltrace/ctf-ir/event.h" and included by "babeltrace/ctf-writer/event.h".
I've got it running... make LIBBABELTRACE_DIR=/opt/libbabeltrace/ ugh ;-)
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib /opt/libbabeltrace/bin/babeltrace ./ctf-data/
[04:41:11.445378840] (+?.?????????) sched:sched_switch: { }, { pid = 5782, comm = "ls", prev_comm = [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ], prev_pid = 5782, prev_prio = 120, prev_state = 0x1, next_comm = [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ], next_pid = 2376, next_prio = 120 } [04:41:11.445390175] (+0.000011335) sched:sched_switch: { }, { pid = 5782, comm = "ls", prev_comm = [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ], prev_pid = 5782, prev_prio = 120, prev_state = 0x1, next_comm = [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ], next_pid = 2376, next_prio = 120 }
...
looks like we need some better string arg handling, but we'll get there soon or later
Quoted 6 lines
Quoted 2 lines
feel free to use/change my commits as you wish ;-)
Thank you.
Quoted 1 line
Quoted 6 lines
Quoted 2 lines
I think the best would be to merge your changes with mine into some meaningful patchset, before it goes to review
Yeah. I merged some of the fixups back and for the
data-bt.cchanges, dunno. Either all my changes as a single patch or merge it back into yours as it doesn't make sense to post each incremental change.
we'd like to have single/compact change commits.. so far I think we can go with following commits:
perf tools: VF=2 code
perf tools: Feature check for libbabeltrace
perf tools: Add new data command
and merge your changes/fixies into:
perf tools: Add perf data to CTF convertion support
if we could break the last one into more logical changes it'd be nicer.. but if not, let's got from here
I'll prepare those 1st 3 patches.. and review your code while getting familiar with ctf again ;-)
thanks,
jirka
Quoted 11 lines · Jiri Olsa
I've got it running... make LIBBABELTRACE_DIR=/opt/libbabeltrace/ ugh ;-)
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib
/opt/libbabeltrace/bin/babeltrace./ctf-data/[04:41:11.445378840] (+?.?????????) sched:sched_switch: { }, { pid = 5782, comm = "ls",
prev_comm= [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ],prev_pid= 5782,prev_prio= 120,prev_state=0x1,next_comm= [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ],next_pid= 2376,next_prio= 120 } [04:41:11.445390175] (+0.000011335) sched:sched_switch: { }, { pid = 5782, comm = "ls",prev_comm= [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ],prev_pid= 5782,prev_prio= 120,prev_state=0x1,next_comm= [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ],next_pid= 2376,next_prio= 120 }...
looks like we need some better string arg handling, but we'll get there soon or later
I'm looking at how python gets it nicer…
Quoted 7 lines
we'd like to have single/compact change commits.. so far I think we can go with following commits:
perf tools: VF=2 code
perf tools: Feature check for libbabeltrace
perf tools: Add new data command
Quoted 8 lines
and merge your changes/fixies into:
perf tools: Add perf data to CTF convertion supportif we could break the last one into more logical changes it'd be nicer.. but if not, let's got from here
I'll prepare those 1st 3 patches.. and review your code while getting familiar with ctf again ;-)
Okay. I rebased and re-pushed perf_ctf_3. It is down to 3 + 2 patches. Can merge the last two patches into one.
Quoted 2 lines
thanks,
jirka
Sebastian
Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 9 lines
[jolsa@krava perf]$ LD_LIBRARY_PATH=/opt/libbabeltrace/lib
/opt/libbabeltrace/bin/babeltrace./ctf-data/[04:41:11.445378840] (+?.?????????) sched:sched_switch: { }, { pid = 5782, comm = "ls",
prev_comm= [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ],prev_pid= 5782,prev_prio= 120,prev_state=0x1,next_comm= [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ],next_pid= 2376,next_prio= 120 } [04:41:11.445390175] (+0.000011335) sched:sched_switch: { }, { pid = 5782, comm = "ls",prev_comm= [ [0] = "ls", [1] = "s", [2] = "", [3] = "f", [4] = "", [5] = "", [6] = "-x86_64-l", [7] = "x86_64-l", [8] = "86_64-l", [9] = "6_64-l", [10] = "_64-l", [11] = "64-l", [12] = "4-l", [13] = "-l", [14] = "l", [15] = "" ],prev_pid= 5782,prev_prio= 120,prev_state=0x1,next_comm= [ [0] = "kworker/0:0", [1] = "worker/0:0", [2] = "orker/0:0", [3] = "rker/0:0", [4] = "ker/0:0", [5] = "er/0:0", [6] = "r/0:0", [7] = "/0:0", [8] = "0:0", [9] = ":0", [10] = "0", [11] = "", [12] = "", [13] = "", [14] = "", [15] = "" ],next_pid= 2376,next_prio= 120 }...
looks like we need some better string arg handling, but we'll get there soon or later
I'm looking at how python gets it nicer…
Okay, fixed. The problem was that a string has the STRING and the ARRAY bit set. So I used the length of the array and made an array of strings. Not really what was expected :) I removed this nonsense. Now I get:
[06:37:09.867844231] (+0.000005597) sched:sched_switch: { cpu_id = 0 }, { common_pid = 14068, common_tid = 14068, common_comm = "ls", prev_comm = "ls", prev_pid = 14068, prev_prio = 120, prev_state = 2, next_comm = "swapper/5", next_pid = 0, next_prio = 120 }
So the strings look good now. I also renamed "pid" to "common_pid" because
[06:37:09.885385418] (+0.017541187) sched:sched_wakeup: { cpu_id = 0 }, { common_pid = 179, common_tid = 179, common_comm = ":179", comm = "ls", pid = 14068, prio = 120, success = 1, target_cpu = 5 }
that thing brings its own pid & comm.
And while looking at the data types and dropped that & LONG since it is not set for 64bit data types as I assumed. I do now consider ->size and the result is
[06:37:09.867838634] (+0.000253941) sched:sched_stat_runtime: { cpu_id = 0 }, { common_pid = 14068, common_tid = 14068, common_comm = "ls", comm = "ls", pid = 14068, runtime = 2020750, vruntime = 76395575003 }
that means vruntime is 64bit as it should and decimal might be nice. \o/
Sebastian
On Tue, Jul 22, 2014 at 03:31:28PM +0200, Sebastian Andrzej Siewior wrote:
SNIP
Quoted 13 lines
So the strings look good now. I also renamed "pid" to "
common_pid" because[06:37:09.885385418] (+0.017541187) sched:sched_wakeup: {
cpu_id= 0 }, {common_pid= 179,common_tid= 179,common_comm= ":179", comm = "ls", pid = 14068, prio = 120, success = 1,target_cpu= 5 }that thing brings its own pid & comm.
And while looking at the data types and dropped that & LONG since it is not set for 64bit data types as I assumed. I do now consider ->size and the result is
Quoted 8 lines
[06:37:09.867838634] (+0.000253941) sched:sched_stat_runtime: {
cpu_id= 0 }, {common_pid= 14068,common_tid= 14068,common_comm= "ls", comm = "ls", pid = 14068, runtime = 2020750, vruntime = 76395575003 }that means vruntime is 64bit as it should and decimal might be nice. \o/
Sebastian
hi,
I made some changes over your branch:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/ctf_convert
following patches:
(needs to be prettyfied, but hopefuly works and ilustrate the point)
perf tools: Iterate both common_fields and fields for tracepoit perf tools: Use sample_type for generic fields
we now add all fields available in the perf.data (via perf_event_attr:sample_type) and then for tracepoints we add both common_types and types fields
I'm not sure we want to store made up data like symbol information, which could be obtained from IP anyway - how does babeltrace or any other CTF viewer do that?
---
However I'm still not convinced 100% this is the way we want to go with this. What we do now is:
- read/parse perf.data events and convert them into ctf events
What I/we do originally wanted to do was to use CTF to describe perf event as is defined via perf interface, so we dont need to parse data stream, but just store it.
This would be handy/needed for CTF record support, where we _want_ to store raw data from kernel and not parse it before that.
I'll recheck and get back with issues I found with this approach.
Meanwhile I think we can continue with what we have now, I just wanted to share the other approach with you ;-)
thanks,
jirka
Quoted 1 line · Jiri Olsa
hi,
Hi,
Quoted 9 lines
I made some changes over your branch:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/ctf_convertfollowing patches:
(needs to be prettyfied, but hopefuly works and ilustrate the point)
perf tools: Iterate bothcommon_fieldsand fields for tracepoit perf tools: Usesample_typefor generic fields
Quoted 6 lines
we now add all fields available in the perf.data (via perf_event_attr:sample_type) and then for tracepoints we add both
common_typesand types fieldsI'm not sure we want to store made up data like symbol information, which could be obtained from IP anyway - how does babeltrace or any other CTF viewer do that?
This is probably for Mathieu best to answer. I *think* that it records pointers and does not resolve symbols if possible.
I assumed it to be a good thing to do because you don't get the symbols anymore once the library/binary is upgraded unless you look for the old files. But then the very same problem you have perf since it does not record the symbol.
So we could drop the symbol resolution but then we would have to teach babeltrace and friends to resolve it.
Quoted 5 lines
---
However I'm still not convinced 100% this is the way we want to go with this. What we do now is:
- read/parse perf.data events and convert them into ctf events
Quoted 8 lines
What I/we do originally wanted to do was to use CTF to describe perf event as is defined via perf interface, so we dont need to parse data stream, but just store it.
This would be handy/needed for CTF record support, where we _want_ to store raw data from kernel and not parse it before that.
I'll recheck and get back with issues I found with this approach.
Okay.
Quoted 1 line
Meanwhile I think we can continue with what we have now,
Good :)
Quoted 4 lines
I just wanted to share the other approach with you ;-)
thanks,
jirka
Sebastian