Skip to contentSkip to conversation

perf: introduce perf based task analyzer

PATCH In review Petar Gligoric 3 participants 12 messages Dec 6, 2022 – Dec 12, 2022 perf events

lore
Share & tools
lore
#0 lore raw

v2 cover perf: introduce perf based task analyzer

From: Petar Gligoric <[email protected]>

This patch-series introduces the task analyzer and adds the feature to output csv files for further analysis in thirds party script (e.g. pandas and friends).

The task analyzer dissects recorded perf.data files based on sched:sched_switch events. It outputs useful information for the user of each task, like times between schedule in/schedule out of the same task.

Switched-In Switched-Out CPU PID TID Comm Runtime Time Out-In

15576.658891407   15576.659156086   4     2412     2428            gdbus        265            1949
15576.659111320   15576.659455410   0     2412     2412      gnome-shell        344            2267
15576.659491326   15576.659506173   2       74       74      kworker/2:1         15           13145

[...]

The user can modify the output to his liking and his necessity. He can either limit the output to tasks he wants or filter tasks he does not present in the output. The output can also be limited via the time or specific tasks can be highlight via colors. A combination of those options is also possible.
Additionally the user can print out a summary of all tasks, which is a table of information from all tasks throughout the whole trace. Information, like total runtime, how many times the tasks have been scheduled in, what the max runtime was and when it occurred, just to name few.

Summary
Task Information Runtime Information
PID TID Comm Runs Accumulated Mean Median Min Max Max At

 14    14     ksoftirqd/0   13         334      26      15    9   127 15571.621211956
 15    15     rcu_preempt  133        1778      13      13    2    33 15572.581176024
 16    16     migration/0    3          49      16      13   12    24 15571.608915425

[...]

The standard task as well as the summary can be printed in either nanoseconds, milliseconds or microseconds(standard).
Both standard as well as summary can be saved in a user specified file in csv format.

v2:
- Add tests for new perf task-analyzer suggested by Ian Rogers

Hagen Paul Pfeifer (1):
perf script: introduce task analyzer

Petar Gligoric (2):
perf script: task-analyzer add csv support
perf test: Add new task-analyzer tests

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Namhyung Kim <[email protected]>
 .../scripts/python/bin/tasks-analyzer-record  |   2 +
 .../scripts/python/bin/tasks-analyzer-report  |   3 +
 tools/perf/scripts/python/tasks-analyzer.py   | 937 ++++++++++++++++++
 3 files changed, 942 insertions(+)
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-record
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-report
 create mode 100755 tools/perf/scripts/python/tasks-analyzer.py
Signature · 1 line
2.30.2
#1 lore raw

PATCH v2 1/3 perf script: introduce task analyzer

From: Hagen Paul Pfeifer <[email protected]>

Introduce a new perf script to analyze task scheduling behavior.

During the task analysis, some data is always needed - which goes beyond the simple time of switching on and off a task (process/thread). This concerns for example the runtime of a process or the frequency with which the process was called. This script serves to simplify this recurring analyze process. It immediately provides the user with helpful task characteristic information about the tasks runtimes.

Usage:

Recorded can be in two ways:

$ perf script record tasks-analyzer -- sleep 10
$ perf record -e sched:sched_switch -a -- sleep 10

The script can parse all perf.data files, most important: sched:sched_switch events are mandatory, other events will be ignored.

Most simple report use case is to just call the script without arguments:

$ perf script report tasks-analyzer
    Switched-In      Switched-Out CPU      PID      TID             Comm    Runtime     Time Out-In
15576.658891407   15576.659156086   4     2412     2428            gdbus        265            1949
15576.659111320   15576.659455410   0     2412     2412      gnome-shell        344            2267
15576.659491326   15576.659506173   2       74       74      kworker/2:1         15           13145
15576.659506173   15576.659825748   2     2858     2858  gnome-terminal-        320           63263
15576.659871270   15576.659902872   6    20932    20932    kworker/u16:0         32         2314582
15576.659909951   15576.659945501   3    27264    27264               sh         36              -1
15576.659853285   15576.659971052   7    27265    27265             perf        118         5050741
[...]

What is not shown here are the ASCII color sequences. For example, if the task consists of only one thread, the TID is grayed out.

Runtime is the time the task was running on the CPU, Time Out-In is the time between the process being scheduled *out* and scheduled back *in*. So the last time span between two executions. If -1 is printed, then the task simply ran the first time in the measurements - a Out-In delta could not be calculated.

In addition to the chronological representation, there is a summary on task level. This output can be additionally switched on via the --summary option and provides information such as max, min & average runtime per process. The maximum runtime is often important for debugging. The call looks like this:

$ perf script report tasks-analyzer --summary
Summary
     Task Information                       Runtime Information
  PID   TID            Comm Runs Accumulated    Mean  Median  Min   Max          Max At
   14    14     ksoftirqd/0   13         334      26      15    9   127 15571.621211956
   15    15     rcu_preempt  133        1778      13      13    2    33 15572.581176024
   16    16     migration/0    3          49      16      13   12    24 15571.608915425
   20    20     migration/1    3          34      11      13    8    13 15571.639101555
   25    25     migration/2    3          32      11      12    9    12 15575.639239896
[...]

Besides these two options, there are a number of other options that change the output and behavior. This can be queried via --help. Options worth mentioning include:

- filter-tasks         - filter out unneeded tasks, --filter-task 1337,/sbin/init
- highlight-tasks      - more pleasant focusing, --highlight-tasks 1:red,mutt:yellow
- extended-times       - show combinations of elapsed times between schedule in/schedule out
- summary-extended     - summary with additional information, like maximum delta time statistics

- rename-comms-by-tids - handy for inexpressive processnames like python, --rename 1337:my-python-app
- ms - show timestamps in milliseconds, nanoseconds is also possible (--ns)
- time-limit - limit the analyzer to a time range, --time-limit 15576.0:15576.1

Script is tested and prime time ready for python2 & python3:
- make PYTHON=python3 prefix=/usr/local install
- make PYTHON=python2 prefix=/usr/local install

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 .../scripts/python/bin/task-analyzer-record   |   2 +
 .../scripts/python/bin/task-analyzer-report   |   3 +
 tools/perf/scripts/python/task-analyzer.py    | 838 ++++++++++++++++++
 3 files changed, 843 insertions(+)
 create mode 100755 tools/perf/scripts/python/bin/task-analyzer-record
 create mode 100755 tools/perf/scripts/python/bin/task-analyzer-report
 create mode 100755 tools/perf/scripts/python/task-analyzer.py
tools/perf/scripts/python/bin/task-analyzer-record new file +2 -0
new file mode 100755
index 000000000000..0f6b51bb2767
@@ -0,0 +1,2 @@
+#!/bin/bash
+perf record -e sched:sched_switch -e sched:sched_migrate_task "$@"
tools/perf/scripts/python/bin/task-analyzer-report new file +3 -0
new file mode 100755
index 000000000000..4b16a8cc40a0
@@ -0,0 +1,3 @@
+#!/bin/bash
+# description: analyze timings of tasks
+perf script -s "$PERF_EXEC_PATH"/scripts/python/task-analyzer.py -- "$@"
tools/perf/scripts/python/task-analyzer.py new file +838 -0

838 lines not on this page · raw message

Signature · 1 line
2.30.2
#2 lore raw

PATCH v2 2/3 perf script: task-analyzer add csv support

In reply to Petar Gligoric

From: Petar Gligoric <[email protected]>

This patch adds the possibility to write the trace and the summary as csv files to a user specified file. A format as such simplifies further data processing. This is achieved by having ";" as separators instead of spaces and solely one header per file.
Additional parameters are being considered, like in the normal usage of the script. Colors are turned off in the case of a csv output, thus the highlight option is also being ignored.

Usage:

Write standard task to csv file:

$ perf script report tasks-analyzer --csv <file>
write limited output to csv file in nanoseconds:
$ perf script report tasks-analyzer --csv <file> --ns --limit-to-tasks 1337

Write summary to a csv file:

$ perf script report tasks-analyzer --csv-summary <file>
Write summary to csv file with additional schedule information:
$ perf script report tasks-analyzer --csv-summary <file> --summary-extended

Write both summary and standard task to a csv file:

$ perf script report tasks-analyzer --csv --csv-summary

The following examples illustrate what is possible with the CSV output. The first command sequence will record all scheduler switch events for 10 seconds, the task-analyzer calculates task information like runtimes as CSV. A small python snippet using pandas and matplotlib will visualize the most frequent task (e.g. kworker/1:1) runtimes - each runtime as a bar in a bar chart:

$ perf record -e sched:sched_switch -a -- sleep 10
$ perf script report tasks-analyzer --ns --csv tasks.csv
$ cat << EOF > /tmp/freq-comm-runtimes-bar.py
  import pandas as pd
  import matplotlib.pyplot as plt

df = pd.read_csv("tasks.csv", sep=';')
most_freq_comm = df["COMM"].value_counts().idxmax() most_freq_runtimes = df[df["COMM"]==most_freq_comm]["Runtime"] plt.title(f"Runtimes for Task {most_freq_comm} in Nanoseconds") plt.bar(range(len(most_freq_runtimes)), most_freq_runtimes) plt.show()

$ python3 /tmp/freq-comm-runtimes-bar.py

As a seconds example, the subsequent script generates a pie chart of all accumulated tasks runtimes for 10 seconds of system recordings:

$ perf record -e sched:sched_switch -a -- sleep 10
$ perf script report tasks-analyzer --csv-summary task-summary.csv
$ cat << EOF > /tmp/accumulated-task-pie.py
  import pandas as pd
  from matplotlib.pyplot import pie, axis, show

df = pd.read_csv("task-summary.csv", sep=';') sums = df.groupby(df["Comm"])["Accumulated"].sum() axis("equal")
pie(sums, labels=sums.index);
show()
EOF

$ python3 /tmp/accumulated-task-pie.py

A variety of other visualizations are possible in matplotlib and other environments. Of course, pandas, numpy and co. also allow easy statistical analysis of the data!

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 tools/perf/scripts/python/task-analyzer.py | 274 ++++++++++++++-------
 1 file changed, 185 insertions(+), 89 deletions(-)
tools/perf/scripts/python/task-analyzer.py +185 -89
index f74abe50f3b2..52e8dae9b1f0 100755
@@ -156,6 +156,18 @@ def _parse_args():
         help="always, never or auto, allowing configuring color output"
             " via the command line",
     )
+    parser.add_argument(
+        "--csv",
+        default="",
+        help="Write trace to file selected by user. Options, like --ns or --extended"
+            "-times are used.",
+    )
+    parser.add_argument(
+        "--csv-summary",
+        default="",
+        help="Write summary to file selected by user. Options, like --ns or"
+            " --summary-extended are used.",
+    )
     args = parser.parse_args()
     args.tid_renames = dict()
 
@@ -275,7 +287,6 @@ class Timespans(object):
 
 
 
-
 class Summary(object):
     """
     Primary instance for calculating the summary output. Processes the whole trace to
@@ -327,7 +338,7 @@ class Summary(object):
                 sum(db["inter_times"].values()) - 4 * decimal_precision
                 )
             _header += ("Max Inter Task Times",)
-        print(fmt.format(*_header))
+        fd_sum.write(fmt.format(*_header) +  "\n")
 
     def _column_titles(self):
         """
@@ -336,34 +347,58 @@ class Summary(object):
         values are being displayed in grey. Thus in their format two additional {},
         are placed for color set and reset.
         """
+        separator, fix_csv_align = _prepare_fmt_sep()
         decimal_precision, time_precision = _prepare_fmt_precision()
-        fmt = " {{:>{}}}".format(db["task_info"]["pid"])
-        fmt += " {{:>{}}}".format(db["task_info"]["tid"])
-        fmt += " {{:>{}}}".format(db["task_info"]["comm"])
-        fmt += " {{:>{}}}".format(db["runtime_info"]["runs"])
-        fmt += " {{:>{}}}".format(db["runtime_info"]["acc"])
-        fmt += " {{:>{}}}".format(db["runtime_info"]["mean"])
-        fmt += " {{:>{}}}".format(db["runtime_info"]["median"])
-        fmt += " {{:>{}}}".format(db["runtime_info"]["min"] - decimal_precision)
-        fmt += " {{:>{}}}".format(db["runtime_info"]["max"] - decimal_precision)
-        fmt += " {{}}{{:>{}}}{{}}".format(db["runtime_info"]["max_at"] - time_precision)
+        fmt = "{{:>{}}}".format(db["task_info"]["pid"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, db["task_info"]["tid"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, db["task_info"]["comm"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["runs"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["acc"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["mean"] * fix_csv_align)
+        fmt += "{}{{:>{}}}".format(
+            separator, db["runtime_info"]["median"] * fix_csv_align
+        )
+        fmt += "{}{{:>{}}}".format(
+            separator, (db["runtime_info"]["min"] - decimal_precision) * fix_csv_align
+        )
+        fmt += "{}{{:>{}}}".format(
+            separator, (db["runtime_info"]["max"] - decimal_precision) * fix_csv_align
+        )
+        fmt += "{}{{}}{{:>{}}}{{}}".format(
+            separator, (db["runtime_info"]["max_at"] - time_precision) * fix_csv_align
+        )
 
         column_titles = ("PID", "TID", "Comm")
         column_titles += ("Runs", "Accumulated", "Mean", "Median", "Min", "Max")
-        column_titles += (_COLORS["grey"], "At", _COLORS["reset"])
+        column_titles += (_COLORS["grey"], "Max At", _COLORS["reset"])
 
         if args.summary_extended:
-            fmt += " {{:>{}}}".format(db["inter_times"]["out_in"] - decimal_precision)
-            fmt += " {{}}{{:>{}}}{{}}".format(
-                db["inter_times"]["inter_at"] - time_precision
+            fmt += "{}{{:>{}}}".format(
+                separator,
+                (db["inter_times"]["out_in"] - decimal_precision) * fix_csv_align
+            )
+            fmt += "{}{{}}{{:>{}}}{{}}".format(
+                separator,
+                (db["inter_times"]["inter_at"] - time_precision) * fix_csv_align
+            )
+            fmt += "{}{{:>{}}}".format(
+                separator,
+                (db["inter_times"]["out_out"] - decimal_precision) * fix_csv_align
+            )
+            fmt += "{}{{:>{}}}".format(
+                separator,
+                (db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align
+            )
+            fmt += "{}{{:>{}}}".format(
+                separator,
+                (db["inter_times"]["in_out"] - decimal_precision) * fix_csv_align
             )
-            fmt += " {{:>{}}}".format(db["inter_times"]["out_out"] - decimal_precision)
-            fmt += " {{:>{}}}".format(db["inter_times"]["in_in"] - decimal_precision)
-            fmt += " {{:>{}}}".format(db["inter_times"]["in_out"] - decimal_precision)
 
             column_titles += ("Out-In", _COLORS["grey"], "Max At", _COLORS["reset"],
                         "Out-Out", "In-In", "In-Out")
-        print(fmt.format(*column_titles))
+
+        fd_sum.write(fmt.format(*column_titles) + "\n")
+
 
     def _task_stats(self):
         """calculates the stats of every task and constructs the printable summary"""
@@ -414,39 +449,53 @@ class Summary(object):
             self._calc_alignments_summary(align_helper)
 
     def _format_stats(self):
+        separator, fix_csv_align = _prepare_fmt_sep()
         decimal_precision, time_precision = _prepare_fmt_precision()
-        fmt = " {{:>{}d}}".format(db["task_info"]["pid"])
-        fmt += " {{:>{}d}}".format(db["task_info"]["tid"])
-        fmt += " {{:>{}}}".format(db["task_info"]["comm"])
-        fmt += " {{:>{}d}}".format(db["runtime_info"]["runs"])
-        fmt += " {{:>{}.{}f}}".format(db["runtime_info"]["acc"], time_precision)
-        fmt += " {{}}{{:>{}.{}f}}".format(db["runtime_info"]["mean"], time_precision)
-        fmt += " {{:>{}.{}f}}".format(db["runtime_info"]["median"], time_precision)
-        fmt += " {{:>{}.{}f}}".format(
-            db["runtime_info"]["min"] - decimal_precision, time_precision
-            )
-        fmt += " {{:>{}.{}f}}".format(
-            db["runtime_info"]["max"] - decimal_precision, time_precision
-            )
-        fmt += " {{}}{{:>{}.{}f}}{{}}{{}}".format(
-            db["runtime_info"]["max_at"] - time_precision, decimal_precision
+        len_pid = db["task_info"]["pid"] * fix_csv_align
+        len_tid = db["task_info"]["tid"] * fix_csv_align
+        len_comm = db["task_info"]["comm"] * fix_csv_align
+        len_runs = db["runtime_info"]["runs"] * fix_csv_align
+        len_acc = db["runtime_info"]["acc"] * fix_csv_align
+        len_mean = db["runtime_info"]["mean"] * fix_csv_align
+        len_median = db["runtime_info"]["median"] * fix_csv_align
+        len_min = (db["runtime_info"]["min"] - decimal_precision) * fix_csv_align
+        len_max = (db["runtime_info"]["max"] - decimal_precision) * fix_csv_align
+        len_max_at = (db["runtime_info"]["max_at"] - time_precision) * fix_csv_align
+        if args.summary_extended:
+            len_out_in = (
+                db["inter_times"]["out_in"] - decimal_precision
+            ) * fix_csv_align
+            len_inter_at = (
+                db["inter_times"]["inter_at"] - time_precision
+            ) * fix_csv_align
+            len_out_out = (
+                db["inter_times"]["out_out"] - decimal_precision
+            ) * fix_csv_align
+            len_in_in = (db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align
+            len_in_out = (
+                db["inter_times"]["in_out"] - decimal_precision
+            ) * fix_csv_align
+
+        fmt = "{{:{}d}}".format(len_pid)
+        fmt += "{}{{:{}d}}".format(separator, len_tid)
+        fmt += "{}{{:>{}}}".format(separator, len_comm)
+        fmt += "{}{{:{}d}}".format(separator, len_runs)
+        fmt += "{}{{:{}.{}f}}".format(separator, len_acc, time_precision)
+        fmt += "{}{{}}{{:{}.{}f}}".format(separator, len_mean, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, len_median, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, len_min, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, len_max, time_precision)
+        fmt += "{}{{}}{{:{}.{}f}}{{}}{{}}".format(
+            separator, len_max_at, decimal_precision
         )
         if args.summary_extended:
-            fmt += " {{:>{}.{}f}}".format(
-                db["inter_times"]["out_in"] - decimal_precision, time_precision
-            )
-            fmt += " {{}}{{:>{}.{}f}}{{}}".format(
-                db["inter_times"]["inter_at"] - time_precision, decimal_precision
-            )
-            fmt += " {{:>{}.{}f}}".format(
-                db["inter_times"]["out_out"] - decimal_precision, time_precision
-            )
-            fmt += " {{:>{}.{}f}}".format(
-                db["inter_times"]["in_in"] - decimal_precision, time_precision
-            )
-            fmt += " {{:>{}.{}f}}".format(
-                db["inter_times"]["in_out"] - decimal_precision, time_precision
+            fmt += "{}{{:{}.{}f}}".format(separator, len_out_in, time_precision)
+            fmt += "{}{{}}{{:{}.{}f}}{{}}".format(
+                separator, len_inter_at, decimal_precision
             )
+            fmt += "{}{{:{}.{}f}}".format(separator, len_out_out, time_precision)
+            fmt += "{}{{:{}.{}f}}".format(separator, len_in_in, time_precision)
+            fmt += "{}{{:{}.{}f}}".format(separator, len_in_out, time_precision)
         return fmt
 
 
@@ -467,13 +516,15 @@ class Summary(object):
 
 
     def print(self):
-        print("\nSummary")
         self._task_stats()
-        self._print_header()
-        self._column_titles()
         fmt = self._format_stats()
+
+        if not args.csv_summary:
+            print("\nSummary")
+            self._print_header()
+        self._column_titles()
         for i in range(len(self._body)):
-            print(fmt.format(*tuple(self._body[i])))
+            fd_sum.write(fmt.format(*tuple(self._body[i])) + "\n")
 
 
 
@@ -531,37 +582,45 @@ def _filter_non_printable(unfiltered):
 
 
 def _fmt_header():
-    fmt = "{{:>{}}}".format(LEN_SWITCHED_IN)
-    fmt += " {{:>{}}}".format(LEN_SWITCHED_OUT)
-    fmt += " {{:>{}}}".format(LEN_CPU)
-    fmt += " {{:>{}}}".format(LEN_PID)
-    fmt += "  {{:>{}}}".format(LEN_TID)
-    fmt += "  {{:>{}}}".format(LEN_COMM)
-    fmt += " {{:>{}}}".format(LEN_RUNTIME)
-    fmt += " {{:>{}}}".format(LEN_OUT_IN)
+    separator, fix_csv_align = _prepare_fmt_sep()
+    fmt = "{{:>{}}}".format(LEN_SWITCHED_IN*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_SWITCHED_OUT*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_CPU*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_PID*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_TID*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_COMM*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_RUNTIME*fix_csv_align)
+    fmt += "{}{{:>{}}}".format(separator, LEN_OUT_IN*fix_csv_align)
     if args.extended_times:
-        fmt += " {{:>{}}}".format(LEN_OUT_OUT)
-        fmt += " {{:>{}}}".format(LEN_IN_IN)
-        fmt += " {{:>{}}}".format(LEN_IN_OUT)
+        fmt += "{}{{:>{}}}".format(separator, LEN_OUT_OUT*fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, LEN_IN_IN*fix_csv_align)
+        fmt += "{}{{:>{}}}".format(separator, LEN_IN_OUT*fix_csv_align)
     return fmt
 
 
 def _fmt_body():
+    separator, fix_csv_align = _prepare_fmt_sep()
     decimal_precision, time_precision = _prepare_fmt_precision()
-    fmt = "{{}}{{:{}.{}f}}".format(LEN_SWITCHED_IN, decimal_precision)
-    fmt += " {{:{}.{}f}}".format(LEN_SWITCHED_OUT, decimal_precision)
-    fmt += " {{:{}d}}".format(LEN_CPU)
-    fmt += " {{:{}d}}".format(LEN_PID)
-    fmt += "  {{}}{{:{}d}}{{}}".format(LEN_TID)
-    fmt += "  {{}}{{:>{}}}".format(LEN_COMM)
-    fmt += " {{:{}.{}f}}".format(LEN_RUNTIME, time_precision)
+    fmt = "{{}}{{:{}.{}f}}".format(LEN_SWITCHED_IN*fix_csv_align, decimal_precision)
+    fmt += "{}{{:{}.{}f}}".format(
+        separator, LEN_SWITCHED_OUT*fix_csv_align, decimal_precision
+    )
+    fmt += "{}{{:{}d}}".format(separator, LEN_CPU*fix_csv_align)
+    fmt += "{}{{:{}d}}".format(separator, LEN_PID*fix_csv_align)
+    fmt += "{}{{}}{{:{}d}}{{}}".format(separator, LEN_TID*fix_csv_align)
+    fmt += "{}{{}}{{:>{}}}".format(separator, LEN_COMM*fix_csv_align)
+    fmt += "{}{{:{}.{}f}}".format(separator, LEN_RUNTIME*fix_csv_align, time_precision)
     if args.extended_times:
-        fmt += " {{:{}.{}f}}".format(LEN_OUT_IN, time_precision)
-        fmt += " {{:{}.{}f}}".format(LEN_OUT_OUT, time_precision)
-        fmt += " {{:{}.{}f}}".format(LEN_IN_IN, time_precision)
-        fmt += " {{:{}.{}f}}{{}}".format(LEN_IN_OUT, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, LEN_OUT_IN*fix_csv_align, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, LEN_OUT_OUT*fix_csv_align, time_precision)
+        fmt += "{}{{:{}.{}f}}".format(separator, LEN_IN_IN*fix_csv_align, time_precision)
+        fmt += "{}{{:{}.{}f}}{{}}".format(
+            separator, LEN_IN_OUT*fix_csv_align, time_precision
+        )
     else:
-        fmt += " {{:{}.{}f}}{{}}".format(LEN_OUT_IN, time_precision)
+        fmt += "{}{{:{}.{}f}}{{}}".format(
+            separator, LEN_OUT_IN*fix_csv_align, time_precision
+        )
     return fmt
 
 
@@ -571,7 +630,8 @@ def _print_header():
             "Time Out-In")
     if args.extended_times:
         header += ("Time Out-Out", "Time In-In", "Time In-Out")
-    print(fmt.format(*header))
+    fd_task.write(fmt.format(*header) + "\n")
+
 
 
 def _print_task_finish(task):
@@ -583,7 +643,6 @@ def _print_task_finish(task):
     in_in = -1
     in_out = -1
     fmt = _fmt_body()
-
     # depending on user provided highlight option we change the color
     # for particular tasks
     if str(task.tid) in args.highlight_tasks_map:
@@ -612,16 +671,22 @@ def _print_task_finish(task):
         out_out = timespan_gap_tid.out_out
         in_in = timespan_gap_tid.in_in
         in_out = timespan_gap_tid.in_out
+
+
     if args.extended_times:
-        print(fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu, task.pid,
-                        c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
+        line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu,
+                        task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
                         task.runtime(time_unit), out_in, out_out, in_in, in_out,
-                        c_row_reset))
+                        c_row_reset) + "\n"
     else:
-        print(fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu, task.pid,
-                        c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
-                        task.runtime(time_unit), out_in, c_row_reset))
-
+        line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu,
+                        task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
+                        task.runtime(time_unit), out_in, c_row_reset) + "\n"
+    try:
+        fd_task.write(line_out)
+    except(IOError):
+        # don't mangle the output if user SIGINT this script
+        sys.exit()
 
 def _record_cleanup(_list):
     """
@@ -733,10 +798,19 @@ def _argument_filter_sanity_check():
         )
     if args.time_limit and (args.summary or args.summary_only or args.summary_extended):
         sys.exit("Error: Cannot set time limit and print summary")
-
+    if args.csv_summary:
+        args.summary = True
+        if args.csv == args.csv_summary:
+            sys.exit("Error: Chosen files for csv and csv summary are the same")
+    if args.csv and (args.summary_extended or args.summary) and not args.csv_summary:
+        sys.exit("Error: No file chosen to write summary to. Choose with --csv-summary "
+        "<file>")
+    if args.csv and args.summary_only:
+        sys.exit("Error: --csv chosen and --summary-only. Standard task would not be"
+            "written to csv file.")
 
 def _argument_prepare_check():
-    global time_unit
+    global time_unit, fd_task, fd_sum
     if args.filter_tasks:
         args.filter_tasks = args.filter_tasks.split(",")
     if args.limit_to_tasks:
@@ -769,6 +843,21 @@ def _argument_prepare_check():
         time_unit = "ms"
 
 
+    fd_task = sys.stdout
+    if args.csv:
+        args.stdio_color = "never"
+        fd_task = open(args.csv, "w")
+        print("generating csv at",args.csv,)
+
+    fd_sum = sys.stdout
+    if args.csv_summary:
+        args.stdio_color = "never"
+        fd_sum = open(args.csv_summary, "w")
+        print("generating csv summary at",args.csv_summary)
+        if not args.csv:
+            args.summary_only = True
+
+
 def _is_within_timelimit(time):
     """
     Check if a time limit was given by parameter, if so ignore the rest. If not,
@@ -801,10 +890,17 @@ def _prepare_fmt_precision():
     decimal_precision = 6
     time_precision = 3
     if args.ns:
-       decimal_precision = 9
-       time_precision = 0
+     decimal_precision = 9
+     time_precision = 0
     return decimal_precision, time_precision
 
+def _prepare_fmt_sep():
+    separator = " "
+    fix_csv_align = 1
+    if args.csv or args.csv_summary:
+        separator = ";"
+        fix_csv_align = 0
+    return separator, fix_csv_align
 
 def trace_unhandled(event_name, context, event_fields_dict, perf_sample_dict):
     pass
Signature · 1 line
2.30.2
#3 lore raw

PATCH v2 3/3 perf test: add new task-analyzer tests

In reply to Petar Gligoric

From: Petar Gligoric <[email protected]>

Provide task-analyzer test cases for all possible arguments and a subset of possible combinations.

12 Tests in total.

test_basic:
- cmd:"perf script report task-analyzer"
- Fundamental test of script without arguments.
- Check for standard output.

test_ns_rename:
- cmd:"perf script report task-analyzer --ns --rename-comms-by-tids 0:random"
- Standard task with timestamps in nanoseconds and comm renamed.
- Check for standard output.

test_ms_filtertasks_highlight:
- cmd:"perf script report task-analyzer --ms --filter-tasks perf --highlight-tasks perf"
- Standard task with timestamps in milliseconds, task filtered out and highlighted.
- Check for standard output.

test_extended_times_timelimit_limittasks:
- cmd "perf script report task-analyzer --extended-times --time-limit :99999"
- Standard task with additional schedule out/in info and timlimit active at 99999.
- Check for extended table output.

test_summary:
- cmd:"perf script report task-analyzer --summary"
- Standard task with additional summary output.
- Check for summary print.

test_summary_extended:
- cmd:"perf script report task-analyzer --summary-extended"
- Standard task with summary and additional schedule in/out info.
- Chceck for extended table print.

test_summaryonly:
- cmd:"perf script report task-analyzer --summary-only"
- Only summary should be printed.
- Check for summary print.

test_extended_times_summary_ns:
- cmd:"perf script report task-analyzer --extended-times --summary --ns"
- Standard task with extended schedule in/out information and summary in ns.
- Check for extended table and summary.

test_csv:
- cmd:"perf script report task-analyzer --csv csv"
- Print standard task to csv file in csv format.
- Check for csv format.

test_csv_extended_times:
- cmd:"perf script report task-analyzer --csv csv --extended-times"
- Print standard task to csv file in csv format with additional schedule in/out information.
- Check for additional information and csv format.

test_csvsummary:
- cmd:"perf script report task-analyzer --csv-summary csvsummary"
- Print summary to csvsummary file in csv format.
- Check for csv format.

test_csvsummary_extended:
- cmd:"perf script report task-analyzer --csv-summary csvsummary --summary-extended"
- Print summary to csvsummary file in csv format with additional schedule in/out information.
- Check for additional information and csv format.

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 tools/perf/tests/shell/test_task_analyzer.sh | 151 +++++++++++++++++++
 1 file changed, 151 insertions(+)
 create mode 100755 tools/perf/tests/shell/test_task_analyzer.sh
tools/perf/tests/shell/test_task_analyzer.sh new file +151 -0
new file mode 100755
index 000000000000..a98e4ab66040
@@ -0,0 +1,151 @@
+#!/bin/bash
+# perf script task-analyzer tests
+# SPDX-License-Identifier: GPL-2.0
+
+tmpdir=$(mktemp -d /tmp/perf-script-task-analyzer-XXXXX)
+err=0
+
+cleanup() {
+  rm -f perf.data
+  rm -f perf.data.old
+  rm -f csv
+  rm -f csvsummary
+  rm -rf $tmpdir
+  trap - exit term int
+}
+
+trap_cleanup() {
+  cleanup
+  exit 1
+}
+trap trap_cleanup exit term int
+
+report() {
+	if [ $1 = 0 ]; then
+		echo "PASS: \"$2\""
+	else
+		echo "FAIL: \"$2\" Error message: \"$3\""
+		err=1
+	fi
+}
+
+check_exec_0() {
+	if [ $? != 0 ]; then
+		report 1 "invokation of ${$1} command failed"
+	fi
+}
+
+find_str_or_fail() {
+	grep -q "$1" $2
+	if [ $? != 0 ]; then
+		report 1 $3 "Failed to find required string:'${1}'."
+	else
+		report 0 $3
+	fi
+}
+
+prepare_perf_data() {
+	# 1s should be sufficient to catch at least some switches
+	perf record -e sched:sched_switch -a -- sleep 1 > /dev/null 2>&1
+}
+
+# check standard inkvokation with no arguments
+test_basic() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_ns_rename(){
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --ns --rename-comms-by-tids 0:random > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_ms_filtertasks_highlight(){
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --ms --filter-tasks perf --highlight-tasks perf \
+	> $out
+	check_exec_0 "perf"
+	find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_extended_times_timelimit_limittasks() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --extended-times --time-limit :99999 \
+	--limit-to-tasks perf > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Out-Out" $out ${FUNCNAME[0]}
+}
+
+test_summary() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --summary > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_summaryextended() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --summary-extended > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Inter Task Times" $out ${FUNCNAME[0]}
+}
+
+test_summaryonly() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --summary-only > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_extended_times_summary_ns() {
+	out="$tmpdir/perf.out"
+	perf script report task-analyzer --extended-times --summary --ns > $out
+	check_exec_0 "perf"
+	find_str_or_fail "Out-Out" $out ${FUNCNAME[0]}
+	find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_csv() {
+	perf script report task-analyzer --csv csv > /dev/null
+	check_exec_0 "perf"
+	find_str_or_fail "Comm;" csv ${FUNCNAME[0]}
+}
+
+test_csv_extended_times() {
+	perf script report task-analyzer --csv csv --extended-times > /dev/null
+	check_exec_0 "perf"
+	find_str_or_fail "Out-Out;" csv ${FUNCNAME[0]}
+}
+
+test_csvsummary() {
+	perf script report task-analyzer --csv-summary csvsummary > /dev/null
+	check_exec_0 "perf"
+	find_str_or_fail "Comm;" csvsummary ${FUNCNAME[0]}
+}
+
+test_csvsummary_extended() {
+	perf script report task-analyzer --csv-summary csvsummary --summary-extended \
+	>/dev/null
+	check_exec_0 "perf"
+	find_str_or_fail "Out-Out;" csvsummary ${FUNCNAME[0]}
+}
+
+prepare_perf_data
+test_basic
+test_ns_rename
+test_ms_filtertasks_highlight
+test_extended_times_timelimit_limittasks
+test_summary
+test_summaryextended
+test_summaryonly
+test_extended_times_summary_ns
+test_csv
+test_csvsummary
+test_csv_extended_times
+test_csvsummary_extended
+cleanup
+exit $err
Signature · 1 line
2.30.2
maintainer
#4 lore raw
In reply to Petar Gligoric

Hello,

Quoted 35 lines · Petar Gligoric

From: Petar Gligoric <[email protected]>

This patch-series introduces the task analyzer and adds the feature to output csv files for further analysis in thirds party script (e.g. pandas and friends).

The task analyzer dissects recorded perf.data files based on sched:sched_switch events. It outputs useful information for the user of each task, like times between schedule in/schedule out of the same task.

Switched-In Switched-Out CPU PID TID Comm Runtime Time Out-In

15576.658891407   15576.659156086   4     2412     2428            gdbus        265            1949
15576.659111320   15576.659455410   0     2412     2412      gnome-shell        344            2267
15576.659491326   15576.659506173   2       74       74      kworker/2:1         15           13145

[...]

The user can modify the output to his liking and his necessity. He can either limit the output to tasks he wants or filter tasks he does not present in the output. The output can also be limited via the time or specific tasks can be highlight via colors. A combination of those options is also possible.
Additionally the user can print out a summary of all tasks, which is a table of information from all tasks throughout the whole trace. Information, like total runtime, how many times the tasks have been scheduled in, what the max runtime was and when it occurred, just to name few.

Summary
Task Information Runtime Information
PID TID Comm Runs Accumulated Mean Median Min Max Max At

 14    14     ksoftirqd/0   13         334      26      15    9   127 15571.621211956
 15    15     rcu_preempt  133        1778      13      13    2    33 15572.581176024
 16    16     migration/0    3          49      16      13   12    24 15571.608915425
Quoted 6 lines

[...]

The standard task as well as the summary can be printed in either nanoseconds, milliseconds or microseconds(standard).
Both standard as well as summary can be saved in a user specified file in csv format.

Have you looked at 'perf sched timehist' ?
I think it has the common functionality and can be easily extended if there's missing one.

Thanks,
Namhyung

Quoted 27 lines

v2:
- Add tests for new perf task-analyzer suggested by Ian Rogers

Hagen Paul Pfeifer (1):
perf script: introduce task analyzer

Petar Gligoric (2):
perf script: task-analyzer add csv support
perf test: Add new task-analyzer tests

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ian Rogers <[email protected]>
Cc Namhyung Kim <[email protected]>
 .../scripts/python/bin/tasks-analyzer-record  |   2 +
 .../scripts/python/bin/tasks-analyzer-report  |   3 +
 tools/perf/scripts/python/tasks-analyzer.py   | 937 ++++++++++++++++++
 3 files changed, 942 insertions(+)
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-record
 create mode 100755 tools/perf/scripts/python/bin/tasks-analyzer-report
 create mode 100755 tools/perf/scripts/python/tasks-analyzer.py

--
2.30.2

#5 lore raw
Quoted 9 lines · Namhyung Kim

Hello,

Have you looked at 'perf sched timehist' ?
I think it has the common functionality and can be easily extended if there's missing one.

Thanks,
Namhyung

Thanks for the input! For this patchset we explicitly decided against extending "perf sched timehist" - after some pros and cons. Mainly we didn't want to break existing programs (which might parse the output of perf sched) and also the goal of the task-analyzer is a bit different. E.g what will follow as a follow-up patch, is to show IRQs visually pleasing intermixed with tasks to show potential sources of task latency. This will be offered as an option for the task-analyzer, but would be too much functionality for "perf sched timehist". This was the main reason why we decided against the extension.

Best Regards,
Petar

maintainer
#6 lore raw
Quoted 12 lines · Petar Gligoric
Quoted 9 lines · Namhyung Kim

Hello,

Have you looked at 'perf sched timehist' ?
I think it has the common functionality and can be easily extended if there's missing one.

Thanks,
Namhyung

Quoted 9 lines

Thanks for the input! For this patchset we explicitly decided against extending "perf sched timehist" - after some pros and cons. Mainly we didn't want to break existing programs (which might parse the output of perf sched) and also the goal of the task-analyzer is a bit different. E.g what will follow as a follow-up patch, is to show IRQs visually pleasing intermixed with tasks to show potential sources of task latency. This will be offered as an option for the task-analyzer, but would be too much functionality for "perf sched timehist". This was the main reason why we decided against the extension.

Then you might want to add a new sub-command under perf sched. But I guess we can just add a new option for the different output format in the perf sched timehist.

Anyway, "perf script" is a generic tool not targeting specific events. This functionality requires sched_switch (and more?) then we need the record part to make sure the data has the events. That's why it's natural to have it in perf sched IMHO.

Thanks,
Namhyung

#7 lore raw
Quoted 14 lines
Quoted 9 lines

Thanks for the input! For this patchset we explicitly decided against extending "perf sched timehist" - after some pros and cons. Mainly we didn't want to break existing programs (which might parse the output of perf sched) and also the goal of the task-analyzer is a bit different. E.g what will follow as a follow-up patch, is to show IRQs visually pleasing intermixed with tasks to show potential sources of task latency. This will be offered as an option for the task-analyzer, but would be too much functionality for "perf sched timehist". This was the main reason why we decided against the extension.

Then you might want to add a new sub-command under perf sched. But I guess we can just add a new option for the different output format in the perf sched timehist.

Quoted 7 lines

Anyway, "perf script" is a generic tool not targeting specific events. This functionality requires sched_switch (and more?) then we need the record part to make sure the data has the events. That's why it's natural to have it in perf sched IMHO.

Thanks,
Namhyung

We assumed that python scripts should not only be used as a "generic tool". There are a number of tools like flamegraph, netdev-times, dropmonitor and other scripts that analyze and look at *very specific* events. The question is rather, why should this be implemented in C? When using Python - with batteries included - less code has to be written for the identical result, and it is less error-prone than C (less technical debt). We have measured the performance, even for very large perf.data files, and again we see no problems.

But maybe this should be clarified in principle! What do you say Arnaldo, Ian, ...?

Petar

maintainer
#8 lore raw
Quoted 20 lines · Petar Gligoric
Quoted 20 lines
Quoted 10 lines

Thanks for the input! For this patchset we explicitly decided against extending "perf sched timehist" - after some pros and cons. Mainly we didn't want to break existing programs (which might parse the output of perf sched) and also the goal of the task-analyzer is a bit different. E.g what will follow as a follow-up patch, is to show IRQs visually pleasing intermixed with tasks to show potential sources of task latency. This will be offered as an option for the task-analyzer, but would be too much functionality for "perf sched timehist". This was the main reason why we decided against the extension.

Then you might want to add a new sub-command under perf sched. But I guess we can just add a new option for the different output format in the perf sched timehist.

Anyway, "perf script" is a generic tool not targeting specific events. This functionality requires sched_switch (and more?) then we need the record part to make sure the data has the events. That's why it's natural to have it in perf sched IMHO.

Quoted 4 lines
Quoted 2 lines

Thanks,
Namhyung

We assumed that python scripts should not only be used as a "generic tool".

Oh, I didn't know you worked on a separate python script. Sorry, I just read the cover letter (but not the diffstat) and I thought you added it to the builtin-script.c. ;-)

Quoted 6 lines

There are a number of tools like flamegraph, netdev-times, dropmonitor and other scripts that analyze and look at *very specific* events. The question is rather, why should this be implemented in C? When using Python - with batteries included - less code has to be written for the identical result, and it is less error-prone than C (less technical debt). We have measured the performance, even for very large perf.data files, and again we see no problems.

Then I have no problem with python. Please take a look at the scripts/python/bin directory to provide shortcuts for record/report.

Thanks,
Namhyung

Quoted 6 lines

But maybe this should be clarified in principle! What do you say Arnaldo, Ian, ...?

Petar

#9 lore raw
Quoted 9 lines · Namhyung Kim

Then I have no problem with python. Please take a look at the scripts/python/bin directory to provide shortcuts for record/report.

Thanks,
Namhyung

Great that we could resolve it! Regarding wrapper: I thought I added them or did you mean something else?

.../scripts/python/bin/tasks-analyzer-record |  2 +
.../scripts/python/bin/tasks-analyzer-report |  3 +

Best Regards,
Petar

maintainer
#10 lore raw
Quoted 11 lines · Petar Gligoric
Quoted 5 lines · Namhyung Kim

Then I have no problem with python. Please take a look at the scripts/python/bin directory to provide shortcuts for record/report.

Great that we could resolve it! Regarding wrapper: I thought I added them or did you mean something else?

.../scripts/python/bin/tasks-analyzer-record |  2 +
.../scripts/python/bin/tasks-analyzer-report |  3 +

Right, that's what I meant. You got it already! :)

Thanks,
Namhyung

#11 lore raw

Em Thu, Dec 08, 2022 at 12:15:13PM -0800, Namhyung Kim escreveu:

Quoted 7 lines · Petar Gligoric
Quoted 6 lines
Quoted 5 lines · Namhyung Kim

Then I have no problem with python. Please take a look at the scripts/python/bin directory to provide shortcuts for record/report.

Quoted 7 lines
Quoted 5 lines

Great that we could resolve it! Regarding wrapper: I thought I added them or did you mean something else?

.../scripts/python/bin/tasks-analyzer-record |  2 +
.../scripts/python/bin/tasks-analyzer-report |  3 +

Right, that's what I meant. You got it already! :)

I don't see any objection to merging this, comes with a set of 'perf test' to make sure we don't regress, I see no reason not to merge, done.

- Arnaldo