Add ability to generate some graphs
v2 cover Add ability to generate some graphs
From: "Yann E. MORIN" <[email protected]>
Hello All!
Here is a small series that adds some graphing abilities to Buildroot:
- graph build-times, originally by Thomas, revamped now that our step-hooks
generate $(O)/build/build-time.log for all builds
- expose this, graph-depends and <package>-graph-depends as make targets
- allow generating either PDF or PNG graphs
Regards,
Yann E. MORIN.
Changes v1 -> v2:
- add --input to graph-build time (Thomas)
- move %-graph-depends to package/pkg-generic (Thomas)
- document <package>-graph-depends (Thomas)
- document GRAPH_ALT and GRAPH_OUT in manual (Thomas)
- misc typoes
Regards,
Yann E. MORIN.
----------------------------------------------------------------
Thomas Petazzoni (1):
graph-build-time: generate graphs based on timing data
Yann E. MORIN (3):
Makefile: expose target 'graph-build' to generate the build-time graphs
Makefile: expose 'graph-depends' to generate a graph of the dependency tree
graphs: support generating png graphs
Makefile | 22 +++ docs/manual/common-usage.txt | 4 + package/pkg-generic.mk | 5 + support/scripts/graph-build-time | 290 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 321 insertions(+) create mode 100755 support/scripts/graph-build-time
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/4 graph-build-time: generate graphs based on timing data
From: Thomas Petazzoni <[email protected]>
This script generates graphs of packages build time, from the timing data generated by Buildroot in the $(O)/build-time.log file.
Example usage:
cat $(O)/build-time.log | \
./support/scripts/graph-build-time \
--type=histogram --output=foobar.pdfThree graph types are available :
* histogram, which creates an histogram of the build time for each package, decomposed by each step (extract, patch, configure, etc.). The order in which the packages are shown is
configurable: by package name, by build order, or by duration order. See the --order option.
* pie-packages, which creates a pie chart of the build time of each package (without decomposition in steps). Packages that contributed to less than 1% of the overall build time are all grouped together in an "Other" entry.
* pie-steps, which creates a pie chart of the time spent globally on each step (extract, patch, configure, etc...)
The default is to generate an histogram ordered by package name.
[yann.morin.1998 at free.fr: adapt to the format of the step-hooks build-time.log,
add sort order by name, default to name-ordered histogram, use our colours for pie-charts, add alternate color-scheme, add short-options, add --input/-i]
--- support/scripts/graph-build-time | 291 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100755 support/scripts/graph-build-time
support/scripts/graph-build-time new file +291 -0
@@ -0,0 +1,291 @@ +#!/usr/bin/env python + +# Copyright (C) 2011 by Thomas Petazzoni <[email protected]> +# Copyright (C) 2013 by Yann E. MORIN <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# This script generates graphs of packages build time, from the timing +# data generated by Buildroot in the $(O)/build-time.log file. +# +# Example usage: +# +# cat $(O)/build-time.log | ./support/scripts/graph-build-time --type=histogram --output=foobar.pdf +# +# Three graph types are available : +# +# * histogram, which creates an histogram of the build time for each +# package, decomposed by each step (extract, patch, configure, +# etc.). The order in which the packages are shown is +# configurable: by package name, by build order, or by duration +# order. See the --order option. +# +# * pie-packages, which creates a pie chart of the build time of +# each package (without decomposition in steps). Packages that +# contributed to less than 1% of the overall build time are all +# grouped together in an "Other" entry. +# +# * pie-steps, which creates a pie chart of the time spent globally +# on each step (extract, patch, configure, etc...) +# +# The default is to generate an histogram ordered by package name. +# +# Requirements: +# +# * matplotlib (python-matplotlib on Debian/Ubuntu systems) +# * numpy (python-numpy on Debian/Ubuntu systems) +# * argparse (by default in Python 2.7, requires python-argparse if +# Python 2.6 is used) + +import matplotlib +import numpy +matplotlib.use('PDF') + +import matplotlib.pyplot as plt +import matplotlib.font_manager as fm +import csv +import argparse +import sys + +steps = [ 'extract', 'patch', 'configure', 'build', + 'install-target', 'install-staging', 'install-images', + 'install-host'] + +default_colors = ['#e60004', '#009836', '#2e1d86', '#ffed00', + '#0068b5', '#f28e00', '#940084', '#97c000'] + +alternate_colors = ['#00e0e0', '#3f7f7f', '#ff0000', '#00c000', + '#0080ff', '#c000ff', '#00eeee', '#e0e000'] + +class Package: + def __init__(self, name): + self.name = name + self.steps_duration = {} + self.steps_start = {} + self.steps_end = {} + + def add_step(self, step, state, time): + if state == "start": + self.steps_start[step] = time + else: + self.steps_end[step] = time + if self.steps_start.has_key(step) and self.steps_end.has_key(step): + self.steps_duration[step] = self.steps_end[step] - self.steps_start[step] + + def get_duration(self, step=None): + if step == None: + duration = 0 + for step in self.steps_duration.keys(): + duration += self.steps_duration[step] + return duration + if self.steps_duration.has_key(step): + return self.steps_duration[step] + return 0 + +# Generate an histogram of the time spent in each step of each +# package. +def pkg_histogram(data, output, order="build"): + n_pkgs = len(data) + ind = numpy.arange(n_pkgs) + + if order == "duration": + data = sorted(data, key=lambda p: p.get_duration(), reverse=True) + elif order == "name": + data = sorted(data, key=lambda p: p.name, reverse=False) + + # Prepare the vals array, containing one entry for each step + vals = [] + for step in steps: + val = [] + for p in data: + val.append(p.get_duration(step)) + vals.append(val) + + bottom = [0] * n_pkgs + legenditems = [] + + plt.figure() + + # Draw the bars, step by step + for i in range(0, len(vals)): + b = plt.bar(ind+0.1, vals[i], width=0.8, color=colors[i], bottom=bottom, linewidth=0.25) + legenditems.append(b[0]) + bottom = [ bottom[j] + vals[i][j] for j in range(0, len(vals[i])) ] + + # Draw the package names + plt.xticks(ind + .6, [ p.name for p in data ], rotation=-60, rotation_mode="anchor", fontsize=8, ha='left') + + # Adjust size of graph (double the width) + sz = plt.gcf().get_size_inches() + plt.gcf().set_size_inches(sz[0] * 2, sz[1]) + + # Add more space for the package names at the bottom + plt.gcf().subplots_adjust(bottom=0.2) + + # Remove ticks in the graph for each package + axes = plt.gcf().gca() + for line in axes.get_xticklines(): + line.set_markersize(0) + + axes.set_ylabel('Time (seconds)') + + # Reduce size of legend text + leg_prop = fm.FontProperties(size=6) + + # Draw legend + plt.legend(legenditems, steps, prop=leg_prop) + + if order == "name": + plt.title('Build time of packages\n') + elif order == "build": + plt.title('Build time of packages, by build order\n') + elif order == "duration": + plt.title('Build time of packages, by duration order\n') + + # Save graph + plt.savefig(output) + +# Generate a pie chart with the time spent building each package. +def pkg_pie_time_per_package(data, output): + # Compute total build duration + total = 0 + for p in data: + total += p.get_duration() + + # Build the list of labels and values, and filter the packages + # that account for less than 1% of the build time. + labels = [] + values = [] + other_value = 0 + for p in data: + if p.get_duration() < (total * 0.01): + other_value += p.get_duration() + else: + labels.append(p.name) + values.append(p.get_duration()) + + labels.append('Other') + values.append(other_value) + + plt.figure() + + # Draw pie graph + patches, texts, autotexts = plt.pie(values, labels=labels, + autopct='%1.1f%%', shadow=True, + colors=colors) + + # Reduce text size + proptease = fm.FontProperties() + proptease.set_size('xx-small') + plt.setp(autotexts, fontproperties=proptease) + plt.setp(texts, fontproperties=proptease) + + plt.title('Build time per package') + plt.savefig(output) + +# Generate a pie chart with a portion for the overall time spent in +# each step for all packages. +def pkg_pie_time_per_step(data, output): + steps_values = [] + for step in steps: + val = 0 + for p in data: + val += p.get_duration(step) + steps_values.append(val) + + plt.figure() + + # Draw pie graph + patches, texts, autotexts = plt.pie(steps_values, labels=steps, + autopct='%1.1f%%', shadow=True, + colors=colors) + + # Reduce text size + proptease = fm.FontProperties() + proptease.set_size('xx-small') + plt.setp(autotexts, fontproperties=proptease) + plt.setp(texts, fontproperties=proptease) + + plt.title('Build time per step') + plt.savefig(output) + +# Parses the csv file passed on standard input and returns a list of +# Package objects, filed with the duration of each step and the total +# duration of the package. +def read_data(input_file): + if input_file is None: + input_file = sys.stdin + else: + input_file = open(input_file) + reader = csv.reader(input_file, delimiter=':') + pkgs = [] + + # Auxilliary function to find a package by name in the list. + def getpkg(name): + for p in pkgs: + if p.name == name: + return p + return None + + for row in reader: + time = int(row[0].strip()) + state = row[1].strip() + step = row[2].strip() + pkg = row[3].strip() + + p = getpkg(pkg) + if p is None: + p = Package(pkg) + pkgs.append(p) + + p.add_step(step, state, time) + + return pkgs + +parser = argparse.ArgumentParser(description='Draw build time graphs') +parser.add_argument("--type", '-t', metavar="GRAPH_TYPE", + help="Type of graph (histogram, pie-packages, pie-steps)") +parser.add_argument("--order", '-O', metavar="GRAPH_ORDER", + help="Ordering of packages: build or duration (for histogram only)") +parser.add_argument("--alternate-colors", '-c', action="store_true", + help="Use alternate colour-scheme") +parser.add_argument("--input", '-i', metavar="OUTPUT", + help="Input file (usually $(O)/build/build-time.log)") +parser.add_argument("--output", '-o', metavar="OUTPUT", required=True, + help="Output file (PDF extension)") +args = parser.parse_args() + +d = read_data(args.input) + +if args.alternate_colors: + colors = alternate_colors +else: + colors = default_colors + +if args.type == "histogram" or args.type == None: + if args.order == "build" or args.order == "duration" or args.order == "name": + pkg_histogram(d, args.output, args.order) + elif args.order == None: + pkg_histogram(d, args.output, "name") + else: + sys.stderr.write("Unknown ordering: %s\n" % args.order) + exit(1) +elif args.type == "pie-packages": + pkg_pie_time_per_package(d, args.output) +elif args.type == "pie-steps": + pkg_pie_time_per_step(d, args.output) +else: + sys.stderr.write("Unknown type: %s\n" % args.type) + exit(1)
Signature · 1 line
1.8.1.2
PATCH v2 2/4 Makefile: expose target 'graph-build' to generate the build-time graphs
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Generate the build-time graphs by calling:
make graph-build
This generates the graphs in $(O)/graphs/
It is possible to use the alternate color-scheme by setting the variable GRAPH_ALT=1 on the command line:
make GRAPH_ALT=1 graph-build
--- Makefile | 12 ++++++++++++ docs/manual/common-usage.txt | 2 ++ 2 files changed, 14 insertions(+)
Makefile +12 -0
@@ -631,6 +631,17 @@ legal-info: dirs legal-info-clean legal-info-prepare $(TARGETS_LEGAL_INFO) \ show-targets: @echo $(TARGETS) +graph-build: $(O)/build/build-time.log + @install -d $(O)/graphs + $(foreach o,name build duration,./support/scripts/graph-build-time \ + --type=histogram --order=$(o) --input=$(<) \ + --output=$(O)/graphs/build.hist-$(o).pdf \ + $(if $(GRAPH_ALT),--alternate-colors)$(sep)) + $(foreach t,packages steps,./support/scripts/graph-build-time \ + --type=pie-$(t) --input=$(<) \ + --output=$(O)/graphs/build.pie-$(t).pdf \ + $(if $(GRAPH_ALT),--alternate-colors)$(sep)) + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) all: menuconfig
@@ -843,6 +854,7 @@ endif @echo ' manual-pdf - build manual in PDF' @echo ' manual-text - build manual in text' @echo ' manual-epub - build manual in ePub' + @echo ' graph-build - generate graphs of the build times' @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build'
docs/manual/common-usage.txt +2 -0
@@ -86,6 +86,8 @@ to +make+ or set in the environment: Note that the Buildroot download directory can also be set from the configuration interface, so through the Buildroot +.config+ file; this is the recommended way of setting it. +* +GRAPH_ALT+, if set and non-empty, to use an alternate color-scheme in + build-time graphs An example that uses config files located in the toplevel directory and in your $HOME:
Signature · 1 line
1.8.1.2
PATCH v2 3/4 Makefile: expose 'graph-depends' to generate a graph of the dependency tree
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Generate the graph of the complete dependency tree by calling:
make graph-depends
It's also possible to generate the graph-depends for a single package:
make PKG-graph-depends
The graphs are generated in $(O)/graphs/
--- Makefile | 7 +++++++ package/pkg-generic.mk | 5 +++++ 2 files changed, 12 insertions(+)
Makefile +7 -0
@@ -642,6 +642,11 @@ graph-build: $(O)/build/build-time.log --output=$(O)/graphs/build.pie-$(t).pdf \ $(if $(GRAPH_ALT),--alternate-colors)$(sep)) +graph-depends: + @$(INSTALL) -d $(O)/graphs + @./support/scripts/graph-depends \ + |dot -Tpdf -o $(O)/graphs/$(@).pdf + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) all: menuconfig
@@ -814,6 +819,7 @@ help: @echo ' toolchain - build toolchain' @echo ' <package>-rebuild - force recompile <package>' @echo ' <package>-reconfigure - force reconfigure <package>' + @echo ' <package>-graph-depends - generate graph of the dependency tree for package' @echo @echo 'Configuration:' @echo ' menuconfig - interactive curses-based configurator'
@@ -855,6 +861,7 @@ endif @echo ' manual-text - build manual in text' @echo ' manual-epub - build manual in ePub' @echo ' graph-build - generate graphs of the build times' + @echo ' graph-depends - generate graph of the dependency tree' @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build'
package/pkg-generic.mk +5 -0
@@ -468,6 +468,11 @@ endif $(1)-show-depends: @echo $$($(2)_DEPENDENCIES) +$(1)-graph-depends: + @$(INSTALL) -d $(O)/graphs + @./support/scripts/graph-depends $(1) \ + |dot -Tpdf -o $(O)/graphs/$$(@).pdf + $(1)-dirclean: $$($(2)_TARGET_DIRCLEAN) $(1)-clean-for-rebuild:
Signature · 1 line
1.8.1.2
PATCH v2 4/4 graphs: support generating png graphs
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
PDF files can not be easily embedded in other documents (eg. ODT, or HTML).
Add support for generating PNG graphs, by setting the GRAPH_OUT=pdf|png on the command line:
make GRAPH_OUT=png graph-build graph-depends
The default is still to generate PDF graphs.
--- Makefile | 9 ++++++--- docs/manual/common-usage.txt | 2 ++ package/pkg-generic.mk | 2 +- support/scripts/graph-build-time | 3 +-- 4 files changed, 10 insertions(+), 6 deletions(-)
Makefile +6 -3
@@ -121,6 +121,9 @@ else $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE)) endif +# Need that early, before we scan packages +# Avoids doing the $(or...) everytime +BR2_GRAPH_OUT := $(or $(GRAPH_OUT),pdf) BUILD_DIR:=$(BASE_DIR)/build STAMP_DIR:=$(BASE_DIR)/stamps
@@ -635,17 +638,17 @@ graph-build: $(O)/build/build-time.log @install -d $(O)/graphs $(foreach o,name build duration,./support/scripts/graph-build-time \ --type=histogram --order=$(o) --input=$(<) \ - --output=$(O)/graphs/build.hist-$(o).pdf \ + --output=$(O)/graphs/build.hist-$(o).$(BR2_GRAPH_OUT) \ $(if $(GRAPH_ALT),--alternate-colors)$(sep)) $(foreach t,packages steps,./support/scripts/graph-build-time \ --type=pie-$(t) --input=$(<) \ - --output=$(O)/graphs/build.pie-$(t).pdf \ + --output=$(O)/graphs/build.pie-$(t).$(BR2_GRAPH_OUT) \ $(if $(GRAPH_ALT),--alternate-colors)$(sep)) graph-depends: @$(INSTALL) -d $(O)/graphs @./support/scripts/graph-depends \ - |dot -Tpdf -o $(O)/graphs/$(@).pdf + |dot -T$(BR2_GRAPH_OUT) -o $(O)/graphs/$(@).$(BR2_GRAPH_OUT) else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
docs/manual/common-usage.txt +2 -0
@@ -88,6 +88,8 @@ to +make+ or set in the environment: is the recommended way of setting it. * +GRAPH_ALT+, if set and non-empty, to use an alternate color-scheme in build-time graphs +* +GRAPH_OUT+ to set the filetype of generated graphs, either +pdf+ (the + default), or +png+. An example that uses config files located in the toplevel directory and in your $HOME:
package/pkg-generic.mk +1 -1
@@ -471,7 +471,7 @@ $(1)-show-depends: $(1)-graph-depends: @$(INSTALL) -d $(O)/graphs @./support/scripts/graph-depends $(1) \ - |dot -Tpdf -o $(O)/graphs/$$(@).pdf + |dot -T$(BR2_GRAPH_OUT) -o $(O)/graphs/$$(@).$(BR2_GRAPH_OUT) $(1)-dirclean: $$($(2)_TARGET_DIRCLEAN)
support/scripts/graph-build-time +1 -2
@@ -51,7 +51,6 @@ import matplotlib import numpy -matplotlib.use('PDF') import matplotlib.pyplot as plt import matplotlib.font_manager as fm
@@ -264,7 +263,7 @@ parser.add_argument("--alternate-colors", '-c', action="store_true", parser.add_argument("--input", '-i', metavar="OUTPUT", help="Input file (usually $(O)/build/build-time.log)") parser.add_argument("--output", '-o', metavar="OUTPUT", required=True, - help="Output file (PDF extension)") + help="Output file (.pdf or .png extension)") args = parser.parse_args() d = read_data(args.input)
Signature · 1 line
1.8.1.2
Thomas Petazzoni
Appliedv2 cover Add ability to generate some graphs
In reply to Yann E. MORINDear Yann E. MORIN,
Quoted 7 lines · Yann E. MORIN
Thomas Petazzoni (1):
graph-build-time: generate graphs based on timing dataYann E. MORIN (3):
Makefile: expose target 'graph-build' to generate the build-time graphs
Makefile: expose 'graph-depends' to generate a graph of the dependency tree
graphs: support generating png graphs
Applied, thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com