graph-depends improvements (branch yem/graphs)
v2 cover graph-depends improvements (branch yem/graphs)
From: "Yann E. MORIN" <[email protected]>
Hello All!
This series brings a few improvements to the graph-depends utility:
- add option to remove transitive dependencies
- add option to pas arbitrary args to the dot utility
- cleanup and re-arrange the code
This series is open for discusion, especially on how we pass options to the graph generators:
- support/scripts/graph-depends: needs to know whether to generate transitive dependencies or not, and how deep to limit the dependencies
- the 'dot' utility: needs to know whether to graph top-down or left-right (or whatever), the type of graph (tree, circle...) and so on...
Currently, environment variables are used, but that is not in the tastes of some (me including), but I can't see a better way.
Peter suggested using kconfig knobs, but it does not scale when you want different settings for different graphs (e.g, different depth level for different packages).
Samuel suggested using a config file, but it has the same issues as the kconfig options. He also suggested pasing a single environment variable, but that does not work since some options are for graph-depends shile others are for dot.
Any other suggestion? ;-)
Regards,
Yann E. MORIN.
The following changes since commit ebb5692f358833e4fb6f3ed5f3075241d80c6667:
zeromq: explicitly add -lstdc++ for static build (2014-05-09 16:31:42 +0200)
are available in the git repository at:
git://gitorious.org/buildroot/buildroot.git yem/graphs
for you to fetch changes up to ccbf2dd69529d62cc925cc891016988e2cb59bbb:
manual: document the new graph-depends options (2014-05-09 16:51:14 +0200)
----------------------------------------------------------------
Yann E. MORIN (7):
graphs: add option to remove transitive dependencies in dependency graph
graphs-depends: merge redundant-dependencies elimination
graph-depends: handle the depth argument in a more pythonic way
graph-depends: document the max_depth variable
graph-depends: rename the mode constants
graph-depends: add option to pass arbitrary dot options
manual: document the new graph-depends options
Makefile | 7 ++- docs/manual/common-usage.txt | 6 ++ package/pkg-generic.mk | 4 +- support/scripts/graph-depends | 126 +++++++++++++++++++++++++----------------- 4 files changed, 86 insertions(+), 57 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/7 graphs: add option to remove transitive dependencies in dependency graph
From: "Yann E. MORIN" <[email protected]>
Currently, all the dependencies of a package are drawn on the dependency graph, including transitive dependencies (e.g. A->B->C and A->C).
Very big graphs, with lots of packages with lots of dependencies, the dependency graph can be very dense, and transitive dependencies are cluttering the graph.
In some cases, only getting the "build-order" dependencies is enough (e.g. to see what impact a package rebuild would have).
Add a new environment variable to disable drawing transitive dependencies.
Basically, it would turn this graph:
pkg1 ---> pkg2 ---> pkg3 -------------------.
|\__________/ \ \
|\____________________ \ \
| \ \ \
`-> pkg4 ---> pkg5 ---> pkg6 ---> pkg7 ---> pkg8
\__________/into that graph:
pkg1 ---> pkg2 ---> pkg3 -----------.
| \
`-> pkg4 ---> pkg5 ---> pkg6 ---> pkg7 ---> pkg8---
Note: some graphs showing the results are there:
http://ymorin.is-a-geek.org/download/tmp/graphs/
--- Makefile | 5 +++-- package/pkg-generic.mk | 2 +- support/scripts/graph-depends | 49 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 3 deletions(-)
Makefile +3 -2
@@ -150,7 +150,8 @@ endif # Need that early, before we scan packages # Avoids doing the $(or...) everytime BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) -BR_GRAPH_DEPTH := $(or $(BR2_GRAPH_DEPTH),0) +BR_GRAPH_DEPS_OPTS := --depth $(or $(BR2_GRAPH_DEPTH),0) +BR_GRAPH_DEPS_OPTS += $(if $(BR2_GRAPH_NO_TRANSITIVE),--no-transitive,--transitive) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images
@@ -674,7 +675,7 @@ graph-build: $(O)/build/build-time.log graph-depends: @$(INSTALL) -d $(O)/graphs @cd "$(CONFIG_DIR)"; \ - $(TOPDIR)/support/scripts/graph-depends -d $(BR_GRAPH_DEPTH) \ + $(TOPDIR)/support/scripts/graph-depends $(BR_GRAPH_DEPS_OPTS) \ |tee $(O)/graphs/$(@).dot \ |dot -T$(BR_GRAPH_OUT) -o $(O)/graphs/$(@).$(BR_GRAPH_OUT)
package/pkg-generic.mk +1 -1
@@ -512,7 +512,7 @@ $(1)-show-depends: $(1)-graph-depends: @$(INSTALL) -d $(O)/graphs @cd "$(CONFIG_DIR)"; \ - $(TOPDIR)/support/scripts/graph-depends -p $(1) -d $(BR_GRAPH_DEPTH) \ + $(TOPDIR)/support/scripts/graph-depends -p $(1) $(BR_GRAPH_DEPS_OPTS) \ |tee $(O)/graphs/$$(@).dot \ |dot -T$(BR_GRAPH_OUT) -o $(O)/graphs/$$(@).$(BR_GRAPH_OUT)
support/scripts/graph-depends +49 -0
67 lines not on this page · raw message
Signature · 1 line
1.8.3.2
PATCH v2 2/7 graphs-depends: merge redundant-dependencies elimination
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Merge the redundant-dependencies elimination into the newly introduced transitive-dependencies elimination.
This makes the code cleaner and much shorter, because:
- the ('all',pkg) redundant dependency is in fact a transitive dependency, and we now have code to deal with that
- the (pkg,'toolchain') dependency is easy enough to deal with that having a separate function for that is overkill
--- support/scripts/graph-depends | 46 +++++++++---------------------------------- 1 file changed, 9 insertions(+), 37 deletions(-)
support/scripts/graph-depends +9 -37
@@ -146,40 +146,6 @@ def get_all_depends(pkgs): def pkg_node_name(pkg): return pkg.replace("-","") -# Helper function for remove_redundant_deps(). This function tells -# whether package "pkg" is the dependency of another package that is -# not the special "all" package. -def has_redundant_deps(deps, pkg): - for dep in deps: - if dep[0] != "all" and dep[1] == pkg: - return True - return False - -# This function removes redundant dependencies of the special "all" -# package. This "all" package is created to reflect the origin of the -# selection for all packages that are not themselves selected by any -# other package. So for example if you enable libpng, zlib is enabled -# as a dependency. But zlib being selected by libpng, it also appears -# as a dependency of the "all" package. This needlessly complicates -# the generated dependency graph. So when we have the dependency list -# (all -> zlib, all -> libpn, libpng -> zlib), we get rid of the 'all -# -> zlib' dependency, because zlib is already a dependency of a -# regular package. -def remove_redundant_deps(deps): - newdeps = [] - for dep in deps: - if dep[0] == "all" and dep[1] == "toolchain": - newdeps.append(dep) - continue - if dep[0] != "all" and dep[1] != "toolchain": - newdeps.append(dep) - continue - if not has_redundant_deps(deps, dep[1]): - newdeps.append(dep) - continue - sys.stderr.write("Removing redundant dep %s -> %s\n" % (dep[0],dep[1])) - return newdeps - TARGET_EXCEPTIONS = [ "target-generic-securetty", "target-generic-issue",
@@ -217,8 +183,6 @@ if mode == FULL_MODE: elif mode == PKG_MODE: dependencies = get_all_depends([rootpkg]) -dependencies = remove_redundant_deps(dependencies) - # Make the dependencies a dictionnary { 'pkg':[dep1, dep2, ...] } dict_deps = {} for dep in dependencies:
@@ -260,10 +224,18 @@ def remove_transitive_deps(pkg,deps): new_d.append(d[i]) return new_d +# This function removes the dependency on the 'toolchain' package +def remove_toolchain_deps(pkg,deps): + return [p for p in deps[pkg] if not p == 'toolchain'] + # This functions trims down the dependency list of all packages. +# It applies in sequence all the dependency-elimination methods. def remove_extra_deps(deps): for pkg in deps.keys(): - if not transitive: + if not pkg == 'all': + deps[pkg] = remove_toolchain_deps(pkg,deps) + for pkg in deps.keys(): + if not transitive or pkg == 'all': deps[pkg] = remove_transitive_deps(pkg,deps) return deps
Signature · 1 line
1.8.3.2
PATCH v2 3/7 graph-depends: handle the depth argument in a more pythonic way
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Add some comment as well, enhance help text.
--- support/scripts/graph-depends | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
support/scripts/graph-depends +6 -4
@@ -33,13 +33,16 @@ FULL_MODE = 1 PKG_MODE = 2 mode = 0 + +# Limit drawing the dependency graph to this depth. 0 means 'no limit'. max_depth = 0 parser = argparse.ArgumentParser(description="Graph pacakges dependencies") parser.add_argument("--package", '-p', metavar="PACKAGE", help="Graph the dependencies of PACKAGE") -parser.add_argument("--depth", '-d', metavar="DEPTH", - help="Limit the dependency graph to DEPTH levels") +parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth", + help="Limit the dependency graph to DEPTH levels; 0 means no limit.") +parser.set_defaults(depth=0) parser.add_argument("--transitive", dest="transitive", action='store_true') parser.add_argument("--no-transitive", dest="transitive", action='store_false', help="Draw (do not draw) transitive dependencies")
@@ -52,8 +55,7 @@ else: mode = PKG_MODE rootpkg = args.package -if args.depth is not None: - max_depth = int(args.depth) +max_depth = int(args.depth) transitive = args.transitive
Signature · 1 line
1.8.3.2
PATCH 3/7 graph-depends: handle the depth argument in a more pythonic way
Yann, all,
Quoted 20 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
Add some comment as well, enhance help text.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/scripts/graph-depends | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)support/scripts/graph-depends +2
-0@@ -33,13 +33,16 @@ FULL_MODE = 1 PKG_MODE = 2 mode = 0 + +# Limit drawing the dependency graph to this depth. 0 means 'no limit'.
support/scripts/graph-depends FULL_MODE = 1 · line 45
Quoted 10 lines
support/scripts/graph-depends +3
-2max_depth = 0 parser = argparse.ArgumentParser(description="Graph pacakges dependencies") parser.add_argument("--package", '-p', metavar="PACKAGE", help="Graph the dependencies of PACKAGE") -parser.add_argument("--depth", '-d', metavar="DEPTH", - help="Limit the dependency graph to DEPTH levels") +parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth", + help="Limit the dependency graph to DEPTH levels; 0 means no limit.") +parser.set_defaults(depth=0)
You can also do:
parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth",
help="Limit the dependency graph to DEPTH levels; 0 means no limit.", type=int, default=0)
Quoted 20 lines
support/scripts/graph-depends +1
-3parser.add_argument("--transitive", dest="transitive", action='store_true') parser.add_argument("--no-transitive", dest="transitive", action='store_false', help="Draw (do not draw) transitive dependencies")@@ -52,8 +55,7 @@ else: mode = PKG_MODE rootpkg = args.package -if args.depth is not None: - max_depth = int(args.depth) +max_depth = int(args.depth) transitive = args.transitive --1.8.3.2
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 1 line
Samuel
PATCH v2 3/7 graph-depends: handle the depth argument in a more pythonic way
Samuel, All,
On 2014-05-09 17:14 +0200, Samuel Martin spake thusly:
Quoted 36 lines
Yann, all,
Quoted 20 lines · Yann E. MORIN
From: "Yann E. MORIN" <[email protected]>
Add some comment as well, enhance help text.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>--- support/scripts/graph-depends | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)support/scripts/graph-depends +2
-0@@ -33,13 +33,16 @@ FULL_MODE = 1 PKG_MODE = 2 mode = 0 + +# Limit drawing the dependency graph to this depth. 0 means 'no limit'.support/scripts/graph-depends FULL_MODE = 1 · line 45
Quoted 10 lines
support/scripts/graph-depends +3
-2max_depth = 0 parser = argparse.ArgumentParser(description="Graph pacakges dependencies") parser.add_argument("--package", '-p', metavar="PACKAGE", help="Graph the dependencies of PACKAGE") -parser.add_argument("--depth", '-d', metavar="DEPTH", - help="Limit the dependency graph to DEPTH levels") +parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth", + help="Limit the dependency graph to DEPTH levels; 0 means no limit.") +parser.set_defaults(depth=0)You can also do:
Quoted 3 lines
parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth",
help="Limit the dependency graph to DEPTH levels; 0 means no limit.", type=int, default=0)
Aha! Nice! Thank you. :-)
Then I suppose that: max_depth = int(args.depth) has to be just: max_depth = args.depth
Right?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 3/7 graph-depends: handle the depth argument in a more pythonic way
On Fri, May 9, 2014 at 5:28 PM, Yann E. MORIN <[email protected]> wrote:
[...]
Quoted 12 lines
Quoted 11 lines
support/scripts/graph-depends FULL_MODE = 1 · line 45
Quoted 5 lines
support/scripts/graph-depends +3
-2-parser.add_argument("--depth", '-d', metavar="DEPTH", - help="Limit the dependency graph to DEPTH levels") +parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth", + help="Limit the dependency graph to DEPTH levels; 0 means no limit.") +parser.set_defaults(depth=0)You can also do:
parser.add_argument("--depth", '-d', metavar="DEPTH", dest="depth",
help="Limit the dependency graph to DEPTH levels; 0 means no limit.", type=int, default=0)
Quoted 5 lines
Aha! Nice! Thank you. :-)
Then I suppose that: max_depth = int(args.depth) has to be just: max_depth = args.depthRight?
yes :-)
(see, you start loving python ;-])
Signature · 1 line
Samuel
PATCH v2 4/7 graph-depends: document the max_depth variable
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Although unnecessary (we already have initialisation via the parser), initialise the max_depth option, and document it at the same time.
--- support/scripts/graph-depends | 3 +++ 1 file changed, 3 insertions(+)
support/scripts/graph-depends +3 -0
@@ -37,6 +37,9 @@ mode = 0 # Limit drawing the dependency graph to this depth. 0 means 'no limit'. max_depth = 0 +# Whether to draw the transitive dependencies +transitive = True + parser = argparse.ArgumentParser(description="Graph pacakges dependencies") parser.add_argument("--package", '-p', metavar="PACKAGE", help="Graph the dependencies of PACKAGE")
Signature · 1 line
1.8.3.2
PATCH v2 5/7 graph-depends: rename the mode constants
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
This is ugly, since Python does not have enum constructs, so by moving the 'type' of the constant ('MODE' here) to the beginning, we get an artificial 'namespace' for the constants.
--- support/scripts/graph-depends | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
support/scripts/graph-depends +8 -12
@@ -25,13 +25,9 @@ import sys import subprocess import argparse -# In FULL_MODE, we draw the full dependency graph for all selected -# packages -FULL_MODE = 1 - -# In PKG_MODE, we only draw the dependency graph for a given package -PKG_MODE = 2 - +# Modes of operation: +MODE_FULL = 1 # draw full dependency graph for all selected packages +MODE_PKG = 2 # draw dependency graph for a given package mode = 0 # Limit drawing the dependency graph to this depth. 0 means 'no limit'.
@@ -53,9 +49,9 @@ parser.set_defaults(transitive=True) args = parser.parse_args() if args.package is None: - mode = FULL_MODE + mode = MODE_FULL else: - mode = PKG_MODE + mode = MODE_PKG rootpkg = args.package max_depth = int(args.depth)
@@ -167,7 +163,7 @@ TARGET_EXCEPTIONS = [ # In full mode, start with the result of get_targets() to get the main # targets and then use get_all_depends() for all targets -if mode == FULL_MODE: +if mode == MODE_FULL: targets = get_targets() dependencies = [] allpkgs.append('all')
@@ -185,7 +181,7 @@ if mode == FULL_MODE: # In pkg mode, start directly with get_all_depends() on the requested # package -elif mode == PKG_MODE: +elif mode == MODE_PKG: dependencies = get_all_depends([rootpkg]) # Make the dependencies a dictionnary { 'pkg':[dep1, dep2, ...] }
@@ -253,7 +249,7 @@ def print_attrs(pkg): print "all [color=lightblue,style=filled]" return print "%s [label = \"%s\"]" % (pkg_node_name(pkg), pkg) - if mode == PKG_MODE and pkg == rootpkg: + if mode == MODE_PKG and pkg == rootpkg: print "%s [color=lightblue,style=filled]" % pkg_node_name(rootpkg) else: print "%s [color=grey,style=filled]" % pkg_node_name(pkg)
Signature · 1 line
1.8.3.2
PATCH v2 6/7 graph-depends: add option to pass arbitrary dot options
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
Kids nowaday seem to prefer a left-to-right drawing rather than the more conventional and historical top-down drawing.
Rather than multiply the number of environment variables, just add a single one where the user can pass arbitrary dot options, such as:
make BR2_GRAPH_DOT_OPTS=-Grankdir=LR graph-depends
--- Makefile | 2 +- package/pkg-generic.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
Makefile +1 -1
@@ -677,7 +677,7 @@ graph-depends: @cd "$(CONFIG_DIR)"; \ $(TOPDIR)/support/scripts/graph-depends $(BR_GRAPH_DEPS_OPTS) \ |tee $(O)/graphs/$(@).dot \ - |dot -T$(BR_GRAPH_OUT) -o $(O)/graphs/$(@).$(BR_GRAPH_OUT) + |dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(O)/graphs/$(@).$(BR_GRAPH_OUT) else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
package/pkg-generic.mk +1 -1
@@ -514,7 +514,7 @@ $(1)-graph-depends: @cd "$(CONFIG_DIR)"; \ $(TOPDIR)/support/scripts/graph-depends -p $(1) $(BR_GRAPH_DEPS_OPTS) \ |tee $(O)/graphs/$$(@).dot \ - |dot -T$(BR_GRAPH_OUT) -o $(O)/graphs/$$(@).$(BR_GRAPH_OUT) + |dot $(BR2_GRAPH_DOT_OPTS) -T$(BR_GRAPH_OUT) -o $(O)/graphs/$$(@).$(BR_GRAPH_OUT) $(1)-dirclean: $$($(2)_TARGET_DIRCLEAN)
Signature · 1 line
1.8.3.2
PATCH v2 7/7 manual: document the new graph-depends options
In reply to Yann E. MORINFrom: "Yann E. MORIN" <[email protected]>
--- docs/manual/common-usage.txt | 6 ++++++ 1 file changed, 6 insertions(+)
docs/manual/common-usage.txt +6 -0
@@ -94,6 +94,10 @@ to +make+ or set in the environment: default), or +png+. * +BR2_GRAPH_DEPTH+ (an integer) to limit the depth of the dependency graph. The default, +0+, is to not limit the depth. +* +BR2_GRAPH_NO_TRANSITIVE+, if set and non-empty, to not draw the transitive + dependencies. The default is to draw transitive dependencies. +* +BR2_GRAPH_DOT_OPTS+ is passed verbatim as options to the +dot+ utility to + draw the dependency graph. An example that uses config files located in the toplevel directory and in your $HOME:
@@ -198,6 +202,8 @@ supported. BR2_GRAPH_OUT=svg make graph-depends -------------------------------- +See xref:env-vars[] for other ways to customize the dependency graph. + === Graphing the build duration [[graph-duration]]
Signature · 1 line
1.8.3.2
PATCH 7/7 manual: document the new graph-depends options
"Yann E. MORIN" <[email protected]> schreef:
Quoted 32 lines
From: "Yann E. MORIN" <[email protected]>
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Maxime Hadjinlian <[email protected]>---
docs/manual/common-usage.txt| 6 ++++++1 file changed, 6 insertions(+)docs/manual/common-usage.txt +4
-0@@ -94,6 +94,10 @@ to +make+ or set in the environment: default), or +png+. * +BR2_GRAPH_DEPTH+ (an integer) to limit the depth of the dependency graph. The default, +0+, is to not limit the depth. +* +BR2_GRAPH_NO_TRANSITIVE+, if set and non-empty, to not draw the transitive + dependencies. The default is to draw transitive dependencies. +* +BR2_GRAPH_DOT_OPTS+ is passed verbatim as options to the +dot+ utility to + draw the dependency graph.An example that uses config files located in the toplevel directory and in your $HOME:
docs/manual/common-usage.txt +0
-0@@ -198,6 +202,8 @@ supported.BR2_GRAPH_OUT=svg make graph-depends
--------------------------------docs/manual/common-usage.txt +2
-0+See xref:env-vars[] for other ways to customize the dependency graph. +=== Graphing the build duration
Quoted 1 line
[[graph-duration]]
(Checked for spelling only)