support/graphs: fixup and speedup graph dependencies (branch yem/graphs-from-make-2)

Hello All!

This series is a joint effort with Thomas, to get a graph-depends that is more reliable (does not forget some dependencies), and faster.

TL;DR: we fixed it, and it is faster. ;-)

Long story:

Currently, graph-depends is used in three ways: to graph the dependency tree (which is its prime objective, hence the name), to check for circular dependencies, and to compute the list of recursive direct or reverse dependencies.

In so doing, graph-depends needs to build up the dependency tree, so it calls to make to get an initial list of packages, and then iteratively calls it again and again to get the dependencies, until none are returned anymore. Running make is rather costly, because it scans the entire collection of .mk; running it iteratively is thus in O(n), with n the dependency depth.

This series was started with the realisation that 'make' already knows about the dependency tree (that's its job, damned!).

So, this series starts with some commits which actually fix graph-depends to not forget any dependency, then cut down on madatory dependencies.

Building upon that, we introduce the necesary infrastructure in our Makefiles, to return the dependency tree, and use that from graph-depends.

Incidentally, we also notice that recursive, direct and reverse, dependencies can also be had from make in a much less costly manner than with graph-depends.

Changes v1 -> v2:
- use memoisation to further speed-up recursive reverse dependencies
- update timings accordingly

Regards,
Yann E. MORIN.

The following changes since commit bdfea8428f1d89ec79e0253b61fe86eb5437532d

Update for 2019.02-rc3 (2019-03-01 12:57:30 +0100)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to ae655a96d9f6c640f9cf16c273c732fe2009ce57

support/graph-depends: use the new make-based dependency tree (2019-03-03 11:06:38 +0100)

----------------------------------------------------------------
Thomas Petazzoni (1):
support/graph-depends: make sure mandatory deps are displayed

Yann E. MORIN (7):
support/graph-depends: add option to exclude mandatory deps support/graph-depends: also cut on host-skeleton support/graph-depends: also cut on host-tar support/graph-depends: also cut on host-gzip infra/pkg-generic: use pure Makefile-based recursive dependencies
core: add make-based full-dependency list support/graph-depends: use the new make-based dependency tree

 Makefile                      |   4 ++
 fs/common.mk                  |   8 +++
 package/pkg-generic.mk        |  22 +++++--
 support/scripts/brpkgutil.py  | 107 ++++++++++++++++-----------------
 support/scripts/graph-depends | 133 +++++++++---------------------------------
 5 files changed, 109 insertions(+), 165 deletions(-)

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

From: Thomas Petazzoni <[email protected]>

The current graph-depends implementation filters out a number of "mandatory" dependencies that all packages have: dependency on "toolchain" and dependency on "skeleton".

Despite this filtering, in full graph dependencies, "toolchain" and "skeleton" are still shown, because they are target packages, and therefore appear in the result of "make show-targets". Thanks to this, they will be visible as dependencies of the "ALL" node, which is the root of the dependency tree.

However, as we are going to introduce host-skeleton as a "mandatory dependency" to be filtered out, this is no longer going to work.

This commit adjusts the remove_extra_deps() function to ensure that when a mandatory dependency is removed, this dependency exists between the root of the dependency tree and the mandatory dependency.

This issue was noticed by Yann E. Morin, and this commit provides a different implementation than what Yann proposed in
https://patchwork.ozlabs.org/patch/910453/.

[yann.morin.1998 at free.fr:
- list mandatory deps before removing them
- fix flake8 warnings
]

---
 support/scripts/graph-depends | 9 +++++++++
 1 file changed, 9 insertions(+)

index d2b100f385..5a6f6930e9 100755
@@ -181,6 +181,12 @@ def remove_mandatory_deps(pkg, deps):
     return [p for p in deps[pkg] if p not in MANDATORY_DEPS]
 
 
+# This function returns all dependencies of pkg that are part of the
+# mandatory dependencies:
+def get_mandatory_deps(pkg, deps):
+    return [p for p in deps[pkg] if p in MANDATORY_DEPS]
+
+
 # This function will check that there is no loop in the dependency chain
 # As a side effect, it builds up the dependency cache.
 def check_circular_deps(deps):

@@ -213,6 +219,9 @@ def check_circular_deps(deps):
 def remove_extra_deps(deps, rootpkg, transitive):
     for pkg in list(deps.keys()):
         if not pkg == rootpkg:
+            for d in get_mandatory_deps(pkg, deps):
+                if d not in deps[rootpkg]:
+                    deps[rootpkg].append(d)
             deps[pkg] = remove_mandatory_deps(pkg, deps)
     for pkg in list(deps.keys()):
         if not transitive or pkg == rootpkg:

2.14.1

Some times, multiple dependency graphs for a set of packages (mostly the application-level packages for the project) are included in reports (e.g. delivery notes). Repeating the mandatory dependencies on all those graphs is useless and clutters the important dependencies.

When we had only two such mandatory dependencies (toolchain, skeleton), it was manageable to list them as manual exclusions:
-x toolchain -x skeleton

But we now have quite a few such dependencies, and it becomes a bit more cumbersome to manage, not counting the ones we may add in the future.

Add an option to exclude all those mandatory dependencies, to generate neat graphs.

---
 support/scripts/graph-depends | 5 +++++
 1 file changed, 5 insertions(+)

index 5a6f6930e9..5c5de7dd0b 100755
@@ -311,6 +311,8 @@ def parse_args():
                         "'host' to stop on host packages.")
     parser.add_argument("--exclude", "-x", metavar="PACKAGE", dest="exclude_list", action="append",
                         help="Like --stop-on, but do not add PACKAGE to the graph.")
+    parser.add_argument("--exclude-mandatory", "-X", action="store_true",
+                        help="Like if -x was passed for all mandatory dependencies.")
     parser.add_argument("--colors", "-c", metavar="COLOR_LIST", dest="colors",
                         default="lightblue,grey,gainsboro",
                         help="Comma-separated list of the three colors to use" +

@@ -364,6 +366,9 @@ def main():
     else:
         exclude_list = args.exclude_list
 
+    if args.exclude_mandatory:
+        exclude_list += MANDATORY_DEPS
+
     if args.direct:
         get_depends_func = brpkgutil.get_depends
         arrow_dir = "forward"

2.14.1

host-skeleton is a dependency of almost all packages, except a very few. As such, it clutters the dependency graph uselessly.

Do with it as we do for the skeleton: cut the dependency chains.

---
 support/scripts/graph-depends | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 5c5de7dd0b..c3e5d83ee8 100755
@@ -172,7 +172,7 @@ def remove_transitive_deps(pkg, deps):
 
 # List of dependencies that all/many packages have, and that we want
 # to trim when generating the dependency graph.
-MANDATORY_DEPS = ['toolchain', 'skeleton']
+MANDATORY_DEPS = ['toolchain', 'skeleton', 'host-skeleton']
 
 
 # This function removes the dependency on some 'mandatory' package, like the

2.14.1

When host-tar is needed, it is a mandatory dependency of all packages. As such, drawing the dependency lines toward host-tar would uselessly clutter the graph.

So, like for the skeleton and host-skeleton, we cut the dependency chains toward host-tar.

---
 support/scripts/graph-depends | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index c3e5d83ee8..3143c61cc4 100755
@@ -172,7 +172,7 @@ def remove_transitive_deps(pkg, deps):
 
 # List of dependencies that all/many packages have, and that we want
 # to trim when generating the dependency graph.
-MANDATORY_DEPS = ['toolchain', 'skeleton', 'host-skeleton']
+MANDATORY_DEPS = ['toolchain', 'skeleton', 'host-skeleton', 'host-tar']
 
 
 # This function removes the dependency on some 'mandatory' package, like the

2.14.1

When host-gzip is needed, it is a mandatory dependency of all packages. As such, drawing the dependency lines toward host-gzip would uselessly clutter the graph.

So, like for the skeleton, host-skeleton, and host-tar, we cut the dependency chains toward host-gzip.

---
 support/scripts/graph-depends | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 3143c61cc4..29134c8237 100755
@@ -172,7 +172,7 @@ def remove_transitive_deps(pkg, deps):
 
 # List of dependencies that all/many packages have, and that we want
 # to trim when generating the dependency graph.
-MANDATORY_DEPS = ['toolchain', 'skeleton', 'host-skeleton', 'host-tar']
+MANDATORY_DEPS = ['toolchain', 'skeleton', 'host-skeleton', 'host-tar', 'host-gzip']
 
 
 # This function removes the dependency on some 'mandatory' package, like the

2.14.1

Calling to the graph-depends script is very costly, as it calls back to 'make' a lot of time.

It turns out that we already have the list of recursive dependencies, so we can just print that.

As for the recursive reverse dependencies, we use the same memoisation technique to cut-down on the expansion cost, which would otherwise be on the order of ?(??) (with ? enabled packages).

From a defconfig, modified to use glibc, C++, wchar, locales, ssp, and allyespackageconfig (tweaked to avoid multi providers, etc...), the timings for X-show-recursive-rdepends are:

                    before      after       speedup     #rdeps
    libnss          0m22.932s   0m5.775s     3.97x      3
    qt5base         0m41.176s   0m5.781s     7.12x      67
    libjpeg         0m56.185s   0m5.749s     9.71x      228
    libxml2         0m54.964s   0m5.795s     9.48x      271
    freetype        0m46.754s   0m5.819s     8.07x      287
    libpng          0m53.577s   0m5.760s     9.30x      303
    sqlite          1m15.222s   0m5.807s    12.95x      801
    libopenssl      1m25.471s   0m5.844s    14.63x      931
    readline        1m13.805s   0m5.775s    12.78x      958
    libzlib         1m11.807s   0m5.820s    12.34x      1039
    toolchain       1m23.712s   0m6.080s    13.77x      2107
    skeleton        1m27.839s   0m6.293s    13.96x      2111 (+1)
    host-skeleton   1m27.405s   0m6.350s    13.76x      2172 (+2)

- speedup: ratio before/after
- #rdeps: number of recursive reverse dependencies, with the extra dependencies returned with this patch, see below for the reason.

So, for a low-level package with a lot of reverse dependencies, like libzlibz, libopenssl or readline are, the timings are already very much in favour of the change. This is less impressive with packages that have few dependencies (libnss), but still much faster.

Also, remember that the config tested has as much packages enabled as possible, so is in itself a degenerate case. With simpler and more realistic configurations, the gains would probably be a bit lower than reported above, but various tests still report good improvements overall (note: coming up with a 'realistic' configuration is pretty hard, as everyone and their dog have their notion of what is realistic in their context, so nothing displayed here; timings are left as an exercise for the interested parties to report aggravation in their cases should they notice some regression).

Note that, more recursive reverse dependencies may be displayed now, since we do not apply the exceptions applied in graph-depends. For example, host-skeleton gains two new recursive reverse dependencies:
skeleton and toolchain, which are both exceptions in graph-depends.

As for direct (not reverse) dependencies: the gain is not as fantastic as for reverse ones, but it is still noticeable, especially thanks to a21212fb7cf (package/pkg-generic: speed up RECURSIVE_FINAL_DEPENDENCIES); just a few examples for %-show-recursive-depends:

                    before      after       speedup     #deps
    libzlib         0m46.864s   0m5.902s     7.94x      17
    qt5base         0m57.590s   0m5.848s     9.85x      190
    sqlite          0m46.601s   0m5.816s     8.01x      24

Basically, displaying recursive dependencies, direct or reverse, is almost a constant now: it only slightly varies by about 10% depending on the complexity of the dependency chain, with the parsing of the Makefiles still accounting for the large majority of the time.

(PS. Thanks to Joseph for suggesting a list of interesting packages to test, and thanks to Trent for his example of memoisation!)

---
Changes v1 -> v2:
- use memoisation (Thanks Trent!)
- redo the timings
- drop the comments about skeleton and host-skeleton, that are no longer degenerate cases

---
 package/pkg-generic.mk | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

index 4353bd3868..0d17e62a70 100644
@@ -649,6 +649,18 @@ $(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \
 	) \
 	$$($(2)_FINAL_RECURSIVE_DEPENDENCIES__X))
 
+$(2)_FINAL_RECURSIVE_RDEPENDENCIES = $$(sort \
+	$$(if $$(filter undefined,$$(origin $(2)_FINAL_RECURSIVE_RDEPENDENCIES__X)), \
+		$$(eval $(2)_FINAL_RECURSIVE_RDEPENDENCIES__X := \
+			$$(foreach p, \
+				$$($(2)_RDEPENDENCIES), \
+				$$(p) \
+				$$($$(call UPPERCASE,$$(p))_FINAL_RECURSIVE_RDEPENDENCIES) \
+			) \
+		) \
+	) \
+	$$($(2)_FINAL_RECURSIVE_RDEPENDENCIES__X))
+
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
 $(2)_INSTALL_TARGET		?= YES

@@ -826,15 +838,13 @@ $(1)-show-depends:
 			@echo $$($(2)_FINAL_ALL_DEPENDENCIES)
 
 $(1)-show-recursive-depends:
-			@cd "$$(CONFIG_DIR)" && \
-			$$(TOPDIR)/support/scripts/graph-depends -p $(1) -f -q
+			@echo $$($(2)_FINAL_RECURSIVE_DEPENDENCIES)
 
 $(1)-show-rdepends:
 			@echo $$($(2)_RDEPENDENCIES)
 
 $(1)-show-recursive-rdepends:
-			@cd "$$(CONFIG_DIR)" && \
-			$$(TOPDIR)/support/scripts/graph-depends -p $(1) --reverse -f -q
+			@echo $$($(2)_FINAL_RECURSIVE_RDEPENDENCIES)
 
 $(1)-show-build-order: $$(patsubst %,%-show-build-order,$$($(2)_FINAL_ALL_DEPENDENCIES))
 	@:

2.14.1

Currently, when we need to build the full dependency graph, we call make to show the list of packages (make show-targets), and then call it again and again iteratively while it returns new packages.

Since calling make will parse the whole set of our Makefiles, this takes quite a bit of time (~4s each here), and the total can get pretty long.

However, make being make, already builds the whole dependency tree information, so we can just ask for it.

Add a new top-level rule 'show-dependency-tree' that displays the whole set of dependencies for all packages. For each package, its name, type and version is displayed, then all the direct, first-level dependencies are dumped. We choose a format that is not unlike the dot-graph format, because it is both easy to read as a human, and easy to parse as a machine:

    foo: target 1.2.3
    foo -> bar host-meh
    bar: target virtual
    bar -> buz
    buz: target 2.3.4
    buz ->
    host-meh: host virtual
    host-meh -> host-bleark
    host-bleark: host 3.4.5
    host-bleark ->
    rootfs-meh: host
    rootfs-meh -> host-bleark

To be noted: rootfs are currently reported as if they were 'host' packages, to stay aligned with how graph-depends currently treats them. Ideally, graph-depends could be enhanced to recognise them separately, but that is another story.

For just plain defconfig, which is about the smallest config we can have with an internal toolchain, we already have a seven-fold improvement (with the graph-depends rule modified to not run the pdf generation, to be able to just compare the tree generation):

    $ time make graph-depends
    real    0m27.344s
    $ time make show-dependency-tree
    real    0m3.848s

From defconfig, C++, wchar, locales, ssp, and allyespackageconfig, tweaked for even more packages (qt5 not qt4, luajit to avoid multi providers, etc...), the timings are (graph-depends still modified to not generate the pdf):

    $ time make graph-depends
    real    1m56.459s
    $ time make show-dependency-tree
    real    0m5.748s

There. I don't think those numbers need any explanation whatsoever; they do speak on their own. OK, for maths sake, the ratio is about twenty-fold. So, "yeah", I guess... ;-)

---
 Makefile               | 4 ++++
 fs/common.mk           | 8 ++++++++
 package/pkg-generic.mk | 4 ++++
 3 files changed, 16 insertions(+)

index 3d95d066fb..7d127e3799 100644
@@ -876,6 +876,10 @@ graph-depends-requirements:
 	@dot -? >/dev/null 2>&1 || \
 		{ echo "ERROR: The 'dot' program from Graphviz is needed for graph-depends" >&2; exit 1; }
 
+.PHONY: show-dependency-tree
+show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(PACKAGES) $(TARGETS_ROOTFS))
+	@:
+
 .PHONY: graph-depends
 graph-depends: graph-depends-requirements
 	@$(INSTALL) -d $(GRAPHS_DIR)

index a560417c6c..daa43efd75 100644
@@ -47,6 +47,10 @@ ROOTFS_COMMON_DEPENDENCIES = \
 	$(BR2_TAR_HOST_DEPENDENCY) \
 	$(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd)
 
+rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES))
+	$(info rootfs-common: host)
+	$(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
+
 .PHONY: rootfs-common
 rootfs-common: $(ROOTFS_COMMON_DEPENDENCIES) target-finalize
 	@$(call MESSAGE,"Generating root filesystems common tables")

@@ -80,6 +84,10 @@ ROOTFS_$(2)_TARGET_DIR = $$(ROOTFS_$(2)_DIR)/target
 
 ROOTFS_$(2)_DEPENDENCIES += rootfs-common
 
+rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES))
+	$$(info rootfs-$(1): host)
+	$$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
+
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
 ROOTFS_$(2)_COMPRESS_EXT = .gz
 ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c

index 0d17e62a70..62c2e221f7 100644
@@ -850,6 +850,10 @@ $(1)-show-build-order: $$(patsubst %,%-show-build-order,$$($(2)_FINAL_ALL_DEPEND
 	@:
 	$$(info $(1))
 
+$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_ALL_DEPENDENCIES))
+	$$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION)))
+	$$(info $(1) -> $$(foreach d,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(d)))
+
 $(1)-graph-depends: graph-depends-requirements
 	$(call pkg-graph-depends,$(1),--direct)
 

2.14.1

Now that we can get the whole dependency tree from make, use it to speed up things considerably.

So far, we had three functions to get the dependencies information:
get_depends(), get_rdepends(), and, somehow unrelated, get_version().

Because of the way %-show-{,r}depends works, getting the dependency tree was expensive, the three functions all took a set of packages for which to get the dependencies, in an attempt to limit the time it took to get that tree.

Now, getting the tree is much, much less costly, and we can get the whole tree as cheaply as we previously got only the first-level dependencies.

Furthemore, we can now also get the version information at the same time, and that also brings in whether the package is virtual or not, target or host.

So, we drop all three helper functions, and replace them with a single one that returns all that information in one go: full dependency tree, per-package type, and per-package version.

Note: since commit 2d29fd96a (pkg-virtual: remove VERSION/SOURCE), virtual packages are no longer reported as having a 'virtual' version, so have since been displayed as regular packages in the graphs. Although noone complained, this patch incidentally restores the initial behaviour, and virtual packages are now correctly displayed as such again.

---
Note: as I rewrote brpkgutil.py almost entirely now, and as I also substantially modified graph-depends, I added my (C) to them. We don't usually have copyright information in our files (no .mk or no Config.in and such have that), so I think it would be OK to just drop the (C) in there as well, especially since the Berne convention does not require it for a work to be protected and recognised anyway:
https://en.wikipedia.org/wiki/Copyright_symbol

Beside, the authorship information is already present in the git log, and it is much more accurate in there than it is in the files themselves.

---
 support/scripts/brpkgutil.py  | 107 ++++++++++++++++++--------------------
 support/scripts/graph-depends | 117 +++++-------------------------------------
 2 files changed, 64 insertions(+), 160 deletions(-)

index e70d525353..b5ea467bf9 100644
@@ -1,67 +1,60 @@
 # Copyright (C) 2010-2013 Thomas Petazzoni <[email protected]>
+# Copyright (C) 2018 Yann E. MORIN <[email protected]>
 
 import logging
-import sys
+import os
 import subprocess
 
 
-# Execute the "make <pkg>-show-version" command to get the version of a given
-# list of packages, and return the version formatted as a Python dictionary.
-def get_version(pkgs):
-    logging.info("Getting version for %s" % pkgs)
-    cmd = ["make", "-s", "--no-print-directory"]
-    for pkg in pkgs:
-        cmd.append("%s-show-version" % pkg)
-    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
-    output = p.communicate()[0]
-    if p.returncode != 0:
-        logging.error("Error getting version %s" % pkgs)
-        sys.exit(1)
-    output = output.split("\n")
-    if len(output) != len(pkgs) + 1:
-        logging.error("Error getting version")
-        sys.exit(1)
-    version = {}
-    for i in range(0, len(pkgs)):
-        pkg = pkgs[i]
-        version[pkg] = output[i]
-    return version
+# This function returns a tuple of three dictionaries, all using package
+# names as keys:
+# - a dictionary which values are the lists of packages that are the
+#   dependencies of the package used as key;
+# - a dictionary which values are the type of the package used as key;
+# - a dictionary which values are the version of the package used as key,
+#   'virtual' for a virtual package, or the empty string for a rootfs.
+#
+# 'direction' can be either 'direct' or 'forward' to get the direct (aka
+# forward) dependencies, or 'back' or 'reverse' to get the reverse (aka
+# backward) dependencies.
+def get_dependency_tree(direction="direct"):
+    logging.info("Getting dependency tree...")
 
-
-def _get_depends(pkgs, rule):
-    logging.info("Getting dependencies for %s" % pkgs)
-    cmd = ["make", "-s", "--no-print-directory"]
-    for pkg in pkgs:
-        cmd.append("%s-%s" % (pkg, rule))
-    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
-    output = p.communicate()[0]
-    if p.returncode != 0:
-        logging.error("Error getting dependencies %s\n" % pkgs)
-        sys.exit(1)
-    output = output.split("\n")
-    if len(output) != len(pkgs) + 1:
-        logging.error("Error getting dependencies")
-        sys.exit(1)
     deps = {}
-    for i in range(0, len(pkgs)):
-        pkg = pkgs[i]
-        pkg_deps = output[i].split(" ")
-        if pkg_deps == ['']:
-            deps[pkg] = []
+    types = {}
+    versions = {}
+
+    # Special case for the 'all' top-level fake package
+    deps['all'] = []
+    types['all'] = 'target'
+    versions['all'] = ''
+
+    cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"]
+    with open(os.devnull, 'wb') as devnull:
+        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull, universal_newlines=True)
+        output = p.communicate()[0]
+
+    for l in output.splitlines():
+        if " -> " in l:
+            pkg = l.split(" -> ")[0]
+            if direction == "forward" or direction == "direct":
+                try:
+                    deps[pkg] += l.split(" -> ")[1].split()
+                except KeyError:
+                    deps[pkg] = l.split(" -> ")[1].split()
+            elif direction == "back" or direction == "reverse":
+                for p in l.split(" -> ")[1].split():
+                    try:
+                        deps[p].append(pkg)
+                    except KeyError:
+                        deps[p] = [pkg]
+            else:
+                raise ValueError('direction must be one of: direct, forward, reverse, back')
         else:
-            deps[pkg] = pkg_deps
-    return deps
+            pkg, type_version = l.split(": ", 1)
+            t, v = "{} -".format(type_version).split(None, 2)[:2]
+            deps['all'].append(pkg)
+            types[pkg] = t
+            versions[pkg] = v
 
-
-# Execute the "make <pkg>-show-depends" command to get the list of
-# dependencies of a given list of packages, and return the list of
-# dependencies formatted as a Python dictionary.
-def get_depends(pkgs):
-    return _get_depends(pkgs, 'show-depends')
-
-
-# Execute the "make <pkg>-show-rdepends" command to get the list of
-# reverse dependencies of a given list of packages, and return the
-# list of dependencies formatted as a Python dictionary.
-def get_rdepends(pkgs):
-    return _get_depends(pkgs, 'show-rdepends')
+    return (deps, types, versions)

2.14.1

Hello,

Applied series until there.

However, I didn't apply this one, because it breaks "make <pkg>-graph-rdepends", with a "RuntimeError: maximum recursion depth exceeded". The attached defconfig allows to reproduce the issue with "make libusb-graph-rdepends".

Thanks!

Thomas

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: defconfig
Type: application/octet-stream
Size: 746 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190317/43727807/attachment.obj>