graph-depends: split off some functions to pkgutil.py
cover graph-depends: split off some functions to pkgutil.py
From: Thomas De Schampheleire <[email protected]>
The main goal of this patch set is to extract get_version and get_depends from graph-depends into a separate file pkgutil.py, for reuse by other scripts.
The script I need it for is not really upstreamable (opkg generation for some packages), but I think the changes I'm submitting here could be useful in other cases too.
Thomas De Schampheleire (3):
graph-depends: avoid use of global var 'rule' in get_depends
graph-depends: split off get_version/get_depends into pkgutil.py
graph-depends: split off get_rdepends from get_depends
support/scripts/graph-depends | 59 ++++----------------------------------- support/scripts/pkgutil.py | 64 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 53 deletions(-) create mode 100644 support/scripts/pkgutil.py
Signature · 1 line
2.10.2
PATCH 1/3 graph-depends: avoid use of global var 'rule' in get_depends
From: Thomas De Schampheleire <[email protected]>
Function get_depends was recently changed to support both normal dependencies as reverse dependencies, via a global variable 'rule' that equals 'show-depends' or 'show-rdepends'.
As a subsequent function will extract this function get_depends to a separate file, the use of globals is problematic.
Instead, pass the global as an argument.
--- support/scripts/graph-depends | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
support/scripts/graph-depends +2 -2
@@ -161,7 +161,7 @@ def get_targets(): # Execute the "make <pkg>-show-depends" command to get the list of # dependencies of a given list of packages, and return the list of # dependencies formatted as a Python dictionary. -def get_depends(pkgs): +def get_depends(pkgs, rule): sys.stderr.write("Getting dependencies for %s\n" % pkgs) cmd = ["make", "-s", "--no-print-directory" ] for pkg in pkgs:
@@ -204,7 +204,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs) + depends = get_depends(filtered_pkgs, rule) deps = set() for pkg in filtered_pkgs:
Signature · 1 line
2.10.2
Thomas Petazzoni
AppliedPATCH 1/3 graph-depends: avoid use of global var 'rule' in get_depends
Hello,
Quoted 9 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Function
get_dependswas recently changed to support both normal dependencies as reverse dependencies, via a global variable 'rule' that equals 'show-depends' or 'show-rdepends'.As a subsequent function will extract this function
get_dependsto a separate file, the use of globals is problematic.
Quoted 6 lines
Instead, pass the global as an argument.
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied to master, thanks. We really need to clean up this script to move the "global" code into a main() function. Right now there is some global code intermixed with sub-functions, it's horrible.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 1/3 graph-depends: avoid use of global var 'rule' in get_depends
Quoted 14 lines · Thomas Petazzoni
Hello,
Quoted 11 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Function
get_dependswas recently changed to support both normal dependencies as reverse dependencies, via a global variable 'rule' that equals 'show-depends' or 'show-rdepends'.As a subsequent function will extract this function
get_dependsto a separate file, the use of globals is problematic.Instead, pass the global as an argument.
Quoted 8 lines
Quoted 4 lines
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)Applied to master, thanks. We really need to clean up this script to move the "global" code into a
main()function. Right now there is some global code intermixed with sub-functions, it's horrible.
Thanks.
Yes, I agree to move the global code into main. I didn't do it in this series to not wander off too much of the real goal, but since you have the same idea I can do it as a follow-up later.
PATCH 1/1 support/scripts: split off get_version/get_depends into pkgutil.py
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into pkgutil.py.
--- support/scripts/graph-depends | 55 ++++--------------------------------------- support/scripts/pkgutil.py | 55 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 51 deletions(-) create mode 100644 support/scripts/pkgutil.py
support/scripts/graph-depends +4 -51
@@ -26,6 +26,8 @@ import subprocess import argparse from fnmatch import fnmatch +import pkgutil + # Modes of operation: MODE_FULL = 1 # draw full dependency graph for all selected packages MODE_PKG = 2 # draw dependency graph for a given package
@@ -122,28 +124,6 @@ host_colour = colours[2] allpkgs = [] -# Execute the "make <pkg>-show-version" command to get the version of a given -# list of packages, and return the version formatted as a Python dictionary. -def get_version(pkgs): - sys.stderr.write("Getting version for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-show-version" % pkg) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting version %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting version\n") - sys.exit(1) - version = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - version[pkg] = output[i] - return version - # Execute the "make show-targets" command to get the list of the main # Buildroot PACKAGES and return it formatted as a Python list. This # list is used as the starting point for full dependency graphs
@@ -158,33 +138,6 @@ def get_targets(): return [] return output.split(' ') -# Execute the "make <pkg>-show-depends" command to get the list of -# dependencies of a given list of packages, and return the list of -# dependencies formatted as a Python dictionary. -def get_depends(pkgs): - sys.stderr.write("Getting dependencies for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-%s" % (pkg, rule)) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting dependencies %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting dependencies\n") - sys.exit(1) - deps = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - pkg_deps = output[i].split(" ") - if pkg_deps == ['']: - deps[pkg] = [] - else: - deps[pkg] = pkg_deps - return deps - # Recursive function that builds the tree of dependencies for a given # list of packages. The dependencies are built in a list called # 'dependencies', which contains tuples of the form (pkg1 ->
@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs) + depends = pkgutil.get_depends(filtered_pkgs) deps = set() for pkg in filtered_pkgs:
@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgs if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-color
support/scripts/pkgutil.py new file +55 -0
55 lines not on this page · raw message
Signature · 1 line
2.10.2
PATCH 1/1 support/scripts: split off get_version/get_depends into pkgutil.py
Quoted 6 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.
Quoted 7 lines
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 55 ++++--------------------------------------- support/scripts/pkgutil.py | 55 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 51 deletions(-) create mode 100644 support/scripts/pkgutil.py
Please disregard this specific patch, it does not work on current Buildroot. It was accidentally sent along with the real 3-patch series (which does work).
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into pkgutil.py.
--- support/scripts/graph-depends | 55 ++++--------------------------------------- support/scripts/pkgutil.py | 55 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 51 deletions(-) create mode 100644 support/scripts/pkgutil.py
support/scripts/graph-depends +4 -51
@@ -26,6 +26,8 @@ import subprocess import argparse from fnmatch import fnmatch +import pkgutil + # Modes of operation: MODE_FULL = 1 # draw full dependency graph for all selected packages MODE_PKG = 2 # draw dependency graph for a given package
@@ -122,28 +124,6 @@ host_colour = colours[2] allpkgs = [] -# Execute the "make <pkg>-show-version" command to get the version of a given -# list of packages, and return the version formatted as a Python dictionary. -def get_version(pkgs): - sys.stderr.write("Getting version for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-show-version" % pkg) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting version %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting version\n") - sys.exit(1) - version = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - version[pkg] = output[i] - return version - # Execute the "make show-targets" command to get the list of the main # Buildroot PACKAGES and return it formatted as a Python list. This # list is used as the starting point for full dependency graphs
@@ -158,33 +138,6 @@ def get_targets(): return [] return output.split(' ') -# Execute the "make <pkg>-show-depends" command to get the list of -# dependencies of a given list of packages, and return the list of -# dependencies formatted as a Python dictionary. -def get_depends(pkgs, rule): - sys.stderr.write("Getting dependencies for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-%s" % (pkg, rule)) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting dependencies %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting dependencies\n") - sys.exit(1) - deps = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - pkg_deps = output[i].split(" ") - if pkg_deps == ['']: - deps[pkg] = [] - else: - deps[pkg] = pkg_deps - return deps - # Recursive function that builds the tree of dependencies for a given # list of packages. The dependencies are built in a list called # 'dependencies', which contains tuples of the form (pkg1 ->
@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs, rule) + depends = pkgutil.get_depends(filtered_pkgs, rule) deps = set() for pkg in filtered_pkgs:
@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgs if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-color
support/scripts/pkgutil.py new file +55 -0
55 lines not on this page · raw message
Signature · 1 line
2.10.2
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Thomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
[--SNIP--]
Quoted 1 line
+import pkgutil
So, I always claimed I was not a Python expoert, but should we not do:
from okgutil import get_versions get_depends
of something similar?
[--SNIP--]
support/scripts/graph-depends def get_all_depends(pkgs): · line 160
Quoted 6 lines
support/scripts/graph-depends +1
-1@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs, rule) + depends = pkgutil.get_depends(filtered_pkgs, rule)
And then just use get_depends here?
support/scripts/graph-depends if check_only: · line 333
Quoted 8 lines
support/scripts/graph-depends +1
-1deps = set() for pkg in filtered_pkgs:@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgs
Ditto?
support/scripts/pkgutil.py line 1
Quoted 10 lines
support/scripts/graph-depends +0
-0if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-colorsupport/scripts/pkgutil.py new file +1
-0@@ -0,0 +1,55 @@ +#!/usr/bin/env python
As I understand it, this file is not supposed to be standalone, but just a "library" of functions?
So it should not need to have the sha-bang line, yes?
Otherwise, I'm not opposed to this.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Quoted 11 lines · Yann E. MORIN
Thomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
[--SNIP--]Quoted 1 line
+import pkgutil
So, I always claimed I was not a Python expoert, but should we not do:
from okgutil import
get_versionsget_dependsof something similar?
Both methods are technically possible, and I think this is partially a matter of personal/project preference, and partially some general guidelines.
When using 'import foo', any symbol used from foo needs to be qualified explicitly, i.e. foo.one, foo.two. You could consider it as making a namespace accessible but keeping everything inside that namespace.
With 'from foo import one, two' you allow future usages to be done without referring to foo anymore. I consider it as 'using namespace foo' in C++ terminology.
One could argue about performance of one vs the other, but I think it hardly matters (although I did never measure that).
Personally, I'm more in favor of 'import foo' and keeping explicit references for following reasons:
1. it makes it clear inside the code where a symbol comes from -- in general I consider 'using namespace foo' a bad thing.
2. maintainability: if more functions are added to foo and used from a script, the list in the import statement becomes larger and larger, to possibly ridiculous proportions.
There is one case where I do use 'from Foo import Foo' and that is when Foo is a class that is saved in its own equally-named file Foo.py. In this case, Foo.Foo seems a bit redundant and it makes sense to make the class Foo directly accessible.
I found the following 100% reliable reference (ahum) about the topic:
http://stackoverflow.com/questions/710551/import-module-or-from-module-import
Quoted 36 lines
[--SNIP--]
support/scripts/graph-depends def get_all_depends(pkgs): · line 160
Quoted 6 lines
support/scripts/graph-depends +1
-1@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs, rule) + depends = pkgutil.get_depends(filtered_pkgs, rule)And then just use
get_dependshere?support/scripts/graph-depends if check_only: · line 333
Quoted 8 lines
support/scripts/graph-depends +1
-1deps = set() for pkg in filtered_pkgs:@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgsDitto?
support/scripts/pkgutil.py line 1
Quoted 10 lines
support/scripts/graph-depends +0
-0if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-colorsupport/scripts/pkgutil.py new file +1
-0@@ -0,0 +1,55 @@ +#!/usr/bin/env pythonAs I understand it, this file is not supposed to be standalone, but just a "library" of functions?
Quoted 1 line
So it should not need to have the sha-bang line, yes?
Yes, you are right, that line is not necessary.
Quoted 2 lines
Otherwise, I'm not opposed to this.
Thanks!
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Thomas DS, All,
On 2017-02-05 22:27 +0100, Thomas De Schampheleire spake thusly:
Quoted 9 lines · Yann E. MORIN
Quoted 8 lines
Thomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
[--SNIP--]Quoted 1 line
+import pkgutil
So, I always claimed I was not a Python expoert, but should we not do:
Quoted 7 lines
Quoted 3 lines
from okgutil import
get_versionsget_dependsof something similar?
Both methods are technically possible, and I think this is partially a matter of personal/project preference, and partially some general guidelines.
Ok for me, then.
Regards,
Yann E. MORIN.
Quoted 73 lines
When using 'import foo', any symbol used from foo needs to be qualified explicitly, i.e. foo.one, foo.two. You could consider it as making a namespace accessible but keeping everything inside that namespace.
With 'from foo import one, two' you allow future usages to be done without referring to foo anymore. I consider it as 'using namespace foo' in C++ terminology.
One could argue about performance of one vs the other, but I think it hardly matters (although I did never measure that).
Personally, I'm more in favor of 'import foo' and keeping explicit references for following reasons:
1. it makes it clear inside the code where a symbol comes from -- in general I consider 'using namespace foo' a bad thing.
2. maintainability: if more functions are added to foo and used from a script, the list in the import statement becomes larger and larger, to possibly ridiculous proportions.There is one case where I do use 'from Foo import Foo' and that is when Foo is a class that is saved in its own equally-named file
Foo.py. In this case, Foo.Foo seems a bit redundant and it makes sense to make the class Foo directly accessible.I found the following 100% reliable reference (ahum) about the topic:
http://stackoverflow.com/questions/710551/import-module-or-from-module-importQuoted 36 lines
[--SNIP--]
support/scripts/graph-depends def get_all_depends(pkgs): · line 160
Quoted 6 lines
support/scripts/graph-depends +1
-1@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs, rule) + depends = pkgutil.get_depends(filtered_pkgs, rule)And then just use
get_dependshere?support/scripts/graph-depends if check_only: · line 333
Quoted 8 lines
support/scripts/graph-depends +1
-1deps = set() for pkg in filtered_pkgs:@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgsDitto?
support/scripts/pkgutil.py line 1
Quoted 10 lines
support/scripts/graph-depends +0
-0if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-colorsupport/scripts/pkgutil.py new file +1
-0@@ -0,0 +1,55 @@ +#!/usr/bin/env pythonAs I understand it, this file is not supposed to be standalone, but just a "library" of functions?
Quoted 1 line
So it should not need to have the sha-bang line, yes?
Yes, you are right, that line is not necessary.
Quoted 2 lines
Otherwise, I'm not opposed to this.
Thanks!
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Thomas Petazzoni
AppliedPATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
In reply to Thomas De SchampheleireHello,
Quoted 6 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.
Quoted 6 lines
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 55 ++++--------------------------------------- support/scripts/pkgutil.py | 55 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 51 deletions(-) create mode 100644 support/scripts/pkgutil.py
Applied to master, after removing the shebang from pkgutil.py as suggested by Yann.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
In reply to Thomas De SchampheleireThomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.
Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Regards,
Yann E. MORIN.
Quoted 165 lines
Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 55 ++++--------------------------------------- support/scripts/pkgutil.py | 55 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 51 deletions(-) create mode 100644 support/scripts/pkgutil.pysupport/scripts/graph-depends +4
-51@@ -26,6 +26,8 @@ import subprocess import argparse from fnmatch import fnmatch +import pkgutil + # Modes of operation: MODE_FULL = 1 # draw full dependency graph for all selected packages MODE_PKG = 2 # draw dependency graph for a given package@@ -122,28 +124,6 @@ host_colour = colours[2] allpkgs = [] -# Execute the "make <pkg>-show-version" command to get the version of a given -# list of packages, and return the version formatted as a Python dictionary. -def get_version(pkgs): - sys.stderr.write("Getting version for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-show-version" % pkg) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting version %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting version\n") - sys.exit(1) - version = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - version[pkg] = output[i] - return version - # Execute the "make show-targets" command to get the list of the main # Buildroot PACKAGES and return it formatted as a Python list. This # list is used as the starting point for full dependency graphs@@ -158,33 +138,6 @@ def get_targets(): return [] return output.split(' ') -# Execute the "make <pkg>-show-depends" command to get the list of -# dependencies of a given list of packages, and return the list of -# dependencies formatted as a Python dictionary. -def get_depends(pkgs, rule): - sys.stderr.write("Getting dependencies for %s\n" % pkgs) - cmd = ["make", "-s", "--no-print-directory" ] - for pkg in pkgs: - cmd.append("%s-%s" % (pkg, rule)) - p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) - output = p.communicate()[0] - if p.returncode != 0: - sys.stderr.write("Error getting dependencies %s\n" % pkgs) - sys.exit(1) - output = output.split("\n") - if len(output) != len(pkgs) + 1: - sys.stderr.write("Error getting dependencies\n") - sys.exit(1) - deps = {} - for i in range(0, len(pkgs)): - pkg = pkgs[i] - pkg_deps = output[i].split(" ") - if pkg_deps == ['']: - deps[pkg] = [] - else: - deps[pkg] = pkg_deps - return deps - # Recursive function that builds the tree of dependencies for a given # list of packages. The dependencies are built in a list called # 'dependencies', which contains tuples of the form (pkg1 ->@@ -204,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = get_depends(filtered_pkgs, rule) + depends = pkgutil.get_depends(filtered_pkgs, rule) deps = set() for pkg in filtered_pkgs:@@ -377,7 +330,7 @@ if check_only: sys.exit(0) dict_deps = remove_extra_deps(dict_deps) -dict_version = get_version([pkg for pkg in allpkgs +dict_version = pkgutil.get_version([pkg for pkg in allpkgs if pkg != "all" and not pkg.startswith("root")]) # Print the attributes of a node: label and fill-colorsupport/scripts/pkgutil.py new file +55
-0@@ -0,0 +1,55 @@ +#!/usr/bin/env python + +# Copyright (C) 2010-2013 Thomas Petazzoni <[email protected]> + +import sys +import subprocess + +# Execute the "make <pkg>-show-version" command to get the version of a given +# list of packages, and return the version formatted as a Python dictionary. +def get_version(pkgs): + sys.stderr.write("Getting version for %s\n" % pkgs) + cmd = ["make", "-s", "--no-print-directory" ] + for pkg in pkgs: + cmd.append("%s-show-version" % pkg) + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) + output = p.communicate()[0] + if p.returncode != 0: + sys.stderr.write("Error getting version %s\n" % pkgs) + sys.exit(1) + output = output.split("\n") + if len(output) != len(pkgs) + 1: + sys.stderr.write("Error getting version\n") + sys.exit(1) + version = {} + for i in range(0, len(pkgs)): + pkg = pkgs[i] + version[pkg] = output[i] + return version + +# Execute the "make <pkg>-show-depends" command to get the list of +# dependencies of a given list of packages, and return the list of +# dependencies formatted as a Python dictionary. +def get_depends(pkgs, rule): + sys.stderr.write("Getting dependencies for %s\n" % pkgs) + cmd = ["make", "-s", "--no-print-directory" ] + for pkg in pkgs: + cmd.append("%s-%s" % (pkg, rule)) + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True) + output = p.communicate()[0] + if p.returncode != 0: + sys.stderr.write("Error getting dependencies %s\n" % pkgs) + sys.exit(1) + output = output.split("\n") + if len(output) != len(pkgs) + 1: + sys.stderr.write("Error getting dependencies\n") + sys.exit(1) + deps = {} + for i in range(0, len(pkgs)): + pkg = pkgs[i] + pkg_deps = output[i].split(" ") + if pkg_deps == ['']: + deps[pkg] = [] + else: + deps[pkg] = pkg_deps + return depsSignature · 1 line
2.10.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Hi Yann,
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 6 lines
Thomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 2 lines
From: Thomas De Schampheleire <[email protected]>
Quoted 8 lines
Quoted 3 lines
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).
Best regards,
Thomas
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Thomas, All,
On 2018-11-07 20:06 +0100, Thomas De Schampheleire spake thusly:
Quoted 14 lines
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 11 lines
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Quoted 5 lines
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).
So, if you were to get a new function that would return basically the same, but in another format, but much quicker (~4s instead of ~45s), would that be something you could adapt to?
I'm changing the way the dependency tree is extracted from the Makefile data, and the function now has this API (function name yet to be bike-shedded about):
dict_deps, dict_types = get_dependency_tree(direction)
where:
- direction is either string 'forward' or 'back',
- dict_deps is a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, what get_all_depends() currently returns, but the whole dependency tree)
- dict_types is a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.
If needed, I could very easily make it return a three-tuple, with the third one being dict_versions, keyed by package names, with values the package version.
Would that be something that would be usable in your use case?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Hi Yann,
El mi?., 7 nov. 2018 a las 20:39, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 34 lines
Thomas, All,
On 2018-11-07 20:06 +0100, Thomas De Schampheleire spake thusly:
Quoted 14 lines
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 11 lines
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Quoted 5 lines
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).So, if you were to get a new function that would return basically the same, but in another format, but much quicker (~4s instead of ~45s), would that be something you could adapt to?
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):
dict_deps,dict_types=get_dependency_tree(direction)
Quoted 8 lines
where:
- direction is either string 'forward' or 'back',
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)
How is dict_deps different from what get_depends in brpkgutil.py currently returns? Your statement "but the whole dependency tree" is not clear to me.
Quoted 11 lines
-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.If needed, I could very easily make it return a three-tuple, with the third one being
dict_versions, keyed by package names, with values the package version.Would that be something that would be usable in your use case?
To help the discussion, here are some snippets from our code:
Package.versions = brpkgutil.get_version(list(Package.files.keys())) Package.depends = brpkgutil.get_depends(list(Package.files.keys()))
...
self.version = Package.versions[self.pkg] # this
variable will be used in several places, we just need the string representing the version
...
def get_opkg_depends(self):
"""Return the dependencies that need to be declared in the opkg""" deps = set(Package.depends[self.pkg]).intersection(set(Package.requested_pkgs)) if deps is not None:
return ','.join(deps)
else:
return ''
The code above is basically all we do with the version and dependencies. As long as we can get this info in some way, I'm fine with changes. Returning the versions together with the dependencies in one call, removing the call to brpkgutil.get_version is fine by me, if it is better for the in-tree usage (I'm fine with either way).
Thanks,
Thomas
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
On 2018-11-07 21:41 +0100, Thomas De Schampheleire spake thusly:
Quoted 37 lines
El mi?., 7 nov. 2018 a las 20:39, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 35 lines
On 2018-11-07 20:06 +0100, Thomas De Schampheleire spake thusly:
Quoted 14 lines
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 11 lines
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Quoted 5 lines
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).So, if you were to get a new function that would return basically the same, but in another format, but much quicker (~4s instead of ~45s), would that be something you could adapt to?
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):
dict_deps,dict_types=get_dependency_tree(direction)where:
- direction is either string 'forward' or 'back',
Quoted 8 lines
Quoted 4 lines
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)How is
dict_depsdifferent from whatget_dependsinbrpkgutil.pycurrently returns? Your statement "but the whole dependency tree" is not clear to me.
Currently, brpkgutil.get_depends() takes a list of packages for which to get the dependencies. The new get_dependency_tree() does not take such a list; it always returns the whole dependency tree (i.e. from the 'ALL' meta package).
Quoted 9 lines
Quoted 9 lines
-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.If needed, I could very easily make it return a three-tuple, with the third one being
dict_versions, keyed by package names, with values the package version.
Quoted 6 lines
Quoted 1 line
Would that be something that would be usable in your use case?
To help the discussion, here are some snippets from our code:
Package.versions = brpkgutil.get_version(list(Package.files.keys())) Package.depends = brpkgutil.get_depends(list(Package.files.keys()))
So you could write:
Package.depends, Package.versions = brpkgutil.get_dependency_tree('forward')
Quoted 12 lines
self.version = Package.versions[self.pkg] # this
variable will be used in several places, we just need the string representing the version
...
defget_opkg_depends(self):
"""Return the dependencies that need to be declared in the opkg""" deps = set(Package.depends[self.pkg]).intersection(set(Package.requested_pkgs)) if deps is not None:
return ','.join(deps)
else:
return ''
Quoted 5 lines
The code above is basically all we do with the version and dependencies. As long as we can get this info in some way, I'm fine with changes. Returning the versions together with the dependencies in one call, removing the call to brpkgutil.get_version is fine by me, if it is better for the in-tree usage (I'm fine with either way).
OK, good.
Currently, in Buildroot, we use the version string just because we need to know whether a package is virutal or not. Do you really need the version string for what it is, or just to test if it is virtual or not?
Regards,
Yann E. MORIN.
Quoted 2 lines
Thanks,
Thomas
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
El mi?., 7 nov. 2018 a las 22:07, Yann E. MORIN
(<[email protected]>) escribi?:
Quoted 44 lines
On 2018-11-07 21:41 +0100, Thomas De Schampheleire spake thusly:
Quoted 42 lines
El mi?., 7 nov. 2018 a las 20:39, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 39 lines
On 2018-11-07 20:06 +0100, Thomas De Schampheleire spake thusly:
Quoted 14 lines
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 11 lines
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Quoted 5 lines
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).So, if you were to get a new function that would return basically the same, but in another format, but much quicker (~4s instead of ~45s), would that be something you could adapt to?
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):
dict_deps,dict_types=get_dependency_tree(direction)where:
- direction is either string 'forward' or 'back',
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)
Quoted 8 lines
Quoted 3 lines
How is
dict_depsdifferent from whatget_dependsinbrpkgutil.pycurrently returns? Your statement "but the whole dependency tree" is not clear to me.Currently,
brpkgutil.get_depends()takes a list of packages for which to get the dependencies. The newget_dependency_tree()does not take such a list; it always returns the whole dependency tree (i.e. from the 'ALL' meta package).
Ok, thanks.
Quoted 39 lines
Quoted 10 lines
Quoted 10 lines
-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.If needed, I could very easily make it return a three-tuple, with the third one being
dict_versions, keyed by package names, with values the package version.Quoted 6 lines
Quoted 1 line
Would that be something that would be usable in your use case?
To help the discussion, here are some snippets from our code:
Package.versions = brpkgutil.get_version(list(Package.files.keys())) Package.depends = brpkgutil.get_depends(list(Package.files.keys()))
So you could write:
Package.depends, Package.versions =
brpkgutil.get_dependency_tree('forward')Quoted 12 lines
self.version = Package.versions[self.pkg] # this
variable will be used in several places, we just need the string representing the version
...
defget_opkg_depends(self):
"""Return the dependencies that need to be declared in the opkg""" deps = set(Package.depends[self.pkg]).intersection(set(Package.requested_pkgs)) if deps is not None:
return ','.join(deps)
else:
return ''Quoted 5 lines
The code above is basically all we do with the version and dependencies. As long as we can get this info in some way, I'm fine with changes. Returning the versions together with the dependencies in one call, removing the call to brpkgutil.get_version is fine by me, if it is better for the in-tree usage (I'm fine with either way).
Quoted 5 lines
OK, good.
Currently, in Buildroot, we use the version string just because we need to know whether a package is virutal or not. Do you really need the version string for what it is, or just to test if it is virtual or not?
We do need the version, which is used in the .ipk filename as well as in its metadata.
Whether the package is virtual or not is not a concern for us.
Thanks,
Thomas
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
In reply to Yann E. MORINQuoted 23 lines · Yann E. MORIN
Thomas, All,
On 2018-11-07 20:06 +0100, Thomas De Schampheleire spake thusly:
Quoted 14 lines
El mi?., 7 nov. 2018 a las 19:07, Yann E. MORIN
(<[email protected]>) escribi?:Quoted 11 lines
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 5 lines
From: Thomas De Schampheleire <[email protected]>
Functions to obtain the version and dependencies of a package from Python can be useful for several scripts. Extract this logic out of graph-depends into
pkgutil.py.Coming back to this script, because I'm rewriting the way graph-depends gets the dependency tree. When you said "useful for several scripts," did you expect it to be useful to scripts that are not in Buildroot (e.g. user-local scripts)?
Quoted 5 lines
Yes, exactly. We are using the logic from pkgutil from another python script and want to avoid code duplication.
This particular script is not something that upstream accepts, i.e. generating opkg files for specific packages (and for that, we need to know the dependencies and versions of each package).
Quoted 7 lines
So, if you were to get a new function that would return basically the same, but in another format, but much quicker (~4s instead of ~45s), would that be something you could adapt to?
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):
Let me take this opportunity to bikeshed on this right away :-)
Quoted 2 lines
dict_deps,dict_types=get_dependency_tree(direction)
I like the function name! However...
Quoted 4 lines
where:
- direction is either string 'forward' or 'back',
I would make this a boolean argument:
def get_dependency_tree(backward: bool = True) -> Dict[str, Package]:
Note however that it is trivial to populate the backward dependencies once you
have the forward ones, so I'm not sure its worth it to make the distinction.
Quoted 10 lines
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.
I would take this opportunity to create a class Package with members:
name: str # Package name including hsot- prefix
dependencies: Set[Package] # Forward dependenciesdependencies_backwards: Set[Package] # Backward dependencies
virtual: bool # If true, it's a virtual package
host: bool # If true, it's a host package
version: str # Package version
The __init__ would set name, virtual, host, version. The dependencies are added
dynamically, and whenever you add one you do it in both directions.
Regards,
Arnout
Quoted 9 lines
If needed, I could very easily make it return a three-tuple, with the third one being
dict_versions, keyed by package names, with values the package version.Would that be something that would be usable in your use case?
Regards,
Yann E. MORIN.
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Arnout, All,
On 2018-11-07 23:26 +0100, Arnout Vandecappelle spake thusly:
Quoted 5 lines · Yann E. MORIN
Quoted 3 lines
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):Let me take this opportunity to bikeshed on this right away :-)
If that were not you, then who else? ;-)
Quoted 2 lines
Quoted 1 line
dict_deps,dict_types=get_dependency_tree(direction)I like the function name! However...
Oh come on, that *is* the most important thing! ;-)
Quoted 5 lines
Quoted 3 lines
where:
- direction is either string 'forward' or 'back',
I would make this a boolean argument:
defget_dependency_tree(backward: bool = True)-> Dict[str, Package]:
Just for my own understanding: what syntax is that? I was trying to write the protoype of the function, but in Python there is no prototype where you can explain the return type. Yours seems like it is what I was looking for. Is it described somewhere?
(See how I'm trying to sidetrack you into not bikeshedding? ;-))
Quoted 2 lines
Note however that it is trivial to populate the backward dependencies once you
have the forward ones, so I'm not sure its worth it to make the distinction.
I was also thinking about that too.
Quoted 10 lines
Quoted 9 lines
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.
Quoted 8 lines
I would take this opportunity to create a class Package with members:
name: str # Package name including hsot- prefix
dependencies: Set[Package] # Forward dependenciesdependencies_backwards: Set[Package] # Backward dependencies
virtual: bool # If true, it's a virtual package
host: bool # If true, it's a host package
version: str # Package version
Meh, I was thinking about exactly the same thing, but it is quite an invasive change. But let me bikeshed in turn: I would make your proposed 'host' field an enum instead, because we don't have only packages, but also rootfs. So I would have 'kind' (or 'type' if that's not reserved?) which could be one of 'target','host', or 'rootfs'.
Quoted 2 lines
The
__init__would set name, virtual, host, version. The dependencies are added
dynamically, and whenever you add one you do it in both directions.
Yep.
Dang, I'm always getting side-tracked into side topics. All I wanted was a faster graph-depends, and it's already there now! Sob... ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Quoted 18 lines · Yann E. MORIN
Arnout, All,
On 2018-11-07 23:26 +0100, Arnout Vandecappelle spake thusly:
Quoted 5 lines · Yann E. MORIN
Quoted 3 lines
I'm changing the way the dependency tree is extracted from the
Makefiledata, and the function now has this API (function name yet to be bike-shedded about):Let me take this opportunity to bikeshed on this right away :-)
If that were not you, then who else? ;-)
Quoted 2 lines
Quoted 1 line
dict_deps,dict_types=get_dependency_tree(direction)I like the function name! However...
Oh come on, that *is* the most important thing! ;-)
Quoted 2 lines
Quoted 2 lines
where:
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
- direction is either string 'forward' or 'back',
I would make this a boolean argument:
defget_dependency_tree(backward: bool = True)-> Dict[str, Package]:Just for my own understanding: what syntax is that? I was trying to write the protoype of the function, but in Python there is no prototype where you can explain the return type. Yours seems like it is what I was looking for. Is it described somewhere?
Those are type annotations. They don't do anything by themselves, but they can
be used by other tools/libraries to do static type checking.
Of course, they can't be used in Python 2.ancient so you can't actually use
them in graph-depends, but it's a good way of documenting the intention.
For documentation, I'll refer to PEP 484 [1].
Quoted 26 lines
(See how I'm trying to sidetrack you into not bikeshedding? ;-))
Quoted 2 lines
Note however that it is trivial to populate the backward dependencies once you
have the forward ones, so I'm not sure its worth it to make the distinction.I was also thinking about that too.
Quoted 10 lines
Quoted 9 lines
-
dict_depsis a dictionnary, which keys are the package names, and which values are lists of packages that are direct dependencies of the key package (basically, whatget_all_depends()currently returns, but the whole dependency tree)-
dict_typesis a dictionnary, which keys are the package names, and the values are string representing whtehr the packages are target or host, and virtual or not, e.g.: 'target', 'target-virtual', 'host', 'host-virtual'.Quoted 8 lines
I would take this opportunity to create a class Package with members:
name: str # Package name including hsot- prefix
dependencies: Set[Package] # Forward dependenciesdependencies_backwards: Set[Package] # Backward dependencies
virtual: bool # If true, it's a virtual package
host: bool # If true, it's a host package
version: str # Package version
Quoted 2 lines
Meh, I was thinking about exactly the same thing, but it is quite an invasive change.
I would think that the invasive change you're about to do is the ideal occasion
to do this one :-)
Quoted 2 lines
But let me bikeshed in turn: I would make your proposed 'host' field an enum instead,
Python enums require an external package that we can't assume is installed. But
I suppose you didn't mean an actual enum :-P
Quoted 3 lines
because we don't have only packages, but
also rootfs. So I would have 'kind' (or 'type' if that's not reserved?) which could be one of 'target','host', or 'rootfs'.
Sounds good to me. 'kind' is better than 'type' IMHO.
Quoted 7 lines
Quoted 2 lines
The
__init__would set name, virtual, host, version. The dependencies are added
dynamically, and whenever you add one you do it in both directions.Yep.
Dang, I'm always getting side-tracked into side topics. All I wanted was a faster graph-depends, and it's already there now! Sob... ;-)
Feel free to ignore my feedback! I'll take progress over perfection any time.
Regards,
Arnout
PATCH 2/3 graph-depends: split off get_version/get_depends into pkgutil.py
Arnout, All,
On 2018-11-08 09:01 +0100, Arnout Vandecappelle spake thusly:
Quoted 7 lines · Yann E. MORIN
Quoted 5 lines
Quoted 1 line
def
get_dependency_tree(backward: bool = True)-> Dict[str, Package]:Just for my own understanding: what syntax is that? I was trying to write the protoype of the function, but in Python there is no prototype where you can explain the return type. Yours seems like it is what I was looking for. Is it described somewhere?
For documentation, I'll refer to PEP 484 [1].
ACK, thanks. :-)
[--SNIP--]
Quoted 5 lines
Quoted 2 lines
But let me bikeshed in turn: I would make your proposed 'host' field an enum instead,
Python enums require an external package that we can't assume is installed. But
I suppose you didn't mean an actual enum :-P
Yeah, I knew, I was thinking 'emulated' enum, as class-level constants, like:
class Package():
T_UNKNOWN_PKG = 0
T_HOST_PKG = 1
T_TARGET_PK = 2
[...]and then refer to them as Package.T_HOST_PKG etc...
(but of course, they are not constant! they are just class-variables with default values.)
Quoted 3 lines
Quoted 2 lines
Dang, I'm always getting side-tracked into side topics. All I wanted was a faster graph-depends, and it's already there now! Sob... ;-)
Feel free to ignore my feedback! I'll take progress over perfection any time.
Eh! ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 3/3 graph-depends: split off get_rdepends from get_depends
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
The use of a 'rule' variable that can contain 'show-depends' or 'show-rdepends' is not logical if get_depends is considered as a reusable function from various scripts. The name of these rules are too much an implementation detail.
Therefore, split the existing get_depends into two separate functions get_depends and get_rdepends, while keeping code duplication to a minimum.
--- support/scripts/graph-depends | 6 +++--- support/scripts/pkgutil.py | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-)
support/scripts/graph-depends +3 -3
@@ -102,13 +102,13 @@ else: transitive = args.transitive if args.direct: - rule = "show-depends" + get_depends_func = pkgutil.get_depends arrow_dir = "forward" else: if mode == MODE_FULL: sys.stderr.write("--reverse needs a package\n") sys.exit(1) - rule = "show-rdepends" + get_depends_func = pkgutil.get_rdepends arrow_dir = "back" # Get the colours: we need exactly three colours,
@@ -157,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = pkgutil.get_depends(filtered_pkgs, rule) + depends = get_depends_func(filtered_pkgs) deps = set() for pkg in filtered_pkgs:
support/scripts/pkgutil.py +13 -4
@@ -27,10 +27,7 @@ def get_version(pkgs): version[pkg] = output[i] return version -# Execute the "make <pkg>-show-depends" command to get the list of -# dependencies of a given list of packages, and return the list of -# dependencies formatted as a Python dictionary. -def get_depends(pkgs, rule): +def _get_depends(pkgs, rule): sys.stderr.write("Getting dependencies for %s\n" % pkgs) cmd = ["make", "-s", "--no-print-directory" ] for pkg in pkgs:
@@ -53,3 +50,15 @@ def get_depends(pkgs, rule): else: deps[pkg] = pkg_deps return deps + +# Execute the "make <pkg>-show-depends" command to get the list of +# dependencies of a given list of packages, and return the list of +# dependencies formatted as a Python dictionary. +def get_depends(pkgs): + return _get_depends(pkgs, 'show-depends') + +# Execute the "make <pkg>-show-rdepends" command to get the list of +# reverse dependencies of a given list of packages, and return the +# list of dependencies formatted as a Python dictionary. +def get_rdepends(pkgs): + return _get_depends(pkgs, 'show-rdepends')
Signature · 1 line
2.10.2
PATCH 3/3 graph-depends: split off get_rdepends from get_depends
Thomas DS, All,
On 2017-02-03 21:57 +0100, Thomas De Schampheleire spake thusly:
Quoted 11 lines
From: Thomas De Schampheleire <[email protected]>
The use of a 'rule' variable that can contain 'show-depends' or 'show-rdepends' is not logical if
get_dependsis considered as a reusable function from various scripts. The name of these rules are too much an implementation detail.Therefore, split the existing
get_dependsinto two separate functionsget_dependsandget_rdepends, while keeping code duplication to a minimum.Signed-off-by Thomas De Schampheleire <[email protected]>
Regards,
Yann E. MORIN.
Quoted 69 lines
--- support/scripts/graph-depends | 6 +++--- support/scripts/pkgutil.py | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-)support/scripts/graph-depends +3
-3@@ -102,13 +102,13 @@ else: transitive = args.transitive if args.direct: - rule = "show-depends" + get_depends_func = pkgutil.get_depends arrow_dir = "forward" else: if mode == MODE_FULL: sys.stderr.write("--reverse needs a package\n") sys.exit(1) - rule = "show-rdepends" + get_depends_func = pkgutil.get_rdepends arrow_dir = "back" # Get the colours: we need exactly three colours,@@ -157,7 +157,7 @@ def get_all_depends(pkgs): if len(filtered_pkgs) == 0: return [] - depends = pkgutil.get_depends(filtered_pkgs, rule) + depends = get_depends_func(filtered_pkgs) deps = set() for pkg in filtered_pkgs:support/scripts/pkgutil.py +13
-4@@ -27,10 +27,7 @@ def get_version(pkgs): version[pkg] = output[i] return version -# Execute the "make <pkg>-show-depends" command to get the list of -# dependencies of a given list of packages, and return the list of -# dependencies formatted as a Python dictionary. -def get_depends(pkgs, rule): +def _get_depends(pkgs, rule): sys.stderr.write("Getting dependencies for %s\n" % pkgs) cmd = ["make", "-s", "--no-print-directory" ] for pkg in pkgs:@@ -53,3 +50,15 @@ def get_depends(pkgs, rule): else: deps[pkg] = pkg_deps return deps + +# Execute the "make <pkg>-show-depends" command to get the list of +# dependencies of a given list of packages, and return the list of +# dependencies formatted as a Python dictionary. +def get_depends(pkgs): + return _get_depends(pkgs, 'show-depends') + +# Execute the "make <pkg>-show-rdepends" command to get the list of +# reverse dependencies of a given list of packages, and return the +# list of dependencies formatted as a Python dictionary. +def get_rdepends(pkgs): + return _get_depends(pkgs, 'show-rdepends')Signature · 1 line
2.10.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Thomas Petazzoni
AppliedPATCH 3/3 graph-depends: split off get_rdepends from get_depends
In reply to Thomas De SchampheleireHello,
Quoted 7 lines · Thomas De Schampheleire
From: Thomas De Schampheleire <[email protected]>
The use of a 'rule' variable that can contain 'show-depends' or 'show-rdepends' is not logical if
get_dependsis considered as a reusable function from various scripts. The name of these rules are too much an implementation detail.
Quoted 8 lines
Therefore, split the existing
get_dependsinto two separate functionsget_dependsandget_rdepends, while keeping code duplication to a minimum.Signed-off-by Thomas De Schampheleire <[email protected]>--- support/scripts/graph-depends | 6 +++--- support/scripts/pkgutil.py | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 7 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com