support/scripts/pkg-stats: import cve module only when needed
PATCH 1/4 support/scripts/pkg-stats: import cve module only when needed
The cve module needs ijson, which may not be installed. Since cve matching is only enabled when --nvd-path is passed, it is a bit silly to error out about ijson being missing if it's not used.
So instead of unconditionally importing the cve module, only do it conditionally.
However, instead of doing it right at the point where it is used, we do it at the beginning of the main() function. Indeed, if the cve module is needed but cannot be imported, we want to error out immediately rather than doing a whole bunch of things, and failing on the user later on in the middle of the pkg-stats execution.
--- support/scripts/pkg-stats | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
support/scripts/pkg-stats +6 -1
@@ -32,7 +32,6 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "..")) sys.path.append(os.path.join(brpath, "utils")) from getdeveloperlib import parse_developers # noqa: E402 -import cve as cvecheck # noqa: E402 INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
@@ -950,7 +949,13 @@ def parse_args(): def __main__(): + global cvecheck + args = parse_args() + + if args.nvd_path: + import cve as cvecheck + if args.packages: package_list = args.packages.split(",") elif args.configpackages:
Signature · 1 line
2.28.0
PATCH 1/4 support/scripts/pkg-stats: import cve module only when needed
Quoted 13 lines · On Thu, 19 Nov 2020 15:53:51 +0100 Thomas Petazzoni…
The cve module needs ijson, which may not be installed. Since cve matching is only enabled when --nvd-path is passed, it is a bit silly to error out about ijson being missing if it's not used.
So instead of unconditionally importing the cve module, only do it conditionally.
However, instead of doing it right at the point where it is used, we do it at the beginning of the
main()function. Indeed, if the cve module is needed but cannot be imported, we want to error out immediately rather than doing a whole bunch of things, and failing on the user later on in the middle of the pkg-stats execution.
Quoted 4 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- support/scripts/pkg-stats | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
Series applied.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 1/4 support/scripts/pkg-stats: import cve module only when needed
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Quoted 13 lines · On Thu, 19 Nov 2020 15:53:51 +0100 Thomas Petazzoni…
The cve module needs ijson, which may not be installed. Since cve matching is only enabled when --nvd-path is passed, it is a bit silly to error out about ijson being missing if it's not used.
So instead of unconditionally importing the cve module, only do it conditionally.
However, instead of doing it right at the point where it is used, we do it at the beginning of the
main()function. Indeed, if the cve module is needed but cannot be imported, we want to error out immediately rather than doing a whole bunch of things, and failing on the user later on in the middle of the pkg-stats execution.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Thomas Petazzoni <[email protected]>---
support/scripts/pkg-stats | 7 ++++++-1 file changed, 6 insertions(+), 1 deletion(-)Series applied.
Committed to 2020.02.x and 2020.11.x, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/4 support/scripts/pkg-stats: fix Python 3.8 deprecation warning
In reply to Thomas PetazzoniWith Python 3.8, the following deprecation warnings are emitted:
/home/thomas/projets/buildroot/./support/scripts/pkg-stats:418: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
/home/thomas/projets/buildroot/./support/scripts/pkg-stats:536: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11.
The correct way to pass coroutines is to use asyncio.create_task(), but this is rather new method (Python 3.7), and using it breaks compatibility with older Python versions. As suggested at
https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task,
use the more cryptic, but also more compatible asyncio.ensure_future() method.
--- support/scripts/pkg-stats | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +2 -2
@@ -413,7 +413,7 @@ async def check_package_urls(packages): async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess: packages = [p for p in packages if p.status['url'][0] == 'ok'] for pkg in packages: - tasks.append(check_url_status(sess, pkg, len(packages))) + tasks.append(asyncio.ensure_future(check_url_status(sess, pkg, len(packages)))) await asyncio.wait(tasks)
@@ -531,7 +531,7 @@ async def check_package_latest_version(packages): async with aiohttp.ClientSession(connector=connector, trust_env=True) as sess: packages = [p for p in packages if p.has_valid_infra] for pkg in packages: - tasks.append(check_package_latest_version_get(sess, pkg, len(packages))) + tasks.append(asyncio.ensure_future(check_package_latest_version_get(sess, pkg, len(packages)))) await asyncio.wait(tasks)
Signature · 1 line
2.28.0
PATCH 3/4 utils/get-developers: use Developers.hasfile() where appropriate
In reply to Thomas PetazzoniInstead of open-coding Developers.hasfile() in utils/get-developers, use it directly.
--- utils/get-developers | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
utils/get-developers +2 -3
@@ -77,9 +77,8 @@ def __main__(): if args.files is not None: args.files = [os.path.abspath(f) for f in args.files] for dev in devs: - for devfile in dev.files: - commonfiles = [f for f in args.files if f.startswith(devfile)] - if commonfiles: + for f in args.files: + if dev.hasfile(f): print(dev.name) break
Signature · 1 line
2.28.0
PATCH 3/4 utils/get-developers: use Developers.hasfile() where appropriate
Hi Thomas,
Am Do., 19. Nov. 2020 um 15:54 Uhr schrieb Thomas Petazzoni <[email protected]>:
Quoted 5 lines
Instead of open-coding
Developers.hasfile()in utils/get-developers, use it directly.Signed-off-by Thomas Petazzoni <[email protected]>
Quoted 10 lines
--- utils/get-developers | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)utils/get-developers +0
-0@@ -77,9 +77,8 @@ def __main__(): if args.files is not None:
utils/get-developers def __main__(): · line 84
Quoted 10 lines
utils/get-developers +2
-3args.files = [os.path.abspath(f) for f in args.files] for dev in devs: - for devfile in dev.files: - commonfiles = [f for f in args.files if f.startswith(devfile)] - if commonfiles: + for f in args.files: + if dev.hasfile(f): print(dev.name) break
Thank you
Signature · 1 line
Heiko
PATCH 1/4 support/scripts/pkg-stats: import cve module only when needed
In reply to Thomas PetazzoniHi Thomas,
Am Do., 19. Nov. 2020 um 15:54 Uhr schrieb Thomas Petazzoni <[email protected]>:
Quoted 8 lines
The cve module needs ijson, which may not be installed. Since cve matching is only enabled when --nvd-path is passed, it is a bit silly to error out about ijson being missing if it's not used.
So instead of unconditionally importing the cve module, only do it conditionally.
Quoted 7 lines
However, instead of doing it right at the point where it is used, we do it at the beginning of the
main()function. Indeed, if the cve module is needed but cannot be imported, we want to error out immediately rather than doing a whole bunch of things, and failing on the user later on in the middle of the pkg-stats execution.Signed-off-by Thomas Petazzoni <[email protected]>
Quoted 20 lines
--- support/scripts/pkg-stats | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)support/scripts/pkg-stats +0
-1@@ -32,7 +32,6 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "..")) sys.path.append(os.path.join(brpath, "utils")) from getdeveloperlib import parse_developers # noqa: E402 -import cve as cvecheck # noqa: E402 INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")@@ -950,7 +949,13 @@ def parse_args(): def __main__():
support/scripts/pkg-stats def parse_args(): · line 961
Quoted 10 lines
support/scripts/pkg-stats +6
-0+ global cvecheck + args = parse_args() + + if args.nvd_path: + import cve as cvecheck + if args.packages: package_list = args.packages.split(",") elif args.configpackages:
Thank you
Signature · 1 line
Heiko
PATCH 4/4 utils/getdeveloperlib.py: use relative paths for files
In reply to Thomas PetazzoniUsing absolute paths within getdeveloperlib isn't very sensible, it makes a lot more sense to handle everything as relative paths from the top-level Buildroot source directory.
parse_developers() is changed to no longer take the base path as
argument: it is automatically calculated based on the location of utils/getdeveloperlib.py. Then, the rest of the logic is adjusted to use relative paths, and prepend them with the base "brpath" when needed.
This commit allows pkg-stats to report correct developers information even when executed from an out of tree directory.
Before this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json $ cat out.json | jq '.packages.ipmitool.developers' []
$ cat out.json | jq '.defconfigs.stm32f469_disco'
{
"name": "stm32f469_disco",
"path": "configs/stm32f469_disco_defconfig",
"developers": []
}After this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json $ cat out.json | jq '.packages.ipmitool.developers' [ "Floris Bos <[email protected]>", "Heiko Thiery <[email protected]>" ] $ cat out.json | jq '.defconfigs.stm32f469_disco' { "name": "stm32f469_disco", "path": "configs/stm32f469_disco_defconfig", "developers": [ "Christophe Priouzeau <[email protected]>" ] }
--- support/scripts/pkg-stats | 2 +- utils/get-developers | 5 ----- utils/getdeveloperlib.py | 26 ++++++++++++-------------- 3 files changed, 13 insertions(+), 20 deletions(-)
support/scripts/pkg-stats +1 -1
@@ -969,7 +969,7 @@ def __main__(): print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") - developers = parse_developers(brpath) + developers = parse_developers() print("Build defconfig list ...") defconfigs = get_defconfig_list() for d in defconfigs:
utils/get-developers +0 -5
@@ -45,10 +45,6 @@ def __main__(): print("No action specified") return - # getdeveloperlib expects to be executed from the toplevel buildroot - # directory, which is one level up from this script - os.chdir(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..')) - devs = getdeveloperlib.parse_developers() if devs is None: sys.exit(1)
@@ -75,7 +71,6 @@ def __main__(): # Handle the files action if args.files is not None: - args.files = [os.path.abspath(f) for f in args.files] for dev in devs: for f in args.files: if dev.hasfile(f):
utils/getdeveloperlib.py +12 -14
@@ -6,6 +6,8 @@ import subprocess import sys import unittest +brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..")) + # # Patch parsing functions #
@@ -94,14 +96,14 @@ def get_all_test_cases(suite): yield (suite.__module__, suite.__class__.__name__) -def list_unittests(path): +def list_unittests(): """Use the unittest module to retreive all test cases from a given directory""" loader = unittest.TestLoader() - suite = loader.discover(path) + suite = loader.discover(os.path.join(brpath, "support", "testing")) tests = {} for module, test in get_all_test_cases(suite): - module_path = os.path.join(path, *module.split('.')) + module_path = os.path.join("support", "testing", *module.split('.')) tests.setdefault(module_path, []).append('%s.%s' % (module, test)) return tests
@@ -124,9 +126,7 @@ class Developer: self.defconfigs = parse_developer_defconfigs(files) def hasfile(self, f): - f = os.path.abspath(f) for fs in self.files: - fs = os.path.abspath(fs) if f.startswith(fs): return True return False
@@ -158,7 +158,7 @@ def parse_developer_packages(fnames): patterns, and return a list of those packages.""" packages = set() for fname in fnames: - for root, dirs, files in os.walk(fname): + for root, dirs, files in os.walk(os.path.join(brpath, fname)): for f in files: path = os.path.join(root, f) if fname_get_package_infra(path):
@@ -223,7 +223,7 @@ def parse_developer_runtime_tests(fnames): # List all files recursively for fname in fnames: if os.path.isdir(fname): - for root, _dirs, files in os.walk(fname): + for root, _dirs, files in os.walk(os.path.join(brpath, fname)): all_files += [os.path.join(root, f) for f in files] else: all_files.append(fname)
@@ -237,15 +237,13 @@ def parse_developer_runtime_tests(fnames): return runtimes -def parse_developers(basepath=None): +def parse_developers(): """Parse the DEVELOPERS file and return a list of Developer objects.""" developers = [] linen = 0 - if basepath is None: - basepath = os.getcwd() global unittests - unittests = list_unittests(os.path.join(basepath, 'support/testing')) - with open(os.path.join(basepath, "DEVELOPERS"), "r") as f: + unittests = list_unittests() + with open(os.path.join(brpath, "DEVELOPERS"), "r") as f: files = [] name = None for line in f:
@@ -259,11 +257,11 @@ def parse_developers(basepath=None): name = line[2:].strip() elif line.startswith("F:"): fname = line[2:].strip() - dev_files = glob.glob(os.path.join(basepath, fname)) + dev_files = glob.glob(os.path.join(brpath, fname)) if len(dev_files) == 0: print("WARNING: '%s' doesn't match any file" % fname, file=sys.stderr) - files += dev_files + files += [os.path.relpath(f, brpath) for f in dev_files] elif line == "": if not name: continue
Signature · 1 line
2.28.0
PATCH 4/4 utils/getdeveloperlib.py: use relative paths for files
Hi Thomas,
Am Do., 19. Nov. 2020 um 15:54 Uhr schrieb Thomas Petazzoni <[email protected]>:
Quoted 5 lines
Using absolute paths within getdeveloperlib isn't very sensible, it makes a lot more sense to handle everything as relative paths from the top-level Buildroot source directory.
Quoted 8 lines
parse_developers()is changed to no longer take the base path as
argument: it is automatically calculated based on the location ofutils/getdeveloperlib.py. Then, the rest of the logic is adjusted to use relative paths, and prepend them with the base "brpath" when needed.This commit allows pkg-stats to report correct developers information even when executed from an out of tree directory.
Just tried this but either I do not completely understand what it should do or there is an issue.
I do as follows:
--- >8 ---
# mkdir b
# cd b
# make ../buildroot/ defconfig
# ../buildroot/support/scripts/pkg-stats -c --json out.json
make: *** Keine Regel, um ?show-info? zu erstellen. Schluss.
Traceback (most recent call last):
File "../buildroot/support/scripts/pkg-stats", line 1008, in <module>
__main__()
File "../buildroot/support/scripts/pkg-stats", line 962, in __main__
package_list = get_config_packages()
File "../buildroot/support/scripts/pkg-stats", line 337, in
get_config_packages
js = json.loads(subprocess.check_output(cmd))
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['make',
'--no-print-directory', 'show-info']' returned non-zero exit status 2.--- >8 ---
Quoted 16 lines
Before this patch:
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json $ cat out.json | jq '.packages.ipmitool.developers' []$ cat out.json | jq '.defconfigs.stm32f469_disco' { "name": "stm32f469_disco", "path": "configs/stm32f469_disco_defconfig", "developers": [] }After this patch:
Quoted 14 lines
$ ~/buildroot/support/scripts/pkg-stats -p ipmitool --json out.json $ cat out.json | jq '.packages.ipmitool.developers' [ "Floris Bos <[email protected]>", "Heiko Thiery <[email protected]>" ] $ cat out.json | jq '.defconfigs.stm32f469_disco' { "name": "stm32f469_disco", "path": "configs/stm32f469_disco_defconfig", "developers": [ "Christophe Priouzeau <[email protected]>" ] }
In-tree this change works as expected.
Quoted 2 lines
Signed-off-by Thomas Petazzoni <[email protected]>
Quoted 124 lines
--- support/scripts/pkg-stats | 2 +- utils/get-developers | 5 ----- utils/getdeveloperlib.py | 26 ++++++++++++-------------- 3 files changed, 13 insertions(+), 20 deletions(-)support/scripts/pkg-stats +1
-1@@ -969,7 +969,7 @@ def __main__(): print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") - developers = parse_developers(brpath) + developers = parse_developers() print("Build defconfig list ...") defconfigs = get_defconfig_list() for d in defconfigs:utils/get-developers +0
-5@@ -45,10 +45,6 @@ def __main__(): print("No action specified") return - # getdeveloperlib expects to be executed from the toplevel buildroot - # directory, which is one level up from this script - os.chdir(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..')) - devs = getdeveloperlib.parse_developers() if devs is None: sys.exit(1)@@ -75,7 +71,6 @@ def __main__(): # Handle the files action if args.files is not None: - args.files = [os.path.abspath(f) for f in args.files] for dev in devs: for f in args.files: if dev.hasfile(f):utils/getdeveloperlib.py +10
-12@@ -6,6 +6,8 @@ import subprocess import sys import unittest +brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..")) + # # Patch parsing functions #@@ -94,14 +96,14 @@ def get_all_test_cases(suite): yield (suite.__module__, suite.__class__.__name__) -def list_unittests(path): +def list_unittests(): """Use the unittest module to retreive all test cases from a given directory""" loader = unittest.TestLoader() - suite = loader.discover(path) + suite = loader.discover(os.path.join(brpath, "support", "testing")) tests = {} for module, test in get_all_test_cases(suite): - module_path = os.path.join(path, *module.split('.')) + module_path = os.path.join("support", "testing", *module.split('.')) tests.setdefault(module_path, []).append('%s.%s' % (module, test)) return tests@@ -124,9 +126,7 @@ class Developer: self.defconfigs = parse_developer_defconfigs(files) def hasfile(self, f): - f = os.path.abspath(f) for fs in self.files: - fs = os.path.abspath(fs) if f.startswith(fs): return True return False@@ -158,7 +158,7 @@ def parse_developer_packages(fnames): patterns, and return a list of those packages.""" packages = set() for fname in fnames: - for root, dirs, files in os.walk(fname): + for root, dirs, files in os.walk(os.path.join(brpath, fname)): for f in files: path = os.path.join(root, f) if fname_get_package_infra(path):@@ -223,7 +223,7 @@ def parse_developer_runtime_tests(fnames): # List all files recursively for fname in fnames: if os.path.isdir(fname): - for root, _dirs, files in os.walk(fname): + for root, _dirs, files in os.walk(os.path.join(brpath, fname)): all_files += [os.path.join(root, f) for f in files] else: all_files.append(fname)@@ -237,15 +237,13 @@ def parse_developer_runtime_tests(fnames): return runtimes -def parse_developers(basepath=None): +def parse_developers(): """Parse the DEVELOPERS file and return a list of Developer objects.""" developers = [] linen = 0 - if basepath is None: - basepath = os.getcwd() global unittests - unittests = list_unittests(os.path.join(basepath, 'support/testing')) - with open(os.path.join(basepath, "DEVELOPERS"), "r") as f: + unittests = list_unittests() + with open(os.path.join(brpath, "DEVELOPERS"), "r") as f: files = [] name = None for line in f:@@ -259,11 +257,11 @@ def parse_developers(basepath=None): name = line[2:].strip() elif line.startswith("F:"): fname = line[2:].strip()
utils/getdeveloperlib.py def parse_developers(basepath=None): · line 267
Quoted 10 lines
utils/getdeveloperlib.py +2
-2- dev_files = glob.glob(os.path.join(basepath, fname)) + dev_files = glob.glob(os.path.join(brpath, fname)) if len(dev_files) == 0: print("WARNING: '%s' doesn't match any file" % fname, file=sys.stderr) - files += dev_files + files += [os.path.relpath(f, brpath) for f in dev_files] elif line == "": if not name: continue
Thank you.
--
Heiko
PATCH 4/4 utils/getdeveloperlib.py: use relative paths for files
Hello Heiko,
Thanks for the review and feedback!
Quoted 8 lines · On Fri, 20 Nov 2020 11:03:45 +0100 Heiko Thiery <[email protected]> wrote:
I do as follows:
--- >8 ---
# mkdir b # cd b # make ../buildroot/ defconfig # ../buildroot/support/scripts/pkg-stats -c --json out.json make: *** Keine Regel, um ?show-info? zu erstellen. Schluss.
The current folder must be a Builroot output directory:
$ mkdir foobar/ $ cd foobar/ $ make -C /path/to/buildroot O=$(pwd) blabla_defconfig $ make pkg-stats or $ /path/to/buildroot/support/scripts/pkg-stats -c --json out.json
Or this should also work (but not tested)
$ cd /path/to/buildroot $ make O=/output blabla_defconfig $ make O=/output pkg-stats
In any case: "pkg-stats -c" runs "make show-info", so it needs to be from a place where you can run Buildroot make targets.
Does this clarifies the situation ?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/4 utils/getdeveloperlib.py: use relative paths for files
Hi Thomas,
Am Fr., 20. Nov. 2020 um 15:14 Uhr schrieb Thomas Petazzoni <[email protected]>:
Quoted 28 lines
Hello Heiko,
Thanks for the review and feedback!
Quoted 8 lines · On Fri, 20 Nov 2020 11:03:45 +0100 Heiko Thiery <[email protected]> wrote:
I do as follows:
--- >8 ---
# mkdir b # cd b # make ../buildroot/ defconfig # ../buildroot/support/scripts/pkg-stats -c --json out.json make: *** Keine Regel, um ?show-info? zu erstellen. Schluss.The current folder must be a Builroot output directory:
$ mkdir foobar/ $ cd foobar/ $ make -C /path/to/buildroot O=$(pwd) blabla_defconfig $ make pkg-stats or $ /path/to/buildroot/support/scripts/pkg-stats -c --json out.jsonOr this should also work (but not tested)
Quoted 8 lines
$ cd /path/to/buildroot $ make O=/output blabla_defconfig $ make O=/output pkg-statsIn any case: "pkg-stats -c" runs "make show-info", so it needs to be from a place where you can run Buildroot make targets.
Does this clarifies the situation ?
Oh yes .. that was my fault. I completely forgot to add the O=$PWD argument.
Forget my previous mail ;-)
Signature · 1 line
Heiko