Extend pkg-stats to replace cve-checker
cover Extend pkg-stats to replace cve-checker
Hello,
We recently introduced cve-checker (commit fafa3e4e293faabc0d38a714eb88a25252936a99). But thinking more about it, what it does is in fact very similar to pkg-stats. It even largely borrows from pkg-stats structure and logic.
The main difference is that pkg-stats was originally written as a Buildroot maintainer-oriented tool, where the goal is to keep an eye on the entire set of Buildroot packages. On the other hand, cve-checker was written as a tool mainly for a Buildroot user, to keep an eye on the CVEs affecting just the packages currently enabled in the current configuration.
So, what this patch series does is extend pkg-stats so that instead of producing its output only for all Buildroot packages, it can be done just for the set of packages enabled in the current configuration.
Here is how it goes:
- PATCH 1 makes pkg-stats usable outside of the Buildroot top-level directory. This will be useful to be able to run it from any output directory.
- PATCH 2 really allows pkg-satts to generate its details based on the set of currently configured packages. This mode is enabled using the new -c option.
- PATCH 3 drops cve-checker
- PATCH 4 promotes the pkg-stats functionality as a Makefile target. Note that only the "pkg-stats -c" mode is used here: we target the use of pkg-stats by Buildroot users, who want results based on their configuration. The use of pkg-stats as a maintainer tool is different, and we assume maintainers will know how to run pkg-stats.
- PATCH 5 adds some mentions of "make show-info" and "make pkg-stats" in the Buildroot manual.
Note: I think this series should be merged in master, not in next. Indeed, cve-checker is new in 2020.11-rc1, so it would probably be a bit silly to release 2020.11 with cve-checker and remove it right after.
Thanks,
Thomas
Thomas Petazzoni (5):
support/scripts/pkg-stats: allow to run script outside of the
top-level directory
support/scripts/pkg-stats: support generating stats based on
configured packages
support/scripts/cve-checker: remove script
Makefile: add pkg-stats target
docs/manual: add some minimal documentation about show-info and
pkg-stats
Makefile | 9 ++ docs/manual/common-usage.txt | 23 ++++ support/scripts/cve-checker | 196 ----------------------------------- support/scripts/pkg-stats | 48 ++++++--- 4 files changed, 63 insertions(+), 213 deletions(-) delete mode 100755 support/scripts/cve-checker
Signature · 1 line
2.26.2
PATCH 1/5 support/scripts/pkg-stats: allow to run script outside of the top-level directory
Currently, pkg-stats expects being executed from Buildroot's top-level source directory. As we are going to extend pkg-stats to cover only the packages available in the current configuration, it makes sense to be able to run it from the output directory, which can be anywhere compared to Buildroot's top-level directory.
This commit adjusts pkg-stats to this, by inferring all Buildroot paths based on the location of the pkg-stats script itself.
--- support/scripts/pkg-stats | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
support/scripts/pkg-stats +21 -17
@@ -28,7 +28,9 @@ import subprocess import json import sys -sys.path.append('utils/') +brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "..")) + +sys.path.append(os.path.join(brpath, "utils")) from getdeveloperlib import parse_developers # noqa: E402 import cve as cvecheck # noqa: E402
@@ -66,7 +68,7 @@ def get_defconfig_list(): """ return [ Defconfig(name[:-len('_defconfig')], os.path.join('configs', name)) - for name in os.listdir('configs') + for name in os.listdir(os.path.join(brpath, 'configs')) if name.endswith('_defconfig') ]
@@ -108,9 +110,10 @@ class Package: Fills in the .url field """ self.status['url'] = ("warning", "no Config.in") - for filename in os.listdir(os.path.dirname(self.path)): + pkgdir = os.path.dirname(os.path.join(brpath, self.path)) + for filename in os.listdir(pkgdir): if fnmatch.fnmatch(filename, 'Config.*'): - fp = open(os.path.join(os.path.dirname(self.path), filename), "r") + fp = open(os.path.join(pkgdir, filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip()
@@ -138,7 +141,7 @@ class Package: Fills in the .infras field """ self.infras = list() - with open(self.path, 'r') as f: + with open(os.path.join(brpath, self.path), 'r') as f: lines = f.readlines() for l in lines: match = INFRA_RE.match(l)
@@ -178,7 +181,7 @@ class Package: return hashpath = self.path.replace(".mk", ".hash") - if os.path.exists(hashpath): + if os.path.exists(os.path.join(brpath, hashpath)): self.status['hash'] = ("ok", "found") else: self.status['hash'] = ("error", "missing")
@@ -191,7 +194,7 @@ class Package: self.status['patches'] = ("na", "no valid package infra") return - pkgdir = os.path.dirname(self.path) + pkgdir = os.path.dirname(os.path.join(brpath, self.path)) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
@@ -214,8 +217,8 @@ class Package: """ Fills in the .warnings and .status['pkg-check'] fields """ - cmd = ["./utils/check-package"] - pkgdir = os.path.dirname(self.path) + cmd = [os.path.join(brpath, "utils/check-package")] + pkgdir = os.path.dirname(os.path.join(brpath, self.path)) self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files:
@@ -300,11 +303,12 @@ def get_pkglist(npackages, package_list): "toolchain/toolchain-wrapper.mk"] packages = list() count = 0 - for root, dirs, files in os.walk("."): + for root, dirs, files in os.walk(brpath): + root = os.path.relpath(root, brpath) rootdir = root.split("/") - if len(rootdir) < 2: + if len(rootdir) < 1: continue - if rootdir[1] not in WALK_USEFUL_SUBDIRS: + if rootdir[0] not in WALK_USEFUL_SUBDIRS: continue for f in files: if not f.endswith(".mk"):
@@ -316,8 +320,7 @@ def get_pkglist(npackages, package_list): pkgpath = os.path.join(root, f) skip = False for exclude in WALK_EXCLUDES: - # pkgpath[2:] strips the initial './' - if re.match(exclude, pkgpath[2:]): + if re.match(exclude, pkgpath): skip = True continue if skip:
@@ -678,7 +681,7 @@ def boolean_str(b): def dump_html_pkg(f, pkg): f.write(" <tr>\n") - f.write(" <td>%s</td>\n" % pkg.path[2:]) + f.write(" <td>%s</td>\n" % pkg.path) # Patch count td_class = ["centered"]
@@ -945,12 +948,13 @@ def __main__(): else: package_list = None date = datetime.datetime.utcnow() - commit = subprocess.check_output(['git', 'rev-parse', + commit = subprocess.check_output(['git', '-C', brpath, + 'rev-parse', 'HEAD']).splitlines()[0].decode() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") - developers = parse_developers() + developers = parse_developers(brpath) print("Build defconfig list ...") defconfigs = get_defconfig_list() for d in defconfigs:
Signature · 1 line
2.26.2
PATCH 1/5 support/scripts/pkg-stats: allow to run script outside of the top-level directory
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Currently, pkg-stats expects being executed from Buildroot's top-level source directory. As we are going to extend pkg-stats to cover only the packages available in the current configuration, it makes sense to be able to run it from the output directory, which can be anywhere compared to Buildroot's top-level directory.
Quoted 4 lines
This commit adjusts pkg-stats to this, by inferring all Buildroot paths based on the location of the pkg-stats script itself.
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 2/5 support/scripts/pkg-stats: support generating stats based on configured packages
In reply to Thomas Petazzonipkg-stats was initially a Buildroot maintenance oriented tool: it was designed to examine all Buildroot packages and provide statistics/details about them.
However, it turns out that a number of details provided by pkg-stats, especially CVEs, are relevant also for Buildroot users, who would like to check regularly if their specific Buildroot configuration is affected by CVEs or not, and possibly check if all packages have license information, license files, etc.
The cve-checker script was recently introduced to provide an output relatively similar to pkg-stats, but focused on CVEs only.
But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses "make show-info" to only consider packages enabled in the current configuration.
So, this commit introduces a -c option to pkg-stats, to tell pkg-stats to generate its output based on the list of configured packages. -c is mutually exclusive with the -p option (explicit list of packages) and -n option (a number of packages, picked randomly).
--- support/scripts/pkg-stats | 10 ++++++++++ 1 file changed, 10 insertions(+)
support/scripts/pkg-stats +10 -0
@@ -333,6 +333,12 @@ def get_pkglist(npackages, package_list): return packages +def get_config_packages(): + cmd = ["make", "--no-print-directory", "show-info"] + js = json.loads(subprocess.check_output(cmd)) + return js.keys() + + def package_init_make_info(): # Fetch all variables at once variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars",
@@ -929,6 +935,8 @@ def parse_args(): output.add_argument('--json', dest='json', type=resolvepath, help='JSON output file') packages = parser.add_mutually_exclusive_group() + packages.add_argument('-c', dest='configpackages', action='store_true', + help='Apply to packages enabled in current configuration') packages.add_argument('-n', dest='npackages', type=int, action='store', help='Number of packages') packages.add_argument('-p', dest='packages', action='store',
@@ -945,6 +953,8 @@ def __main__(): args = parse_args() if args.packages: package_list = args.packages.split(",") + elif args.configpackages: + package_list = get_config_packages() else: package_list = None date = datetime.datetime.utcnow()
Signature · 1 line
2.26.2
PATCH 2/5 support/scripts/pkg-stats: support generating stats based on configured packages
Quoted 20 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
pkg-stats was initially a Buildroot maintenance oriented tool: it was designed to examine all Buildroot packages and provide statistics/details about them.
However, it turns out that a number of details provided by pkg-stats, especially CVEs, are relevant also for Buildroot users, who would like to check regularly if their specific Buildroot configuration is affected by CVEs or not, and possibly check if all packages have license information, license files, etc.
The cve-checker script was recently introduced to provide an output relatively similar to pkg-stats, but focused on CVEs only.
But in fact, its main difference is on the set of packages that we
consider: pkg-stats considers all packages, while cve-checker uses "make show-info" to only consider packages enabled in the current configuration.
Quoted 5 lines
So, this commit introduces a -c option to pkg-stats, to tell pkg-stats to generate its output based on the list of configured packages. -c is mutually exclusive with the -p option (explicit list of packages) and -n option (a number of packages, picked randomly).
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 3/5 support/scripts/cve-checker: remove script
In reply to Thomas PetazzoniNow that pkg-stats is able to generate its output based on the list of packages enabled in the current configuration, cve-checker doesn't serve any purpose.
--- support/scripts/cve-checker | 196 ------------------------------------ 1 file changed, 196 deletions(-) delete mode 100755 support/scripts/cve-checker
support/scripts/cve-checker deleted +0 -196
@@ -1,196 +0,0 @@ -#!/usr/bin/env python - -# Copyright (C) 2009 by Thomas Petazzoni <[email protected]> -# Copyright (C) 2020 by Gregory CLEMENT <[email protected]> -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -import argparse -import datetime -import os -import json -import sys -import cve as cvecheck - - -class Package: - def __init__(self, name, version, ignored_cves): - self.name = name - self.version = version - self.cves = list() - self.ignored_cves = ignored_cves - - -def check_package_cves(nvd_path, packages): - if not os.path.isdir(nvd_path): - os.makedirs(nvd_path) - - for cve in cvecheck.CVE.read_nvd_dir(nvd_path): - for pkg_name in cve.pkg_names: - pkg = packages.get(pkg_name, '') - if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves) == cve.CVE_AFFECTS: - pkg.cves.append(cve.identifier) - - -html_header = """ -<head> -<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script> -<style type=\"text/css\"> -table { - width: 100%; -} -td { - border: 1px solid black; -} -td.centered { - text-align: center; -} -td.wrong { - background: #ff9a69; -} -td.correct { - background: #d2ffc4; -} - -</style> -<title>CVE status for Buildroot configuration</title> -</head> - -<p id=\"sortable_hint\"></p> -""" - - -html_footer = """ -</body> -<script> -if (typeof sorttable === \"object\") { - document.getElementById(\"sortable_hint\").innerHTML = - \"hint: the table can be sorted by clicking the column headers\" -} -</script> -</html> -""" - - -def dump_html_pkg(f, pkg): - f.write(" <tr>\n") - f.write(" <td>%s</td>\n" % pkg.name) - - # Current version - if len(pkg.version) > 20: - version = pkg.version[:20] + "..." - else: - version = pkg.version - f.write(" <td class=\"centered\">%s</td>\n" % version) - - # CVEs - td_class = ["centered"] - if len(pkg.cves) == 0: - td_class.append("correct") - else: - td_class.append("wrong") - f.write(" <td class=\"%s\">\n" % " ".join(td_class)) - for cve in pkg.cves: - f.write(" <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve)) - f.write(" </td>\n") - - f.write(" </tr>\n") - - -def dump_html_all_pkgs(f, packages): - f.write(""" -<table class=\"sortable\"> -<tr> -<td>Package</td> -<td class=\"centered\">Version</td> -<td class=\"centered\">CVEs</td> -</tr> -""") - for pkg in packages: - dump_html_pkg(f, pkg) - f.write("</table>") - - -def dump_html_gen_info(f, date): - f.write("<p><i>Generated on %s</i></p>\n" % (str(date))) - - -def dump_html(packages, date, output): - with open(output, 'w') as f: - f.write(html_header) - dump_html_all_pkgs(f, packages) - dump_html_gen_info(f, date) - f.write(html_footer) - - -def dump_json(packages, date, output): - # Format packages as a dictionnary instead of a list - pkgs = { - pkg.name: { - "version": pkg.version, - "cves": pkg.cves, - } for pkg in packages - } - # The actual structure to dump, add date to it - final = {'packages': pkgs, - 'date': str(date)} - with open(output, 'w') as f: - json.dump(final, f, indent=2, separators=(',', ': ')) - f.write('\n') - - -def resolvepath(path): - return os.path.abspath(os.path.expanduser(path)) - - -def parse_args(): - parser = argparse.ArgumentParser() - output = parser.add_argument_group('output', 'Output file(s)') - output.add_argument('--html', dest='html', type=resolvepath, - help='HTML output file') - output.add_argument('--json', dest='json', type=resolvepath, - help='JSON output file') - parser.add_argument('--nvd-path', dest='nvd_path', - help='Path to the local NVD database', type=resolvepath, - required=True) - args = parser.parse_args() - if not args.html and not args.json: - parser.error('at least one of --html or --json (or both) is required') - return args - - -def __main__(): - packages = list() - content = json.load(sys.stdin) - for item in content: - pkg = content[item] - p = Package(item, pkg.get('version', ''), pkg.get('ignore_cves', '')) - packages.append(p) - - args = parse_args() - date = datetime.datetime.utcnow() - - print("Checking packages CVEs") - check_package_cves(args.nvd_path, {p.name: p for p in packages}) - - if args.html: - print("Write HTML") - dump_html(packages, date, args.html) - if args.json: - print("Write JSON") - dump_json(packages, date, args.json) - - -__main__()
Signature · 1 line
2.26.2
PATCH 3/5 support/scripts/cve-checker: remove script
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Now that pkg-stats is able to generate its output based on the list of packages enabled in the current configuration, cve-checker doesn't serve any purpose.
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 4/5 Makefile: add pkg-stats target
In reply to Thomas PetazzoniNow that pkg-stats is not just a maintainer-oriented tool, but a tool generally useful to users, introduce a make target to run pkg-stats. Of course, it is run with the newly introduced -c option, which produces a pkg-stats output for just the selection of packages of the currently defined configuration.
--- Makefile | 9 +++++++++ 1 file changed, 9 insertions(+)
Makefile +9 -0
@@ -937,6 +937,14 @@ show-info: ) \ ) +.PHONY: pkg-stats +pkg-stats: + @cd "$(CONFIG_DIR)" ; \ + $(TOPDIR)/support/scripts/pkg-stats -c \ + --json $(O)/pkg-stats.json \ + --html $(O)/pkg-stats.html \ + --nvd-path $(DL_DIR)/buildroot-nvd + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) # Some subdirectories are also package names. To avoid that "make linux"
@@ -1156,6 +1164,7 @@ help: @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' show-info - generate info about packages, as a JSON blurb' + @echo ' pkg-stats - generate info about packages as JSON and HTML' @echo ' printvars - dump internal variables selected with VARS=...' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
Signature · 1 line
2.26.2
PATCH 4/5 Makefile: add pkg-stats target
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Now that pkg-stats is not just a maintainer-oriented tool, but a tool generally useful to users, introduce a make target to run pkg-stats. Of course, it is run with the newly introduced -c option, which produces a pkg-stats output for just the selection of packages of the currently defined configuration.
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 5/5 docs/manual: add some minimal documentation about show-info and pkg-stats
In reply to Thomas Petazzoni--- docs/manual/common-usage.txt | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
docs/manual/common-usage.txt +23 -0
@@ -157,6 +157,29 @@ your filesystem, those parts may not be all-zeroes when read back). You should only use sparse files when handling files on the build machine, not when transferring them to an actual device that will be used on the target. +=== Details about packages + +[[package-details]] + +Buildroot can produce a JSON blurb that describes the set of enabled +packages in the current configuration, together with their +dependencies, licenses and other metadata. This JSON blurb is produced +by using the +show-info+ make target: + +------------------------ +make show-info +------------------------ + +Buildroot can also produce details about packages as HTML and JSON +output using the +pkg-stats+ make target. Amongst other things, these +details include whether known CVEs (security vulnerabilities) affect +the packages in your current configuration. It also shows if there is +a newer upstream version for those packages. + +------------------------ +make pkg-stats +------------------------ + === Graphing the dependencies between packages [[graph-depends]]
Signature · 1 line
2.26.2
PATCH 5/5 docs/manual: add some minimal documentation about show-info and pkg-stats
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Signed-off-by Thomas Petazzoni <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
cover Extend pkg-stats to replace cve-checker
In reply to Thomas PetazzoniHello Thomas,
Quoted 14 lines
Hello,
We recently introduced cve-checker (commit
fafa3e4e293faabc0d38a714eb88a25252936a99). But thinking more about it, what it does is in fact very similar to pkg-stats. It even largely borrows from pkg-stats structure and logic.The main difference is that pkg-stats was originally written as a Buildroot maintainer-oriented tool, where the goal is to keep an eye on the entire set of Buildroot packages. On the other hand, cve-checker was written as a tool mainly for a Buildroot user, to keep an eye on the CVEs affecting just the packages currently enabled in the current configuration.
Quoted 3 lines
So, what this patch series does is extend pkg-stats so that instead of producing its output only for all Buildroot packages, it can be done just for the set of packages enabled in the current configuration.
Your series looks good and I think it is a good things to finally remove cve-checker. I introduced cve.py in order to share code between cve-checker and pkg-stats but in the end there was sill a lot of duplicate code, and each evolution was really painful, as it was needed to duplicate it in each script but with very few difference.
Gregory
Quoted 52 lines
Here is how it goes:
- PATCH 1 makes pkg-stats usable outside of the Buildroot top-level directory. This will be useful to be able to run it from any output directory.
- PATCH 2 really allows pkg-satts to generate its details based on the set of currently configured packages. This mode is enabled using the new -c option.
- PATCH 3 drops cve-checker
- PATCH 4 promotes the pkg-stats functionality as a
Makefiletarget. Note that only the "pkg-stats -c" mode is used here: we target the use of pkg-stats by Buildroot users, who want results based on their configuration. The use of pkg-stats as a maintainer tool is different, and we assume maintainers will know how to run pkg-stats.- PATCH 5 adds some mentions of "make show-info" and "make pkg-stats" in the Buildroot manual.
Note: I think this series should be merged in master, not in next. Indeed, cve-checker is new in 2020.11-rc1, so it would probably be a bit silly to release 2020.11 with cve-checker and remove it right after.
Thanks,
Thomas
Thomas Petazzoni (5):
support/scripts/pkg-stats: allow to run script outside of the
top-level directory
support/scripts/pkg-stats: support generating stats based on
configured packages
support/scripts/cve-checker: remove script
Makefile: add pkg-stats target
docs/manual: add some minimal documentation about show-info and
pkg-statsMakefile | 9 ++ docs/manual/common-usage.txt | 23 ++++ support/scripts/cve-checker | 196 ----------------------------------- support/scripts/pkg-stats | 48 ++++++--- 4 files changed, 63 insertions(+), 213 deletions(-) delete mode 100755 support/scripts/cve-checkerSignature · 1 line
2.26.2
Signature · 3 lines
Gregory Clement, Bootlin Embedded Linux and Kernel engineering http://bootlin.com