pkg-stats json output improvements
v5 cover pkg-stats json output improvements
- add developers information to the packages
- add supported defconfigs to json
- add license information to json
- add license file information to json
- add patch files to json
- add a more generic check status field to the packages for easier post processing. The following check status are availble: checks: cve, license, license-files, hash, hash-license,patches, version, url, developers, pkg-check
---
v4 -> v5:
- fix failure while applying to master
v3 -> v4:
Included the review comments from Titouan
- use only one signed-off mail address
- do some things in a more pythonic way
- remove the patch_count attribute and use a class property instead (Note that the patch_count value will not go into the json output anymore,but then coutn can be computed by "len(patch_files)".)
- remove the status initializer in the package class that was introduced in v3
- update the cve status at the end of the cve check loop
- change the is_valid_infra -> has_valid_infra and use it as property
v2 -> v3:
- keep variable latest_release but change format
- add check for license file hashes
- introduce a na status for the checks
- add a list of all possible checks to the json output
- add the cve check status
v1 -> v2:
- cleanup and recreation of patches
- remove pkg name from dumping to json
- use patch_files instead of combine count and files in dict
- include getdevelopers.py to reuse Developers class
---
Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve check
support/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++--------- 1 file changed, 211 insertions(+), 60 deletions(-)
Signature · 1 line
2.20.1
PATCH v5 1/12 support/scripts/pkg-stats: store latest version in a dict
This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output.
--- support/scripts/pkg-stats | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
support/scripts/pkg-stats +15 -16
@@ -71,7 +71,7 @@ class Package: self.url_status = None self.url_worker = None self.cves = list() - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_")
@@ -462,9 +462,8 @@ def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects - This field has a special format: - (status, version, id) - with: + This field is a dict and has the following keys: + - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND
@@ -480,7 +479,7 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) worker_pool.terminate() del http_pool
@@ -519,13 +518,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.latest_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.latest_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1
@@ -691,29 +690,29 @@ def dump_html_pkg(f, pkg): f.write(" <td class=\"centered\">%s</td>\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.latest_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "<b>Error</b>" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "<b>Not found</b>" else: - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: latest_version_text = "<b>Found, but no version</b>" else: latest_version_text = "<a href=\"https://release-monitoring.org/project/%s\"><b>%s</b></a>" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.latest_version['id'], str(pkg.latest_version['version'])) latest_version_text += "<br/>" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by <a href=\"https://release-monitoring.org/distro/Buildroot/\">distro</a>" else: latest_version_text += "found by guess"
Signature · 1 line
2.20.1
PATCH v5 2/12 support/scripts/pkg-stats: store patch files for the package
In reply to Heiko ThieryRemove the patch_count attribute and use a class property instead.
--- support/scripts/pkg-stats | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
support/scripts/pkg-stats +6 -3
@@ -64,7 +64,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False - self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None
@@ -93,6 +93,10 @@ class Package: self.url_status = "Missing" fp.close() + @property + def patch_count(self): + return len(self.patch_files) + def set_infra(self): """ Fills in the .infras field
@@ -131,10 +135,9 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') def set_current_version(self): """
Signature · 1 line
2.20.1
PATCH v5 3/12 support/scripts/pkg-stats: set developers info
In reply to Heiko ThieryUse the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package.
--- support/scripts/pkg-stats | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
support/scripts/pkg-stats +17 -0
@@ -30,10 +30,14 @@ import certifi import distutils.version import time import gzip +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + NVD_START_YEAR = 2002 NVD_JSON_VERSION = "1.0" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
@@ -171,6 +175,16 @@ class Package: """ return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + dev.name + for dev in developers + if dev.hasfile(self.path) + ] + def __eq__(self, other): return self.path == other.path
@@ -894,6 +908,8 @@ def __main__(): 'HEAD']).splitlines()[0].decode() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...")
@@ -905,6 +921,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...")
Signature · 1 line
2.20.1
PATCH v5 4/12 support/scripts/pkg-stats: store licences of package
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +4 -2
@@ -56,7 +56,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() all_ignored_cves = dict()
@@ -65,6 +65,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False
@@ -125,6 +126,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True
@@ -387,7 +389,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
Signature · 1 line
2.20.1
PATCH v5 5/12 support/scripts/pkg-stats: add package status
In reply to Heiko ThieryUnify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message.
The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check
With that status information the following variables are replaced:has_license, has_license_files, has_hash, url_status
--- support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 32 deletions(-)
support/scripts/pkg-stats +69 -32
@@ -73,10 +73,10 @@ class Package: self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_")
@@ -85,17 +85,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("error", "missing") fp.close() @property
@@ -121,30 +121,42 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field
@@ -155,10 +167,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
@@ -169,6 +182,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line.decode()) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def is_cve_ignored(self, cve):
@@ -179,7 +196,7 @@ class Package: def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = [ dev.name
@@ -187,6 +204,14 @@ class Package: if dev.hasfile(self.path) ] + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + else: + self.status['developers'] = ("warning", "no developers") + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path
@@ -195,7 +220,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) class CVE:
@@ -409,23 +434,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) del pkg.url_worker pool.terminate()
@@ -499,6 +524,18 @@ def check_package_latest_version(packages): results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', "Release Monitoring API error") + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', "Package not found on Release Monitoring") + + if pkg.latest_version['version'] is None: + pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring") + elif pkg.latest_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version'])) + else: + pkg.status['version'] = ('ok', 'up-to-date') worker_pool.terminate() del http_pool
@@ -525,15 +562,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1
@@ -676,30 +713,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20:
@@ -750,12 +787,12 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] in ("error", "warning"): td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): + if pkg.status['url'][0] == "error": td_class.append("invalid_url") - url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.url_status) + url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "<a href=%s>Link</a>" % pkg.url
Signature · 1 line
2.20.1
PATCH v5 6/12 support/scripts/pkg-stats: add package count to stats
In reply to Heiko Thiery--- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
@@ -552,6 +552,7 @@ def check_package_cves(nvd_path, packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the
Signature · 1 line
2.20.1
PATCH v5 7/12 support/scripts/pkg-stats: store pkg dir path
In reply to Heiko ThieryThis value can be used for later processing.
In the buildroot-stats application this is used to create links pointing to the git repo of buildroot.
--- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
@@ -64,6 +64,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.has_license = False
Signature · 1 line
2.20.1
PATCH v5 8/12 support/scripts/pkg-stats: add defconfig support
In reply to Heiko ThieryScan configs directory and create Defconfig objects.
--- support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +41 -2
@@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + developer.name + for developer in developers + if developer.hasfile(self.path) + ] + + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + return [ + Defconfig(name[:-len('_defconfig')], os.path.join('configs', name)) + for name in os.listdir('configs') + if name.endswith('_defconfig') + ] + class Package: all_licenses = dict()
@@ -889,7 +917,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name']
@@ -900,6 +928,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v
@@ -910,6 +944,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)}
@@ -951,6 +986,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...")
@@ -977,7 +1016,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__()
Signature · 1 line
2.20.1
PATCH v5 9/12 support/scripts/pkg-stats: add support for license hash check
In reply to Heiko ThieryStore the names of license files and check if they are in the hash file.
--- support/scripts/pkg-stats | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +14 -2
@@ -85,7 +85,7 @@ def get_defconfig_list(): class Package: all_licenses = dict() - all_license_files = list() + all_license_files = dict() all_versions = dict() all_ignored_cves = dict()
@@ -98,6 +98,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False + self.license_files = None self.patch_files = [] self.warnings = 0 self.current_version = None
@@ -159,6 +160,7 @@ class Package: self.license = self.all_licenses[var] self.status['license'] = ("ok", "found") if var in self.all_license_files: + self.license_files = self.all_license_files[var].split(' ') self.status['license-files'] = ("ok", "found") def set_hash_info(self):
@@ -166,8 +168,18 @@ class Package: Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") + self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): self.status['hash'] = ("ok", "found") + self.status['hash-license'] = ("error", "no license in hash file") + # check if license files are in hash file + if self.license_files is not None: + self.status['hash-license'] = ("ok", "found") + with open(hashpath) as f: + content = f.read() + for license in self.license_files: + if content.find(license) == -1: + self.status['hash-license'] = ("error", "license missing in hash file") else: self.status['hash'] = ("error", "missing")
@@ -449,7 +461,7 @@ def package_init_make_info(): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): continue pkgvar = pkgvar[:-14] - Package.all_license_files.append(pkgvar) + Package.all_license_files[pkgvar] = value elif pkgvar.endswith("_VERSION"): if pkgvar.endswith("_DL_VERSION"):
Signature · 1 line
2.20.1
PATCH v5 9/12 support/scripts/pkg-stats: add support for license hash check
Quoted 3 lines · On Sat, 7 Mar 2020 08:56:30 +0100 Heiko Thiery <[email protected]> wrote:
Store the names of license files and check if they are in the hash file.
Signed-off-by Heiko Thiery <[email protected]>
Checking that the license files are in the hash file should not be done in pkg-stats. pkg-stats is about statistics, not about error reporting. If anything, this should be checked by check-package, whose results are anyway taken into account by pkg-stats.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v5 9/12 support/scripts/pkg-stats: add support for license hash check
Hi Thomas,
Am Sa., 7. M?rz 2020 um 16:08 Uhr schrieb Thomas Petazzoni <[email protected]>:
Quoted 4 lines
On Sat, 7 Mar 2020 08:56:30 +0100
Heiko Thiery <[email protected]> wrote:
Quoted 8 lines
Quoted 3 lines
Store the names of license files and check if they are in the hash file.
Signed-off-by Heiko Thiery <[email protected]>Checking that the license files are in the hash file should not be done in pkg-stats. pkg-stats is about statistics, not about error reporting. If anything, this should be checked by check-package, whose results are anyway taken into account by pkg-stats.
I will take a look on it and try to integrate the test in check-package.
Quoted 8 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
BR,
Heiko
PATCH v5 10/12 support/scripts/pkg-stats: set status to 'na' for virtual packages
In reply to Heiko ThieryIf there is no infra set or infra is virtual the status is set to 'na'.
This is done for the follwing checks:
- license
- license-files
- hash
- hash-license
- patches
- version
--- support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
support/scripts/pkg-stats +27 -0
@@ -132,6 +132,15 @@ class Package: def patch_count(self): return len(self.patch_files) + @property + def has_valid_infra(self): + try: + if self.infras[0][1] == 'virtual': + return False + except IndexError: + return False + return True + def set_infra(self): """ Fills in the .infras field
@@ -153,6 +162,11 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ + if not self.has_valid_infra: + self.status['license'] = ("na", "no valid package infra") + self.status['license-files'] = ("na", "no valid package infra") + return + var = self.pkgvar() self.status['license'] = ("error", "missing") self.status['license-files'] = ("error", "missing")
@@ -167,6 +181,11 @@ class Package: """ Fills in the .status['hash'] field """ + if not self.has_valid_infra: + self.status['hash'] = ("na", "no valid package infra") + self.status['hash-license'] = ("na", "no valid package infra") + return + hashpath = self.path.replace(".mk", ".hash") self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath):
@@ -187,6 +206,10 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ + if not self.has_valid_infra: + self.status['patches'] = ("na", "no valid package infra") + return + pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
@@ -566,6 +589,10 @@ def check_package_latest_version(packages): for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + if not pkg.has_valid_infra: + pkg.status['version'] = ("na", "no valid package infra") + continue + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: pkg.status['version'] = ('warning', "Release Monitoring API error") elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
Signature · 1 line
2.20.1
PATCH v5 11/12 support/scripts/pkg-stats: add list of status checks to the json output
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
@@ -88,6 +88,11 @@ class Package: all_license_files = dict() all_versions = dict() all_ignored_cves = dict() + # This is the list of all possible checks. Add new checks to this list so + # a tool that post-processeds the json output knows the checks before + # iterating over the packages. + status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license', + 'license-files', 'patches', 'pkg-check', 'url', 'version'] def __init__(self, name, path): self.name = name
@@ -984,6 +989,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output): final = {'packages': pkgs, 'stats': statistics, 'defconfigs': defconfigs, + 'package_status_checks': Package.status_checks, 'commit': commit, 'date': str(date)}
Signature · 1 line
2.20.1
PATCH v5 12/12 support/scripts/pkg-stats: add status for cve check
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
@@ -622,6 +622,12 @@ def check_package_cves(nvd_path, packages): if pkg_name in packages and cve.affects(packages[pkg_name]): packages[pkg_name].cves.append(cve.identifier) + for pkg_name, pkg in packages.items(): + if len(pkg.cves) > 0: + pkg.status['cve'] = ('error', 'affected by cve') + else: + pkg.status['cve'] = ('ok', 'no cve found') + def calculate_stats(packages): stats = defaultdict(int)
Signature · 1 line
2.20.1
v5 cover pkg-stats json output improvements
In reply to Heiko ThieryHello,
Quoted 5 lines · On Sat, 7 Mar 2020 08:56:21 +0100 Heiko Thiery <[email protected]> wrote:
Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package
Quoted 8 lines
support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve check
I've applied the entire series to next, except "support/scripts/pkg-stats: add support for license hash check" as I explained as a reply to this patch.
Also, I fixed the daily-mail script from the buildroot-test repo to be compatible with the new JSON format.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com