pkg-stats json output improvements
v4 cover pkg-stats json output improvements
- add developers information to the packages
- add supported defconfigs to json
- add license information to json
- add license file information to json
- add patch files to json
- add a more generic check status field to the packages for easier post processing. The following check status are availble: checks: cve, license, license-files, hash, hash-license,patches, version, url, developers, pkg-check
---
v3 -> v4:
Included the review comments from Titouan
- use only one signed-off mail address
- do some things in a more pythonic way
- remove the patch_count attribute and use a class property instead (Note that the patch_count value will not go into the json output anymore,but then coutn can be computed by "len(patch_files)".)
- remove the status initializer in the package class that was introduced in v3
- update the cve status at the end of the cve check loop
- change the is_valid_infra -> has_valid_infra and use it as property
v2 -> v3:
- keep variable latest_release but change format
- add check for license file hashes
- introduce a na status for the checks
- add a list of all possible checks to the json output
- add the cve check status
v1 -> v2:
- cleanup and recreation of patches
- remove pkg name from dumping to json
- use patch_files instead of combine count and files in dict
- include getdevelopers.py to reuse Developers class
---
Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve check
support/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++--------- 1 file changed, 211 insertions(+), 60 deletions(-)
Signature · 1 line
2.20.1
PATCH v4 1/12 support/scripts/pkg-stats: store latest version in a dict
This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output.
--- support/scripts/pkg-stats | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-)
support/scripts/pkg-stats +15 -16
@@ -71,7 +71,7 @@ class Package: self.url_status = None self.url_worker = None self.cves = list() - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_")
@@ -460,9 +460,8 @@ def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects - This field has a special format: - (status, version, id) - with: + This field is a dict and has the following keys: + - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND
@@ -478,7 +477,7 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) del http_pool
@@ -516,13 +515,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.latest_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.latest_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1
@@ -688,29 +687,29 @@ def dump_html_pkg(f, pkg): f.write(" <td class=\"centered\">%s</td>\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.latest_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "<b>Error</b>" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "<b>Not found</b>" else: - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: latest_version_text = "<b>Found, but no version</b>" else: latest_version_text = "<a href=\"https://release-monitoring.org/project/%s\"><b>%s</b></a>" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.latest_version['id'], str(pkg.latest_version['version'])) latest_version_text += "<br/>" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by <a href=\"https://release-monitoring.org/distro/Buildroot/\">distro</a>" else: latest_version_text += "found by guess"
Signature · 1 line
2.20.1
PATCH v4 2/12 support/scripts/pkg-stats: store patch files for the package
In reply to Heiko ThieryRemove the patch_count attribute and use a class property instead.
--- support/scripts/pkg-stats | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
support/scripts/pkg-stats +6 -3
@@ -64,7 +64,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False - self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None
@@ -93,6 +93,10 @@ class Package: self.url_status = "Missing" fp.close() + @property + def patch_count(self): + return len(self.patch_files) + def set_infra(self): """ Fills in the .infras field
@@ -131,10 +135,9 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') def set_current_version(self): """
Signature · 1 line
2.20.1
PATCH v4 3/12 support/scripts/pkg-stats: set developers info
In reply to Heiko ThieryUse the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package.
--- support/scripts/pkg-stats | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
support/scripts/pkg-stats +17 -0
@@ -30,10 +30,14 @@ import certifi import distutils.version import time import gzip +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + NVD_START_YEAR = 2002 NVD_JSON_VERSION = "1.0" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
@@ -171,6 +175,16 @@ class Package: """ return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + dev.name + for dev in developers + if dev.hasfile(self.path) + ] + def __eq__(self, other): return self.path == other.path
@@ -891,6 +905,8 @@ def __main__(): 'HEAD']).splitlines()[0] print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...")
@@ -902,6 +918,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...")
Signature · 1 line
2.20.1
PATCH v4 4/12 support/scripts/pkg-stats: store licences of package
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +4 -2
@@ -56,7 +56,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() all_ignored_cves = dict()
@@ -65,6 +65,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False
@@ -125,6 +126,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True
@@ -387,7 +389,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
Signature · 1 line
2.20.1
PATCH v4 5/12 support/scripts/pkg-stats: add package status
In reply to Heiko ThieryUnify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message.
The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check
With that status information the following variables are replaced:has_license, has_license_files, has_hash, url_status
--- support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 32 deletions(-)
support/scripts/pkg-stats +69 -32
@@ -73,10 +73,10 @@ class Package: self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_")
@@ -85,17 +85,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("error", "missing") fp.close() @property
@@ -121,30 +121,42 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field
@@ -155,10 +167,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
@@ -169,6 +182,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def is_cve_ignored(self, cve):
@@ -179,7 +196,7 @@ class Package: def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = [ dev.name
@@ -187,6 +204,14 @@ class Package: if dev.hasfile(self.path) ] + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + else: + self.status['developers'] = ("warning", "no developers") + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path
@@ -195,7 +220,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) class CVE:
@@ -409,23 +434,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): Package.pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) def release_monitoring_get_latest_version_by_distro(pool, name):
@@ -497,6 +522,18 @@ def check_package_latest_version(packages): results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', "Release Monitoring API error") + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', "Package not found on Release Monitoring") + + if pkg.latest_version['version'] is None: + pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring") + elif pkg.latest_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version'])) + else: + pkg.status['version'] = ('ok', 'up-to-date') del http_pool
@@ -522,15 +559,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1
@@ -673,30 +710,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" <td class=\"%s\">%s</td>\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20:
@@ -747,12 +784,12 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] in ("error", "warning"): td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): + if pkg.status['url'][0] == "error": td_class.append("invalid_url") - url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.url_status) + url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "<a href=%s>Link</a>" % pkg.url
Signature · 1 line
2.20.1
PATCH v4 6/12 support/scripts/pkg-stats: add package count to stats
In reply to Heiko Thiery--- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
@@ -549,6 +549,7 @@ def check_package_cves(nvd_path, packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the
Signature · 1 line
2.20.1
PATCH v4 7/12 support/scripts/pkg-stats: store pkg dir path
In reply to Heiko ThieryThis value can be used for later processing.
In the buildroot-stats application this is used to create links pointing to the git repo of buildroot.
--- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
@@ -64,6 +64,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.has_license = False
Signature · 1 line
2.20.1
PATCH v4 8/12 support/scripts/pkg-stats: add defconfig support
In reply to Heiko ThieryScan configs directory and create Defconfig objects.
--- support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +41 -2
@@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + developer.name + for developer in developers + if developer.hasfile(self.path) + ] + + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + return [ + Defconfig(name[:-len('_defconfig')], os.path.join('configs', name)) + for name in os.listdir('configs') + if name.endswith('_defconfig') + ] + class Package: all_licenses = dict()
@@ -886,7 +914,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name']
@@ -897,6 +925,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v
@@ -907,6 +941,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)}
@@ -948,6 +983,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...")
@@ -974,7 +1013,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__()
Signature · 1 line
2.20.1
PATCH v4 9/12 support/scripts/pkg-stats: add support for license hash check
In reply to Heiko ThieryStore the names of license files and check if they are in the hash file.
--- support/scripts/pkg-stats | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +14 -2
@@ -85,7 +85,7 @@ def get_defconfig_list(): class Package: all_licenses = dict() - all_license_files = list() + all_license_files = dict() all_versions = dict() all_ignored_cves = dict()
@@ -98,6 +98,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False + self.license_files = None self.patch_files = [] self.warnings = 0 self.current_version = None
@@ -159,6 +160,7 @@ class Package: self.license = self.all_licenses[var] self.status['license'] = ("ok", "found") if var in self.all_license_files: + self.license_files = self.all_license_files[var].split(' ') self.status['license-files'] = ("ok", "found") def set_hash_info(self):
@@ -166,8 +168,18 @@ class Package: Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") + self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): self.status['hash'] = ("ok", "found") + self.status['hash-license'] = ("error", "no license in hash file") + # check if license files are in hash file + if self.license_files is not None: + self.status['hash-license'] = ("ok", "found") + with open(hashpath) as f: + content = f.read() + for license in self.license_files: + if content.find(license) == -1: + self.status['hash-license'] = ("error", "license missing in hash file") else: self.status['hash'] = ("error", "missing")
@@ -449,7 +461,7 @@ def package_init_make_info(): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): continue pkgvar = pkgvar[:-14] - Package.all_license_files.append(pkgvar) + Package.all_license_files[pkgvar] = value elif pkgvar.endswith("_VERSION"): if pkgvar.endswith("_DL_VERSION"):
Signature · 1 line
2.20.1
PATCH v4 10/12 support/scripts/pkg-stats: set status to 'na' for virtual packages
In reply to Heiko ThieryIf there is no infra set or infra is virtual the status is set to 'na'.
This is done for the follwing checks:
- license
- license-files
- hash
- hash-license
- patches
- version
--- support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
support/scripts/pkg-stats +27 -0
@@ -132,6 +132,15 @@ class Package: def patch_count(self): return len(self.patch_files) + @property + def has_valid_infra(self): + try: + if self.infras[0][1] == 'virtual': + return False + except IndexError: + return False + return True + def set_infra(self): """ Fills in the .infras field
@@ -153,6 +162,11 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ + if not self.has_valid_infra: + self.status['license'] = ("na", "no valid package infra") + self.status['license-files'] = ("na", "no valid package infra") + return + var = self.pkgvar() self.status['license'] = ("error", "missing") self.status['license-files'] = ("error", "missing")
@@ -167,6 +181,11 @@ class Package: """ Fills in the .status['hash'] field """ + if not self.has_valid_infra: + self.status['hash'] = ("na", "no valid package infra") + self.status['hash-license'] = ("na", "no valid package infra") + return + hashpath = self.path.replace(".mk", ".hash") self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath):
@@ -187,6 +206,10 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ + if not self.has_valid_infra: + self.status['patches'] = ("na", "no valid package infra") + return + pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
@@ -564,6 +587,10 @@ def check_package_latest_version(packages): for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + if not pkg.has_valid_infra: + pkg.status['version'] = ("na", "no valid package infra") + continue + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: pkg.status['version'] = ('warning', "Release Monitoring API error") elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
Signature · 1 line
2.20.1
PATCH v4 11/12 support/scripts/pkg-stats: add list of status checks to the json output
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
@@ -88,6 +88,11 @@ class Package: all_license_files = dict() all_versions = dict() all_ignored_cves = dict() + # This is the list of all possible checks. Add new checks to this list so + # a tool that post-processeds the json output knows the checks before + # iterating over the packages. + status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license', + 'license-files', 'patches', 'pkg-check', 'url', 'version'] def __init__(self, name, path): self.name = name
@@ -981,6 +986,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output): final = {'packages': pkgs, 'stats': statistics, 'defconfigs': defconfigs, + 'package_status_checks': Package.status_checks, 'commit': commit, 'date': str(date)}
Signature · 1 line
2.20.1
PATCH v4 12/12 support/scripts/pkg-stats: add status for cve check
In reply to Heiko Thiery--- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
@@ -619,6 +619,12 @@ def check_package_cves(nvd_path, packages): if pkg_name in packages and cve.affects(packages[pkg_name]): packages[pkg_name].cves.append(cve.identifier) + for pkg_name, pkg in packages.items(): + if len(pkg.cves) > 0: + pkg.status['cve'] = ('error', 'affected by cve') + else: + pkg.status['cve'] = ('ok', 'no cve found') + def calculate_stats(packages): stats = defaultdict(int)
Signature · 1 line
2.20.1
v4 cover pkg-stats json output improvements
In reply to Heiko ThieryHello Heiko,
Thanks for sending this updated series.
However it does not apply nicely on top of master (probably because of the last patch I sent to make pkg-stats compatible with Python3).
Would you care to resend ?
Titouan
Quoted 57 lines · Heiko Thiery
- add developers information to the packages
- add supported defconfigs to json
- add license information to json
- add license file information to json
- add patch files to json
- add a more generic check status field to the packages for easier post processing. The following check status are availble: checks: cve, license, license-files, hash, hash-license,patches, version, url, developers, pkg-check---
v3 -> v4:Included the review comments from Titouan
- use only one signed-off mail address
- do some things in a more pythonic way
- remove thepatch_countattribute and use a class property instead (Note that thepatch_countvalue will not go into the json output anymore,but then coutn can be computed by "len(patch_files)".)
- remove the status initializer in the package class that was introduced in v3
- update the cve status at the end of the cve check loop
- change theis_valid_infra->has_valid_infraand use it as propertyv2 -> v3:
- keep variablelatest_releasebut change format
- add check for license file hashes
- introduce a na status for the checks
- add a list of all possible checks to the json output
- add the cve check statusv1 -> v2:
- cleanup and recreation of patches
- remove pkg name from dumping to json
- usepatch_filesinstead of combine count and files in dict
- includegetdevelopers.pyto reuse Developers class---
Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve checksupport/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++--------- 1 file changed, 211 insertions(+), 60 deletions(-)
v4 cover pkg-stats json output improvements
Hi Tiouan and all,
Am Fr., 6. M?rz 2020 um 17:46 Uhr schrieb Titouan Christophe <[email protected]>:
Quoted 9 lines
Hello Heiko,
Thanks for sending this updated series.
However it does not apply nicely on top of master (probably because of the last patch I sent to make pkg-stats compatible with Python3).
Would you care to resend ?
I will rebase to the master and come back with a v5.
Quoted 2 lines
Titouan
BR,
Heiko