Skip to contentSkip to conversation

pkg-stats json output improvements

PATCH Superseded Heiko Thiery 2 participants 15 messages Mar 2, 2020 – Mar 6, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover pkg-stats json output improvements

- add developers information to the packages
- add supported defconfigs to json
- add license information to json
- add license file information to json
- add patch files to json
- add a more generic check status field to the packages for easier post processing. The following check status are availble: checks: cve, license, license-files, hash, hash-license,patches, version, url, developers, pkg-check

---
v3 -> v4:

Included the review comments from Titouan

- use only one signed-off mail address
- do some things in a more pythonic way
- remove the patch_count attribute and use a class property instead (Note that the patch_count value will not go into the json output anymore,but then coutn can be computed by "len(patch_files)".)
- remove the status initializer in the package class that was introduced in v3
- update the cve status at the end of the cve check loop
- change the is_valid_infra -> has_valid_infra and use it as property

v2 -> v3:
- keep variable latest_release but change format
- add check for license file hashes
- introduce a na status for the checks
- add a list of all possible checks to the json output
- add the cve check status

v1 -> v2:
- cleanup and recreation of patches
- remove pkg name from dumping to json
- use patch_files instead of combine count and files in dict
- include getdevelopers.py to reuse Developers class

---

Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve check

 support/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++---------
 1 file changed, 211 insertions(+), 60 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v4 1/12 support/scripts/pkg-stats: store latest version in a dict

This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output.

---
 support/scripts/pkg-stats | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
support/scripts/pkg-stats +15 -16
index 7721d98459..727b203032 100755
@@ -71,7 +71,7 @@ class Package:
         self.url_status = None
         self.url_worker = None
         self.cves = list()
-        self.latest_version = (RM_API_STATUS_ERROR, None, None)
+        self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
@@ -460,9 +460,8 @@ def check_package_latest_version(packages):
     """
     Fills in the .latest_version field of all Package objects
 
-    This field has a special format:
-      (status, version, id)
-    with:
+    This field is a dict and has the following keys:
+
     - status: one of RM_API_STATUS_ERROR,
       RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN,
       RM_API_STATUS_NOT_FOUND
@@ -478,7 +477,7 @@ def check_package_latest_version(packages):
     worker_pool = Pool(processes=64)
     results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
     for pkg, r in zip(packages, results):
-        pkg.latest_version = r
+        pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
     del http_pool
 
 
@@ -516,13 +515,13 @@ def calculate_stats(packages):
             stats["hash"] += 1
         else:
             stats["no-hash"] += 1
-        if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+        if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO:
             stats["rmo-mapping"] += 1
         else:
             stats["rmo-no-mapping"] += 1
-        if not pkg.latest_version[1]:
+        if not pkg.latest_version['version']:
             stats["version-unknown"] += 1
-        elif pkg.latest_version[1] == pkg.current_version:
+        elif pkg.latest_version['version'] == pkg.current_version:
             stats["version-uptodate"] += 1
         else:
             stats["version-not-uptodate"] += 1
@@ -688,29 +687,29 @@ def dump_html_pkg(f, pkg):
     f.write("  <td class=\"centered\">%s</td>\n" % current_version)
 
     # Latest version
-    if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+    if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
         td_class.append("version-error")
-    if pkg.latest_version[1] is None:
+    if pkg.latest_version['version'] is None:
         td_class.append("version-unknown")
-    elif pkg.latest_version[1] != pkg.current_version:
+    elif pkg.latest_version['version'] != pkg.current_version:
         td_class.append("version-needs-update")
     else:
         td_class.append("version-good")
 
-    if pkg.latest_version[0] == RM_API_STATUS_ERROR:
+    if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
         latest_version_text = "<b>Error</b>"
-    elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND:
+    elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
         latest_version_text = "<b>Not found</b>"
     else:
-        if pkg.latest_version[1] is None:
+        if pkg.latest_version['version'] is None:
             latest_version_text = "<b>Found, but no version</b>"
         else:
             latest_version_text = "<a href=\"https://release-monitoring.org/project/%s\"><b>%s</b></a>" % \
-                (pkg.latest_version[2], str(pkg.latest_version[1]))
+                (pkg.latest_version['id'], str(pkg.latest_version['version']))
 
         latest_version_text += "<br/>"
 
-        if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO:
+        if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO:
             latest_version_text += "found by <a href=\"https://release-monitoring.org/distro/Buildroot/\">distro</a>"
         else:
             latest_version_text += "found by guess"
Signature · 1 line
2.20.1
#2 lore raw

PATCH v4 2/12 support/scripts/pkg-stats: store patch files for the package

In reply to Heiko Thiery

Remove the patch_count attribute and use a class property instead.

---
 support/scripts/pkg-stats | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
support/scripts/pkg-stats +6 -3
index 727b203032..4116622c98 100755
@@ -64,7 +64,7 @@ class Package:
         self.has_license = False
         self.has_license_files = False
         self.has_hash = False
-        self.patch_count = 0
+        self.patch_files = []
         self.warnings = 0
         self.current_version = None
         self.url = None
@@ -93,6 +93,10 @@ class Package:
                 self.url_status = "Missing"
                 fp.close()
 
+    @property
+    def patch_count(self):
+        return len(self.patch_files)
+
     def set_infra(self):
         """
         Fills in the .infras field
@@ -131,10 +135,9 @@ class Package:
         """
         Fills in the .patch_count field
         """
-        self.patch_count = 0
         pkgdir = os.path.dirname(self.path)
         for subdir, _, _ in os.walk(pkgdir):
-            self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch'))
+            self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
 
     def set_current_version(self):
         """
Signature · 1 line
2.20.1
#3 lore raw

PATCH v4 3/12 support/scripts/pkg-stats: set developers info

In reply to Heiko Thiery

Use the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package.

---
 support/scripts/pkg-stats | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
support/scripts/pkg-stats +17 -0
index 4116622c98..c40fe61d9e 100755
@@ -30,10 +30,14 @@ import certifi
 import distutils.version
 import time
 import gzip
+import sys
 from urllib3 import HTTPSConnectionPool
 from urllib3.exceptions import HTTPError
 from multiprocessing import Pool
 
+sys.path.append('utils/')
+from getdeveloperlib import parse_developers
+
 NVD_START_YEAR = 2002
 NVD_JSON_VERSION = "1.0"
 NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
@@ -171,6 +175,16 @@ class Package:
         """
         return cve in self.all_ignored_cves.get(self.pkgvar(), [])
 
+    def set_developers(self, developers):
+        """
+        Fills in the .developers field
+        """
+        self.developers = [
+            dev.name
+            for dev in developers
+            if dev.hasfile(self.path)
+        ]
+
     def __eq__(self, other):
         return self.path == other.path
 
@@ -891,6 +905,8 @@ def __main__():
                                       'HEAD']).splitlines()[0]
     print("Build package list ...")
     packages = get_pkglist(args.npackages, package_list)
+    print("Getting developers ...")
+    developers = parse_developers()
     print("Getting package make info ...")
     package_init_make_info()
     print("Getting package details ...")
@@ -902,6 +918,7 @@ def __main__():
         pkg.set_check_package_warnings()
         pkg.set_current_version()
         pkg.set_url()
+        pkg.set_developers(developers)
     print("Checking URL status")
     check_package_urls(packages)
     print("Getting latest versions ...")
Signature · 1 line
2.20.1
#4 lore raw

PATCH v4 4/12 support/scripts/pkg-stats: store licences of package

In reply to Heiko Thiery
---
 support/scripts/pkg-stats | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +4 -2
index c40fe61d9e..b0ee4f3bac 100755
@@ -56,7 +56,7 @@ http_pool = None
 
 
 class Package:
-    all_licenses = list()
+    all_licenses = dict()
     all_license_files = list()
     all_versions = dict()
     all_ignored_cves = dict()
@@ -65,6 +65,7 @@ class Package:
         self.name = name
         self.path = path
         self.infras = None
+        self.license = None
         self.has_license = False
         self.has_license_files = False
         self.has_hash = False
@@ -125,6 +126,7 @@ class Package:
         var = self.pkgvar()
         if var in self.all_licenses:
             self.has_license = True
+            self.license = self.all_licenses[var]
         if var in self.all_license_files:
             self.has_license_files = True
 
@@ -387,7 +389,7 @@ def package_init_make_info():
             if value == "unknown":
                 continue
             pkgvar = pkgvar[:-8]
-            Package.all_licenses.append(pkgvar)
+            Package.all_licenses[pkgvar] = value
 
         elif pkgvar.endswith("_LICENSE_FILES"):
             if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
Signature · 1 line
2.20.1
#5 lore raw

PATCH v4 5/12 support/scripts/pkg-stats: add package status

In reply to Heiko Thiery

Unify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message.

The following checks are performed:
- url: status of the URL check
- license: status of the license presence check
- license-files: status of the license file check
- hash: status of the hash file presence check
- patches: status of the patches count check
- pkg-check: status of the check-package script result
- developers: status if a package has developers in the DEVELOPERS file
- version: status of the version check

With that status information the following variables are replaced:
has_license, has_license_files, has_hash, url_status

---
 support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------
 1 file changed, 69 insertions(+), 32 deletions(-)
support/scripts/pkg-stats +69 -32
index b0ee4f3bac..47f7507540 100755
@@ -73,10 +73,10 @@ class Package:
         self.warnings = 0
         self.current_version = None
         self.url = None
-        self.url_status = None
         self.url_worker = None
         self.cves = list()
         self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
+        self.status = {}
 
     def pkgvar(self):
         return self.name.upper().replace("-", "_")
@@ -85,17 +85,17 @@ class Package:
         """
         Fills in the .url field
         """
-        self.url_status = "No Config.in"
+        self.status['url'] = ("warning", "no Config.in")
         for filename in os.listdir(os.path.dirname(self.path)):
             if fnmatch.fnmatch(filename, 'Config.*'):
                 fp = open(os.path.join(os.path.dirname(self.path), filename), "r")
                 for config_line in fp:
                     if URL_RE.match(config_line):
                         self.url = config_line.strip()
-                        self.url_status = "Found"
+                        self.status['url'] = ("ok", "found")
                         fp.close()
                         return
-                self.url_status = "Missing"
+                self.status['url'] = ("error", "missing")
                 fp.close()
 
     @property
@@ -121,30 +121,42 @@ class Package:
 
     def set_license(self):
         """
-        Fills in the .has_license and .has_license_files fields
+        Fills in the .status['license'] and .status['license-files'] fields
         """
         var = self.pkgvar()
+        self.status['license'] = ("error", "missing")
+        self.status['license-files'] = ("error", "missing")
         if var in self.all_licenses:
-            self.has_license = True
             self.license = self.all_licenses[var]
+            self.status['license'] = ("ok", "found")
         if var in self.all_license_files:
-            self.has_license_files = True
+            self.status['license-files'] = ("ok", "found")
 
     def set_hash_info(self):
         """
-        Fills in the .has_hash field
+        Fills in the .status['hash'] field
         """
         hashpath = self.path.replace(".mk", ".hash")
-        self.has_hash = os.path.exists(hashpath)
+        if os.path.exists(hashpath):
+            self.status['hash'] = ("ok", "found")
+        else:
+            self.status['hash'] = ("error", "missing")
 
     def set_patch_count(self):
         """
-        Fills in the .patch_count field
+        Fills in the .patch_count, .patch_files and .status['patches'] fields
         """
         pkgdir = os.path.dirname(self.path)
         for subdir, _, _ in os.walk(pkgdir):
             self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
 
+        if self.patch_count == 0:
+            self.status['patches'] = ("ok", "no patches")
+        elif self.patch_count < 5:
+            self.status['patches'] = ("warning", "some patches")
+        else:
+            self.status['patches'] = ("error", "lots of patches")
+
     def set_current_version(self):
         """
         Fills in the .current_version field
@@ -155,10 +167,11 @@ class Package:
 
     def set_check_package_warnings(self):
         """
-        Fills in the .warnings field
+        Fills in the .warnings and .status['pkg-check'] fields
         """
         cmd = ["./utils/check-package"]
         pkgdir = os.path.dirname(self.path)
+        self.status['pkg-check'] = ("error", "Missing")
         for root, dirs, files in os.walk(pkgdir):
             for f in files:
                 if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host":
@@ -169,6 +182,10 @@ class Package:
             m = re.match("^([0-9]*) warnings generated", line)
             if m:
                 self.warnings = int(m.group(1))
+                if self.warnings == 0:
+                    self.status['pkg-check'] = ("ok", "no warnings")
+                else:
+                    self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings))
                 return
 
     def is_cve_ignored(self, cve):
@@ -179,7 +196,7 @@ class Package:
 
     def set_developers(self, developers):
         """
-        Fills in the .developers field
+        Fills in the .developers and .status['developers'] field
         """
         self.developers = [
             dev.name
@@ -187,6 +204,14 @@ class Package:
             if dev.hasfile(self.path)
         ]
 
+        if self.developers:
+            self.status['developers'] = ("ok", "{} developers".format(len(self.developers)))
+        else:
+            self.status['developers'] = ("warning", "no developers")
+
+    def is_status_ok(self, name):
+        return self.status[name][0] == 'ok'
+
     def __eq__(self, other):
         return self.path == other.path
 
@@ -195,7 +220,7 @@ class Package:
 
     def __str__(self):
         return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \
-            (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count)
+            (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count)
 
 
 class CVE:
@@ -409,23 +434,23 @@ def package_init_make_info():
 
 
 def check_url_status_worker(url, url_status):
-    if url_status != "Missing" and url_status != "No Config.in":
+    if url_status[0] == 'ok':
         try:
             url_status_code = requests.head(url, timeout=30).status_code
             if url_status_code >= 400:
-                return "Invalid(%s)" % str(url_status_code)
+                return ("error", "invalid {}".format(url_status_code))
         except requests.exceptions.RequestException:
-            return "Invalid(Err)"
-        return "Ok"
+            return ("error", "invalid (err)")
+        return ("ok", "valid")
     return url_status
 
 
 def check_package_urls(packages):
     Package.pool = Pool(processes=64)
     for pkg in packages:
-        pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status))
+        pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url']))
     for pkg in packages:
-        pkg.url_status = pkg.url_worker.get(timeout=3600)
+        pkg.status['url'] = pkg.url_worker.get(timeout=3600)
 
 
 def release_monitoring_get_latest_version_by_distro(pool, name):
@@ -497,6 +522,18 @@ def check_package_latest_version(packages):
     results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages))
     for pkg, r in zip(packages, results):
         pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
+
+        if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
+            pkg.status['version'] = ('warning', "Release Monitoring API error")
+        elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
+            pkg.status['version'] = ('warning', "Package not found on Release Monitoring")
+
+        if pkg.latest_version['version'] is None:
+            pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring")
+        elif pkg.latest_version['version'] != pkg.current_version:
+            pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version']))
+        else:
+            pkg.status['version'] = ('ok', 'up-to-date')
     del http_pool
 
 
@@ -522,15 +559,15 @@ def calculate_stats(packages):
             stats["infra-%s" % infra] += 1
         else:
             stats["infra-unknown"] += 1
-        if pkg.has_license:
+        if pkg.is_status_ok('license'):
             stats["license"] += 1
         else:
             stats["no-license"] += 1
-        if pkg.has_license_files:
+        if pkg.is_status_ok('license-files'):
             stats["license-files"] += 1
         else:
             stats["no-license-files"] += 1
-        if pkg.has_hash:
+        if pkg.is_status_ok('hash'):
             stats["hash"] += 1
         else:
             stats["no-hash"] += 1
@@ -673,30 +710,30 @@ def dump_html_pkg(f, pkg):
 
     # License
     td_class = ["centered"]
-    if pkg.has_license:
+    if pkg.is_status_ok('license'):
         td_class.append("correct")
     else:
         td_class.append("wrong")
     f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_license)))
+            (" ".join(td_class), boolean_str(pkg.is_status_ok('license'))))
 
     # License files
     td_class = ["centered"]
-    if pkg.has_license_files:
+    if pkg.is_status_ok('license-files'):
         td_class.append("correct")
     else:
         td_class.append("wrong")
     f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_license_files)))
+            (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files'))))
 
     # Hash
     td_class = ["centered"]
-    if pkg.has_hash:
+    if pkg.is_status_ok('hash'):
         td_class.append("correct")
     else:
         td_class.append("wrong")
     f.write("  <td class=\"%s\">%s</td>\n" %
-            (" ".join(td_class), boolean_str(pkg.has_hash)))
+            (" ".join(td_class), boolean_str(pkg.is_status_ok('hash'))))
 
     # Current version
     if len(pkg.current_version) > 20:
@@ -747,12 +784,12 @@ def dump_html_pkg(f, pkg):
 
     # URL status
     td_class = ["centered"]
-    url_str = pkg.url_status
-    if pkg.url_status == "Missing" or pkg.url_status == "No Config.in":
+    url_str = pkg.status['url'][1]
+    if pkg.status['url'][0] in ("error", "warning"):
         td_class.append("missing_url")
-    elif pkg.url_status.startswith("Invalid"):
+    if pkg.status['url'][0] == "error":
         td_class.append("invalid_url")
-        url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.url_status)
+        url_str = "<a href=%s>%s</a>" % (pkg.url, pkg.status['url'][1])
     else:
         td_class.append("good_url")
         url_str = "<a href=%s>Link</a>" % pkg.url
Signature · 1 line
2.20.1
#6 lore raw

PATCH v4 6/12 support/scripts/pkg-stats: add package count to stats

In reply to Heiko Thiery
---
 support/scripts/pkg-stats | 1 +
 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
index 47f7507540..2847cb82f3 100755
@@ -549,6 +549,7 @@ def check_package_cves(nvd_path, packages):
 
 def calculate_stats(packages):
     stats = defaultdict(int)
+    stats['packages'] = len(packages)
     for pkg in packages:
         # If packages have multiple infra, take the first one. For the
         # vast majority of packages, the target and host infra are the
Signature · 1 line
2.20.1
#7 lore raw

PATCH v4 7/12 support/scripts/pkg-stats: store pkg dir path

In reply to Heiko Thiery

This value can be used for later processing.

In the buildroot-stats application this is used to create links pointing to the git repo of buildroot.

---
 support/scripts/pkg-stats | 1 +
 1 file changed, 1 insertion(+)
support/scripts/pkg-stats +1 -0
index 2847cb82f3..1fd306a6d5 100755
@@ -64,6 +64,7 @@ class Package:
     def __init__(self, name, path):
         self.name = name
         self.path = path
+        self.pkg_path = os.path.dirname(path)
         self.infras = None
         self.license = None
         self.has_license = False
Signature · 1 line
2.20.1
#8 lore raw

PATCH v4 8/12 support/scripts/pkg-stats: add defconfig support

In reply to Heiko Thiery

Scan configs directory and create Defconfig objects.

---
 support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +41 -2
index 1fd306a6d5..432090c251 100755
@@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4
 # because it's used by sub-processes.
 http_pool = None
 
+class Defconfig:
+    def __init__(self, name, path):
+        self.name = name
+        self.path = path
+        self.developers = None
+
+    def set_developers(self, developers):
+        """
+        Fills in the .developers field
+        """
+        self.developers = [
+            developer.name
+            for developer in developers
+            if developer.hasfile(self.path)
+        ]
+
+
+def get_defconfig_list():
+    """
+    Builds the list of Buildroot defconfigs, returning a list of Defconfig
+    objects.
+    """
+    return [
+        Defconfig(name[:-len('_defconfig')], os.path.join('configs', name))
+        for name in os.listdir('configs')
+        if name.endswith('_defconfig')
+    ]
+
 
 class Package:
     all_licenses = dict()
@@ -886,7 +914,7 @@ def dump_html(packages, stats, date, commit, output):
         f.write(html_footer)
 
 
-def dump_json(packages, stats, date, commit, output):
+def dump_json(packages, defconfigs, stats, date, commit, output):
     # Format packages as a dictionnary instead of a list
     # Exclude local field that does not contains real date
     excluded_fields = ['url_worker', 'name']
@@ -897,6 +925,12 @@ def dump_json(packages, stats, date, commit, output):
             if k not in excluded_fields
         } for pkg in packages
     }
+    defconfigs = {
+        d.name: {
+            k: v
+            for k, v in d.__dict__.items()
+        } for d in defconfigs
+    }
     # Aggregate infrastructures into a single dict entry
     statistics = {
         k: v
@@ -907,6 +941,7 @@ def dump_json(packages, stats, date, commit, output):
     # The actual structure to dump, add commit and date to it
     final = {'packages': pkgs,
              'stats': statistics,
+             'defconfigs': defconfigs,
              'commit': commit,
              'date': str(date)}
 
@@ -948,6 +983,10 @@ def __main__():
     packages = get_pkglist(args.npackages, package_list)
     print("Getting developers ...")
     developers = parse_developers()
+    print("Build defconfig list ...")
+    defconfigs = get_defconfig_list()
+    for d in defconfigs:
+        d.set_developers(developers)
     print("Getting package make info ...")
     package_init_make_info()
     print("Getting package details ...")
@@ -974,7 +1013,7 @@ def __main__():
         dump_html(packages, stats, date, commit, args.html)
     if args.json:
         print("Write JSON")
-        dump_json(packages, stats, date, commit, args.json)
+        dump_json(packages, defconfigs, stats, date, commit, args.json)
 
 
 __main__()
Signature · 1 line
2.20.1
#9 lore raw

PATCH v4 9/12 support/scripts/pkg-stats: add support for license hash check

In reply to Heiko Thiery

Store the names of license files and check if they are in the hash file.

---
 support/scripts/pkg-stats | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
support/scripts/pkg-stats +14 -2
index 432090c251..c9287b416e 100755
@@ -85,7 +85,7 @@ def get_defconfig_list():
 
 class Package:
     all_licenses = dict()
-    all_license_files = list()
+    all_license_files = dict()
     all_versions = dict()
     all_ignored_cves = dict()
 
@@ -98,6 +98,7 @@ class Package:
         self.has_license = False
         self.has_license_files = False
         self.has_hash = False
+        self.license_files = None
         self.patch_files = []
         self.warnings = 0
         self.current_version = None
@@ -159,6 +160,7 @@ class Package:
             self.license = self.all_licenses[var]
             self.status['license'] = ("ok", "found")
         if var in self.all_license_files:
+            self.license_files = self.all_license_files[var].split(' ')
             self.status['license-files'] = ("ok", "found")
 
     def set_hash_info(self):
@@ -166,8 +168,18 @@ class Package:
         Fills in the .status['hash'] field
         """
         hashpath = self.path.replace(".mk", ".hash")
+        self.status['hash-license'] = ("na", "no hash file")
         if os.path.exists(hashpath):
             self.status['hash'] = ("ok", "found")
+            self.status['hash-license'] = ("error", "no license in hash file")
+            # check if license files are in hash file
+            if self.license_files is not None:
+                self.status['hash-license'] = ("ok", "found")
+                with open(hashpath) as f:
+                    content = f.read()
+                    for license in self.license_files:
+                        if content.find(license) == -1:
+                            self.status['hash-license'] = ("error", "license missing in hash file")
         else:
             self.status['hash'] = ("error", "missing")
 
@@ -449,7 +461,7 @@ def package_init_make_info():
             if pkgvar.endswith("_MANIFEST_LICENSE_FILES"):
                 continue
             pkgvar = pkgvar[:-14]
-            Package.all_license_files.append(pkgvar)
+            Package.all_license_files[pkgvar] = value
 
         elif pkgvar.endswith("_VERSION"):
             if pkgvar.endswith("_DL_VERSION"):
Signature · 1 line
2.20.1
#10 lore raw

PATCH v4 10/12 support/scripts/pkg-stats: set status to 'na' for virtual packages

In reply to Heiko Thiery

If there is no infra set or infra is virtual the status is set to 'na'.

This is done for the follwing checks:
- license
- license-files
- hash
- hash-license
- patches
- version

---
 support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
support/scripts/pkg-stats +27 -0
index c9287b416e..4cf1a919f3 100755
@@ -132,6 +132,15 @@ class Package:
     def patch_count(self):
         return len(self.patch_files)
 
+    @property
+    def has_valid_infra(self):
+        try:
+            if self.infras[0][1] == 'virtual':
+                return False
+        except IndexError:
+                return False
+        return True
+
     def set_infra(self):
         """
         Fills in the .infras field
@@ -153,6 +162,11 @@ class Package:
         """
         Fills in the .status['license'] and .status['license-files'] fields
         """
+        if not self.has_valid_infra:
+            self.status['license'] = ("na", "no valid package infra")
+            self.status['license-files'] = ("na", "no valid package infra")
+            return
+
         var = self.pkgvar()
         self.status['license'] = ("error", "missing")
         self.status['license-files'] = ("error", "missing")
@@ -167,6 +181,11 @@ class Package:
         """
         Fills in the .status['hash'] field
         """
+        if not self.has_valid_infra:
+            self.status['hash'] = ("na", "no valid package infra")
+            self.status['hash-license'] = ("na", "no valid package infra")
+            return
+
         hashpath = self.path.replace(".mk", ".hash")
         self.status['hash-license'] = ("na", "no hash file")
         if os.path.exists(hashpath):
@@ -187,6 +206,10 @@ class Package:
         """
         Fills in the .patch_count, .patch_files and .status['patches'] fields
         """
+        if not self.has_valid_infra:
+            self.status['patches'] = ("na", "no valid package infra")
+            return
+
         pkgdir = os.path.dirname(self.path)
         for subdir, _, _ in os.walk(pkgdir):
             self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch')
@@ -564,6 +587,10 @@ def check_package_latest_version(packages):
     for pkg, r in zip(packages, results):
         pkg.latest_version = dict(zip(['status', 'version', 'id'], r))
 
+        if not pkg.has_valid_infra:
+            pkg.status['version'] = ("na", "no valid package infra")
+            continue
+
         if pkg.latest_version['status'] == RM_API_STATUS_ERROR:
             pkg.status['version'] = ('warning', "Release Monitoring API error")
         elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND:
Signature · 1 line
2.20.1
#11 lore raw

PATCH v4 11/12 support/scripts/pkg-stats: add list of status checks to the json output

In reply to Heiko Thiery
---
 support/scripts/pkg-stats | 6 ++++++
 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
index 4cf1a919f3..9d058ca67d 100755
@@ -88,6 +88,11 @@ class Package:
     all_license_files = dict()
     all_versions = dict()
     all_ignored_cves = dict()
+    # This is the list of all possible checks. Add new checks to this list so
+    # a tool that post-processeds the json output knows the checks before
+    # iterating over the packages.
+    status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license',
+                     'license-files', 'patches', 'pkg-check', 'url', 'version']
 
     def __init__(self, name, path):
         self.name = name
@@ -981,6 +986,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output):
     final = {'packages': pkgs,
              'stats': statistics,
              'defconfigs': defconfigs,
+             'package_status_checks': Package.status_checks,
              'commit': commit,
              'date': str(date)}
 
Signature · 1 line
2.20.1
#12 lore raw

PATCH v4 12/12 support/scripts/pkg-stats: add status for cve check

In reply to Heiko Thiery
---
 support/scripts/pkg-stats | 6 ++++++
 1 file changed, 6 insertions(+)
support/scripts/pkg-stats +6 -0
index 9d058ca67d..b0ae681c22 100755
@@ -619,6 +619,12 @@ def check_package_cves(nvd_path, packages):
             if pkg_name in packages and cve.affects(packages[pkg_name]):
                 packages[pkg_name].cves.append(cve.identifier)
 
+    for pkg_name, pkg in packages.items():
+        if len(pkg.cves) > 0:
+            pkg.status['cve'] = ('error', 'affected by cve')
+        else:
+            pkg.status['cve'] = ('ok', 'no cve found')
+
 
 def calculate_stats(packages):
     stats = defaultdict(int)
Signature · 1 line
2.20.1
#13 lore raw

v4 cover pkg-stats json output improvements

In reply to Heiko Thiery

Hello Heiko,

Thanks for sending this updated series.

However it does not apply nicely on top of master (probably because of the last patch I sent to make pkg-stats compatible with Python3).

Would you care to resend ?

Titouan

Quoted 57 lines · Heiko Thiery

- add developers information to the packages
- add supported defconfigs to json
- add license information to json
- add license file information to json
- add patch files to json
- add a more generic check status field to the packages for easier post processing. The following check status are availble: checks: cve, license, license-files, hash, hash-license,patches, version, url, developers, pkg-check

---
v3 -> v4:

Included the review comments from Titouan

- use only one signed-off mail address
- do some things in a more pythonic way
- remove the patch_count attribute and use a class property instead (Note that the patch_count value will not go into the json output anymore,but then coutn can be computed by "len(patch_files)".)
- remove the status initializer in the package class that was introduced in v3
- update the cve status at the end of the cve check loop
- change the is_valid_infra -> has_valid_infra and use it as property

v2 -> v3:
- keep variable latest_release but change format
- add check for license file hashes
- introduce a na status for the checks
- add a list of all possible checks to the json output
- add the cve check status

v1 -> v2:
- cleanup and recreation of patches
- remove pkg name from dumping to json
- use patch_files instead of combine count and files in dict
- include getdevelopers.py to reuse Developers class

---

Heiko Thiery (12):
support/scripts/pkg-stats: store latest version in a dict support/scripts/pkg-stats: store patch files for the package support/scripts/pkg-stats: set developers info support/scripts/pkg-stats: store licences of package support/scripts/pkg-stats: add package status support/scripts/pkg-stats: add package count to stats support/scripts/pkg-stats: store pkg dir path support/scripts/pkg-stats: add defconfig support support/scripts/pkg-stats: add support for license hash check support/scripts/pkg-stats: set status to 'na' for virtual packages support/scripts/pkg-stats: initialize all package status checks support/scripts/pkg-stats: add status for cve check

  support/scripts/pkg-stats | 271 +++++++++++++++++++++++++++++---------
  1 file changed, 211 insertions(+), 60 deletions(-)
#14 lore raw

v4 cover pkg-stats json output improvements

Hi Tiouan and all,

Am Fr., 6. M?rz 2020 um 17:46 Uhr schrieb Titouan Christophe <[email protected]>:

Quoted 9 lines

Hello Heiko,

Thanks for sending this updated series.

However it does not apply nicely on top of master (probably because of the last patch I sent to make pkg-stats compatible with Python3).

Would you care to resend ?

I will rebase to the master and come back with a v5.

Quoted 2 lines

Titouan

BR,
Heiko