Skip to contentSkip to conversation

Improving CVE reporting

PATCH Superseded Gregory CLEMENT 4 participants 24 messages Jul 10, 2020 – Jul 23, 2020

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Improving CVE reporting

Hello,

The purpose of this series is to improve the CVE reporting in order to be usable for a project.

Until know the CVE affecting the packages were reported for the buildroot project using pkg-stat. With this series it is now possible to report the packages affected by CVEs for a given configuration.

While I was on CVE I switched to the support of the JSON 1.1 for the NVDE database.

In this series I also added a new state for the CVE status of the packages. This new state will be used to emphasize that the automatic check has failed and it was needed to be verified manually. The idea behind this was to be as much accurate as possible to avoid any false positive. It will also help to improve the meta-data of the package.

The next step will be to reuse the works done by Matthew Weber [1] to use the cpeid and only use the package name and the package version as fall back.

In this series there is at least one open point about the packages excluded from the cve check. For now I excluded the kernel and gcc as there are also excluded by the pkg-stats script but this list could (should ?) be extended or modified.

changelog:
remove commit info
pick bf89a711a3 cve-checker: add default path for nvd database pick 3940917b4a cve-checker: remove unused argument use macro
move fix first
s/ignored/ignore/

TODO: remove import
remove debug message

Gregory

[1]: http://patchwork.ozlabs.org/project/buildroot/list/?series=183798

Gregory CLEMENT (9):
support/scripts: Turn CVE check into a module support/scripts/cve.py: Switch to JSON 1.1 package/pkg-utils: show-info: report the list of the CVEs ignored package/pkg-utils: Make CVE class independent of the Pacakage class support/scripts: Add a per configuration CVE checker package/pkg-utils: cve.py: Handle exception when version comparison
fails
support/script/pkg-stats: Manage the CVEs that need to be check support/script/cve-checker: Manage the CVEs that need to be check package/pkg-utils/cve.py: Manage case when package version doesn't
exist

 package/pkg-utils.mk        |   5 +-
 support/scripts/cve-checker | 308 ++++++++++++++++++++++++++++++++++++
 support/scripts/cve.py      | 229 +++++++++++++++++++++++++++
 support/scripts/pkg-stats   | 161 +++++--------------
 4 files changed, 581 insertions(+), 122 deletions(-)
 create mode 100755 support/scripts/cve-checker
 create mode 100755 support/scripts/cve.py
Signature · 1 line
2.27.0
#1 lore raw

PATCH 1/9 support/scripts: Turn CVE check into a module

In order to be able to do CVE checking outside of pkg-stat, move the CVE class in a module that can be used by other scripts.

---
 support/scripts/cve.py    | 141 ++++++++++++++++++++++++++++++++++++++
 support/scripts/pkg-stats | 115 +------------------------------
 2 files changed, 144 insertions(+), 112 deletions(-)
 create mode 100755 support/scripts/cve.py
support/scripts/cve.py new file +141 -0
new file mode 100755
index 0000000000..874ab4482d
@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <[email protected]>
+# Copyright (C) 2020 by Gregory CLEMENT <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import datetime
+import os
+import re
+import requests  # URL checking
+import json
+import ijson
+import distutils.version
+import time
+import gzip
+import sys
+
+sys.path.append('utils/')
+
+NVD_START_YEAR = 2002
+NVD_JSON_VERSION = "1.0"
+NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
+
+class CVE:
+    """An accessor class for CVE Items in NVD files"""
+    def __init__(self, nvd_cve):
+        """Initialize a CVE from its NVD JSON representation"""
+        self.nvd_cve = nvd_cve
+
+    @staticmethod
+    def download_nvd_year(nvd_path, year):
+        metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
+        path_metaf = os.path.join(nvd_path, metaf)
+        jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
+        path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
+
+        # If the database file is less than a day old, we assume the NVD data
+        # locally available is recent enough.
+        if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
+            return path_jsonf_gz
+
+        # If not, we download the meta file
+        url = "%s/%s" % (NVD_BASE_URL, metaf)
+        print("Getting %s" % url)
+        page_meta = requests.get(url)
+        page_meta.raise_for_status()
+
+        # If the meta file already existed, we compare the existing
+        # one with the data newly downloaded. If they are different,
+        # we need to re-download the database.
+        # If the database does not exist locally, we need to redownload it in
+        # any case.
+        if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
+            meta_known = open(path_metaf, "r").read()
+            if page_meta.text == meta_known:
+                return path_jsonf_gz
+
+        # Grab the compressed JSON NVD, and write files to disk
+        url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
+        print("Getting %s" % url)
+        page_json = requests.get(url)
+        page_json.raise_for_status()
+        open(path_jsonf_gz, "wb").write(page_json.content)
+        open(path_metaf, "w").write(page_meta.text)
+        return path_jsonf_gz
+
+    @classmethod
+    def read_nvd_dir(cls, nvd_dir):
+        """
+        Iterate over all the CVEs contained in NIST Vulnerability Database
+        feeds since NVD_START_YEAR. If the files are missing or outdated in
+        nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
+        """
+        for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
+            filename = CVE.download_nvd_year(nvd_dir, year)
+            try:
+                content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
+            except:  # noqa: E722
+                print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
+                raise
+            for cve in content:
+                yield cls(cve['cve'])
+
+    def each_product(self):
+        """Iterate over each product section of this cve"""
+        for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+            for product in vendor['product']['product_data']:
+                yield product
+
+    @property
+    def identifier(self):
+        """The CVE unique identifier"""
+        return self.nvd_cve['CVE_data_meta']['ID']
+
+    @property
+    def pkg_names(self):
+        """The set of package names referred by this CVE definition"""
+        return set(p['product_name'] for p in self.each_product())
+
+    def affects(self, br_pkg):
+        """
+        True if the Buildroot Package object passed as argument is affected
+        by this CVE.
+        """
+        if br_pkg.is_cve_ignored(self.identifier):
+            return False
+
+        for product in self.each_product():
+            if product['product_name'] != br_pkg.name:
+                continue
+
+            for v in product['version']['version_data']:
+                if v["version_affected"] == "=":
+                    if br_pkg.current_version == v["version_value"]:
+                        return True
+                elif v["version_affected"] == "<=":
+                    pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+                    if not hasattr(pkg_version, "version"):
+                        print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                        continue
+                    cve_affected_version = distutils.version.LooseVersion(v["version_value"])
+                    if not hasattr(cve_affected_version, "version"):
+                        print("Cannot parse CVE affected version '%s'" % v["version_value"])
+                        continue
+                    return pkg_version <= cve_affected_version
+                else:
+                    print("version_affected: %s" % v['version_affected'])
+        return False
support/scripts/pkg-stats +3 -112

133 lines not on this page · raw message

Signature · 1 line
2.27.0
#2 lore raw

PATCH 2/9 support/scripts/cve.py: Switch to JSON 1.1

In reply to Gregory CLEMENT

In 2019, the JSON vulnerability feeds switched from version 1.0 to 1.1.

The main difference is the removal of the affects element that was used to check if a package was affected by a CVE.

This information is duplicated in the configuration element which contains in the end the cpeid as well as properties about the versions affected. Instead of having a list of the versions affected, with these properties, it is possible to have a range of versions.

---
 support/scripts/cve.py | 119 +++++++++++++++++++++++++++++++++--------
 1 file changed, 96 insertions(+), 23 deletions(-)
support/scripts/cve.py +96 -23
index 874ab4482d..e911fe0c65 100755
@@ -31,9 +31,19 @@ import sys
 sys.path.append('utils/')
 
 NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.0"
+NVD_JSON_VERSION = "1.1"
 NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
 
+import operator
+
+ops = {
+    '>=' : operator.ge,
+    '>' : operator.gt,
+    '<=' : operator.le,
+    '<' : operator.lt,
+    '=' : operator.eq
+}
+
 class CVE:
     """An accessor class for CVE Items in NVD files"""
     def __init__(self, nvd_cve):
@@ -92,23 +102,83 @@ class CVE:
                 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
                 raise
             for cve in content:
-                yield cls(cve['cve'])
+                yield cls(cve)
 
     def each_product(self):
         """Iterate over each product section of this cve"""
-        for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+        for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']:
             for product in vendor['product']['product_data']:
                 yield product
 
+    def parse_node(self, node):
+        """
+        Parse the node inside the configurations section to extract the
+        cpe information usefull to know if a product is affected by
+        the CVE. Actually only the product name and the version
+        descriptor are needed, but we also provide the vendor name.
+        """
+
+        # The node containing the cpe entries matching the CVE can also
+        # contain sub-nodes, so we need to manage it.
+        for child in node.get('children', ()):
+            self.parse_node(child)
+
+        for cpe in node.get('cpe_match', ()):
+            if not cpe['vulnerable']:
+                return
+            cpe23 = cpe['cpe23Uri'].split(':')
+            vendor = cpe23[3]
+            product = cpe23[4]
+            version = cpe23[5]
+            op_start = ''
+            op_end = ''
+            v_start = ''
+            v_end = ''
+
+            if version != '*' and version != '-':
+                # Version is defined, this is a '=' match
+                op_start = '='
+                v_start = version
+            elif version == '-':
+                # no version information is available
+                op_start = '='
+                v_start = version
+            else:
+                # Parse start version, end version and operators
+                if 'versionStartIncluding' in cpe:
+                    op_start = '>='
+                    v_start = cpe['versionStartIncluding']
+
+                if 'versionStartExcluding' in cpe:
+                    op_start = '>'
+                    v_start = cpe['versionStartExcluding']
+
+                if 'versionEndIncluding' in cpe:
+                    op_end = '<='
+                    v_end = cpe['versionEndIncluding']
+
+                if 'versionEndExcluding' in cpe:
+                    op_end = '<'
+                    v_end = cpe['versionEndExcluding']
+
+            key =['vendor', 'product', 'v_start', 'op_start', 'v_end', 'op_end']
+            val = [vendor, product, v_start, op_start, v_end, op_end]
+            yield dict(zip(key, val))
+
+    def each_cpe(self):
+        for node in self.nvd_cve['configurations']['nodes']:
+            for cpe in self.parse_node(node):
+                yield cpe
+
     @property
     def identifier(self):
         """The CVE unique identifier"""
-        return self.nvd_cve['CVE_data_meta']['ID']
+        return self.nvd_cve['cve']['CVE_data_meta']['ID']
 
     @property
     def pkg_names(self):
         """The set of package names referred by this CVE definition"""
-        return set(p['product_name'] for p in self.each_product())
+        return set(p['product'] for p in self.each_cpe())
 
     def affects(self, br_pkg):
         """
@@ -118,24 +188,27 @@ class CVE:
         if br_pkg.is_cve_ignored(self.identifier):
             return False
 
-        for product in self.each_product():
-            if product['product_name'] != br_pkg.name:
+        for cpe in self.each_cpe():
+            affected = True
+            if cpe['product'] != br_pkg.name:
+                continue
+            if cpe['v_start'] == '-':
+                return True
+            if not (cpe['v_start'] or cpe['v_end']):
+                print("No CVE affected version")
                 continue
+            pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+            if not hasattr(pkg_version, "version"):
+                print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                continue
+
+            if cpe['v_start']:
+                    cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
+                    affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
 
-            for v in product['version']['version_data']:
-                if v["version_affected"] == "=":
-                    if br_pkg.current_version == v["version_value"]:
-                        return True
-                elif v["version_affected"] == "<=":
-                    pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
-                    if not hasattr(pkg_version, "version"):
-                        print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
-                        continue
-                    cve_affected_version = distutils.version.LooseVersion(v["version_value"])
-                    if not hasattr(cve_affected_version, "version"):
-                        print("Cannot parse CVE affected version '%s'" % v["version_value"])
-                        continue
-                    return pkg_version <= cve_affected_version
-                else:
-                    print("version_affected: %s" % v['version_affected'])
+            if (affected and cpe['v_end']):
+                    cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
+                    affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
+            if (affected):
+                return True
         return False
Signature · 1 line
2.27.0
#3 lore raw

PATCH 3/9 package/pkg-utils: show-info: report the list of the CVEs ignored

In reply to Gregory CLEMENT

Add the list of the CVEs to ignore for each package because they already have a fix for it.

This information will be useful for a cve-checker.

---
 package/pkg-utils.mk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
package/pkg-utils.mk +4 -1
index d88a14ab0f..49ce6dc6f1 100644
@@ -117,7 +117,10 @@ define _json-info-pkg
 		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
 	"reverse_dependencies": [
-		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))),
+	],
+	"ignored_cves": [
+		$(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
 	]
 endef
 
Signature · 1 line
2.27.0
#4 lore raw

PATCH 4/9 package/pkg-utils: Make CVE class independent of the Pacakage class

In reply to Gregory CLEMENT

The affects method of the CVE use the Package class defined in pkg-stats. The purpose of migrating the CVE class outside of pkg-stats was to be able to reuse it from other scripts. So let's remove the Package dependency and only use the needed information.

---
 support/scripts/cve.py    | 10 +++++-----
 support/scripts/pkg-stats | 14 ++++++++------
 2 files changed, 13 insertions(+), 11 deletions(-)
support/scripts/cve.py +5 -5
index e911fe0c65..b754a17991 100755
@@ -180,26 +180,26 @@ class CVE:
         """The set of package names referred by this CVE definition"""
         return set(p['product'] for p in self.each_cpe())
 
-    def affects(self, br_pkg):
+    def affects(self, name, version, cve_ignore_list):
         """
         True if the Buildroot Package object passed as argument is affected
         by this CVE.
         """
-        if br_pkg.is_cve_ignored(self.identifier):
+        if (self.identifier in cve_ignore_list):
             return False
 
         for cpe in self.each_cpe():
             affected = True
-            if cpe['product'] != br_pkg.name:
+            if cpe['product'] != name:
                 continue
             if cpe['v_start'] == '-':
                 return True
             if not (cpe['v_start'] or cpe['v_end']):
                 print("No CVE affected version")
                 continue
-            pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+            pkg_version = distutils.version.LooseVersion(version)
             if not hasattr(pkg_version, "version"):
-                print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                print("Cannot parse package '%s' version '%s'" % (name, version))
                 continue
 
             if cpe['v_start']:
support/scripts/pkg-stats +8 -6
index 1c941104fe..883a5bd2be 100755
@@ -245,11 +245,12 @@ class Package:
                     self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings))
                 return
 
-    def is_cve_ignored(self, cve):
+    def cve_ignored_list(self):
         """
-        Tells if the CVE is ignored by the package
+        Give the list of CVEs ignored by the package
         """
-        return cve in self.all_ignored_cves.get(self.pkgvar(), [])
+        print(self.all_ignored_cves.get(self.pkgvar(), []))
+        return list(self.all_ignored_cves.get(self.pkgvar(), []))
 
     def set_developers(self, developers):
         """
@@ -501,9 +502,10 @@ def check_package_cves(nvd_path, packages):
 
     for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
         for pkg_name in cve.pkg_names:
-            if pkg_name in packages and cve.affects(packages[pkg_name]):
-                packages[pkg_name].cves.append(cve.identifier)
-
+            if pkg_name in packages:
+                pkg = packages[pkg_name]
+                if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()):
+                    pkg.cves.append(cve.identifier)
 
 def calculate_stats(packages):
     stats = defaultdict(int)
Signature · 1 line
2.27.0
#5 lore raw

PATCH 5/9 support/scripts: Add a per configuration CVE checker

In reply to Gregory CLEMENT

This scripts takes as entry on stdin a JSON description of the package used for a given configuration. This description is the one generated by "make show-info".

The script generates the list of all the package used and if they are affected by a CVE. The output is either a JSON or an HTML file similar to the one generated by pkg-stats.

---
 support/scripts/cve-checker | 291 ++++++++++++++++++++++++++++++++++++
 1 file changed, 291 insertions(+)
 create mode 100755 support/scripts/cve-checker
support/scripts/cve-checker new file +291 -0
new file mode 100755
index 0000000000..db8497d7aa
@@ -0,0 +1,291 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <[email protected]>
+# Copyright (C) 2020 by Gregory CLEMENT <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import fnmatch
+import os
+from collections import defaultdict
+import re
+import subprocess
+import requests  # URL checking
+import json
+import ijson
+import certifi
+import distutils.version
+import time
+import gzip
+import sys
+from urllib3 import HTTPSConnectionPool
+from urllib3.exceptions import HTTPError
+from multiprocessing import Pool
+
+sys.path.append('utils/')
+
+import cve as cvecheck
+
+
+INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
+URL_RE = re.compile(r"\s*https?://\S*\s*$")
+
+RM_API_STATUS_ERROR = 1
+RM_API_STATUS_FOUND_BY_DISTRO = 2
+RM_API_STATUS_FOUND_BY_PATTERN = 3
+RM_API_STATUS_NOT_FOUND = 4
+
+# Used to make multiple requests to the same host. It is global
+# because it's used by sub-processes.
+http_pool = None
+
+
+class Package:
+    def __init__(self, name, version, ignored_cves):
+        self.name = name
+        self.version = version
+        self.cves = list()
+        self.ignored_cves = ignored_cves
+
+def check_package_cves(nvd_path, packages):
+    if not os.path.isdir(nvd_path):
+        os.makedirs(nvd_path)
+
+    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+        for pkg_name in cve.pkg_names:
+            pkg = packages.get(pkg_name, '')
+            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
+                pkg.cves.append(cve.identifier)
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+  width: 100%;
+}
+td {
+  border: 1px solid black;
+}
+td.centered {
+  text-align: center;
+}
+td.wrong {
+  background: #ff9a69;
+}
+td.correct {
+  background: #d2ffc4;
+}
+td.nopatches {
+  background: #d2ffc4;
+}
+td.somepatches {
+  background: #ffd870;
+}
+td.lotsofpatches {
+  background: #ff9a69;
+}
+
+td.good_url {
+  background: #d2ffc4;
+}
+td.missing_url {
+  background: #ffd870;
+}
+td.invalid_url {
+  background: #ff9a69;
+}
+
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
+</style>
+<title>CVE status for Buildroot packages</title>
+</head>
+
+<a href=\"#results\">CVE Status</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+  document.getElementById(\"sortable_hint\").innerHTML =
+  \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+    if not infra_list:
+        return "Unknown"
+    elif len(infra_list) == 1:
+        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+    elif infra_list[0][1] == infra_list[1][1]:
+        return "<b>%s</b><br/>%s + %s" % \
+            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+    else:
+        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+            (infra_list[0][1], infra_list[0][0],
+             infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+    if b:
+        return "Yes"
+    else:
+        return "No"
+
+
+def dump_html_pkg(f, pkg):
+    f.write(" <tr>\n")
+    f.write("  <td>%s</td>\n" % pkg.name)
+
+    # Current version
+    if len(pkg.version) > 20:
+        version = pkg.version[:20] + "..."
+    else:
+        version = pkg.version
+    f.write("  <td class=\"centered\">%s</td>\n" % version)
+
+    # CVEs
+    td_class = ["centered"]
+    if len(pkg.cves) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
+    f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+    f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Version</td>
+<td class=\"centered\">CVEs</td>
+</tr>
+""")
+    for pkg in packages:
+        dump_html_pkg(f, pkg)
+    f.write("</table>")
+
+
+def dump_html_gen_info(f, date, commit):
+    # Updated on Mon Feb 19 08:12:08 CET 2018, Git commit aa77030b8f5e41f1c53eb1c1ad664b8c814ba032
+    f.write("<p><i>Updated on %s, git commit %s</i></p>\n" % (str(date), commit))
+
+
+def dump_html(packages, date, commit, output):
+    with open(output, 'w') as f:
+        f.write(html_header)
+        dump_html_all_pkgs(f, packages)
+        dump_html_gen_info(f, date, commit)
+        f.write(html_footer)
+
+
+def dump_json(packages, date, commit, output):
+    # Format packages as a dictionnary instead of a list
+    # Exclude local field that does not contains real date
+    excluded_fields = ['url_worker', 'name']
+    pkgs = {
+        pkg.name: {
+            k: v
+            for k, v in pkg.__dict__.items()
+            if k not in excluded_fields
+        } for pkg in packages
+    }
+     # The actual structure to dump, add commit and date to it
+    final = {'packages': pkgs,
+             'commit': commit,
+             'date': str(date)}
+    with open(output, 'w') as f:
+        json.dump(final, f, indent=2, separators=(',', ': '))
+        f.write('\n')
+
+
+def resolvepath(path):
+        return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    output = parser.add_argument_group('output', 'Output file(s)')
+    output.add_argument('--html', dest='html', type=resolvepath,
+                        help='HTML output file')
+    output.add_argument('--json', dest='json', type=resolvepath,
+                        help='JSON output file')
+    packages = parser.add_mutually_exclusive_group()
+    packages.add_argument('-n', dest='npackages', type=int, action='store',
+                          help='Number of packages')
+    packages.add_argument('-p', dest='packages', action='store',
+                          help='List of packages (comma separated)')
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database', type=resolvepath)
+    args = parser.parse_args()
+    if not args.html and not args.json:
+        parser.error('at least one of --html or --json (or both) is required')
+    return args
+
+
+def __main__():
+    packages = list()
+    exclude_pacakges = ["linux", "gcc"]
+    content = json.load(sys.stdin)
+    for item in content:
+        if item in exclude_pacakges:
+            continue
+        pkg = content[item]
+        p = Package(item, pkg.get('version', ''), pkg.get('ignored_cves', ''))
+        packages.append(p)
+
+    args = parse_args()
+    date = datetime.datetime.utcnow()
+    commit = subprocess.check_output(['git', 'rev-parse',
+                                      'HEAD']).splitlines()[0].decode()
+
+    if args.nvd_path:
+        print("Checking packages CVEs")
+        check_package_cves(args.nvd_path, {p.name: p for p in packages})
+    if args.html:
+        print("Write HTML")
+        dump_html(packages, date, commit, args.html)
+    if args.json:
+        print("Write JSON")
+        dump_json(packages, date, commit, args.json)
+
+__main__()
Signature · 1 line
2.27.0
#6 lore raw

PATCH 6/9 package/pkg-utils: cve.py: Handle exception when version comparison fails

In reply to Gregory CLEMENT

With python 3, when a package has a version number x-y-z instead of x.y.z, then the version returned by LooseVersion can't be compared which raises an exception.

This patch handles this exception by adding a new return value when the comparison can't be done.

---
 support/scripts/cve.py | 9 +++++++++
 1 file changed, 9 insertions(+)
support/scripts/cve.py +9 -0
index b754a17991..03afdeb54a 100755
@@ -203,12 +203,21 @@ class CVE:
                 continue
 
             if cpe['v_start']:
+                try:
                     cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
                     affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
+                    break
+                except:
+                    return 'Unknown'
 
             if (affected and cpe['v_end']):
+                try:
                     cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
                     affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
+                    break
+                except:
+                    return 'Unknown'
+
             if (affected):
                 return True
         return False
Signature · 1 line
2.27.0
#7 lore raw

PATCH 7/9 support/script/pkg-stats: Manage the CVEs that need to be check

In reply to Gregory CLEMENT

When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version.

This patch exposes this new information in json and html format.

---
 support/scripts/pkg-stats | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)
support/scripts/pkg-stats +30 -4
index 883a5bd2be..e033e15e07 100755
@@ -106,9 +106,11 @@ class Package:
         self.patch_files = []
         self.warnings = 0
         self.current_version = None
+        self.unknown_cve = False
         self.url = None
         self.url_worker = None
         self.cves = list()
+        self.cves_to_check = list()
         self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
         self.status = {}
 
@@ -504,7 +506,12 @@ def check_package_cves(nvd_path, packages):
         for pkg_name in cve.pkg_names:
             if pkg_name in packages:
                 pkg = packages[pkg_name]
-                if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()):
+                affected = cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list())
+                print(affected)
+                if (affected == 'Unknown'):
+                    pkg.cves_to_check.append(cve.identifier)
+                elif affected == True:
+                    print(cve.identifier)
                     pkg.cves.append(cve.identifier)
 
 def calculate_stats(packages):
@@ -544,8 +551,11 @@ def calculate_stats(packages):
             stats["version-not-uptodate"] += 1
         stats["patches"] += pkg.patch_count
         stats["total-cves"] += len(pkg.cves)
+        stats["total-cves-to-check"] += len(pkg.cves_to_check)
         if len(pkg.cves) != 0:
             stats["pkg-cves"] += 1
+        if len(pkg.cves_to_check) != 0:
+            stats["pkg-cves_to_check"] += 1
     return stats
 
 
@@ -763,11 +773,22 @@ def dump_html_pkg(f, pkg):
         td_class.append("correct")
     else:
         td_class.append("wrong")
-    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+        f.write("  <td class=\"%s\">\n" % " ".join(td_class))
     for cve in pkg.cves:
         f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
     f.write("  </td>\n")
 
+    # CVEs to check
+    td_class = ["centered"]
+    if len(pkg.cves_to_check) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+        f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves_to_check:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
     f.write(" </tr>\n")
 
 
@@ -786,6 +807,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">Warnings</td>
 <td class=\"centered\">Upstream URL</td>
 <td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
 </tr>
 """)
     for pkg in sorted(packages):
@@ -824,10 +846,14 @@ def dump_html_stats(f, stats):
             stats["version-not-uptodate"])
     f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
             stats["version-unknown"])
-    f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+    f.write("<tr><td>Packages might affected by CVEs, where version needed to be checked</td><td>%s</td></tr>\n" %
             stats["pkg-cves"])
-    f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+    f.write("<tr><td>Total number of CVEs that might affect all packages, where version needed to be checked</td><td>%s</td></tr>\n" %
             stats["total-cves"])
+    f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+            stats["pkg-cves_to_check"])
+    f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+            stats["total-cves_to_check"])
     f.write("</table>\n")
 
 
Signature · 1 line
2.27.0
#8 lore raw

PATCH 8/9 support/script/cve-checker: Manage the CVEs that need to be check

In reply to Gregory CLEMENT

When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version.

This patch exposes this new information in json and html format.

---
 support/scripts/cve-checker | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
support/scripts/cve-checker +19 -2
index db8497d7aa..2a8a2b4d18 100755
@@ -59,6 +59,7 @@ class Package:
         self.name = name
         self.version = version
         self.cves = list()
+        self.cves_to_check = list()
         self.ignored_cves = ignored_cves
 
 def check_package_cves(nvd_path, packages):
@@ -68,8 +69,12 @@ def check_package_cves(nvd_path, packages):
     for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
         for pkg_name in cve.pkg_names:
             pkg = packages.get(pkg_name, '')
-            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
-                pkg.cves.append(cve.identifier)
+            if pkg:
+                affected = cve.affects(pkg.name, pkg.version, pkg.ignored_cves)
+                if (affected == 'Unknown'):
+                    pkg.cves_to_check.append(cve.identifier)
+                elif affected == True:
+                    pkg.cves.append(cve.identifier)
 
 html_header = """
 <head>
@@ -188,6 +193,17 @@ def dump_html_pkg(f, pkg):
         f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
     f.write("  </td>\n")
 
+    # CVEs to check
+    td_class = ["centered"]
+    if len(pkg.cves_to_check) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+        f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves_to_check:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
     f.write(" </tr>\n")
 
 
@@ -198,6 +214,7 @@ def dump_html_all_pkgs(f, packages):
 <td>Package</td>
 <td class=\"centered\">Version</td>
 <td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
 </tr>
 """)
     for pkg in packages:
Signature · 1 line
2.27.0
#9 lore raw

PATCH 9/9 package/pkg-utils/cve.py: Manage case when package version doesn't exist

In reply to Gregory CLEMENT

Until now, when a package didn't report a version, then the CVE comparison was just skipped. It leads most of the time to declare the package not affected by the CVE.

Instead of it, report the 'Unknown' status in order to be aware that the CVE related to this package has to be checked.

---
 support/scripts/cve.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
support/scripts/cve.py +7 -1
index 03afdeb54a..d3480d68dd 100755
@@ -188,6 +188,7 @@ class CVE:
         if (self.identifier in cve_ignore_list):
             return False
 
+        unknown_pkg_version = False
         for cpe in self.each_cpe():
             affected = True
             if cpe['product'] != name:
@@ -200,6 +201,7 @@ class CVE:
             pkg_version = distutils.version.LooseVersion(version)
             if not hasattr(pkg_version, "version"):
                 print("Cannot parse package '%s' version '%s'" % (name, version))
+                unknown_pkg_version = True
                 continue
 
             if cpe['v_start']:
@@ -220,4 +222,8 @@ class CVE:
 
             if (affected):
                 return True
-        return False
+
+        if unknown_pkg_version:
+            return  'Unknown'
+        else:
+            return False
Signature · 1 line
2.27.0
#10 lore raw

v2 cover Improving CVE reporting

In reply to Gregory CLEMENT

Hello,

The purpose of this series is to improve the CVE reporting in order to be usable for a project.

Until know the CVE affecting the packages were reported for the buildroot project using pkg-stat. With this series it is now possible to report the packages affected by CVEs for a given configuration.

While I was on CVE I switched to the support of the JSON 1.1 for the NVDE database.

In this series I also added a new state for the CVE status of the packages. This new state will be used to emphasize that the automatic check has failed and it was needed to be verified manually. The idea behind this was to be as much accurate as possible to avoid any false positive. It will also help to improve the meta-data of the package.

The next step will be to reuse the works done by Matthew Weber [1] to use the cpeid and only use the package name and the package version as fall back.

In this series there is at least one open point about the packages excluded from the cve check. For now I excluded the kernel and gcc as there are also excluded by the pkg-stats script but this list could (should ?) be extended or modified.

In this second version the following changes have been done:
v1 -> v2

- Port the version fix to pkg-stat from cve.py and move this patch as the first one

- Remove debug message

- Remove unused argument -p and -n in cve-checker

- Remove the information about the commit used in the output for the cve-checker

- Remove all the unnecessary import

- Add a default path to the download directory for nvd for the cve-checker

- Do not use boolean anymore for the affected status

- Use ignore_cves instead of ignored_cves in pkg-utils

- Fix the html output for cve-checker and pkg-stat

- Check if ijson is present on the host

Gregory

Gregory CLEMENT (9):
support/script/pkg-stat: Handle exception when version comparison
fails
support/scripts: Turn CVE check into a module support/scripts/cve.py: Switch to JSON 1.1 package/pkg-utils: show-info: report the list of the CVEs ignored support/script: Make CVE class independent of the Pacakage class support/scripts: Add a per configuration CVE checker support/script/pkg-stats: Manage the CVEs that need to be check support/script/cve-checker: Manage the CVEs that need to be check package/pkg-utils/cve.py: Manage case when package version doesn't
exist

 package/pkg-utils.mk        |   5 +-
 support/scripts/cve-checker | 275 ++++++++++++++++++++++++++++++++++++
 support/scripts/cve.py      | 235 ++++++++++++++++++++++++++++++
 support/scripts/pkg-stats   | 159 +++++----------------
 4 files changed, 550 insertions(+), 124 deletions(-)
 create mode 100755 support/scripts/cve-checker
 create mode 100755 support/scripts/cve.py
Signature · 1 line
2.27.0
#11 lore raw

PATCH v2 1/9 support/script/pkg-stat: Handle exception when version comparison fails

In reply to Gregory CLEMENT

With python 3, when a package has a version number x-y-z instead of x.y.z, then the version returned by LooseVersion can't be compared which raises an exception.

This patch handles this exception by adding a new return value when the comparison can't be done. As a third value has been introduce, the booelan are no more used.

---
 support/scripts/pkg-stats | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)
support/scripts/pkg-stats +17 -5
index c1f41fc9e8..a75cb68581 100755
@@ -50,6 +50,10 @@ RM_API_STATUS_FOUND_BY_DISTRO = 2
 RM_API_STATUS_FOUND_BY_PATTERN = 3
 RM_API_STATUS_NOT_FOUND = 4
 
+CVE_AFFECTS = 1
+CVE_DOESNT_AFFECT = 2
+CVE_UNKNOWN = 3
+
 # Used to make multiple requests to the same host. It is global
 # because it's used by sub-processes.
 http_pool = None
@@ -364,7 +368,7 @@ class CVE:
         by this CVE.
         """
         if br_pkg.is_cve_ignored(self.identifier):
-            return False
+            return CVE_DOESNT_AFFECT
 
         for product in self.each_product():
             if product['product_name'] != br_pkg.name:
@@ -373,7 +377,7 @@ class CVE:
             for v in product['version']['version_data']:
                 if v["version_affected"] == "=":
                     if br_pkg.current_version == v["version_value"]:
-                        return True
+                        return CVE_AFFECTS
                 elif v["version_affected"] == "<=":
                     pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
                     if not hasattr(pkg_version, "version"):
@@ -383,10 +387,18 @@ class CVE:
                     if not hasattr(cve_affected_version, "version"):
                         print("Cannot parse CVE affected version '%s'" % v["version_value"])
                         continue
-                    return pkg_version <= cve_affected_version
+                    try:
+                        affected = pkg_version <= cve_affected_version
+                        break
+                    except:
+                        return CVE_UNKNOWN
+                    if affected:
+                        return CVE_AFFECTS
+                    else:
+                        return CVE_DOESNT_AFFECT
                 else:
                     print("version_affected: %s" % v['version_affected'])
-        return False
+        return CVE_DOESNT_AFFECT
 
 
 def get_pkglist(npackages, package_list):
@@ -610,7 +622,7 @@ def check_package_cves(nvd_path, packages):
 
     for cve in CVE.read_nvd_dir(nvd_path):
         for pkg_name in cve.pkg_names:
-            if pkg_name in packages and cve.affects(packages[pkg_name]):
+            if pkg_name in packages and cve.affects(packages[pkg_name]) == CVE_AFFECTS:
                 packages[pkg_name].cves.append(cve.identifier)
 
 
Signature · 1 line
2.27.0
#12 lore raw

PATCH v2 1/9 support/script/pkg-stat: Handle exception when version comparison fails

Hello,

Quoted 9 lines · On Fri, 10 Jul 2020 13:22:37 +0200 Gregory CLEMENT…

With python 3, when a package has a version number x-y-z instead of x.y.z, then the version returned by LooseVersion can't be compared which raises an exception.

This patch handles this exception by adding a new return value when the comparison can't be done. As a third value has been introduce, the booelan are no more used.

Typos in the commit title:

support/scripts/pkg-stats

also, we generally don't use a capital letter to start the short description.

support/scripts/pkg-stats class CVE: · line 393

Quoted 4 lines
support/scripts/pkg-stats +4 -0
+                    try:
+                        affected = pkg_version <= cve_affected_version
+                        break
+                    except:

This kind of "catch all exceptions" is not recommended in Python, and flake8 is not happy about it, so I changed it to:

except TypeError:

I fixed up this issue, the commit log issue, and applied. Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#13 lore raw

PATCH v2 2/9 support/scripts: Turn CVE check into a module

In reply to Gregory CLEMENT

In order to be able to do CVE checking outside of pkg-stat, move the CVE class in a module that can be used by other scripts.

---
 support/scripts/cve.py    | 156 ++++++++++++++++++++++++++++++++++++++
 support/scripts/pkg-stats | 132 +-------------------------------
 2 files changed, 160 insertions(+), 128 deletions(-)
 create mode 100755 support/scripts/cve.py
support/scripts/cve.py new file +156 -0
new file mode 100755
index 0000000000..8fc552b74d
@@ -0,0 +1,156 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <[email protected]>
+# Copyright (C) 2020 by Gregory CLEMENT <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import datetime
+import os
+import requests  # URL checking
+import distutils.version
+import time
+import gzip
+import sys
+
+try:
+    import ijson
+except ImportError:
+    sys.stderr.write("You need ijson to parse NVD for CVE check\n")
+    exit(1)
+
+sys.path.append('utils/')
+
+NVD_START_YEAR = 2002
+NVD_JSON_VERSION = "1.0"
+NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
+
+class CVE:
+    """An accessor class for CVE Items in NVD files"""
+    CVE_AFFECTS = 1
+    CVE_DOESNT_AFFECT = 2
+    CVE_UNKNOWN = 3
+
+    def __init__(self, nvd_cve):
+        """Initialize a CVE from its NVD JSON representation"""
+        self.nvd_cve = nvd_cve
+
+    @staticmethod
+    def download_nvd_year(nvd_path, year):
+        metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year)
+        path_metaf = os.path.join(nvd_path, metaf)
+        jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year)
+        path_jsonf_gz = os.path.join(nvd_path, jsonf_gz)
+
+        # If the database file is less than a day old, we assume the NVD data
+        # locally available is recent enough.
+        if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400:
+            return path_jsonf_gz
+
+        # If not, we download the meta file
+        url = "%s/%s" % (NVD_BASE_URL, metaf)
+        print("Getting %s" % url)
+        page_meta = requests.get(url)
+        page_meta.raise_for_status()
+
+        # If the meta file already existed, we compare the existing
+        # one with the data newly downloaded. If they are different,
+        # we need to re-download the database.
+        # If the database does not exist locally, we need to redownload it in
+        # any case.
+        if os.path.exists(path_metaf) and os.path.exists(path_jsonf_gz):
+            meta_known = open(path_metaf, "r").read()
+            if page_meta.text == meta_known:
+                return path_jsonf_gz
+
+        # Grab the compressed JSON NVD, and write files to disk
+        url = "%s/%s" % (NVD_BASE_URL, jsonf_gz)
+        print("Getting %s" % url)
+        page_json = requests.get(url)
+        page_json.raise_for_status()
+        open(path_jsonf_gz, "wb").write(page_json.content)
+        open(path_metaf, "w").write(page_meta.text)
+        return path_jsonf_gz
+
+    @classmethod
+    def read_nvd_dir(cls, nvd_dir):
+        """
+        Iterate over all the CVEs contained in NIST Vulnerability Database
+        feeds since NVD_START_YEAR. If the files are missing or outdated in
+        nvd_dir, a fresh copy will be downloaded, and kept in .json.gz
+        """
+        for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1):
+            filename = CVE.download_nvd_year(nvd_dir, year)
+            try:
+                content = ijson.items(gzip.GzipFile(filename), 'CVE_Items.item')
+            except:  # noqa: E722
+                print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
+                raise
+            for cve in content:
+                yield cls(cve['cve'])
+
+    def each_product(self):
+        """Iterate over each product section of this cve"""
+        for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+            for product in vendor['product']['product_data']:
+                yield product
+
+    @property
+    def identifier(self):
+        """The CVE unique identifier"""
+        return self.nvd_cve['CVE_data_meta']['ID']
+
+    @property
+    def pkg_names(self):
+        """The set of package names referred by this CVE definition"""
+        return set(p['product_name'] for p in self.each_product())
+
+    def affects(self, br_pkg):
+        """
+        True if the Buildroot Package object passed as argument is affected
+        by this CVE.
+        """
+        if br_pkg.is_cve_ignored(self.identifier):
+            return self.CVE_DOESNT_AFFECT
+
+        for product in self.each_product():
+            if product['product_name'] != br_pkg.name:
+                continue
+
+            for v in product['version']['version_data']:
+                if v["version_affected"] == "=":
+                    if br_pkg.current_version == v["version_value"]:
+                        return self.CVE_AFFECTS
+                elif v["version_affected"] == "<=":
+                    pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+                    if not hasattr(pkg_version, "version"):
+                        print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                        continue
+                    cve_affected_version = distutils.version.LooseVersion(v["version_value"])
+                    if not hasattr(cve_affected_version, "version"):
+                        print("Cannot parse CVE affected version '%s'" % v["version_value"])
+                        continue
+                    try:
+                        affected = pkg_version <= cve_affected_version
+                        break
+                    except:
+                        return self.CVE_UNKNOWN
+                    if affected:
+                        return self.CVE_AFFECTS
+                    else:
+                        return self.CVE_DOESNT_AFFECT
+                else:
+                    print("version_affected: %s" % v['version_affected'])
+        return self.CVE_DOESNT_AFFECT
support/scripts/pkg-stats +4 -128

165 lines not on this page · raw message

Signature · 1 line
2.27.0
#14 lore raw

PATCH v2 3/9 support/scripts/cve.py: Switch to JSON 1.1

In reply to Gregory CLEMENT

In 2019, the JSON vulnerability feeds switched from version 1.0 to 1.1.

The main difference is the removal of the affects element that was used to check if a package was affected by a CVE.

This information is duplicated in the configuration element which contains in the end the cpeid as well as properties about the versions affected. Instead of having a list of the versions affected, with these properties, it is possible to have a range of versions.

---
 support/scripts/cve.py | 127 ++++++++++++++++++++++++++++++++---------
 1 file changed, 100 insertions(+), 27 deletions(-)
support/scripts/cve.py +100 -27
index 8fc552b74d..4d47d2f768 100755
@@ -34,9 +34,19 @@ except ImportError:
 sys.path.append('utils/')
 
 NVD_START_YEAR = 2002
-NVD_JSON_VERSION = "1.0"
+NVD_JSON_VERSION = "1.1"
 NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION
 
+import operator
+
+ops = {
+    '>=' : operator.ge,
+    '>' : operator.gt,
+    '<=' : operator.le,
+    '<' : operator.lt,
+    '=' : operator.eq
+}
+
 class CVE:
     """An accessor class for CVE Items in NVD files"""
     CVE_AFFECTS = 1
@@ -99,23 +109,83 @@ class CVE:
                 print("ERROR: cannot read %s. Please remove the file then rerun this script" % filename)
                 raise
             for cve in content:
-                yield cls(cve['cve'])
+                yield cls(cve)
 
     def each_product(self):
         """Iterate over each product section of this cve"""
-        for vendor in self.nvd_cve['affects']['vendor']['vendor_data']:
+        for vendor in self.nvd_cve['cve']['affects']['vendor']['vendor_data']:
             for product in vendor['product']['product_data']:
                 yield product
 
+    def parse_node(self, node):
+        """
+        Parse the node inside the configurations section to extract the
+        cpe information usefull to know if a product is affected by
+        the CVE. Actually only the product name and the version
+        descriptor are needed, but we also provide the vendor name.
+        """
+
+        # The node containing the cpe entries matching the CVE can also
+        # contain sub-nodes, so we need to manage it.
+        for child in node.get('children', ()):
+            self.parse_node(child)
+
+        for cpe in node.get('cpe_match', ()):
+            if not cpe['vulnerable']:
+                return
+            cpe23 = cpe['cpe23Uri'].split(':')
+            vendor = cpe23[3]
+            product = cpe23[4]
+            version = cpe23[5]
+            op_start = ''
+            op_end = ''
+            v_start = ''
+            v_end = ''
+
+            if version != '*' and version != '-':
+                # Version is defined, this is a '=' match
+                op_start = '='
+                v_start = version
+            elif version == '-':
+                # no version information is available
+                op_start = '='
+                v_start = version
+            else:
+                # Parse start version, end version and operators
+                if 'versionStartIncluding' in cpe:
+                    op_start = '>='
+                    v_start = cpe['versionStartIncluding']
+
+                if 'versionStartExcluding' in cpe:
+                    op_start = '>'
+                    v_start = cpe['versionStartExcluding']
+
+                if 'versionEndIncluding' in cpe:
+                    op_end = '<='
+                    v_end = cpe['versionEndIncluding']
+
+                if 'versionEndExcluding' in cpe:
+                    op_end = '<'
+                    v_end = cpe['versionEndExcluding']
+
+            key =['vendor', 'product', 'v_start', 'op_start', 'v_end', 'op_end']
+            val = [vendor, product, v_start, op_start, v_end, op_end]
+            yield dict(zip(key, val))
+
+    def each_cpe(self):
+        for node in self.nvd_cve['configurations']['nodes']:
+            for cpe in self.parse_node(node):
+                yield cpe
+
     @property
     def identifier(self):
         """The CVE unique identifier"""
-        return self.nvd_cve['CVE_data_meta']['ID']
+        return self.nvd_cve['cve']['CVE_data_meta']['ID']
 
     @property
     def pkg_names(self):
         """The set of package names referred by this CVE definition"""
-        return set(p['product_name'] for p in self.each_product())
+        return set(p['product'] for p in self.each_cpe())
 
     def affects(self, br_pkg):
         """
@@ -125,32 +195,35 @@ class CVE:
         if br_pkg.is_cve_ignored(self.identifier):
             return self.CVE_DOESNT_AFFECT
 
-        for product in self.each_product():
-            if product['product_name'] != br_pkg.name:
+        for cpe in self.each_cpe():
+            affected = True
+            if cpe['product'] != br_pkg.name:
+                continue
+            if cpe['v_start'] == '-':
+                return self.CVE_AFFECTS
+            if not (cpe['v_start'] or cpe['v_end']):
+                print("No CVE affected version")
                 continue
+            pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+            if not hasattr(pkg_version, "version"):
+                print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                continue
+
+            if cpe['v_start']:
+                    try:
+                        cve_affected_version = distutils.version.LooseVersion(cpe['v_start'])
+                        affected = ops.get(cpe['op_start'])(pkg_version, cve_affected_version)
+                        break
+                    except:
+                        return self.CVE_UNKNOWN
 
-            for v in product['version']['version_data']:
-                if v["version_affected"] == "=":
-                    if br_pkg.current_version == v["version_value"]:
-                        return self.CVE_AFFECTS
-                elif v["version_affected"] == "<=":
-                    pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
-                    if not hasattr(pkg_version, "version"):
-                        print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
-                        continue
-                    cve_affected_version = distutils.version.LooseVersion(v["version_value"])
-                    if not hasattr(cve_affected_version, "version"):
-                        print("Cannot parse CVE affected version '%s'" % v["version_value"])
-                        continue
+            if (affected and cpe['v_end']):
                     try:
-                        affected = pkg_version <= cve_affected_version
+                        cve_affected_version = distutils.version.LooseVersion(cpe['v_end'])
+                        affected = ops.get(cpe['op_end'])(pkg_version, cve_affected_version)
                         break
                     except:
                         return self.CVE_UNKNOWN
-                    if affected:
-                        return self.CVE_AFFECTS
-                    else:
-                        return self.CVE_DOESNT_AFFECT
-                else:
-                    print("version_affected: %s" % v['version_affected'])
+            if (affected):
+                return self.CVE_AFFECTS
         return self.CVE_DOESNT_AFFECT
Signature · 1 line
2.27.0
#15 lore raw

PATCH v2 3/9 support/scripts/cve.py: Switch to JSON 1.1

Hello Gregory,

Quoted 2 lines · Gregory CLEMENT

In 2019, the JSON vulnerability feeds switched from version 1.0 to 1.1.

[--SNIP--]

support/scripts/cve.py class CVE: · line 131

Quoted 12 lines
support/scripts/cve.py +12 -0
+    def parse_node(self, node):
+        """
+        Parse the node inside the configurations section to extract the
+        cpe information usefull to know if a product is affected by
+        the CVE. Actually only the product name and the version
+        descriptor are needed, but we also provide the vendor name.
+        """
+
+        # The node containing the cpe entries matching the CVE can also
+        # contain sub-nodes, so we need to manage it.
+        for child in node.get('children', ()):
+            self.parse_node(child)

This doesn't do anything, because the values yielded in the recursive call to self.parse_node() are not used. The generator should be consumed like this (Python 2 and 3)

for child in node.get('children', ()):
for parsed_node in self.parse_node(child):
yield parsed_node

or with a more recent syntax (Python >=3.4)

for child in node.get('children', ()):
yield from self.parse_node(child)

Also, if I understand correctly, this does not check if the CPE nodes have to be ORed or ANDed.

Some time ago, I looked into the switch to the v1.1 of the NVD files, but somehow lamely forgot about it afterwards. This is the function I came up with to determine if a package at a given version would match a certain tree of CPE rules: http://paste.awesom.eu/Dxcv , maybe that could help.

Best regards,

Titouan

#16 lore raw

PATCH v2 3/9 support/scripts/cve.py: Switch to JSON 1.1

Quoted 3 lines

Hello Gregory,

Hello Titouan,

Quoted 35 lines · Gregory CLEMENT
Quoted 2 lines

In 2019, the JSON vulnerability feeds switched from version 1.0 to 1.1.

[--SNIP--]

support/scripts/cve.py class CVE: · line 131

Quoted 12 lines
support/scripts/cve.py +12 -0
+    def parse_node(self, node):
+        """
+        Parse the node inside the configurations section to extract the
+        cpe information usefull to know if a product is affected by
+        the CVE. Actually only the product name and the version
+        descriptor are needed, but we also provide the vendor name.
+        """
+
+        # The node containing the cpe entries matching the CVE can also
+        # contain sub-nodes, so we need to manage it.
+        for child in node.get('children', ()):
+            self.parse_node(child)

This doesn't do anything, because the values yielded in the recursive call to self.parse_node() are not used. The generator should be consumed like this (Python 2 and 3)

for child in node.get('children', ()):
for parsed_node in self.parse_node(child):
yield parsed_node

or with a more recent syntax (Python >=3.4)

for child in node.get('children', ()):
yield from self.parse_node(child)

Quoted 8 lines

Also, if I understand correctly, this does not check if the CPE nodes have to be ORed or ANDed.

Some time ago, I looked into the switch to the v1.1 of the NVD files, but somehow lamely forgot about it afterwards. This is the function I came up with to determine if a package at a given version would match a certain tree of CPE rules: http://paste.awesom.eu/Dxcv , maybe that could help.

Thanks for the feedback, I will fix this according your remarks.

Gregory

Quoted 4 lines

Best regards,

Titouan

Signature · 3 lines
Gregory Clement, Bootlin
Embedded Linux and Kernel engineering
http://bootlin.com
#17 lore raw

PATCH v2 4/9 package/pkg-utils: show-info: report the list of the CVEs ignored

In reply to Gregory CLEMENT

Add the list of the CVEs to ignore for each package because they already have a fix for it.

This information will be useful for a cve-checker.

---
 package/pkg-utils.mk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
package/pkg-utils.mk +4 -1
index d88a14ab0f..21b415cbf3 100644
@@ -117,7 +117,10 @@ define _json-info-pkg
 		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
 	"reverse_dependencies": [
-		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES))),
+	],
+	"ignore_cves": [
+		$(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
 	]
 endef
 
Signature · 1 line
2.27.0
#18 lore raw

PATCH v2 5/9 support/script: Make CVE class independent of the Pacakage class

In reply to Gregory CLEMENT

The affects method of the CVE use the Package class defined in pkg-stats. The purpose of migrating the CVE class outside of pkg-stats was to be able to reuse it from other scripts. So let's remove the Package dependency and only use the needed information.

---
 support/scripts/cve.py    | 10 +++++-----
 support/scripts/pkg-stats | 14 ++++++++------
 2 files changed, 13 insertions(+), 11 deletions(-)
support/scripts/cve.py +5 -5
index 4d47d2f768..8b3712af98 100755
@@ -187,26 +187,26 @@ class CVE:
         """The set of package names referred by this CVE definition"""
         return set(p['product'] for p in self.each_cpe())
 
-    def affects(self, br_pkg):
+    def affects(self, name, version, cve_ignore_list):
         """
         True if the Buildroot Package object passed as argument is affected
         by this CVE.
         """
-        if br_pkg.is_cve_ignored(self.identifier):
+        if (self.identifier in cve_ignore_list):
             return self.CVE_DOESNT_AFFECT
 
         for cpe in self.each_cpe():
             affected = True
-            if cpe['product'] != br_pkg.name:
+            if cpe['product'] != name:
                 continue
             if cpe['v_start'] == '-':
                 return self.CVE_AFFECTS
             if not (cpe['v_start'] or cpe['v_end']):
                 print("No CVE affected version")
                 continue
-            pkg_version = distutils.version.LooseVersion(br_pkg.current_version)
+            pkg_version = distutils.version.LooseVersion(version)
             if not hasattr(pkg_version, "version"):
-                print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version))
+                print("Cannot parse package '%s' version '%s'" % (name, version))
                 continue
 
             if cpe['v_start']:
support/scripts/pkg-stats +8 -6
index 988dca7749..7a0eafe0a6 100755
@@ -242,11 +242,12 @@ class Package:
                     self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings))
                 return
 
-    def is_cve_ignored(self, cve):
+    def cve_ignored_list(self):
         """
-        Tells if the CVE is ignored by the package
+        Give the list of CVEs ignored by the package
         """
-        return cve in self.all_ignored_cves.get(self.pkgvar(), [])
+        print(self.all_ignored_cves.get(self.pkgvar(), []))
+        return list(self.all_ignored_cves.get(self.pkgvar(), []))
 
     def set_developers(self, developers):
         """
@@ -498,9 +499,10 @@ def check_package_cves(nvd_path, packages):
 
     for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
         for pkg_name in cve.pkg_names:
-            if pkg_name in packages and cve.affects(packages[pkg_name]) == cve.CVE_AFFECTS:
-                packages[pkg_name].cves.append(cve.identifier)
-
+            if pkg_name in packages:
+                pkg = packages[pkg_name]
+                if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()) == cve.CVE_AFFECTS :
+                    pkg.cves.append(cve.identifier)
 
 def calculate_stats(packages):
     stats = defaultdict(int)
Signature · 1 line
2.27.0
#19 lore raw

PATCH v2 6/9 support/scripts: Add a per configuration CVE checker

In reply to Gregory CLEMENT

This scripts takes as entry on stdin a JSON description of the package used for a given configuration. This description is the one generated by "make show-info".

The script generates the list of all the package used and if they are affected by a CVE. The output is either a JSON or an HTML file similar to the one generated by pkg-stats.

---
 support/scripts/cve-checker | 258 ++++++++++++++++++++++++++++++++++++
 1 file changed, 258 insertions(+)
 create mode 100755 support/scripts/cve-checker
support/scripts/cve-checker new file +258 -0
new file mode 100755
index 0000000000..19fd104b56
@@ -0,0 +1,258 @@
+#!/usr/bin/env python
+
+# Copyright (C) 2009 by Thomas Petazzoni <[email protected]>
+# Copyright (C) 2020 by Gregory CLEMENT <[email protected]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import argparse
+import datetime
+import os
+import json
+import sys
+
+sys.path.append('utils/')
+
+import cve as cvecheck
+
+class Package:
+    def __init__(self, name, version, ignored_cves):
+        self.name = name
+        self.version = version
+        self.cves = list()
+        self.ignored_cves = ignored_cves
+
+def check_package_cves(nvd_path, packages):
+    if not os.path.isdir(nvd_path):
+        os.makedirs(nvd_path)
+
+    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
+        for pkg_name in cve.pkg_names:
+            pkg = packages.get(pkg_name, '')
+            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
+                pkg.cves.append(cve.identifier)
+
+html_header = """
+<head>
+<script src=\"https://www.kryogenix.org/code/browser/sorttable/sorttable.js\"></script>
+<style type=\"text/css\">
+table {
+  width: 100%;
+}
+td {
+  border: 1px solid black;
+}
+td.centered {
+  text-align: center;
+}
+td.wrong {
+  background: #ff9a69;
+}
+td.correct {
+  background: #d2ffc4;
+}
+td.nopatches {
+  background: #d2ffc4;
+}
+td.somepatches {
+  background: #ffd870;
+}
+td.lotsofpatches {
+  background: #ff9a69;
+}
+
+td.good_url {
+  background: #d2ffc4;
+}
+td.missing_url {
+  background: #ffd870;
+}
+td.invalid_url {
+  background: #ff9a69;
+}
+
+td.version-good {
+  background: #d2ffc4;
+}
+td.version-needs-update {
+  background: #ff9a69;
+}
+td.version-unknown {
+ background: #ffd870;
+}
+td.version-error {
+ background: #ccc;
+}
+
+</style>
+<title>CVE status for Buildroot packages</title>
+</head>
+
+<a href=\"#results\">CVE Status</a><br/>
+
+<p id=\"sortable_hint\"></p>
+"""
+
+
+html_footer = """
+</body>
+<script>
+if (typeof sorttable === \"object\") {
+  document.getElementById(\"sortable_hint\").innerHTML =
+  \"hint: the table can be sorted by clicking the column headers\"
+}
+</script>
+</html>
+"""
+
+
+def infra_str(infra_list):
+    if not infra_list:
+        return "Unknown"
+    elif len(infra_list) == 1:
+        return "<b>%s</b><br/>%s" % (infra_list[0][1], infra_list[0][0])
+    elif infra_list[0][1] == infra_list[1][1]:
+        return "<b>%s</b><br/>%s + %s" % \
+            (infra_list[0][1], infra_list[0][0], infra_list[1][0])
+    else:
+        return "<b>%s</b> (%s)<br/><b>%s</b> (%s)" % \
+            (infra_list[0][1], infra_list[0][0],
+             infra_list[1][1], infra_list[1][0])
+
+
+def boolean_str(b):
+    if b:
+        return "Yes"
+    else:
+        return "No"
+
+
+def dump_html_pkg(f, pkg):
+    f.write(" <tr>\n")
+    f.write("  <td>%s</td>\n" % pkg.name)
+
+    # Current version
+    if len(pkg.version) > 20:
+        version = pkg.version[:20] + "..."
+    else:
+        version = pkg.version
+    f.write("  <td class=\"centered\">%s</td>\n" % version)
+
+    # CVEs
+    td_class = ["centered"]
+    if len(pkg.cves) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
+    f.write(" </tr>\n")
+
+
+def dump_html_all_pkgs(f, packages):
+    f.write("""
+<table class=\"sortable\">
+<tr>
+<td>Package</td>
+<td class=\"centered\">Version</td>
+<td class=\"centered\">CVEs</td>
+</tr>
+""")
+    for pkg in packages:
+        dump_html_pkg(f, pkg)
+    f.write("</table>")
+
+
+def dump_html_gen_info(f, date):
+    # Updated on Mon Feb 19 08:12:08 CET 2018
+    f.write("<p><i>Updated on %s</i></p>\n" % (str(date)))
+
+
+def dump_html(packages, date, output):
+    with open(output, 'w') as f:
+        f.write(html_header)
+        dump_html_all_pkgs(f, packages)
+        dump_html_gen_info(f, date)
+        f.write(html_footer)
+
+
+def dump_json(packages, date, output):
+    # Format packages as a dictionnary instead of a list
+    # Exclude local field that does not contains real date
+    excluded_fields = ['url_worker', 'name']
+    pkgs = {
+        pkg.name: {
+            k: v
+            for k, v in pkg.__dict__.items()
+            if k not in excluded_fields
+        } for pkg in packages
+    }
+     # The actual structure to dump, add date to it
+    final = {'packages': pkgs,
+             'date': str(date)}
+    with open(output, 'w') as f:
+        json.dump(final, f, indent=2, separators=(',', ': '))
+        f.write('\n')
+
+
+def resolvepath(path):
+        return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    output = parser.add_argument_group('output', 'Output file(s)')
+    output.add_argument('--html', dest='html', type=resolvepath,
+                        help='HTML output file')
+    output.add_argument('--json', dest='json', type=resolvepath,
+                        help='JSON output file')
+    packages = parser.add_mutually_exclusive_group()
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database',type=resolvepath,
+                        default='./nvd_dl')
+    args = parser.parse_args()
+    if not args.html and not args.json:
+        parser.error('at least one of --html or --json (or both) is required')
+    return args
+
+
+def __main__():
+    packages = list()
+    exclude_pacakges = ["linux", "gcc"]
+    content = json.load(sys.stdin)
+    for item in content:
+        if item in exclude_pacakges:
+            continue
+        pkg = content[item]
+        p = Package(item, pkg.get('version', ''), pkg.get('ignore_cves', ''))
+        packages.append(p)
+
+    args = parse_args()
+    date = datetime.datetime.utcnow()
+
+    if args.nvd_path:
+        print("Checking packages CVEs")
+        check_package_cves(args.nvd_path, {p.name: p for p in packages})
+    if args.html:
+        print("Write HTML")
+        dump_html(packages, date, args.html)
+    if args.json:
+        print("Write JSON")
+        dump_json(packages, date, args.json)
+
+__main__()
Signature · 1 line
2.27.0
#20 lore raw

PATCH v2 6/9 support/scripts: Add a per configuration CVE checker

Gregory,

Quoted 10 lines · Gregory CLEMENT

This scripts takes as entry on stdin a JSON description of the package used for a given configuration. This description is the one generated by "make show-info".

The script generates the list of all the package used and if they are affected by a CVE. The output is either a JSON or an HTML file similar to the one generated by pkg-stats.

- Checked that nvd-path's default works
- HTML report formatting is cleaned up
- Still see a bunch of the parsing package version output but these make sense as they are for virtual packages. This should be fixed when using CPE
Getting https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Cannot parse package 'zlib' version ''
Cannot parse package 'gettext' version ''
Cannot parse package 'openssl' version ''
Cannot parse package 'openssl' version ''

#21 lore raw

PATCH v2 7/9 support/script/pkg-stats: Manage the CVEs that need to be check

In reply to Gregory CLEMENT

When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version.

This patch exposes this new information in json and html format.

---
 support/scripts/pkg-stats | 29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)
support/scripts/pkg-stats +26 -3
index 7a0eafe0a6..89e92dc995 100755
@@ -106,6 +106,7 @@ class Package:
         self.url = None
         self.url_worker = None
         self.cves = list()
+        self.cves_to_check = list()
         self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None}
         self.status = {}
 
@@ -501,7 +502,10 @@ def check_package_cves(nvd_path, packages):
         for pkg_name in cve.pkg_names:
             if pkg_name in packages:
                 pkg = packages[pkg_name]
-                if cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list()) == cve.CVE_AFFECTS :
+                affected = cve.affects(pkg.name, pkg.current_version, pkg.cve_ignored_list())
+                if (affected == cve.CVE_UNKNOWN):
+                    pkg.cves_to_check.append(cve.identifier)
+                elif affected == cve.CVE_AFFECTS:
                     pkg.cves.append(cve.identifier)
 
 def calculate_stats(packages):
@@ -541,8 +545,11 @@ def calculate_stats(packages):
             stats["version-not-uptodate"] += 1
         stats["patches"] += pkg.patch_count
         stats["total-cves"] += len(pkg.cves)
+        stats["total-cves-to-check"] += len(pkg.cves_to_check)
         if len(pkg.cves) != 0:
             stats["pkg-cves"] += 1
+        if len(pkg.cves_to_check) != 0:
+            stats["pkg-cves_to_check"] += 1
     return stats
 
 
@@ -765,6 +772,17 @@ def dump_html_pkg(f, pkg):
         f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
     f.write("  </td>\n")
 
+    # CVEs to check
+    td_class = ["centered"]
+    if len(pkg.cves_to_check) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves_to_check:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
     f.write(" </tr>\n")
 
 
@@ -783,6 +801,7 @@ def dump_html_all_pkgs(f, packages):
 <td class=\"centered\">Warnings</td>
 <td class=\"centered\">Upstream URL</td>
 <td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
 </tr>
 """)
     for pkg in sorted(packages):
@@ -821,10 +840,14 @@ def dump_html_stats(f, stats):
             stats["version-not-uptodate"])
     f.write("<tr><td>Packages with no known upstream version</td><td>%s</td></tr>\n" %
             stats["version-unknown"])
-    f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+    f.write("<tr><td>"Packages that might be affected by CVEs, where version needs to be checked</td><td>%s</td></tr>\n" %
             stats["pkg-cves"])
-    f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+    f.write("<tr><td>Total number of CVEs that might affect all packages, where version needs to be checked</td><td>%s</td></tr>\n" %
             stats["total-cves"])
+    f.write("<tr><td>Packages affected by CVEs</td><td>%s</td></tr>\n" %
+            stats["pkg-cves_to_check"])
+    f.write("<tr><td>Total number of CVEs affecting all packages</td><td>%s</td></tr>\n" %
+            stats["total-cves_to_check"])
     f.write("</table>\n")
 
 
Signature · 1 line
2.27.0
#22 lore raw

PATCH v2 8/9 support/script/cve-checker: Manage the CVEs that need to be check

In reply to Gregory CLEMENT

When looking for if a package is affected, the version comparison can fail. This means that we don't know if the version of the package used is affected or not and we need to check manually the version.

This patch exposes this new information in json and html format.

---
 support/scripts/cve-checker | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)
support/scripts/cve-checker +19 -2
index 19fd104b56..712ec1ded0 100755
@@ -32,6 +32,7 @@ class Package:
         self.name = name
         self.version = version
         self.cves = list()
+        self.cves_to_check = list()
         self.ignored_cves = ignored_cves
 
 def check_package_cves(nvd_path, packages):
@@ -41,8 +42,12 @@ def check_package_cves(nvd_path, packages):
     for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
         for pkg_name in cve.pkg_names:
             pkg = packages.get(pkg_name, '')
-            if pkg and cve.affects(pkg.name, pkg.version, pkg.ignored_cves):
-                pkg.cves.append(cve.identifier)
+            if pkg:
+                affected = cve.affects(pkg.name, pkg.version, pkg.ignored_cves)
+                if (affected == cve.CVE_UNKNOWN):
+                    pkg.cves_to_check.append(cve.identifier)
+                elif affected == cve.CVE_AFFECTS:
+                    pkg.cves.append(cve.identifier)
 
 html_header = """
 <head>
@@ -161,6 +166,17 @@ def dump_html_pkg(f, pkg):
         f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
     f.write("  </td>\n")
 
+    # CVEs to check
+    td_class = ["centered"]
+    if len(pkg.cves_to_check) == 0:
+        td_class.append("correct")
+    else:
+        td_class.append("wrong")
+    f.write("  <td class=\"%s\">\n" % " ".join(td_class))
+    for cve in pkg.cves_to_check:
+        f.write("   <a href=\"https://security-tracker.debian.org/tracker/%s\">%s<br/>\n" % (cve, cve))
+    f.write("  </td>\n")
+
     f.write(" </tr>\n")
 
 
@@ -171,6 +187,7 @@ def dump_html_all_pkgs(f, packages):
 <td>Package</td>
 <td class=\"centered\">Version</td>
 <td class=\"centered\">CVEs</td>
+<td class=\"centered\">CVEs to check</td>
 </tr>
 """)
     for pkg in packages:
Signature · 1 line
2.27.0
#23 lore raw

PATCH v2 9/9 package/pkg-utils/cve.py: Manage case when package version doesn't exist

In reply to Gregory CLEMENT

Until now, when a package didn't report a version, then the CVE comparison was just skipped. It leads most of the time to declare the package not affected by the CVE.

Instead of it, report the CVE_UNKNOWN status in order to be aware that the CVE related to this package has to be checked.

---
 support/scripts/cve.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
support/scripts/cve.py +7 -1
index 8b3712af98..d7248ec2be 100755
@@ -195,6 +195,7 @@ class CVE:
         if (self.identifier in cve_ignore_list):
             return self.CVE_DOESNT_AFFECT
 
+        unknown_pkg_version = False
         for cpe in self.each_cpe():
             affected = True
             if cpe['product'] != name:
@@ -207,6 +208,7 @@ class CVE:
             pkg_version = distutils.version.LooseVersion(version)
             if not hasattr(pkg_version, "version"):
                 print("Cannot parse package '%s' version '%s'" % (name, version))
+                unknown_pkg_version = True
                 continue
 
             if cpe['v_start']:
@@ -226,4 +228,8 @@ class CVE:
                         return self.CVE_UNKNOWN
             if (affected):
                 return self.CVE_AFFECTS
-        return self.CVE_DOESNT_AFFECT
+
+        if unknown_pkg_version:
+            return self.CVE_UNKNOWN
+        else:
+            return self.CVE_DOESNT_AFFECT
Signature · 1 line
2.27.0