CPE validation
v2 cover CPE validation
Hello,
We have recently merged support to allow packages to express their CPE identifier, which is a unique identifier used by the NIST database to identify software products, and associate CVEs to them.
This patch series extends that by ensuring that CPEs affected to packages in Buildroot do exist in the official CPE dictionary from NIST, and if they do not exist, assist in generating the XML snippet that the CPE dictionary maintainers expect.
In more details:
- PATCH 1 introduces a new "name" field in the show-info JSON output, with the raw name of the package. This is used in PATCH 2.
- PATCH 2 fixes "make pkg-stats" to properly account for host packages, which were currently ignored. It makes use of the new "name" field introduced in PATCH 1 in the show-info JSON output.
- PATCH 3 introduces support/scripts/cpedb.py, which is a new set of helper that is capable of downloading the CPE dictionary, parsing it, and generating XML snippets to submit new CPE entries.
- PATCH 4 modifies support/scripts/pkg-stats to leverage the features of cpedb.py to include validation of CPE identifiers of Buildroot patches.
- PATCH 5 adds support/scripts/gen-missing-cpe, which allows to generate the XML snippets expected by the CPE dictionary mainainers
- PATCH 6 hooks up support/scripts/gen-missing-cpe as the "make missing-cpe" top-level make target
- PATCH 7 extends the documentation with more details about Buildroot features related to vulnerability management.
Changes since v1:
- Drop patches that have been merged upstream.
- New patches adding the "name" property in the show-info JSON output, and using it so that "make pkg-stats" takes into account host packages.
- Fix the parsing of the XML CPE databse for Python versions prior to 3.7. This was reported by Matt, and fixed by Gr?gory Clement.
- Improve how the namespaces are added to the XML file generated by "make missing-cpe", with the proper API instead of using a regexp. This was done by Gr?gory Clement.
- Add encoding in the XML declaration found in the XML file generated by "make missing-cpe". This was done by Gr?gory Clement.
- Reduced the size of the CPE database cache, with the side effect of also having prettier output for "make missing-cpe". This was done by Gr?gory Clement.
This is based on work from Matt Weber, which has been further refactored/rewritten by Gr?gory Clement and myself.
To test this:
$ make pkg-stats $ firefox output/pkg-stats.html $ make missing-cpe $ ls output/cpe-updates/
This work is also available at:
https://github.com/tpetazzoni/buildroot/commits/cpe-validation
Thomas
Matt Weber (2):
support/scripts/cpedb.py: new CPE XML helper
support/scripts/gen-missing-cpe: add new script
Thomas Petazzoni (5):
package/pkg-utils.mk: introduce "name" field in show-info output support/scripts/pkg-stats: properly handle host packages with -c
option
support/scripts/pkg-stats: check CPE existence in CPE dictionnary
Makefile: add new missing-cpe target
docs/manual: add details about vulnerability management
Makefile | 9 ++ docs/manual/common-usage.txt | 42 +++++++ package/pkg-utils.mk | 1 + support/scripts/cpedb.py | 203 ++++++++++++++++++++++++++++++++ support/scripts/gen-missing-cpe | 65 ++++++++++ support/scripts/pkg-stats | 32 +++-- 6 files changed, 342 insertions(+), 10 deletions(-) create mode 100644 support/scripts/cpedb.py create mode 100755 support/scripts/gen-missing-cpe
Signature · 1 line
2.29.2
PATCH v2 1/7 package/pkg-utils.mk: introduce "name" field in show-info output
The keys of the JSON dict returned by "make show-info" is the package name, including the "host-" prefix for host packages.
However, it is sometimes useful to get the actual name of the package, without the "host-" prefix, so we add a "name" property that holds the "raw name" of the package.
--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-utils.mk +1 -0
@@ -96,6 +96,7 @@ endef # $(1): upper-case package or filesystem name define json-info "$($(1)_NAME)": { + "name": "$($(1)_RAWNAME)", "type": "$($(1)_TYPE)", $(if $(filter rootfs,$($(1)_TYPE)), \ $(call _json-info-fs,$(1)), \
Signature · 1 line
2.29.2
PATCH v2 2/7 support/scripts/pkg-stats: properly handle host packages with -c option
In reply to Thomas PetazzoniIn commit 7a607dab336e7f78ab069cff1b503d0688950583 ("support/scripts/pkg-stats: support generating stats based on configured packages"), we added a -c option to pkg-stats to generate a report based on the list of packages enabled in the configuration, rather than for all packages.
This is done based on the list of packages returned in JSON format by "make show-info". However, we use the keys of the JSON dict returned by "make show-info", which include the host- prefix of host packages. Due to this, none of the host packages are currently matching and therefore they are not reported in the pkg-stats -c output.
This commit fixes that by using the recently introduced "name" property in the "make show-info" JSON dict.
--- support/scripts/pkg-stats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/pkg-stats +1 -1
@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list): def get_config_packages(): cmd = ["make", "--no-print-directory", "show-info"] js = json.loads(subprocess.check_output(cmd)) - return js.keys() + return set([v["name"] for k, v in js.items()]) def package_init_make_info():
Signature · 1 line
2.29.2
PATCH v2 2/7 support/scripts/pkg-stats: properly handle host packages with -c option
Quoted 13 lines · Thomas Petazzoni
In commit
7a607dab336e7f78ab069cff1b503d0688950583("support/scripts/pkg-stats: support generating stats based on configured packages"), we added a -c option to pkg-stats to generate a report based on the list of packages enabled in the configuration, rather than for all packages.This is done based on the list of packages returned in JSON format by "make show-info". However, we use the keys of the JSON dict returned by "make show-info", which include the host- prefix of host packages. Due to this, none of the host packages are currently matching and therefore they are not reported in the pkg-stats -c output.
Quoted 4 lines
This commit fixes that by using the recently introduced "name" property in the "make show-info" JSON dict.
Signed-off-by Thomas Petazzoni <[email protected]>
This and the previous patch applied to master, thanks.
The rest will have to wait for Gregory's SoB, but I'll review it.
Regards,
Arnout
Quoted 18 lines
--- support/scripts/pkg-stats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)support/scripts/pkg-stats +1
-1@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list): def get_config_packages(): cmd = ["make", "--no-print-directory", "show-info"] js = json.loads(subprocess.check_output(cmd)) - return js.keys() + return set([v["name"] for k, v in js.items()]) def package_init_make_info():
PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper
In reply to Thomas PetazzoniFrom: Matt Weber <[email protected]>
Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary
--- support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 support/scripts/cpedb.py
support/scripts/cpedb.py new file +203 -0
@@ -0,0 +1,203 @@ +#!/usr/bin/env python3 + +import xml.etree.ElementTree as ET +from xml.etree.ElementTree import Element, SubElement +import gzip +import os +import pickle +import requests +import time +from xml.dom import minidom + +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY'] + +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + +ns = { + '': 'http://cpe.mitre.org/dictionary/2.0', + 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3', + 'xml': 'http://www.w3.org/XML/1998/namespace' +} + + +class CPE: + def __init__(self, cpe_str, titles, refs): + self.cpe_str = cpe_str + self.titles = titles + self.references = refs + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6]) + + def update_xml_dict(self): + ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0') + cpes = Element('cpe-list') + cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3") + cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1") + cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3") + cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1") + cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance") + cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2") + cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd", + "http://cpe.mitre.org/dictionary/2.0", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd", + "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2", + "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd", + "http://scap.nist.gov/schema/scap-core/0.3", + "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd", + "http://scap.nist.gov/schema/configuration/0.1", + "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd", + "http://scap.nist.gov/schema/scap-core/0.1", + "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"])) + item = SubElement(cpes, 'cpe-item') + cpe_short_name = CPE.short_name(self.cpe_str) + cpe_new_ver = CPE.version_update(self.cpe_str) + + item.set('name', 'cpe:/' + cpe_short_name) + self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver) + for title in self.titles: + item.append(title) + if self.references: + item.append(self.references) + cpe23item = SubElement(item, 'cpe-23:cpe23-item') + cpe23item.set('name', self.cpe_str) + + # Generate the XML as a string + xmlstr = ET.tostring(cpes) + + # And use minidom to pretty print the XML + return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8") + + @staticmethod + def version(cpe): + return cpe.split(":")[5] + + @staticmethod + def product(cpe): + return cpe.split(":")[4] + + @staticmethod + def short_name(cpe): + return ":".join(cpe.split(":")[2:6]) + + @staticmethod + def version_update(cpe): + return ":".join(cpe.split(":")[5:6]) + + @staticmethod + def no_version(cpe): + return ":".join(cpe.split(":")[:5]) + + +class CPEDB: + def __init__(self, nvd_path): + self.all_cpes = dict() + self.all_cpes_no_version = dict() + self.nvd_path = nvd_path + + def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version): + print("CPE: Unzipping xml manifest...") + nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb')) + print("CPE: Converting xml manifest to dict...") + tree = ET.parse(nist_cpe_file) + all_cpedb = tree.getroot() + self.parse_dict(all_cpedb) + + print("CPE: Caching dictionary") + cpes_file = open(cache_all_cpes, 'wb') + pickle.dump(self.all_cpes, cpes_file) + cpes_file.close() + cpes_file = open(cache_all_cpes_no_version, 'wb') + pickle.dump(self.all_cpes_no_version, cpes_file) + cpes_file.close() + + def get_xml_dict(self): + print("CPE: Setting up NIST dictionary") + if not os.path.exists(os.path.join(self.nvd_path, "cpe")): + os.makedirs(os.path.join(self.nvd_path, "cpe")) + + cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL)) + if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400: + print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]") + cpe_dict = requests.get(CPEDB_URL) + open(cpe_dict_local, "wb").write(cpe_dict.content) + + cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl") + cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl") + + if not os.path.exists(cache_all_cpes) or \ + not os.path.exists(cache_all_cpes_no_version) or \ + os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \ + os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime: + self.gen_cached_cpedb(cpe_dict_local, + cache_all_cpes, + cache_all_cpes_no_version) + + print("CPE: Loading CACHED dictionary") + cpe_file = open(cache_all_cpes, 'rb') + self.all_cpes = pickle.load(cpe_file) + cpe_file.close() + cpe_file = open(cache_all_cpes_no_version, 'rb') + self.all_cpes_no_version = pickle.load(cpe_file) + cpe_file.close() + + def parse_dict(self, all_cpedb): + # Cycle through the dict and build two dict to be used for custom + # lookups of partial and complete CPE objects + # The objects are then used to create new proposed XML updates if + # if is determined one is required + # Out of the different language titles, select English + for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"): + cpe_titles = [] + for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns): + title.tail = None + cpe_titles.append(title) + + # Some older CPE don't include references, if they do, make + # sure we handle the case of one ref needing to be packed + # in a list + cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references") + if cpe_ref: + for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"): + ref.tail = None + ref.text = ref.text.upper() + if ref.text not in VALID_REFS: + ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501 + cpe_ref.tail = None + cpe_ref.text = None + + cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name') + item = CPE(cpe_str, cpe_titles, cpe_ref) + cpe_str_no_version = CPE.no_version(cpe_str) + # This dict must have a unique key for every CPE version + # which allows matching to the specific obj data of that + # NIST dict entry + self.all_cpes.update({cpe_str: item}) + # This dict has one entry for every CPE (w/o version) to allow + # partial match (no valid version) check (the obj is saved and + # used as seed for suggested xml updates. By updating the same + # non-version'd entry, it assumes the last update here is the + # latest version in the NIST dict) + self.all_cpes_no_version.update({cpe_str_no_version: item}) + + def find_partial(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find_partial_obj(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return self.all_cpes_no_version[cpe_str_no_version] + + def find_partial_latest_version(self, cpe_str_partial): + cpe_obj = self.find_partial_obj(cpe_str_partial) + return cpe_obj.cpe_cur_ver + + def find(self, cpe_str): + if self.find_partial(cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def gen_update_xml(self, cpe_str): + cpe = self.find_partial_obj(cpe_str) + return cpe.update_xml_dict()
Signature · 1 line
2.29.2
PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper
Gr?gory, All,
On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
Quoted 10 lines
From: Matt Weber <[email protected]>
Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionarySigned-off-by Matthew Weber <[email protected]>Co-Developed-by Gr?gory Clement <[email protected]>
Gr?gory, you need to add your SOB line right after your CDB line.
Ditto for the following patch.
Regards,
Yann E. MORIN.
Quoted 219 lines
Co-Developed-by Thomas Petazzoni <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>--- support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 support/scripts/cpedb.pysupport/scripts/cpedb.py new file +203
-0@@ -0,0 +1,203 @@ +#!/usr/bin/env python3 + +import xml.etree.ElementTree as ET +from xml.etree.ElementTree import Element, SubElement +import gzip +import os +import pickle +import requests +import time +from xml.dom import minidom + +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY'] + +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + +ns = { + '': 'http://cpe.mitre.org/dictionary/2.0', + 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3', + 'xml': 'http://www.w3.org/XML/1998/namespace' +} + + +class CPE: + def __init__(self, cpe_str, titles, refs): + self.cpe_str = cpe_str + self.titles = titles + self.references = refs + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6]) + + def update_xml_dict(self): + ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0') + cpes = Element('cpe-list') + cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3") + cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1") + cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3") + cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1") + cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance") + cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2") + cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd", + "http://cpe.mitre.org/dictionary/2.0", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd", + "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2", + "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd", + "http://scap.nist.gov/schema/scap-core/0.3", + "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd", + "http://scap.nist.gov/schema/configuration/0.1", + "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd", + "http://scap.nist.gov/schema/scap-core/0.1", + "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"])) + item = SubElement(cpes, 'cpe-item') + cpe_short_name = CPE.short_name(self.cpe_str) + cpe_new_ver = CPE.version_update(self.cpe_str) + + item.set('name', 'cpe:/' + cpe_short_name) + self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver) + for title in self.titles: + item.append(title) + if self.references: + item.append(self.references) + cpe23item = SubElement(item, 'cpe-23:cpe23-item') + cpe23item.set('name', self.cpe_str) + + # Generate the XML as a string + xmlstr = ET.tostring(cpes) + + # And use minidom to pretty print the XML + return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8") + + @staticmethod + def version(cpe): + return cpe.split(":")[5] + + @staticmethod + def product(cpe): + return cpe.split(":")[4] + + @staticmethod + def short_name(cpe): + return ":".join(cpe.split(":")[2:6]) + + @staticmethod + def version_update(cpe): + return ":".join(cpe.split(":")[5:6]) + + @staticmethod + def no_version(cpe): + return ":".join(cpe.split(":")[:5]) + + +class CPEDB: + def __init__(self, nvd_path): + self.all_cpes = dict() + self.all_cpes_no_version = dict() + self.nvd_path = nvd_path + + def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version): + print("CPE: Unzipping xml manifest...") + nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb')) + print("CPE: Converting xml manifest to dict...") + tree = ET.parse(nist_cpe_file) + all_cpedb = tree.getroot() + self.parse_dict(all_cpedb) + + print("CPE: Caching dictionary") + cpes_file = open(cache_all_cpes, 'wb') + pickle.dump(self.all_cpes, cpes_file) + cpes_file.close() + cpes_file = open(cache_all_cpes_no_version, 'wb') + pickle.dump(self.all_cpes_no_version, cpes_file) + cpes_file.close() + + def get_xml_dict(self): + print("CPE: Setting up NIST dictionary") + if not os.path.exists(os.path.join(self.nvd_path, "cpe")): + os.makedirs(os.path.join(self.nvd_path, "cpe")) + + cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL)) + if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400: + print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]") + cpe_dict = requests.get(CPEDB_URL) + open(cpe_dict_local, "wb").write(cpe_dict.content) + + cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl") + cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl") + + if not os.path.exists(cache_all_cpes) or \ + not os.path.exists(cache_all_cpes_no_version) or \ + os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \ + os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime: + self.gen_cached_cpedb(cpe_dict_local, + cache_all_cpes, + cache_all_cpes_no_version) + + print("CPE: Loading CACHED dictionary") + cpe_file = open(cache_all_cpes, 'rb') + self.all_cpes = pickle.load(cpe_file) + cpe_file.close() + cpe_file = open(cache_all_cpes_no_version, 'rb') + self.all_cpes_no_version = pickle.load(cpe_file) + cpe_file.close() + + def parse_dict(self, all_cpedb): + # Cycle through the dict and build two dict to be used for custom + # lookups of partial and complete CPE objects + # The objects are then used to create new proposed XML updates if + # if is determined one is required + # Out of the different language titles, select English + for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"): + cpe_titles = [] + for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns): + title.tail = None + cpe_titles.append(title) + + # Some older CPE don't include references, if they do, make + # sure we handle the case of one ref needing to be packed + # in a list + cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references") + if cpe_ref: + for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"): + ref.tail = None + ref.text = ref.text.upper() + if ref.text not in VALID_REFS: + ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501 + cpe_ref.tail = None + cpe_ref.text = None + + cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name') + item = CPE(cpe_str, cpe_titles, cpe_ref) + cpe_str_no_version = CPE.no_version(cpe_str) + # This dict must have a unique key for every CPE version + # which allows matching to the specific obj data of that + # NIST dict entry + self.all_cpes.update({cpe_str: item}) + # This dict has one entry for every CPE (w/o version) to allow + # partial match (no valid version) check (the obj is saved and + # used as seed for suggested xml updates. By updating the same + # non-version'd entry, it assumes the last update here is the + # latest version in the NIST dict) + self.all_cpes_no_version.update({cpe_str_no_version: item}) + + def find_partial(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find_partial_obj(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return self.all_cpes_no_version[cpe_str_no_version] + + def find_partial_latest_version(self, cpe_str_partial): + cpe_obj = self.find_partial_obj(cpe_str_partial) + return cpe_obj.cpe_cur_ver + + def find(self, cpe_str): + if self.find_partial(cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def gen_update_xml(self, cpe_str): + cpe = self.find_partial_obj(cpe_str) + return cpe.update_xml_dict()Signature · 1 line
2.29.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper
Gr?gory, All,
On 2021-01-31 23:47 +0100, Yann E. MORIN spake thusly:
Quoted 9 lines
On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
Quoted 8 lines
From: Matt Weber <[email protected]>
Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary
Quoted 6 lines
Quoted 2 lines
Signed-off-by Matthew Weber <[email protected]>Co-Developed-by Gr?gory Clement <[email protected]>Gr?gory, you need to add your SOB line right after your CDB line.
Ditto for the following patch.
No need to resend the series, just reply to each patch with your SoB line, I'll add them when applying.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper
In reply to Thomas PetazzoniQuoted 46 lines · Thomas Petazzoni
From: Matt Weber <[email protected]>
Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.
- Defines the CPE as a object with operations / formats
- Processing of CPE dictionarySigned-off-by Matthew Weber <[email protected]>Co-Developed-by Gr?gory Clement <[email protected]>Co-Developed-by Thomas Petazzoni <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>--- support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 support/scripts/cpedb.pysupport/scripts/cpedb.py new file +23
-0@@ -0,0 +1,203 @@ +#!/usr/bin/env python3 + +import xml.etree.ElementTree as ET +from xml.etree.ElementTree import Element, SubElement +import gzip +import os +import pickle +import requests +import time +from xml.dom import minidom + +VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY'] + +CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + +ns = { + '': 'http://cpe.mitre.org/dictionary/2.0', + 'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3', + 'xml': 'http://www.w3.org/XML/1998/namespace' +} + + +class CPE:
support/scripts/cpedb.py line 33
Quoted 10 lines
support/scripts/cpedb.py new file +10
-0+ def __init__(self, cpe_str, titles, refs): + self.cpe_str = cpe_str + self.titles = titles + self.references = refs + self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6]) + + def update_xml_dict(self): + ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0') + cpes = Element('cpe-list') + cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
Note that instead of all these cpe.set calls, you can also do:
cpes.attrib = {
'xmlns:ns6': "http://scap.nist.gov/schema/scap-core/0.1",
...
}which IMO is more readable. But that's just nitpick.
In fact, I think you can do it in the constructor:
cpes = Element('cpe-list', attrib = { ...but that is probably less readable.
support/scripts/cpedb.py line 40
Quoted 7 lines
support/scripts/cpedb.py new file +7
-0+ cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1") + cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3") + cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1") + cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance") + cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2") + cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
And here instead of join you can use:
cpes.set('xsi:schemaLocation',
"http://scap.nist.gov/schema/cpe-extension/2.3 "
"http://cpe.mitre.org/dictionary/2.0 "
(Python has string concatenation like C).
Quoted 33 lines
support/scripts/cpedb.py new file +33
-0+ "http://cpe.mitre.org/dictionary/2.0", + "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd", + "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2", + "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd", + "http://scap.nist.gov/schema/scap-core/0.3", + "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd", + "http://scap.nist.gov/schema/configuration/0.1", + "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd", + "http://scap.nist.gov/schema/scap-core/0.1", + "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"])) + item = SubElement(cpes, 'cpe-item') + cpe_short_name = CPE.short_name(self.cpe_str) + cpe_new_ver = CPE.version_update(self.cpe_str) + + item.set('name', 'cpe:/' + cpe_short_name) + self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver) + for title in self.titles: + item.append(title) + if self.references: + item.append(self.references) + cpe23item = SubElement(item, 'cpe-23:cpe23-item') + cpe23item.set('name', self.cpe_str) + + # Generate the XML as a string + xmlstr = ET.tostring(cpes) + + # And use minidom to pretty print the XML + return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8") + + @staticmethod + def version(cpe): + return cpe.split(":")[5] +
support/scripts/cpedb.py line 83
Quoted 10 lines
support/scripts/cpedb.py new file +10
-0+ @staticmethod + def product(cpe): + return cpe.split(":")[4] + + @staticmethod + def short_name(cpe): + return ":".join(cpe.split(":")[2:6]) + + @staticmethod + def version_update(cpe):
It's weird to make this a staticmethod if the only caller takes self.cpe_str as
an argument, but OK.
Quoted 26 lines
support/scripts/cpedb.py new file +26
-0+ return ":".join(cpe.split(":")[5:6]) + + @staticmethod + def no_version(cpe): + return ":".join(cpe.split(":")[:5]) + + +class CPEDB: + def __init__(self, nvd_path): + self.all_cpes = dict() + self.all_cpes_no_version = dict() + self.nvd_path = nvd_path + + def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version): + print("CPE: Unzipping xml manifest...") + nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb')) + print("CPE: Converting xml manifest to dict...") + tree = ET.parse(nist_cpe_file) + all_cpedb = tree.getroot() + self.parse_dict(all_cpedb) + + print("CPE: Caching dictionary") + cpes_file = open(cache_all_cpes, 'wb') + pickle.dump(self.all_cpes, cpes_file) + cpes_file.close() + cpes_file = open(cache_all_cpes_no_version, 'wb')
support/scripts/cpedb.py line 119
Quoted 10 lines
support/scripts/cpedb.py new file +10
-0+ pickle.dump(self.all_cpes_no_version, cpes_file) + cpes_file.close() + + def get_xml_dict(self): + print("CPE: Setting up NIST dictionary") + if not os.path.exists(os.path.join(self.nvd_path, "cpe")): + os.makedirs(os.path.join(self.nvd_path, "cpe")) + + cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL)) + if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
This should be a parameter@the beginning of the file rather than a constant
here. Ideally expressed in hours, with *3600 here.
Quoted 9 lines
support/scripts/cpedb.py new file +9
-0+ print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]") + cpe_dict = requests.get(CPEDB_URL) + open(cpe_dict_local, "wb").write(cpe_dict.content) + + cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl") + cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl") + + if not os.path.exists(cache_all_cpes) or \ + not os.path.exists(cache_all_cpes_no_version) or \
support/scripts/cpedb.py line 138
Quoted 10 lines
support/scripts/cpedb.py new file +10
-0+ os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \ + os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime: + self.gen_cached_cpedb(cpe_dict_local, + cache_all_cpes, + cache_all_cpes_no_version) + + print("CPE: Loading CACHED dictionary") + cpe_file = open(cache_all_cpes, 'rb') + self.all_cpes = pickle.load(cpe_file) + cpe_file.close()
The above is better done as
with open(cache_all_cpes, 'rb') as cpe_file:
self.all_cpes = pickle.load(cpe_file)
This do things properly in case of exception.
However, I'm not so sure if the pickled cache is really worth it. I gave it a
go, and the ET.parse + parse_dict took roughly 20 seconds, while just loading the pickle file also took about 20 seconds...
support/scripts/cpedb.py line 148
Quoted 10 lines
support/scripts/cpedb.py new file +10
-0+ cpe_file = open(cache_all_cpes_no_version, 'rb') + self.all_cpes_no_version = pickle.load(cpe_file) + cpe_file.close() + + def parse_dict(self, all_cpedb): + # Cycle through the dict and build two dict to be used for custom + # lookups of partial and complete CPE objects + # The objects are then used to create new proposed XML updates if + # if is determined one is required + # Out of the different language titles, select English
This could have been a docstring.
Actually, all "public" functions should really have a docstring.
So, I accidentally applied this patch... So all the above comments are just
informational - feel free to send follow-up patches, or ignore my feedback.
Regards,
Arnout
Quoted 56 lines
support/scripts/cpedb.py new file +55
-0+ for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"): + cpe_titles = [] + for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns): + title.tail = None + cpe_titles.append(title) + + # Some older CPE don't include references, if they do, make + # sure we handle the case of one ref needing to be packed + # in a list + cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references") + if cpe_ref: + for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"): + ref.tail = None + ref.text = ref.text.upper() + if ref.text not in VALID_REFS: + ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501 + cpe_ref.tail = None + cpe_ref.text = None + + cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name') + item = CPE(cpe_str, cpe_titles, cpe_ref) + cpe_str_no_version = CPE.no_version(cpe_str) + # This dict must have a unique key for every CPE version + # which allows matching to the specific obj data of that + # NIST dict entry + self.all_cpes.update({cpe_str: item}) + # This dict has one entry for every CPE (w/o version) to allow + # partial match (no valid version) check (the obj is saved and + # used as seed for suggested xml updates. By updating the same + # non-version'd entry, it assumes the last update here is the + # latest version in the NIST dict) + self.all_cpes_no_version.update({cpe_str_no_version: item}) + + def find_partial(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find_partial_obj(self, cpe_str): + cpe_str_no_version = CPE.no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return self.all_cpes_no_version[cpe_str_no_version] + + def find_partial_latest_version(self, cpe_str_partial): + cpe_obj = self.find_partial_obj(cpe_str_partial) + return cpe_obj.cpe_cur_ver + + def find(self, cpe_str): + if self.find_partial(cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def gen_update_xml(self, cpe_str): + cpe = self.find_partial_obj(cpe_str) + return cpe.update_xml_dict()
PATCH v2 4/7 support/scripts/pkg-stats: check CPE existence in CPE dictionnary
In reply to Thomas PetazzoniThis commit extends pkg-stats to leverage the recently introduced CPEDB class to verify that the CPEs provided by Buildroot packages are indeed known in the official CPE dictionnary provided by NVD.
--- support/scripts/pkg-stats | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)
support/scripts/pkg-stats +21 -9
@@ -32,7 +32,7 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", "..")) sys.path.append(os.path.join(brpath, "utils")) from getdeveloperlib import parse_developers # noqa: E402 - +from cpedb import CPEDB # noqa: E402 INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile(r"\s*https?://\S*\s*$")
@@ -225,7 +225,8 @@ class Package: if var in self.all_cpeids: self.cpeid = self.all_cpeids[var] - self.status['cpe'] = ("ok", "verified CPE identifier") + # Set a preliminary status, it might be overridden by check_package_cpes() + self.status['cpe'] = ("warning", "not checked against CPE dictionnary") else: self.status['cpe'] = ("error", "no verified CPE identifier")
@@ -600,6 +601,18 @@ def check_package_cves(nvd_path, packages): pkg.status['cve'] = ("ok", "not affected by CVEs") +def check_package_cpes(nvd_path, packages): + cpedb = CPEDB(nvd_path) + cpedb.get_xml_dict() + for p in packages: + if not p.cpeid: + continue + if cpedb.find(p.cpeid): + p.status['cpe'] = ("ok", "verified CPE identifier") + else: + p.status['cpe'] = ("error", "CPE identifier unknown in CPE database") + + def calculate_stats(packages): stats = defaultdict(int) stats['packages'] = len(packages)
@@ -898,19 +911,17 @@ def dump_html_pkg(f, pkg): # CPE ID td_class = ["left"] - if pkg.status['cpe'][0] == "ok": + if pkg.is_status_ok("cpe"): td_class.append("cpe-ok") - elif pkg.status['cpe'][0] == "error": + elif pkg.is_status_error("cpe"): td_class.append("cpe-nok") else: td_class.append("cpe-unknown") f.write(" <td class=\"%s\">\n" % " ".join(td_class)) - if pkg.status['cpe'][0] == "ok": + if pkg.cpeid: f.write(" <code>%s</code>\n" % pkg.cpeid) - elif pkg.status['cpe'][0] == "error": - f.write(" N/A\n") - else: - f.write(" %s\n" % pkg.status['cpe'][1]) + if not pkg.is_status_ok("cpe"): + f.write(" %s%s\n" % ("<br/>" if pkg.cpeid else "", pkg.status['cpe'][1])) f.write(" </td>\n") f.write(" </tr>\n")
@@ -1106,6 +1117,7 @@ def __main__(): if args.nvd_path: print("Checking packages CVEs") check_package_cves(args.nvd_path, packages) + check_package_cpes(args.nvd_path, packages) print("Calculate stats") stats = calculate_stats(packages) if args.html:
Signature · 1 line
2.29.2
PATCH v2 4/7 support/scripts/pkg-stats: check CPE existence in CPE dictionnary
Quoted 6 lines · Thomas Petazzoni
This commit extends pkg-stats to leverage the recently introduced CPEDB class to verify that the CPEs provided by Buildroot packages are indeed known in the official CPE dictionnary provided by NVD.
Co-Developed-by Gr?gory Clement <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>
This one was accidentally applied to master without Gre?gory's SoB as well.
I didn't have any further comments on the patch. Imagine that!
Regards,
Arnout
PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script
In reply to Thomas PetazzoniFrom: Matt Weber <[email protected]>
This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning
- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.
Ref: NIST has a group email (cpe_dictionary at nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.
Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary
--- support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 support/scripts/gen-missing-cpe
support/scripts/gen-missing-cpe new file +65 -0
@@ -0,0 +1,65 @@ +#!/usr/bin/env python3 + +import argparse +import sys +import json +import subprocess +import os +from cpedb import CPEDB, CPE + + +def gen_update_xml_reports(cpes, cpedb, output): + cpe_need_update = [] + + for cpe in cpes: + result = cpedb.find(cpe) + if not result: + result = cpedb.find_partial(CPE.no_version(cpe)) + if result: + cpe_need_update.append(cpe) + else: + print("WARNING: no match found for '%s'" % cpe) + + for cpe in cpe_need_update: + xml = cpedb.gen_update_xml(cpe) + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml' + print("Generating %s" % fname) + fp = open(os.path.join(output, fname), 'w+') + fp.write(xml) + fp.close() + + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes))) + + +def get_cpe_ids(): + print("Getting list of CPE for enabled packages") + cmd = ["make", "--no-print-directory", "show-info"] + js = json.loads(subprocess.check_output(cmd).decode("utf-8")) + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v]) + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path)) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('--output', dest='output', + help='Path to the output CPE update files', type=resolvepath, required=True) + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath, required=True) + return parser.parse_args() + + +def __main__(): + args = parse_args() + if not os.path.isdir(args.output): + print("ERROR: output directory %s does not exist" % args.output) + sys.exit(1) + cpedb = CPEDB(args.nvd_path) + cpedb.get_xml_dict() + cpes = get_cpe_ids() + gen_update_xml_reports(cpes, cpedb, args.output) + + +__main__()
Signature · 1 line
2.29.2
PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script
Quoted 7 lines · Thomas Petazzoni
From: Matt Weber <[email protected]>
This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning
Maybe there should also be a warning for packages which don't have cpeid set at
all...
Quoted 10 lines
- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.
Ref: NIST has a group email (
cpe_dictionaryat nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.
It would be very useful if the script would also print a URL that describes the
submission process.
Quoted 24 lines
Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionarySigned-off-by Matt Weber <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>--- support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 support/scripts/gen-missing-cpesupport/scripts/gen-missing-cpe new file +1
-0@@ -0,0 +1,65 @@ +#!/usr/bin/env python3
support/scripts/gen-missing-cpe line 11
Quoted 10 lines
support/scripts/gen-missing-cpe new file +10
-0+ +import argparse +import sys +import json +import subprocess +import os +from cpedb import CPEDB, CPE + + +def gen_update_xml_reports(cpes, cpedb, output):
cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).
support/scripts/gen-missing-cpe line 14
Quoted 3 lines
support/scripts/gen-missing-cpe new file +3
-0+ cpe_need_update = [] + + for cpe in cpes:
for cpeid in cpeids:
Quoted 5 lines
support/scripts/gen-missing-cpe new file +5
-0+ result = cpedb.find(cpe) + if not result: + result = cpedb.find_partial(CPE.no_version(cpe)) + if result: + cpe_need_update.append(cpe)
support/scripts/gen-missing-cpe line 29
Quoted 10 lines
support/scripts/gen-missing-cpe new file +10
-0+ else: + print("WARNING: no match found for '%s'" % cpe) + + for cpe in cpe_need_update: + xml = cpedb.gen_update_xml(cpe) + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml' + print("Generating %s" % fname) + fp = open(os.path.join(output, fname), 'w+') + fp.write(xml) + fp.close()
This should be
with open(...) as fp:
fp.write(xml)support/scripts/gen-missing-cpe line 42
Quoted 13 lines
support/scripts/gen-missing-cpe new file +13
-0+ + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes))) + + +def get_cpe_ids(): + print("Getting list of CPE for enabled packages") + cmd = ["make", "--no-print-directory", "show-info"] + js = json.loads(subprocess.check_output(cmd).decode("utf-8")) + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v]) + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path))
I don't understand this...
- expanduser should already have been done by the shell. If you call the script as
gen-missing-cpe --output \~/some-path
then I want the output to be in a directory called ~ in the current directory. That's what the rest of the world does. (Yes, I get it, you want to be able to do --output=~/some-path but that's just wrong.)
- abspath shouldn't be needed for anything, we're not doing any changedir or anything like that.
In addition, this script is supposed to be called from make missing-cpe which has already done those expansions.
Quoted 13 lines
support/scripts/gen-missing-cpe new file +13
-0+ + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('--output', dest='output', + help='Path to the output CPE update files', type=resolvepath, required=True) + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath, required=True) + return parser.parse_args() + + +def __main__(): + args = parse_args()
support/scripts/gen-missing-cpe line 65
Quoted 10 lines
support/scripts/gen-missing-cpe new file +10
-0+ if not os.path.isdir(args.output): + print("ERROR: output directory %s does not exist" % args.output) + sys.exit(1) + cpedb = CPEDB(args.nvd_path) + cpedb.get_xml_dict() + cpes = get_cpe_ids() + gen_update_xml_reports(cpes, cpedb, args.output) + + +__main__()
Usually it would be
if __name__ == '__main__':
__main__()
Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just
Regards,
Arnout
PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script
Thomas,
Quoted 6 lines · Arnout Vandecappelle
Quoted 2 lines · Thomas Petazzoni
From: Matt Weber <[email protected]>
Quoted 8 lines
Quoted 5 lines
This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning
Maybe there should also be a warning for packages which don't have cpeid set at
all...
Agree, this would be something that people would miss if we don't list the missing. My original iteration had that as a category that was printed to screen but no XML generated for it.
Regards,
Matt
PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script
In reply to Arnout VandecappelleArnout, All,
On 2021-02-02 22:29 +0100, Arnout Vandecappelle spake thusly:
Quoted 10 lines · Thomas Petazzoni
Quoted 7 lines
From: Matt Weber <[email protected]>
This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning
Maybe there should also be a warning for packages which don't have cpeid set at
all...
I haven't done anything to do so; I though it was better to have at least this script in its current state, rather than nothing at all.
[--SNIP--]
Quoted 2 lines
Quoted 1 line
+def
gen_update_xml_reports(cpes, cpedb, output):cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).
Done.
[--SNIP--]
Quoted 6 lines
support/scripts/gen-missing-cpe line 29
Quoted 3 lines
support/scripts/gen-missing-cpe new file +3
-0+ fp = open(os.path.join(output, fname), 'w+') + fp.write(xml) + fp.close()This should be
with open(...) as fp: fp.write(xml)
Done.
[--SNIP--]
Quoted 13 lines
support/scripts/gen-missing-cpe line 42
Quoted 2 lines
support/scripts/gen-missing-cpe new file +2
-0+def resolvepath(path): + return os.path.abspath(os.path.expanduser(path))I don't understand this...
- expanduser should already have been done by the shell. If you call the script as
gen-missing-cpe --output \~/some-path
then I want the output to be in a directory called ~ in the current directory. That's what the rest of the world does. (Yes, I get it, you want to be able to do --output=~/some-path but that's just wrong.)
Quoted 5 lines
- abspath shouldn't be needed for anything, we're not doing any changedir or anything like that.
In addition, this script is supposed to be called from
make missing-cpewhich has already done those expansions.
It was not obvious to me either that resolvepath() was needed, and as] you said, the usage we have in Buildroot does not need expansion. However, getting rid of it seemed too much, so I left it as-is.
Again, I thought it was better that we have this script rather than nothing.
[--SNIP--]
Quoted 4 lines
Quoted 1 line
+__main__()
Usually it would be
if__name__== '__main__':__main__()
Done.
Quoted 3 lines
Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just
This was sitting in the backlog for quite a while, and we really needed to push a bit on our CPE (and CVE) tooling, so I applied this to master now.
Thanks!
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Yann E. MORIN
AppliedPATCH v2 5/7 support/scripts/gen-missing-cpe: add new script
In reply to Thomas PetazzoniThomas, All,
On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:
Quoted 27 lines
From: Matt Weber <[email protected]>
This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:
- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning
- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.
Ref: NIST has a group email (
cpe_dictionaryat nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary
Quoted 2 lines
Signed-off-by Matt Weber <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>
Applied to master, with most of the changes pointed out by Arnout, thanks. I'll further reply to Arnout's comment...
Regards,
Yann E. MORIN.
Quoted 83 lines
--- support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 support/scripts/gen-missing-cpesupport/scripts/gen-missing-cpe new file +65
-0@@ -0,0 +1,65 @@ +#!/usr/bin/env python3 + +import argparse +import sys +import json +import subprocess +import os +from cpedb import CPEDB, CPE + + +def gen_update_xml_reports(cpes, cpedb, output): + cpe_need_update = [] + + for cpe in cpes: + result = cpedb.find(cpe) + if not result: + result = cpedb.find_partial(CPE.no_version(cpe)) + if result: + cpe_need_update.append(cpe) + else: + print("WARNING: no match found for '%s'" % cpe) + + for cpe in cpe_need_update: + xml = cpedb.gen_update_xml(cpe) + fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml' + print("Generating %s" % fname) + fp = open(os.path.join(output, fname), 'w+') + fp.write(xml) + fp.close() + + print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes))) + + +def get_cpe_ids(): + print("Getting list of CPE for enabled packages") + cmd = ["make", "--no-print-directory", "show-info"] + js = json.loads(subprocess.check_output(cmd).decode("utf-8")) + return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v]) + + +def resolvepath(path): + return os.path.abspath(os.path.expanduser(path)) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('--output', dest='output', + help='Path to the output CPE update files', type=resolvepath, required=True) + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database', type=resolvepath, required=True) + return parser.parse_args() + + +def __main__(): + args = parse_args() + if not os.path.isdir(args.output): + print("ERROR: output directory %s does not exist" % args.output) + sys.exit(1) + cpedb = CPEDB(args.nvd_path) + cpedb.get_xml_dict() + cpes = get_cpe_ids() + gen_update_xml_reports(cpes, cpedb, args.output) + + +__main__()Signature · 6 lines
2.29.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 6/7 Makefile: add new missing-cpe target
In reply to Thomas PetazzoniIt invokes the recently introduced gen-missing-cpe script.
--- Makefile | 9 +++++++++ 1 file changed, 9 insertions(+)
Makefile +9 -0
@@ -945,6 +945,14 @@ pkg-stats: --html $(O)/pkg-stats.html \ --nvd-path $(DL_DIR)/buildroot-nvd +.PHONY: missing-cpe +missing-cpe: + $(Q)mkdir -p $(O)/cpe-updates + $(Q)cd "$(CONFIG_DIR)" ; \ + $(TOPDIR)/support/scripts/gen-missing-cpe \ + --nvd-path $(DL_DIR)/buildroot-nvd \ + --output $(O)/cpe-updates + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) # Some subdirectories are also package names. To avoid that "make linux"
@@ -1163,6 +1171,7 @@ help: @echo ' legal-info - generate info about license compliance' @echo ' show-info - generate info about packages, as a JSON blurb' @echo ' pkg-stats - generate info about packages as JSON and HTML' + @echo ' missing-cpe - generate XML snippets for missing CPE identifiers' @echo ' printvars - dump internal variables selected with VARS=...' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
Signature · 1 line
2.29.2
PATCH v2 6/7 Makefile: add new missing-cpe target
Quoted 3 lines · Thomas Petazzoni
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by Thomas Petazzoni <[email protected]>
Regards,
Arnout
Quoted 32 lines
--- Makefile | 9 +++++++++ 1 file changed, 9 insertions(+)Makefile +9
-0@@ -945,6 +945,14 @@ pkg-stats: --html $(O)/pkg-stats.html \ --nvd-path $(DL_DIR)/buildroot-nvd +.PHONY: missing-cpe +missing-cpe: + $(Q)mkdir -p $(O)/cpe-updates + $(Q)cd "$(CONFIG_DIR)" ; \ + $(TOPDIR)/support/scripts/gen-missing-cpe \ + --nvd-path $(DL_DIR)/buildroot-nvd \ + --output $(O)/cpe-updates + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) # Some subdirectories are also package names. To avoid that "make linux"@@ -1163,6 +1171,7 @@ help: @echo ' legal-info - generate info about license compliance' @echo ' show-info - generate info about packages, as a JSON blurb' @echo ' pkg-stats - generate info about packages as JSON and HTML' + @echo ' missing-cpe - generate XML snippets for missing CPE identifiers' @echo ' printvars - dump internal variables selected with VARS=...' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
PATCH v2 6/7 Makefile: add new missing-cpe target
Thomas,
Quoted 10 lines · Arnout Vandecappelle
Quoted 3 lines · Thomas Petazzoni
It invokes the recently introduced gen-missing-cpe script.
Signed-off-by Thomas Petazzoni <[email protected]>
PATCH v2 7/7 docs/manual: add details about vulnerability management
In reply to Thomas PetazzoniThis addition to the documentation gives some details about CPE/CVE, pkg-stats and missing-cpe.
--- docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)
docs/manual/common-usage.txt +49 -0
@@ -417,6 +417,55 @@ effects: be empty and it's only at the very end of the build that they will be populated. +[[vulnerability-management]] +=== Vulnerability management + +Buildroot integrates a number of features that help tracking whether +known security vulnerabilities are affecting some of the +packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities +and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided +by NIST. + +First, each package in Buildroot can be associated to a _CPE_ +identifier, by means of the various +FOO_CPE_ID_<something>+ variables +that can be provided by each package. See +xref:generic-package-reference[] for more details about those +variables. When the _CPE_ identifier of a package is not defined, a +default value is used, which may or may not be correct. This _CPE_ +identifier is used to find in the NVD database of vulnerabilities +which _CVEs_ are affecting any given package. + +Based on this and the NIST databases, the +make pkg-stats+ target +produces JSON and HTML formatted reports that indicate for each +package of the current Buildroot configuration: + +* If there are known _CVEs_ for each package. Note that CVEs can + manually be ignored on a per-package basis using the + +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch + that fixes the vulnerability, or if the vulnerability is not + applicable to Buildroot for some reason. + +* If the _CPE_ identifier is valid and known in the + https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is + valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or + if +FOO_CPE_ID_VALID = YES+ validating that the default value of the + _CPE_ identifier is correct. In addition, _CPE_ are checked for + existence in the NIST CPE dictionary. If the _CPE_ identifier is not + known to the NIST CPE dictionary, then it is possible to contribute + to the NIST CPE dictionary: one need to submit a XML document that + describes the additional CPE identifier. Buildroot can help + generating such XML documents for missing CPE identifiers using the + +make missing-cpe+ target. The XML documents are generated in + +$(O)/cpe-updates+, will need to be edited before submission to + NIST. In particular, one should verify the version information and + check that the reference URLs are reachable with a valid type + (suggestions for types are provided in the generated file). If a + previous _CPE_ entry was found in the NIST CPE dictionary, the XML + document generation tries to minimize the entry differences between + the existing versions in the dictionary. The XML document + generation uses the XML from the last version of that CPE as a + template for the new submission. + include::eclipse-integration.txt[] include::advanced.txt[]
Signature · 1 line
2.29.2
PATCH v2 7/7 docs/manual: add details about vulnerability management
Quoted 2 lines · Thomas Petazzoni
This addition to the documentation gives some details about CPE/CVE, pkg-stats and missing-cpe.
That was sorely needed!
Quoted 19 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+)docs/manual/common-usage.txt +5
-0@@ -417,6 +417,55 @@ effects: be empty and it's only at the very end of the build that they will be populated. +[[vulnerability-management]] +=== Vulnerability management + +Buildroot integrates a number of features that help tracking whether +known security vulnerabilities are affecting some of the
docs/manual/common-usage.txt effects: · line 434
Quoted 10 lines
docs/manual/common-usage.txt +10
-0+packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities +and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided +by NIST. + +First, each package in Buildroot can be associated to a _CPE_ +identifier, by means of the various +FOO_CPE_ID_<something>+ variables +that can be provided by each package. See +xref:generic-package-reference[] for more details about those +variables. When the _CPE_ identifier of a package is not defined, a +default value is used, which may or may not be correct.
I don't think this reflects the reality very well. cpeid is only used if
*something* is set in the .mk file. Something like:
When no +FOO_CPE_ID_<something>+ variable is set for a package, it is not considered as having a valid _CPE_ identifier.
docs/manual/common-usage.txt effects: · line 436
Quoted 3 lines
This _CPE_
docs/manual/common-usage.txt +2
-0+identifier is used to find in the NVD database of vulnerabilities +which _CVEs_ are affecting any given package.
... If a package doesn't have a valid CPE identifier, a guess is made based on
the package name and version.
Quoted 9 lines
docs/manual/common-usage.txt +9
-0+ +Based on this and the NIST databases, the +make pkg-stats+ target +produces JSON and HTML formatted reports that indicate for each +package of the current Buildroot configuration: + +* If there are known _CVEs_ for each package. Note that CVEs can + manually be ignored on a per-package basis using the + +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch + that fixes the vulnerability, or if the vulnerability is not
docs/manual/common-usage.txt effects: · line 455
Quoted 10 lines
docs/manual/common-usage.txt +10
-0+ applicable to Buildroot for some reason. + +* If the _CPE_ identifier is valid and known in the + https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is + valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or + if +FOO_CPE_ID_VALID = YES+ validating that the default value of the + _CPE_ identifier is correct. In addition, _CPE_ are checked for + existence in the NIST CPE dictionary. If the _CPE_ identifier is not + known to the NIST CPE dictionary, then it is possible to contribute + to the NIST CPE dictionary: one need to submit a XML document that
needs
Quoted 1 line
+ describes the additional CPE identifier.
Can you add a reference to a web page that describes the procedure? Otherwise
it's not terribly helpful IMHO.
docs/manual/common-usage.txt effects: · line 463
Quoted 8 lines
Buildroot can help
docs/manual/common-usage.txt +7
-0+ generating such XML documents for missing CPE identifiers using the + +make missing-cpe+ target. The XML documents are generated in + +$(O)/cpe-updates+, will need to be edited before submission to + NIST. In particular, one should verify the version information and + check that the reference URLs are reachable with a valid type + (suggestions for types are provided in the generated file). If a + previous _CPE_ entry was found in the NIST CPE dictionary, the XML
If no previous CPE entry exists (i.e., the CPE_ID specified in the .mk file is
not valid), then no XML file is generated at all... AFAICS there is no help at the moment for generating the XML for completely new entries.
My brain is too tired ATM to try to formulate the documentation that expresses
that.
I've marked this one changes requested in patchwork.
Regards,
Arnout
Quoted 9 lines
docs/manual/common-usage.txt +5
-0+ document generation tries to minimize the entry differences between + the existing versions in the dictionary. The XML document + generation uses the XML from the last version of that CPE as a + template for the new submission. + include::eclipse-integration.txt[] include::advanced.txt[]