Skip to contentSkip to conversation

CPE validation

PATCH New Thomas Petazzoni 4 participants 20 messages Jan 31, 2021 – May 16, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

v2 cover CPE validation

Hello,

We have recently merged support to allow packages to express their CPE identifier, which is a unique identifier used by the NIST database to identify software products, and associate CVEs to them.

This patch series extends that by ensuring that CPEs affected to packages in Buildroot do exist in the official CPE dictionary from NIST, and if they do not exist, assist in generating the XML snippet that the CPE dictionary maintainers expect.

In more details:

- PATCH 1 introduces a new "name" field in the show-info JSON output, with the raw name of the package. This is used in PATCH 2.

- PATCH 2 fixes "make pkg-stats" to properly account for host packages, which were currently ignored. It makes use of the new "name" field introduced in PATCH 1 in the show-info JSON output.

- PATCH 3 introduces support/scripts/cpedb.py, which is a new set of helper that is capable of downloading the CPE dictionary, parsing it, and generating XML snippets to submit new CPE entries.

- PATCH 4 modifies support/scripts/pkg-stats to leverage the features of cpedb.py to include validation of CPE identifiers of Buildroot patches.

- PATCH 5 adds support/scripts/gen-missing-cpe, which allows to generate the XML snippets expected by the CPE dictionary mainainers

- PATCH 6 hooks up support/scripts/gen-missing-cpe as the "make missing-cpe" top-level make target

- PATCH 7 extends the documentation with more details about Buildroot features related to vulnerability management.

Changes since v1:

- Drop patches that have been merged upstream.

- New patches adding the "name" property in the show-info JSON output, and using it so that "make pkg-stats" takes into account host packages.

- Fix the parsing of the XML CPE databse for Python versions prior to 3.7. This was reported by Matt, and fixed by Gr?gory Clement.

- Improve how the namespaces are added to the XML file generated by "make missing-cpe", with the proper API instead of using a regexp. This was done by Gr?gory Clement.

- Add encoding in the XML declaration found in the XML file generated by "make missing-cpe". This was done by Gr?gory Clement.

- Reduced the size of the CPE database cache, with the side effect of also having prettier output for "make missing-cpe". This was done by Gr?gory Clement.

This is based on work from Matt Weber, which has been further refactored/rewritten by Gr?gory Clement and myself.

To test this:

 $ make pkg-stats
 $ firefox output/pkg-stats.html
 $ make missing-cpe
 $ ls output/cpe-updates/

This work is also available at:

https://github.com/tpetazzoni/buildroot/commits/cpe-validation

Thomas

Matt Weber (2):
support/scripts/cpedb.py: new CPE XML helper
support/scripts/gen-missing-cpe: add new script

Thomas Petazzoni (5):
package/pkg-utils.mk: introduce "name" field in show-info output support/scripts/pkg-stats: properly handle host packages with -c
option
support/scripts/pkg-stats: check CPE existence in CPE dictionnary
Makefile: add new missing-cpe target
docs/manual: add details about vulnerability management

 Makefile                        |   9 ++
 docs/manual/common-usage.txt    |  42 +++++++
 package/pkg-utils.mk            |   1 +
 support/scripts/cpedb.py        | 203 ++++++++++++++++++++++++++++++++
 support/scripts/gen-missing-cpe |  65 ++++++++++
 support/scripts/pkg-stats       |  32 +++--
 6 files changed, 342 insertions(+), 10 deletions(-)
 create mode 100644 support/scripts/cpedb.py
 create mode 100755 support/scripts/gen-missing-cpe
Signature · 1 line
2.29.2
#1 lore raw

PATCH v2 1/7 package/pkg-utils.mk: introduce "name" field in show-info output

The keys of the JSON dict returned by "make show-info" is the package name, including the "host-" prefix for host packages.

However, it is sometimes useful to get the actual name of the package, without the "host-" prefix, so we add a "name" property that holds the "raw name" of the package.

---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-utils.mk +1 -0
index 11090cb628..d94041de24 100644
@@ -96,6 +96,7 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
+		"name": "$($(1)_RAWNAME)",
 		"type": "$($(1)_TYPE)",
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \
Signature · 1 line
2.29.2
#2 lore raw

PATCH v2 2/7 support/scripts/pkg-stats: properly handle host packages with -c option

In reply to Thomas Petazzoni

In commit 7a607dab336e7f78ab069cff1b503d0688950583 ("support/scripts/pkg-stats: support generating stats based on configured packages"), we added a -c option to pkg-stats to generate a report based on the list of packages enabled in the configuration, rather than for all packages.

This is done based on the list of packages returned in JSON format by "make show-info". However, we use the keys of the JSON dict returned by "make show-info", which include the host- prefix of host packages. Due to this, none of the host packages are currently matching and therefore they are not reported in the pkg-stats -c output.

This commit fixes that by using the recently introduced "name" property in the "make show-info" JSON dict.

---
 support/scripts/pkg-stats | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/pkg-stats +1 -1
index d222ba37c3..0d4e781e3b 100755
@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
     js = json.loads(subprocess.check_output(cmd))
-    return js.keys()
+    return set([v["name"] for k, v in js.items()])
 
 
 def package_init_make_info():
Signature · 1 line
2.29.2
#3 lore raw

PATCH v2 2/7 support/scripts/pkg-stats: properly handle host packages with -c option

Quoted 13 lines · Thomas Petazzoni

In commit 7a607dab336e7f78ab069cff1b503d0688950583 ("support/scripts/pkg-stats: support generating stats based on configured packages"), we added a -c option to pkg-stats to generate a report based on the list of packages enabled in the configuration, rather than for all packages.

This is done based on the list of packages returned in JSON format by "make show-info". However, we use the keys of the JSON dict returned by "make show-info", which include the host- prefix of host packages. Due to this, none of the host packages are currently matching and therefore they are not reported in the pkg-stats -c output.

Quoted 4 lines

This commit fixes that by using the recently introduced "name" property in the "make show-info" JSON dict.

This and the previous patch applied to master, thanks.

The rest will have to wait for Gregory's SoB, but I'll review it.

Regards,
Arnout

Quoted 18 lines
---
 support/scripts/pkg-stats | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/pkg-stats +1 -1
index d222ba37c3..0d4e781e3b 100755
@@ -358,7 +358,7 @@ def get_pkglist(npackages, package_list):
 def get_config_packages():
     cmd = ["make", "--no-print-directory", "show-info"]
     js = json.loads(subprocess.check_output(cmd))
-    return js.keys()
+    return set([v["name"] for k, v in js.items()])
 
 
 def package_init_make_info():
#4 lore raw

PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper

In reply to Thomas Petazzoni

From: Matt Weber <[email protected]>

Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.

- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary

---
 support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100644 support/scripts/cpedb.py
support/scripts/cpedb.py new file +203 -0
new file mode 100644
index 0000000000..825ed6cb1e
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+
+import xml.etree.ElementTree as ET
+from xml.etree.ElementTree import Element, SubElement
+import gzip
+import os
+import pickle
+import requests
+import time
+from xml.dom import minidom
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+ns = {
+    '': 'http://cpe.mitre.org/dictionary/2.0',
+    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
+    'xml': 'http://www.w3.org/XML/1998/namespace'
+}
+
+
+class CPE:
+    def __init__(self, cpe_str, titles, refs):
+        self.cpe_str = cpe_str
+        self.titles = titles
+        self.references = refs
+        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+    def update_xml_dict(self):
+        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
+        cpes = Element('cpe-list')
+        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
+        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
+        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
+        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
+        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
+        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
+        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
+                                                 "http://cpe.mitre.org/dictionary/2.0",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
+                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.3",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
+                                                 "http://scap.nist.gov/schema/configuration/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
+        item = SubElement(cpes, 'cpe-item')
+        cpe_short_name = CPE.short_name(self.cpe_str)
+        cpe_new_ver = CPE.version_update(self.cpe_str)
+
+        item.set('name', 'cpe:/' + cpe_short_name)
+        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
+        for title in self.titles:
+            item.append(title)
+        if self.references:
+            item.append(self.references)
+        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
+        cpe23item.set('name', self.cpe_str)
+
+        # Generate the XML as a string
+        xmlstr = ET.tostring(cpes)
+
+        # And use minidom to pretty print the XML
+        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
+
+    @staticmethod
+    def version(cpe):
+        return cpe.split(":")[5]
+
+    @staticmethod
+    def product(cpe):
+        return cpe.split(":")[4]
+
+    @staticmethod
+    def short_name(cpe):
+        return ":".join(cpe.split(":")[2:6])
+
+    @staticmethod
+    def version_update(cpe):
+        return ":".join(cpe.split(":")[5:6])
+
+    @staticmethod
+    def no_version(cpe):
+        return ":".join(cpe.split(":")[:5])
+
+
+class CPEDB:
+    def __init__(self, nvd_path):
+        self.all_cpes = dict()
+        self.all_cpes_no_version = dict()
+        self.nvd_path = nvd_path
+
+    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
+        print("CPE: Unzipping xml manifest...")
+        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
+        print("CPE: Converting xml manifest to dict...")
+        tree = ET.parse(nist_cpe_file)
+        all_cpedb = tree.getroot()
+        self.parse_dict(all_cpedb)
+
+        print("CPE: Caching dictionary")
+        cpes_file = open(cache_all_cpes, 'wb')
+        pickle.dump(self.all_cpes, cpes_file)
+        cpes_file.close()
+        cpes_file = open(cache_all_cpes_no_version, 'wb')
+        pickle.dump(self.all_cpes_no_version, cpes_file)
+        cpes_file.close()
+
+    def get_xml_dict(self):
+        print("CPE: Setting up NIST dictionary")
+        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
+            os.makedirs(os.path.join(self.nvd_path, "cpe"))
+
+        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
+        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
+            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
+            cpe_dict = requests.get(CPEDB_URL)
+            open(cpe_dict_local, "wb").write(cpe_dict.content)
+
+        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
+        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
+
+        if not os.path.exists(cache_all_cpes) or \
+           not os.path.exists(cache_all_cpes_no_version) or \
+           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
+           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
+            self.gen_cached_cpedb(cpe_dict_local,
+                                  cache_all_cpes,
+                                  cache_all_cpes_no_version)
+
+        print("CPE: Loading CACHED dictionary")
+        cpe_file = open(cache_all_cpes, 'rb')
+        self.all_cpes = pickle.load(cpe_file)
+        cpe_file.close()
+        cpe_file = open(cache_all_cpes_no_version, 'rb')
+        self.all_cpes_no_version = pickle.load(cpe_file)
+        cpe_file.close()
+
+    def parse_dict(self, all_cpedb):
+        # Cycle through the dict and build two dict to be used for custom
+        # lookups of partial and complete CPE objects
+        # The objects are then used to create new proposed XML updates if
+        # if is determined one is required
+        # Out of the different language titles, select English
+        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
+            cpe_titles = []
+            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
+                title.tail = None
+                cpe_titles.append(title)
+
+            # Some older CPE don't include references, if they do, make
+            # sure we handle the case of one ref needing to be packed
+            # in a list
+            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
+            if cpe_ref:
+                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
+                    ref.tail = None
+                    ref.text = ref.text.upper()
+                    if ref.text not in VALID_REFS:
+                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
+                cpe_ref.tail = None
+                cpe_ref.text = None
+
+            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
+            item = CPE(cpe_str, cpe_titles, cpe_ref)
+            cpe_str_no_version = CPE.no_version(cpe_str)
+            # This dict must have a unique key for every CPE version
+            # which allows matching to the specific obj data of that
+            # NIST dict entry
+            self.all_cpes.update({cpe_str: item})
+            # This dict has one entry for every CPE (w/o version) to allow
+            # partial match (no valid version) check (the obj is saved and
+            # used as seed for suggested xml updates. By updating the same
+            # non-version'd entry, it assumes the last update here is the
+            # latest version in the NIST dict)
+            self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+    def find_partial(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return cpe_str_no_version
+
+    def find_partial_obj(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return self.all_cpes_no_version[cpe_str_no_version]
+
+    def find_partial_latest_version(self, cpe_str_partial):
+        cpe_obj = self.find_partial_obj(cpe_str_partial)
+        return cpe_obj.cpe_cur_ver
+
+    def find(self, cpe_str):
+        if self.find_partial(cpe_str):
+            if cpe_str in self.all_cpes:
+                return cpe_str
+
+    def gen_update_xml(self, cpe_str):
+        cpe = self.find_partial_obj(cpe_str)
+        return cpe.update_xml_dict()
Signature · 1 line
2.29.2
#5 lore raw

PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper

Gr?gory, All,

On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:

Quoted 10 lines

From: Matt Weber <[email protected]>

Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.

- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary

Gr?gory, you need to add your SOB line right after your CDB line.

Ditto for the following patch.

Regards,
Yann E. MORIN.

Quoted 219 lines
---
 support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100644 support/scripts/cpedb.py
support/scripts/cpedb.py new file +203 -0
new file mode 100644
index 0000000000..825ed6cb1e
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+
+import xml.etree.ElementTree as ET
+from xml.etree.ElementTree import Element, SubElement
+import gzip
+import os
+import pickle
+import requests
+import time
+from xml.dom import minidom
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+ns = {
+    '': 'http://cpe.mitre.org/dictionary/2.0',
+    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
+    'xml': 'http://www.w3.org/XML/1998/namespace'
+}
+
+
+class CPE:
+    def __init__(self, cpe_str, titles, refs):
+        self.cpe_str = cpe_str
+        self.titles = titles
+        self.references = refs
+        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+    def update_xml_dict(self):
+        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
+        cpes = Element('cpe-list')
+        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")
+        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
+        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
+        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
+        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
+        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
+        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",
+                                                 "http://cpe.mitre.org/dictionary/2.0",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
+                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.3",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
+                                                 "http://scap.nist.gov/schema/configuration/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
+        item = SubElement(cpes, 'cpe-item')
+        cpe_short_name = CPE.short_name(self.cpe_str)
+        cpe_new_ver = CPE.version_update(self.cpe_str)
+
+        item.set('name', 'cpe:/' + cpe_short_name)
+        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
+        for title in self.titles:
+            item.append(title)
+        if self.references:
+            item.append(self.references)
+        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
+        cpe23item.set('name', self.cpe_str)
+
+        # Generate the XML as a string
+        xmlstr = ET.tostring(cpes)
+
+        # And use minidom to pretty print the XML
+        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
+
+    @staticmethod
+    def version(cpe):
+        return cpe.split(":")[5]
+
+    @staticmethod
+    def product(cpe):
+        return cpe.split(":")[4]
+
+    @staticmethod
+    def short_name(cpe):
+        return ":".join(cpe.split(":")[2:6])
+
+    @staticmethod
+    def version_update(cpe):
+        return ":".join(cpe.split(":")[5:6])
+
+    @staticmethod
+    def no_version(cpe):
+        return ":".join(cpe.split(":")[:5])
+
+
+class CPEDB:
+    def __init__(self, nvd_path):
+        self.all_cpes = dict()
+        self.all_cpes_no_version = dict()
+        self.nvd_path = nvd_path
+
+    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
+        print("CPE: Unzipping xml manifest...")
+        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
+        print("CPE: Converting xml manifest to dict...")
+        tree = ET.parse(nist_cpe_file)
+        all_cpedb = tree.getroot()
+        self.parse_dict(all_cpedb)
+
+        print("CPE: Caching dictionary")
+        cpes_file = open(cache_all_cpes, 'wb')
+        pickle.dump(self.all_cpes, cpes_file)
+        cpes_file.close()
+        cpes_file = open(cache_all_cpes_no_version, 'wb')
+        pickle.dump(self.all_cpes_no_version, cpes_file)
+        cpes_file.close()
+
+    def get_xml_dict(self):
+        print("CPE: Setting up NIST dictionary")
+        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
+            os.makedirs(os.path.join(self.nvd_path, "cpe"))
+
+        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
+        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:
+            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
+            cpe_dict = requests.get(CPEDB_URL)
+            open(cpe_dict_local, "wb").write(cpe_dict.content)
+
+        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
+        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
+
+        if not os.path.exists(cache_all_cpes) or \
+           not os.path.exists(cache_all_cpes_no_version) or \
+           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
+           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
+            self.gen_cached_cpedb(cpe_dict_local,
+                                  cache_all_cpes,
+                                  cache_all_cpes_no_version)
+
+        print("CPE: Loading CACHED dictionary")
+        cpe_file = open(cache_all_cpes, 'rb')
+        self.all_cpes = pickle.load(cpe_file)
+        cpe_file.close()
+        cpe_file = open(cache_all_cpes_no_version, 'rb')
+        self.all_cpes_no_version = pickle.load(cpe_file)
+        cpe_file.close()
+
+    def parse_dict(self, all_cpedb):
+        # Cycle through the dict and build two dict to be used for custom
+        # lookups of partial and complete CPE objects
+        # The objects are then used to create new proposed XML updates if
+        # if is determined one is required
+        # Out of the different language titles, select English
+        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
+            cpe_titles = []
+            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
+                title.tail = None
+                cpe_titles.append(title)
+
+            # Some older CPE don't include references, if they do, make
+            # sure we handle the case of one ref needing to be packed
+            # in a list
+            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
+            if cpe_ref:
+                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
+                    ref.tail = None
+                    ref.text = ref.text.upper()
+                    if ref.text not in VALID_REFS:
+                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
+                cpe_ref.tail = None
+                cpe_ref.text = None
+
+            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
+            item = CPE(cpe_str, cpe_titles, cpe_ref)
+            cpe_str_no_version = CPE.no_version(cpe_str)
+            # This dict must have a unique key for every CPE version
+            # which allows matching to the specific obj data of that
+            # NIST dict entry
+            self.all_cpes.update({cpe_str: item})
+            # This dict has one entry for every CPE (w/o version) to allow
+            # partial match (no valid version) check (the obj is saved and
+            # used as seed for suggested xml updates. By updating the same
+            # non-version'd entry, it assumes the last update here is the
+            # latest version in the NIST dict)
+            self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+    def find_partial(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return cpe_str_no_version
+
+    def find_partial_obj(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return self.all_cpes_no_version[cpe_str_no_version]
+
+    def find_partial_latest_version(self, cpe_str_partial):
+        cpe_obj = self.find_partial_obj(cpe_str_partial)
+        return cpe_obj.cpe_cur_ver
+
+    def find(self, cpe_str):
+        if self.find_partial(cpe_str):
+            if cpe_str in self.all_cpes:
+                return cpe_str
+
+    def gen_update_xml(self, cpe_str):
+        cpe = self.find_partial_obj(cpe_str)
+        return cpe.update_xml_dict()
Signature · 1 line
2.29.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper

Gr?gory, All,

On 2021-01-31 23:47 +0100, Yann E. MORIN spake thusly:

Quoted 9 lines

On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:

Quoted 8 lines

From: Matt Weber <[email protected]>

Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.

- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary

Quoted 6 lines
Quoted 2 lines

Gr?gory, you need to add your SOB line right after your CDB line.

Ditto for the following patch.

No need to resend the series, just reply to each patch with your SoB line, I'll add them when applying.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH v2 3/7 support/scripts/cpedb.py: new CPE XML helper

In reply to Thomas Petazzoni
Quoted 46 lines · Thomas Petazzoni

From: Matt Weber <[email protected]>

Python class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.

- Defines the CPE as a object with operations / formats
- Processing of CPE dictionary

---
 support/scripts/cpedb.py | 203 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100644 support/scripts/cpedb.py
support/scripts/cpedb.py new file +23 -0
new file mode 100644
index 0000000000..825ed6cb1e
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+
+import xml.etree.ElementTree as ET
+from xml.etree.ElementTree import Element, SubElement
+import gzip
+import os
+import pickle
+import requests
+import time
+from xml.dom import minidom
+
+VALID_REFS = ['VENDOR', 'VERSION', 'CHANGE_LOG', 'PRODUCT', 'PROJECT', 'ADVISORY']
+
+CPEDB_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz"
+
+ns = {
+    '': 'http://cpe.mitre.org/dictionary/2.0',
+    'cpe-23': 'http://scap.nist.gov/schema/cpe-extension/2.3',
+    'xml': 'http://www.w3.org/XML/1998/namespace'
+}
+
+
+class CPE:

support/scripts/cpedb.py line 33

Quoted 10 lines
support/scripts/cpedb.py new file +10 -0
+    def __init__(self, cpe_str, titles, refs):
+        self.cpe_str = cpe_str
+        self.titles = titles
+        self.references = refs
+        self.cpe_cur_ver = "".join(self.cpe_str.split(":")[5:6])
+
+    def update_xml_dict(self):
+        ET.register_namespace('', 'http://cpe.mitre.org/dictionary/2.0')
+        cpes = Element('cpe-list')
+        cpes.set('xmlns:cpe-23', "http://scap.nist.gov/schema/cpe-extension/2.3")

Note that instead of all these cpe.set calls, you can also do:

        cpes.attrib = {
            'xmlns:ns6': "http://scap.nist.gov/schema/scap-core/0.1",
            ...
        }

which IMO is more readable. But that's just nitpick.

In fact, I think you can do it in the constructor:

        cpes = Element('cpe-list', attrib = { ...

but that is probably less readable.

support/scripts/cpedb.py line 40

Quoted 7 lines
support/scripts/cpedb.py new file +7 -0
+        cpes.set('xmlns:ns6', "http://scap.nist.gov/schema/scap-core/0.1")
+        cpes.set('xmlns:scap-core', "http://scap.nist.gov/schema/scap-core/0.3")
+        cpes.set('xmlns:config', "http://scap.nist.gov/schema/configuration/0.1")
+        cpes.set('xmlns:xsi', "http://www.w3.org/2001/XMLSchema-instance")
+        cpes.set('xmlns:meta', "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2")
+        cpes.set('xsi:schemaLocation', " ".join(["http://scap.nist.gov/schema/cpe-extension/2.3",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary-extension_2.3.xsd",

And here instead of join you can use:

cpes.set('xsi:schemaLocation',
"http://scap.nist.gov/schema/cpe-extension/2.3 "

                                       "http://cpe.mitre.org/dictionary/2.0 "

(Python has string concatenation like C).

Quoted 33 lines
support/scripts/cpedb.py new file +33 -0
+                                                 "http://cpe.mitre.org/dictionary/2.0",
+                                                 "https://scap.nist.gov/schema/cpe/2.3/cpe-dictionary_2.3.xsd",
+                                                 "http://scap.nist.gov/schema/cpe-dictionary-metadata/0.2",
+                                                 "https://scap.nist.gov/schema/cpe/2.1/cpe-dictionary-metadata_0.2.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.3",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.3.xsd",
+                                                 "http://scap.nist.gov/schema/configuration/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/configuration_0.1.xsd",
+                                                 "http://scap.nist.gov/schema/scap-core/0.1",
+                                                 "https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd"]))
+        item = SubElement(cpes, 'cpe-item')
+        cpe_short_name = CPE.short_name(self.cpe_str)
+        cpe_new_ver = CPE.version_update(self.cpe_str)
+
+        item.set('name', 'cpe:/' + cpe_short_name)
+        self.titles[0].text.replace(self.cpe_cur_ver, cpe_new_ver)
+        for title in self.titles:
+            item.append(title)
+        if self.references:
+            item.append(self.references)
+        cpe23item = SubElement(item, 'cpe-23:cpe23-item')
+        cpe23item.set('name', self.cpe_str)
+
+        # Generate the XML as a string
+        xmlstr = ET.tostring(cpes)
+
+        # And use minidom to pretty print the XML
+        return minidom.parseString(xmlstr).toprettyxml(encoding="utf-8").decode("utf-8")
+
+    @staticmethod
+    def version(cpe):
+        return cpe.split(":")[5]
+

support/scripts/cpedb.py line 83

Quoted 10 lines
support/scripts/cpedb.py new file +10 -0
+    @staticmethod
+    def product(cpe):
+        return cpe.split(":")[4]
+
+    @staticmethod
+    def short_name(cpe):
+        return ":".join(cpe.split(":")[2:6])
+
+    @staticmethod
+    def version_update(cpe):

It's weird to make this a staticmethod if the only caller takes self.cpe_str as
an argument, but OK.

Quoted 26 lines
support/scripts/cpedb.py new file +26 -0
+        return ":".join(cpe.split(":")[5:6])
+
+    @staticmethod
+    def no_version(cpe):
+        return ":".join(cpe.split(":")[:5])
+
+
+class CPEDB:
+    def __init__(self, nvd_path):
+        self.all_cpes = dict()
+        self.all_cpes_no_version = dict()
+        self.nvd_path = nvd_path
+
+    def gen_cached_cpedb(self, cpedb, cache_all_cpes, cache_all_cpes_no_version):
+        print("CPE: Unzipping xml manifest...")
+        nist_cpe_file = gzip.GzipFile(fileobj=open(cpedb, 'rb'))
+        print("CPE: Converting xml manifest to dict...")
+        tree = ET.parse(nist_cpe_file)
+        all_cpedb = tree.getroot()
+        self.parse_dict(all_cpedb)
+
+        print("CPE: Caching dictionary")
+        cpes_file = open(cache_all_cpes, 'wb')
+        pickle.dump(self.all_cpes, cpes_file)
+        cpes_file.close()
+        cpes_file = open(cache_all_cpes_no_version, 'wb')

support/scripts/cpedb.py line 119

Quoted 10 lines
support/scripts/cpedb.py new file +10 -0
+        pickle.dump(self.all_cpes_no_version, cpes_file)
+        cpes_file.close()
+
+    def get_xml_dict(self):
+        print("CPE: Setting up NIST dictionary")
+        if not os.path.exists(os.path.join(self.nvd_path, "cpe")):
+            os.makedirs(os.path.join(self.nvd_path, "cpe"))
+
+        cpe_dict_local = os.path.join(self.nvd_path, "cpe", os.path.basename(CPEDB_URL))
+        if not os.path.exists(cpe_dict_local) or os.stat(cpe_dict_local).st_mtime < time.time() - 86400:

This should be a parameter@the beginning of the file rather than a constant
here. Ideally expressed in hours, with *3600 here.

Quoted 9 lines
support/scripts/cpedb.py new file +9 -0
+            print("CPE: Fetching xml manifest from [" + CPEDB_URL + "]")
+            cpe_dict = requests.get(CPEDB_URL)
+            open(cpe_dict_local, "wb").write(cpe_dict.content)
+
+        cache_all_cpes = os.path.join(self.nvd_path, "cpe", "all_cpes.pkl")
+        cache_all_cpes_no_version = os.path.join(self.nvd_path, "cpe", "all_cpes_no_version.pkl")
+
+        if not os.path.exists(cache_all_cpes) or \
+           not os.path.exists(cache_all_cpes_no_version) or \

support/scripts/cpedb.py line 138

Quoted 10 lines
support/scripts/cpedb.py new file +10 -0
+           os.stat(cache_all_cpes).st_mtime < os.stat(cpe_dict_local).st_mtime or \
+           os.stat(cache_all_cpes_no_version).st_mtime < os.stat(cpe_dict_local).st_mtime:
+            self.gen_cached_cpedb(cpe_dict_local,
+                                  cache_all_cpes,
+                                  cache_all_cpes_no_version)
+
+        print("CPE: Loading CACHED dictionary")
+        cpe_file = open(cache_all_cpes, 'rb')
+        self.all_cpes = pickle.load(cpe_file)
+        cpe_file.close()

The above is better done as

with open(cache_all_cpes, 'rb') as cpe_file:
self.all_cpes = pickle.load(cpe_file)

This do things properly in case of exception.

However, I'm not so sure if the pickled cache is really worth it. I gave it a
go, and the ET.parse + parse_dict took roughly 20 seconds, while just loading the pickle file also took about 20 seconds...

support/scripts/cpedb.py line 148

Quoted 10 lines
support/scripts/cpedb.py new file +10 -0
+        cpe_file = open(cache_all_cpes_no_version, 'rb')
+        self.all_cpes_no_version = pickle.load(cpe_file)
+        cpe_file.close()
+
+    def parse_dict(self, all_cpedb):
+        # Cycle through the dict and build two dict to be used for custom
+        # lookups of partial and complete CPE objects
+        # The objects are then used to create new proposed XML updates if
+        # if is determined one is required
+        # Out of the different language titles, select English

This could have been a docstring.

Actually, all "public" functions should really have a docstring.

So, I accidentally applied this patch... So all the above comments are just
informational - feel free to send follow-up patches, or ignore my feedback.

Regards,
Arnout

Quoted 56 lines
support/scripts/cpedb.py new file +55 -0
+        for cpe in all_cpedb.findall(".//{http://cpe.mitre.org/dictionary/2.0}cpe-item"):
+            cpe_titles = []
+            for title in cpe.findall('.//{http://cpe.mitre.org/dictionary/2.0}title[@xml:lang="en-US"]', ns):
+                title.tail = None
+                cpe_titles.append(title)
+
+            # Some older CPE don't include references, if they do, make
+            # sure we handle the case of one ref needing to be packed
+            # in a list
+            cpe_ref = cpe.find(".//{http://cpe.mitre.org/dictionary/2.0}references")
+            if cpe_ref:
+                for ref in cpe_ref.findall(".//{http://cpe.mitre.org/dictionary/2.0}reference"):
+                    ref.tail = None
+                    ref.text = ref.text.upper()
+                    if ref.text not in VALID_REFS:
+                        ref.text = ref.text + "-- UPDATE this entry, here are some examples and just one word should be used -- " + ' '.join(VALID_REFS) # noqa E501
+                cpe_ref.tail = None
+                cpe_ref.text = None
+
+            cpe_str = cpe.find(".//{http://scap.nist.gov/schema/cpe-extension/2.3}cpe23-item").get('name')
+            item = CPE(cpe_str, cpe_titles, cpe_ref)
+            cpe_str_no_version = CPE.no_version(cpe_str)
+            # This dict must have a unique key for every CPE version
+            # which allows matching to the specific obj data of that
+            # NIST dict entry
+            self.all_cpes.update({cpe_str: item})
+            # This dict has one entry for every CPE (w/o version) to allow
+            # partial match (no valid version) check (the obj is saved and
+            # used as seed for suggested xml updates. By updating the same
+            # non-version'd entry, it assumes the last update here is the
+            # latest version in the NIST dict)
+            self.all_cpes_no_version.update({cpe_str_no_version: item})
+
+    def find_partial(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return cpe_str_no_version
+
+    def find_partial_obj(self, cpe_str):
+        cpe_str_no_version = CPE.no_version(cpe_str)
+        if cpe_str_no_version in self.all_cpes_no_version:
+            return self.all_cpes_no_version[cpe_str_no_version]
+
+    def find_partial_latest_version(self, cpe_str_partial):
+        cpe_obj = self.find_partial_obj(cpe_str_partial)
+        return cpe_obj.cpe_cur_ver
+
+    def find(self, cpe_str):
+        if self.find_partial(cpe_str):
+            if cpe_str in self.all_cpes:
+                return cpe_str
+
+    def gen_update_xml(self, cpe_str):
+        cpe = self.find_partial_obj(cpe_str)
+        return cpe.update_xml_dict()
#8 lore raw

PATCH v2 4/7 support/scripts/pkg-stats: check CPE existence in CPE dictionnary

In reply to Thomas Petazzoni

This commit extends pkg-stats to leverage the recently introduced CPEDB class to verify that the CPEs provided by Buildroot packages are indeed known in the official CPE dictionnary provided by NVD.

---
 support/scripts/pkg-stats | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
support/scripts/pkg-stats +21 -9
index 0d4e781e3b..f595026336 100755
@@ -32,7 +32,7 @@ brpath = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
 
 sys.path.append(os.path.join(brpath, "utils"))
 from getdeveloperlib import parse_developers  # noqa: E402
-
+from cpedb import CPEDB  # noqa: E402
 
 INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)")
 URL_RE = re.compile(r"\s*https?://\S*\s*$")
@@ -225,7 +225,8 @@ class Package:
 
         if var in self.all_cpeids:
             self.cpeid = self.all_cpeids[var]
-            self.status['cpe'] = ("ok", "verified CPE identifier")
+            # Set a preliminary status, it might be overridden by check_package_cpes()
+            self.status['cpe'] = ("warning", "not checked against CPE dictionnary")
         else:
             self.status['cpe'] = ("error", "no verified CPE identifier")
 
@@ -600,6 +601,18 @@ def check_package_cves(nvd_path, packages):
                 pkg.status['cve'] = ("ok", "not affected by CVEs")
 
 
+def check_package_cpes(nvd_path, packages):
+    cpedb = CPEDB(nvd_path)
+    cpedb.get_xml_dict()
+    for p in packages:
+        if not p.cpeid:
+            continue
+        if cpedb.find(p.cpeid):
+            p.status['cpe'] = ("ok", "verified CPE identifier")
+        else:
+            p.status['cpe'] = ("error", "CPE identifier unknown in CPE database")
+
+
 def calculate_stats(packages):
     stats = defaultdict(int)
     stats['packages'] = len(packages)
@@ -898,19 +911,17 @@ def dump_html_pkg(f, pkg):
 
     # CPE ID
     td_class = ["left"]
-    if pkg.status['cpe'][0] == "ok":
+    if pkg.is_status_ok("cpe"):
         td_class.append("cpe-ok")
-    elif pkg.status['cpe'][0] == "error":
+    elif pkg.is_status_error("cpe"):
         td_class.append("cpe-nok")
     else:
         td_class.append("cpe-unknown")
     f.write("  <td class=\"%s\">\n" % " ".join(td_class))
-    if pkg.status['cpe'][0] == "ok":
+    if pkg.cpeid:
         f.write("  <code>%s</code>\n" % pkg.cpeid)
-    elif pkg.status['cpe'][0] == "error":
-        f.write("  N/A\n")
-    else:
-        f.write("  %s\n" % pkg.status['cpe'][1])
+    if not pkg.is_status_ok("cpe"):
+        f.write("  %s%s\n" % ("<br/>" if pkg.cpeid else "", pkg.status['cpe'][1]))
     f.write("  </td>\n")
 
     f.write(" </tr>\n")
@@ -1106,6 +1117,7 @@ def __main__():
     if args.nvd_path:
         print("Checking packages CVEs")
         check_package_cves(args.nvd_path, packages)
+        check_package_cpes(args.nvd_path, packages)
     print("Calculate stats")
     stats = calculate_stats(packages)
     if args.html:
Signature · 1 line
2.29.2
#9 lore raw

PATCH v2 4/7 support/scripts/pkg-stats: check CPE existence in CPE dictionnary

Quoted 6 lines · Thomas Petazzoni

This commit extends pkg-stats to leverage the recently introduced CPEDB class to verify that the CPEs provided by Buildroot packages are indeed known in the official CPE dictionnary provided by NVD.

This one was accidentally applied to master without Gre?gory's SoB as well.

I didn't have any further comments on the patch. Imagine that!

Regards,
Arnout

#10 lore raw

PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script

In reply to Thomas Petazzoni

From: Matt Weber <[email protected]>

This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:

- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning

- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.

Ref: NIST has a group email (cpe_dictionary at nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.

Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary

---
 support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100755 support/scripts/gen-missing-cpe
support/scripts/gen-missing-cpe new file +65 -0
new file mode 100755
index 0000000000..ed7747295a
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+import json
+import subprocess
+import os
+from cpedb import CPEDB, CPE
+
+
+def gen_update_xml_reports(cpes, cpedb, output):
+    cpe_need_update = []
+
+    for cpe in cpes:
+        result = cpedb.find(cpe)
+        if not result:
+            result = cpedb.find_partial(CPE.no_version(cpe))
+            if result:
+                cpe_need_update.append(cpe)
+            else:
+                print("WARNING: no match found for '%s'" % cpe)
+
+    for cpe in cpe_need_update:
+        xml = cpedb.gen_update_xml(cpe)
+        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
+        print("Generating %s" % fname)
+        fp = open(os.path.join(output, fname), 'w+')
+        fp.write(xml)
+        fp.close()
+
+    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
+
+
+def get_cpe_ids():
+    print("Getting list of CPE for enabled packages")
+    cmd = ["make", "--no-print-directory", "show-info"]
+    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
+    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
+
+
+def resolvepath(path):
+    return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output', dest='output',
+                        help='Path to the output CPE update files', type=resolvepath, required=True)
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database', type=resolvepath, required=True)
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    if not os.path.isdir(args.output):
+        print("ERROR: output directory %s does not exist" % args.output)
+        sys.exit(1)
+    cpedb = CPEDB(args.nvd_path)
+    cpedb.get_xml_dict()
+    cpes = get_cpe_ids()
+    gen_update_xml_reports(cpes, cpedb, args.output)
+
+
+__main__()
Signature · 1 line
2.29.2
#11 lore raw

PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script

Quoted 7 lines · Thomas Petazzoni

From: Matt Weber <[email protected]>

This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:

- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning

Maybe there should also be a warning for packages which don't have cpeid set at
all...

Quoted 10 lines

- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.

Ref: NIST has a group email (cpe_dictionary at nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.

It would be very useful if the script would also print a URL that describes the
submission process.

Quoted 24 lines

Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary

---
 support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100755 support/scripts/gen-missing-cpe
support/scripts/gen-missing-cpe new file +1 -0
new file mode 100755
index 0000000000..ed7747295a
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3

support/scripts/gen-missing-cpe line 11

Quoted 10 lines
support/scripts/gen-missing-cpe new file +10 -0
+
+import argparse
+import sys
+import json
+import subprocess
+import os
+from cpedb import CPEDB, CPE
+
+
+def gen_update_xml_reports(cpes, cpedb, output):

cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).

support/scripts/gen-missing-cpe line 14

Quoted 3 lines
support/scripts/gen-missing-cpe new file +3 -0
+    cpe_need_update = []
+
+    for cpe in cpes:

for cpeid in cpeids:

Quoted 5 lines
support/scripts/gen-missing-cpe new file +5 -0
+        result = cpedb.find(cpe)
+        if not result:
+            result = cpedb.find_partial(CPE.no_version(cpe))
+            if result:
+                cpe_need_update.append(cpe)

support/scripts/gen-missing-cpe line 29

Quoted 10 lines
support/scripts/gen-missing-cpe new file +10 -0
+            else:
+                print("WARNING: no match found for '%s'" % cpe)
+
+    for cpe in cpe_need_update:
+        xml = cpedb.gen_update_xml(cpe)
+        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
+        print("Generating %s" % fname)
+        fp = open(os.path.join(output, fname), 'w+')
+        fp.write(xml)
+        fp.close()

This should be

        with open(...) as fp:
            fp.write(xml)

support/scripts/gen-missing-cpe line 42

Quoted 13 lines
support/scripts/gen-missing-cpe new file +13 -0
+
+    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
+
+
+def get_cpe_ids():
+    print("Getting list of CPE for enabled packages")
+    cmd = ["make", "--no-print-directory", "show-info"]
+    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
+    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
+
+
+def resolvepath(path):
+    return os.path.abspath(os.path.expanduser(path))

I don't understand this...

- expanduser should already have been done by the shell. If you call the script as

gen-missing-cpe --output \~/some-path

then I want the output to be in a directory called ~ in the current directory. That's what the rest of the world does. (Yes, I get it, you want to be able to do --output=~/some-path but that's just wrong.)

- abspath shouldn't be needed for anything, we're not doing any changedir or anything like that.

In addition, this script is supposed to be called from make missing-cpe which has already done those expansions.

Quoted 13 lines
support/scripts/gen-missing-cpe new file +13 -0
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output', dest='output',
+                        help='Path to the output CPE update files', type=resolvepath, required=True)
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database', type=resolvepath, required=True)
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()

support/scripts/gen-missing-cpe line 65

Quoted 10 lines
support/scripts/gen-missing-cpe new file +10 -0
+    if not os.path.isdir(args.output):
+        print("ERROR: output directory %s does not exist" % args.output)
+        sys.exit(1)
+    cpedb = CPEDB(args.nvd_path)
+    cpedb.get_xml_dict()
+    cpes = get_cpe_ids()
+    gen_update_xml_reports(cpes, cpedb, args.output)
+
+
+__main__()

Usually it would be

if __name__ == '__main__':

    __main__()

Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just

Regards,
Arnout

#12 lore raw

PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script

Thomas,

Quoted 6 lines · Arnout Vandecappelle
Quoted 2 lines · Thomas Petazzoni

From: Matt Weber <[email protected]>

Quoted 8 lines
Quoted 5 lines

This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:

- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning

Maybe there should also be a warning for packages which don't have cpeid set at
all...

Agree, this would be something that people would miss if we don't list the missing. My original iteration had that as a category that was printed to screen but no XML generated for it.

Regards,
Matt

#13 lore raw

PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script

In reply to Arnout Vandecappelle

Arnout, All,

On 2021-02-02 22:29 +0100, Arnout Vandecappelle spake thusly:

Quoted 10 lines · Thomas Petazzoni
Quoted 7 lines

From: Matt Weber <[email protected]>

This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:

- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning

Maybe there should also be a warning for packages which don't have cpeid set at
all...

I haven't done anything to do so; I though it was better to have at least this script in its current state, rather than nothing at all.

[--SNIP--]

Quoted 2 lines
Quoted 1 line

+def gen_update_xml_reports(cpes, cpedb, output):

cpes should be cpeids (I first thought it was a list of CPE objects from cpedb).

Done.

[--SNIP--]

Quoted 6 lines

support/scripts/gen-missing-cpe line 29

Quoted 3 lines
support/scripts/gen-missing-cpe new file +3 -0
+        fp = open(os.path.join(output, fname), 'w+')
+        fp.write(xml)
+        fp.close()

This should be

        with open(...) as fp:
            fp.write(xml)

Done.

[--SNIP--]

Quoted 13 lines

support/scripts/gen-missing-cpe line 42

Quoted 2 lines
support/scripts/gen-missing-cpe new file +2 -0
+def resolvepath(path):
+    return os.path.abspath(os.path.expanduser(path))

I don't understand this...

- expanduser should already have been done by the shell. If you call the script as

gen-missing-cpe --output \~/some-path

then I want the output to be in a directory called ~ in the current directory. That's what the rest of the world does. (Yes, I get it, you want to be able to do --output=~/some-path but that's just wrong.)

Quoted 5 lines

- abspath shouldn't be needed for anything, we're not doing any changedir or anything like that.

In addition, this script is supposed to be called from make missing-cpe which has already done those expansions.

It was not obvious to me either that resolvepath() was needed, and as] you said, the usage we have in Buildroot does not need expansion. However, getting rid of it seemed too much, so I left it as-is.

Again, I thought it was better that we have this script rather than nothing.

[--SNIP--]

Quoted 4 lines
Quoted 1 line

+__main__()

Usually it would be
if __name__ == '__main__':

    __main__()

Done.

Quoted 3 lines

Nothing important here, but I want to give you the chance to rework if you
think it's worth it, so just

This was sitting in the backlog for quite a while, and we really needed to push a bit on our CPE (and CVE) tooling, so I applied this to master now.

Thanks!

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#14 lore raw

PATCH v2 5/7 support/scripts/gen-missing-cpe: add new script

In reply to Thomas Petazzoni

Thomas, All,

On 2021-01-31 14:38 +0100, Thomas Petazzoni spake thusly:

Quoted 27 lines

From: Matt Weber <[email protected]>

This script queries the list of CPE IDs for the packages of the current configuration (based on the "make show-info" output), and:

- for CPE IDs that do not have any matching entry in the CPE database, it emits a warning

- for CPE IDs that do have a matching entry, but not with the same version, it generates a snippet of XML that can be used to propose an updated version to NIST.

Ref: NIST has a group email (cpe_dictionary at nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different.

Limitations
- Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE
- Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary

Quoted 2 lines

Applied to master, with most of the changes pointed out by Arnout, thanks. I'll further reply to Arnout's comment...

Regards,
Yann E. MORIN.

Quoted 83 lines
---
 support/scripts/gen-missing-cpe | 65 +++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100755 support/scripts/gen-missing-cpe
support/scripts/gen-missing-cpe new file +65 -0
new file mode 100755
index 0000000000..ed7747295a
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+
+import argparse
+import sys
+import json
+import subprocess
+import os
+from cpedb import CPEDB, CPE
+
+
+def gen_update_xml_reports(cpes, cpedb, output):
+    cpe_need_update = []
+
+    for cpe in cpes:
+        result = cpedb.find(cpe)
+        if not result:
+            result = cpedb.find_partial(CPE.no_version(cpe))
+            if result:
+                cpe_need_update.append(cpe)
+            else:
+                print("WARNING: no match found for '%s'" % cpe)
+
+    for cpe in cpe_need_update:
+        xml = cpedb.gen_update_xml(cpe)
+        fname = CPE.product(cpe) + '-' + CPE.version(cpe) + '.xml'
+        print("Generating %s" % fname)
+        fp = open(os.path.join(output, fname), 'w+')
+        fp.write(xml)
+        fp.close()
+
+    print("Generated %d update files out of %d CPEs" % (len(cpe_need_update), len(cpes)))
+
+
+def get_cpe_ids():
+    print("Getting list of CPE for enabled packages")
+    cmd = ["make", "--no-print-directory", "show-info"]
+    js = json.loads(subprocess.check_output(cmd).decode("utf-8"))
+    return set([v["cpe-id"] for k, v in js.items() if "cpe-id" in v])
+
+
+def resolvepath(path):
+    return os.path.abspath(os.path.expanduser(path))
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output', dest='output',
+                        help='Path to the output CPE update files', type=resolvepath, required=True)
+    parser.add_argument('--nvd-path', dest='nvd_path',
+                        help='Path to the local NVD database', type=resolvepath, required=True)
+    return parser.parse_args()
+
+
+def __main__():
+    args = parse_args()
+    if not os.path.isdir(args.output):
+        print("ERROR: output directory %s does not exist" % args.output)
+        sys.exit(1)
+    cpedb = CPEDB(args.nvd_path)
+    cpedb.get_xml_dict()
+    cpes = get_cpe_ids()
+    gen_update_xml_reports(cpes, cpedb, args.output)
+
+
+__main__()
Signature · 6 lines
2.29.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#15 lore raw

PATCH v2 6/7 Makefile: add new missing-cpe target

In reply to Thomas Petazzoni

It invokes the recently introduced gen-missing-cpe script.

---
 Makefile | 9 +++++++++
 1 file changed, 9 insertions(+)
Makefile +9 -0
index 4d334adcd6..962931e0a5 100644
@@ -945,6 +945,14 @@ pkg-stats:
 		--html $(O)/pkg-stats.html \
 		--nvd-path $(DL_DIR)/buildroot-nvd
 
+.PHONY: missing-cpe
+missing-cpe:
+	$(Q)mkdir -p $(O)/cpe-updates
+	$(Q)cd "$(CONFIG_DIR)" ; \
+	$(TOPDIR)/support/scripts/gen-missing-cpe \
+		--nvd-path $(DL_DIR)/buildroot-nvd \
+		--output $(O)/cpe-updates
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
@@ -1163,6 +1171,7 @@ help:
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  show-info              - generate info about packages, as a JSON blurb'
 	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
+	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
Signature · 1 line
2.29.2
#16 lore raw

PATCH v2 6/7 Makefile: add new missing-cpe target

Quoted 3 lines · Thomas Petazzoni

It invokes the recently introduced gen-missing-cpe script.

Regards,
Arnout

Quoted 32 lines
---
 Makefile | 9 +++++++++
 1 file changed, 9 insertions(+)
Makefile +9 -0
index 4d334adcd6..962931e0a5 100644
@@ -945,6 +945,14 @@ pkg-stats:
 		--html $(O)/pkg-stats.html \
 		--nvd-path $(DL_DIR)/buildroot-nvd
 
+.PHONY: missing-cpe
+missing-cpe:
+	$(Q)mkdir -p $(O)/cpe-updates
+	$(Q)cd "$(CONFIG_DIR)" ; \
+	$(TOPDIR)/support/scripts/gen-missing-cpe \
+		--nvd-path $(DL_DIR)/buildroot-nvd \
+		--output $(O)/cpe-updates
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
@@ -1163,6 +1171,7 @@ help:
 	@echo '  legal-info             - generate info about license compliance'
 	@echo '  show-info              - generate info about packages, as a JSON blurb'
 	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
+	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
#18 lore raw

PATCH v2 7/7 docs/manual: add details about vulnerability management

In reply to Thomas Petazzoni

This addition to the documentation gives some details about CPE/CVE, pkg-stats and missing-cpe.

---
 docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
docs/manual/common-usage.txt +49 -0
index 9ba87a8339..e5d9d79420 100644
@@ -417,6 +417,55 @@ effects:
   be empty and it's only at the very end of the build that they will
   be populated.
 
+[[vulnerability-management]]
+=== Vulnerability management
+
+Buildroot integrates a number of features that help tracking whether
+known security vulnerabilities are affecting some of the
+packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
+and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
+by NIST.
+
+First, each package in Buildroot can be associated to a _CPE_
+identifier, by means of the various +FOO_CPE_ID_<something>+ variables
+that can be provided by each package. See
+xref:generic-package-reference[] for more details about those
+variables. When the _CPE_ identifier of a package is not defined, a
+default value is used, which may or may not be correct. This _CPE_
+identifier is used to find in the NVD database of vulnerabilities
+which _CVEs_ are affecting any given package.
+
+Based on this and the NIST databases, the +make pkg-stats+ target
+produces JSON and HTML formatted reports that indicate for each
+package of the current Buildroot configuration:
+
+* If there are known _CVEs_ for each package. Note that CVEs can
+  manually be ignored on a per-package basis using the
+  +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
+  that fixes the vulnerability, or if the vulnerability is not
+  applicable to Buildroot for some reason.
+
+* If the _CPE_ identifier is valid and known in the
+  https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
+  valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
+  if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
+  _CPE_ identifier is correct. In addition, _CPE_ are checked for
+  existence in the NIST CPE dictionary. If the _CPE_ identifier is not
+  known to the NIST CPE dictionary, then it is possible to contribute
+  to the NIST CPE dictionary: one need to submit a XML document that
+  describes the additional CPE identifier. Buildroot can help
+  generating such XML documents for missing CPE identifiers using the
+  +make missing-cpe+ target. The XML documents are generated in
+  +$(O)/cpe-updates+, will need to be edited before submission to
+  NIST. In particular, one should verify the version information and
+  check that the reference URLs are reachable with a valid type
+  (suggestions for types are provided in the generated file).  If a
+  previous _CPE_ entry was found in the NIST CPE dictionary, the XML
+  document generation tries to minimize the entry differences between
+  the existing versions in the dictionary. The XML document
+  generation uses the XML from the last version of that CPE as a
+  template for the new submission.
+
 include::eclipse-integration.txt[]
 
 include::advanced.txt[]
Signature · 1 line
2.29.2
#19 lore raw

PATCH v2 7/7 docs/manual: add details about vulnerability management

Quoted 2 lines · Thomas Petazzoni

This addition to the documentation gives some details about CPE/CVE, pkg-stats and missing-cpe.

That was sorely needed!

Quoted 19 lines
---
 docs/manual/common-usage.txt | 49 ++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)
docs/manual/common-usage.txt +5 -0
index 9ba87a8339..e5d9d79420 100644
@@ -417,6 +417,55 @@ effects:
   be empty and it's only at the very end of the build that they will
   be populated.
 
+[[vulnerability-management]]
+=== Vulnerability management
+
+Buildroot integrates a number of features that help tracking whether
+known security vulnerabilities are affecting some of the

docs/manual/common-usage.txt effects: · line 434

Quoted 10 lines
docs/manual/common-usage.txt +10 -0
+packages. Buildroot bases this logic on _CVEs_ (Common Vulnerabilities
+and Exposure) and the https://nvd.nist.gov/vuln[NVD database] provided
+by NIST.
+
+First, each package in Buildroot can be associated to a _CPE_
+identifier, by means of the various +FOO_CPE_ID_<something>+ variables
+that can be provided by each package. See
+xref:generic-package-reference[] for more details about those
+variables. When the _CPE_ identifier of a package is not defined, a
+default value is used, which may or may not be correct.

I don't think this reflects the reality very well. cpeid is only used if
*something* is set in the .mk file. Something like:

When no +FOO_CPE_ID_<something>+ variable is set for a package, it is not considered as having a valid _CPE_ identifier.

docs/manual/common-usage.txt effects: · line 436

Quoted 3 lines

This _CPE_

docs/manual/common-usage.txt +2 -0
+identifier is used to find in the NVD database of vulnerabilities
+which _CVEs_ are affecting any given package.

... If a package doesn't have a valid CPE identifier, a guess is made based on
the package name and version.

Quoted 9 lines
docs/manual/common-usage.txt +9 -0
+
+Based on this and the NIST databases, the +make pkg-stats+ target
+produces JSON and HTML formatted reports that indicate for each
+package of the current Buildroot configuration:
+
+* If there are known _CVEs_ for each package. Note that CVEs can
+  manually be ignored on a per-package basis using the
+  +FOO_IGNORED_CVES+ variable, for example if Buildroot has a patch
+  that fixes the vulnerability, or if the vulnerability is not

docs/manual/common-usage.txt effects: · line 455

Quoted 10 lines
docs/manual/common-usage.txt +10 -0
+  applicable to Buildroot for some reason.
+
+* If the _CPE_ identifier is valid and known in the
+  https://nvd.nist.gov/products/cpe[NIST CPE dictionary]. A _CPE_ is
+  valid is one of the +FOO_CPE_ID_<something>+ variable is defined, or
+  if +FOO_CPE_ID_VALID = YES+ validating that the default value of the
+  _CPE_ identifier is correct. In addition, _CPE_ are checked for
+  existence in the NIST CPE dictionary. If the _CPE_ identifier is not
+  known to the NIST CPE dictionary, then it is possible to contribute
+  to the NIST CPE dictionary: one need to submit a XML document that

needs

Quoted 1 line

+ describes the additional CPE identifier.

Can you add a reference to a web page that describes the procedure? Otherwise
it's not terribly helpful IMHO.

docs/manual/common-usage.txt effects: · line 463

Quoted 8 lines

Buildroot can help

docs/manual/common-usage.txt +7 -0
+  generating such XML documents for missing CPE identifiers using the
+  +make missing-cpe+ target. The XML documents are generated in
+  +$(O)/cpe-updates+, will need to be edited before submission to
+  NIST. In particular, one should verify the version information and
+  check that the reference URLs are reachable with a valid type
+  (suggestions for types are provided in the generated file).  If a
+  previous _CPE_ entry was found in the NIST CPE dictionary, the XML

If no previous CPE entry exists (i.e., the CPE_ID specified in the .mk file is
not valid), then no XML file is generated at all... AFAICS there is no help at the moment for generating the XML for completely new entries.

My brain is too tired ATM to try to formulate the documentation that expresses
that.

I've marked this one changes requested in patchwork.

Regards,
Arnout

Quoted 9 lines
docs/manual/common-usage.txt +5 -0
+  document generation tries to minimize the entry differences between
+  the existing versions in the dictionary. The XML document
+  generation uses the XML from the last version of that CPE as a
+  template for the new submission.
+
 include::eclipse-integration.txt[]
 
 include::advanced.txt[]