CPE ID Support
v5 cover CPE ID Support
This series begins adding CPE identifier support to Buildroot. The intent is to establish and maintain a baseline of CPE IDs, one for each package. Each of these IDs ties back to a NIST database entry for the respective piece of software, which is linked to specific vunderabilities.
Within Buildroot, a CPE report can be generated (like legal-info) that captures a target build's list of CPE IDs. This report can then be checked for validity using the new support/scripts/cpe-report or another third party tool. The script uses a new cpedb.py helper to search and identify the validity of the CPE strings.
The pkgstats script has been extended to provide CPE ID checking of matching/requires update/new as part of its html output. It also uses the new cpedb.py helper.
As part of testing this series, the following branch contains a series of fixups required to make these specific packages match the database. (I can submit these to the mailing list but there are ~70 of them)
https://github.com/rc-matthew-l-weber/buildroot/tree/cpe-info-github
Commit 14c3ee6 to 567732d
A follow-on patchset will be submitted adding support for pkgstat generation of CPE updates in XML and Buildroot manual updates for guidance on submission of those XML database updates to the NIST organization. (We'd like to get feedback on this series first to save us effort on the update XML stuff)
Matt Weber (7):
cpe-info: new make target
cpe-info: id prefix/suffix
cpe-info: only report target pkgs
cpe-info: update manual for new pkg vars
support/scripts/cpedb.py: new CPE XML helper
support/scripts/pkg-stats: add CPE reporting
support/scripts/cpe-report: new script
Makefile | 17 ++++- docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------ package/Makefile.in | 4 ++ package/pkg-generic.mk | 21 ++++++ package/pkg-utils.mk | 8 +++ support/scripts/cpe-report | 53 +++++++++++++++ support/scripts/cpedb.py | 52 ++++++++++++++ support/scripts/pkg-stats | 80 ++++++++++++++++++++-- 8 files changed, 302 insertions(+), 50 deletions(-) create mode 100755 support/scripts/cpe-report create mode 100644 support/scripts/cpedb.py
Signature · 1 line
1.9.1
PATCH v5 1/7 cpe-info: new make target
Similar to make legal-info, produce a csv delimited file containing all selected packages CPE identification.
Have the pkg infra define CPE_ID_* defaults using the package name for the vendor and name as most CPE IDs seem to align with that assumption. Also use the pkg version as the CPE ID's version field.
---
Changes
v2
[Thomas P
- Moved comment on conditionals back to this patchset where the conditional is created vs later
v3
[Thomas P
- Merged infra define CPE_ID_* into this patch
- Report all packages vs restricting to just allowing based on if the VENDOR was set (v2). This now represents Thomas P's original idea to report everything. At first I felt I should restrict the reporting to those CPE IDs we had made sure were correct. Turns out we should have actually let the script handle fixing the CPEs and just make a complete design of this up front.
[Matt
- Moved to using the _project on all vendors instead of just name
--- Makefile | 17 ++++++++++++++++- package/pkg-generic.mk | 13 +++++++++++++ package/pkg-utils.mk | 8 ++++++++ 3 files changed, 37 insertions(+), 1 deletion(-)
Makefile +16 -1
@@ -146,7 +146,7 @@ nobuild_targets := source %-source \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \ graph-build graph-size list-defconfigs \ - savedefconfig printvars + savedefconfig printvars cpe-info %-cpe-info ifeq ($(MAKECMDGOALS),) BR_BUILDING = y else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)
@@ -233,6 +233,7 @@ LEGAL_MANIFEST_CSV_TARGET = $(LEGAL_INFO_DIR)/manifest.csv LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings LEGAL_REPORT = $(LEGAL_INFO_DIR)/README +CPE_MANIFEST_CSV = $(BASE_DIR)/cpe-manifest.csv BR2_CONFIG = $(CONFIG_DIR)/.config
@@ -802,6 +803,19 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p mv .legal-info.sha256 legal-info.sha256) @echo "Legal info produced in $(LEGAL_INFO_DIR)" +.PHONY: cpe-info-clean +cpe-info-clean: + @rm -f $(CPE_MANIFEST_CSV) + +.PHONY: cpe-info-prepare +cpe-info-prepare: + @$(call MESSAGE,"Gathering CPE info") + @$(call cpe-manifest,CPE ID,CVE PATCHED,PACKAGE,VERSION,SOURCE SITE) + +.PHONY: cpe-info +cpe-info: cpe-info-clean cpe-info-prepare $(foreach p,$(PACKAGES),$(p)-cpe-info) + @echo "CPE info produced in $(CPE_MANIFEST_CSV)" + .PHONY: show-targets show-targets: @echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS))
@@ -1070,6 +1084,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' cpe-info - generate info about security CPE identification' @echo ' printvars - dump all the internal variables' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
package/pkg-generic.mk +13 -0
@@ -861,6 +861,18 @@ else $(2)_KCONFIG_VAR = BR2_PACKAGE_$(2) endif +$(2)_CPE_ID_VENDOR ?= $$($(2)_NAME)_project +$(2)_CPE_ID_NAME ?= $$($(2)_NAME) +$(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) +$(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) + +$(1)-cpe-info: PKG=$(2) +$(1)-cpe-info: +ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + @$$(call MESSAGE,"Collecting cpe info") + $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) +endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) + # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),) $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
@@ -1002,6 +1014,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-clean-for-reconfigure \ $(1)-clean-for-reinstall \ $(1)-configure \ + $(1)-cpe-info \ $(1)-depends \ $(1)-dirclean \ $(1)-external-deps \
package/pkg-utils.mk +8 -0
@@ -95,3 +95,11 @@ define legal-license-file # pkgname, pkgname-pkgver, pkgdir, filename, file-full } && \ cp $(5) $(LICENSE_FILES_DIR_$(6))/$(2)/$(4) endef + +# +# cpe-info helper functions +# + +define cpe-manifest # cpe, cve patched, pkg name, version, url + echo '"$(1)","$(2)","$(3)","$(4)","$(5)"' >>$(CPE_MANIFEST_CSV) +endef
Signature · 1 line
1.9.1
PATCH v5 2/7 cpe-info: id prefix/suffix
In reply to Matt WeberThere are two types of software cpe prefixes, one for applications and one for operating systems. Note: There is a third type for hardware.
This patchset determines which should be used and stores that information with the package for later use when assembling the CPE report.
There is also a suffix which we just default to wildcards at this point.
Refs:
https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf
https://cpe.mitre.org/specification/
---
Changes
v1 -> v2
[Thomas P
- Change to using a filter on pkg name value vs ifelse
v3
[Arnout
- Moved CPE prefix and suffix defines to package/Makefile.in
--- package/Makefile.in | 4 ++++ package/pkg-generic.mk | 8 +++++++- 2 files changed, 11 insertions(+), 1 deletion(-)
package/Makefile.in +4 -0
@@ -395,6 +395,10 @@ TARGET_CONFIGURE_ARGS = \ ################################################################################ +CPE_PREFIX_OS = cpe:2.3:o +CPE_PREFIX_APP = cpe:2.3:a +CPE_SUFFIX = *:*:*:*:*:*:* + ifeq ($(BR2_SYSTEM_ENABLE_NLS),y) NLS_OPTS = --enable-nls TARGET_NLS_DEPENDENCIES = host-gettext
package/pkg-generic.mk +7 -1
@@ -866,11 +866,17 @@ $(2)_CPE_ID_NAME ?= $$($(2)_NAME) $(2)_CPE_ID_VERSION ?= $$($(2)_VERSION) $(2)_CPE_ID ?= $$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION) +ifneq ($(filter linux linux-headers,$(1)),) +$(2)_CPE_PREFIX = $(CPE_PREFIX_OS) +else +$(2)_CPE_PREFIX = $(CPE_PREFIX_APP) +endif + $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") - $(Q)$$(call cpe-manifest,$$($(2)_CPE_ID),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) + $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) # legal-info: declare dependencies and set values used later for the manifest
Signature · 1 line
1.9.1
PATCH v5 3/7 cpe-info: only report target pkgs
In reply to Matt WeberThe reporting of host packages causes some duplication and complicates what is really in the targets configuration. For the purpose of the first version of this patchset, its assumed that host packages aren't relevant for the configuration and we only report the target's contents.
---
Changes
v1 -> v2
[Thomas P
- select if target vs selecting not host
v3
- Fixed host build error because cpe-info wasn't defined
--- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+)
package/pkg-generic.mk +2 -0
@@ -874,10 +874,12 @@ endif $(1)-cpe-info: PKG=$(2) $(1)-cpe-info: +ifeq ($$($(2)_TYPE),target) ifneq ($$(call qstrip,$$($(2)_SOURCE)),) @$$(call MESSAGE,"Collecting cpe info") $(Q)$$(call cpe-manifest,$$($(2)_CPE_PREFIX):$$($(2)_CPE_ID):$(CPE_SUFFIX),$$($(2)_CVE_PATCHED),$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_ACTUAL_SOURCE_SITE)) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) +endif # ifeq ($$($(2)_TYPE),target) # legal-info: declare dependencies and set values used later for the manifest ifneq ($$($(2)_LICENSE_FILES),)
Signature · 1 line
1.9.1
PATCH v5 4/7 cpe-info: update manual for new pkg vars
In reply to Matt WeberProvide guidance on setting up the *_CPE_* and *_CVE_* variables.
---
Changes
v2
[Thomas P
- Reworded LIBFOO_CVE_PATCHED description
[Matt W
- Added definition for new preset variables to auto-gen the CPE ID
- Added example LIBFOO_CPE_ID_VENDOR to LIBFOO
v3
- Updated to make *_CPE_VENDOR optional
- Changed wording around _CPE_ID as there is only one defined now
--- docs/manual/adding-packages-generic.txt | 117 ++++++++++++++++++++------------ 1 file changed, 74 insertions(+), 43 deletions(-)
docs/manual/adding-packages-generic.txt +74 -43
@@ -24,57 +24,59 @@ system is based on hand-written Makefiles or shell scripts. 09: LIBFOO_SITE = http://www.foosoftware.org/download 10: LIBFOO_LICENSE = GPL-3.0+ 11: LIBFOO_LICENSE_FILES = COPYING -12: LIBFOO_INSTALL_STAGING = YES -13: LIBFOO_CONFIG_SCRIPTS = libfoo-config -14: LIBFOO_DEPENDENCIES = host-libaaa libbbb -15: -16: define LIBFOO_BUILD_CMDS -17: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all -18: endef -19: -20: define LIBFOO_INSTALL_STAGING_CMDS -21: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a -22: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h -23: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib -24: endef -25: -26: define LIBFOO_INSTALL_TARGET_CMDS -27: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib -28: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d -29: endef -30: -31: define LIBFOO_USERS -32: foo -1 libfoo -1 * - - - LibFoo daemon -33: endef -34: -35: define LIBFOO_DEVICES -36: /dev/foo c 666 0 0 42 0 - - - -37: endef -38: -39: define LIBFOO_PERMISSIONS -40: /bin/foo f 4755 foo libfoo - - - - - -41: endef -42: -43: $(eval $(generic-package)) +12: LIBFOO_CPE_ID_VENDOR = foosoftware +13: LIBFOO_INSTALL_STAGING = YES +14: LIBFOO_CONFIG_SCRIPTS = libfoo-config +15: LIBFOO_DEPENDENCIES = host-libaaa libbbb +16: +17: define LIBFOO_BUILD_CMDS +18: $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(@D) all +19: endef +20: +21: define LIBFOO_INSTALL_STAGING_CMDS +22: $(INSTALL) -D -m 0755 $(@D)/libfoo.a $(STAGING_DIR)/usr/lib/libfoo.a +23: $(INSTALL) -D -m 0644 $(@D)/foo.h $(STAGING_DIR)/usr/include/foo.h +24: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(STAGING_DIR)/usr/lib +25: endef +26: +27: define LIBFOO_INSTALL_TARGET_CMDS +28: $(INSTALL) -D -m 0755 $(@D)/libfoo.so* $(TARGET_DIR)/usr/lib +29: $(INSTALL) -d -m 0755 $(TARGET_DIR)/etc/foo.d +30: endef +31: +32: define LIBFOO_USERS +33: foo -1 libfoo -1 * - - - LibFoo daemon +34: endef +35: +36: define LIBFOO_DEVICES +37: /dev/foo c 666 0 0 42 0 - - - +38: endef +39: +40: define LIBFOO_PERMISSIONS +41: /bin/foo f 4755 foo libfoo - - - - - +42: endef +43: +44: $(eval $(generic-package)) -------------------------------- -The Makefile begins on line 7 to 11 with metadata information: the +The Makefile begins on line 7 to 12 with metadata information: the version of the package (+LIBFOO_VERSION+), the name of the tarball containing the package (+LIBFOO_SOURCE+) (xz-ed tarball recommended) the Internet location at which the tarball can be downloaded from -(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+) and file with the -license text (+LIBFOO_LICENSE_FILES+). All variables must start with +(+LIBFOO_SITE+), the license (+LIBFOO_LICENSE+), the file with the +license text (+LIBFOO_LICENSE_FILES+) and the vendor for vunerability +analysis (+LIBFOO_CPE_ID_VENDOR+). All variables must start with the same prefix, +LIBFOO_+ in this case. This prefix is always the uppercased version of the package name (see below to understand where the package name is defined). -On line 12, we specify that this package wants to install something to +On line 13, we specify that this package wants to install something to the staging space. This is often needed for libraries, since they must install header files and other development files in the staging space. This will ensure that the commands listed in the +LIBFOO_INSTALL_STAGING_CMDS+ variable will be executed. -On line 13, we specify that there is some fixing to be done to some +On line 14, we specify that there is some fixing to be done to some of the 'libfoo-config' files that were installed during +LIBFOO_INSTALL_STAGING_CMDS+ phase. These *-config files are executable shell script files that are
@@ -122,14 +124,14 @@ IMAGEMAGICK_CONFIG_SCRIPTS = \ -------------------------------- ================================ -On line 14, we specify the list of dependencies this package relies +On line 15, we specify the list of dependencies this package relies on. These dependencies are listed in terms of lower-case package names, which can be packages for the target (without the +host-+ prefix) or packages for the host (with the +host-+) prefix). Buildroot will ensure that all these packages are built and installed 'before' the current package starts its configuration. -The rest of the Makefile, lines 16..29, defines what should be done +The rest of the Makefile, lines 17..29, defines what should be done at the different steps of the package configuration, compilation and installation. +LIBFOO_BUILD_CMDS+ tells what steps should be performed to
@@ -142,16 +144,16 @@ All these steps rely on the +$(@D)+ variable, which contains the directory where the source code of the package has been extracted. -On lines 31..43, we define a user that is used by this package (e.g. +On lines 32..44, we define a user that is used by this package (e.g. to run a daemon as non-root) (+LIBFOO_USERS+). -On line 35..37, we define a device-node file used by this package +On line 36..38, we define a device-node file used by this package (+LIBFOO_DEVICES+). -On line 39..41, we define the permissions to set to specific files +On line 40..42, we define the permissions to set to specific files installed by this package (+LIBFOO_PERMISSIONS+). -Finally, on line 43, we call the +generic-package+ function, which +Finally, on line 44, we call the +generic-package+ function, which generates, according to the variables defined previously, all the Makefile code necessary to make your package working.
@@ -469,6 +471,35 @@ information is (assuming the package name is +libfoo+) : locations, `/lib/firmware`, `/usr/lib/firmware`, `/lib/modules`, `/usr/lib/modules`, and `/usr/share`, which are automatically excluded. +* +LIBFOO_CPE_ID_VENDOR+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the vendor. By default it's set + to <pkg-name>_project. + +* +LIBFOO_CPE_ID_NAME+ + This variable is optional. It only must be defined if the package name + does not match what the CPE ID uses for the name. By default it's set + to <pkg-name>. + +* +LIBFOO_CPE_ID_VERSION+ + This variable is optional. By default it's set to <pkg-version>. + +* +LIBFOO_CPE_ID+ is optional, as the package infrastructure hangles the + default case of a single package's Common Product Enumeration (CPE) + identification string. +make cpe-info+ copies all of these into a + +cpe-manifest.csv+ file. To identify a package's possible CPE, + the National Vunerability Database can be searched at + https://nvd.nist.gov/products/cpe/search. + +* +LIBFOO_CVE_PATCHED+ is a space-separated list of the package's Common + Vunerability Enumeration (CVE) identification strings. This list + enumerates CVEs which are fixed by patches added in Buildroot. This + allows the CPE reporting to provide additional detail on CVEs which + have been fixed, even if Buildroot is not yet using an updated upstream + release including the fix. This variable is optional. If it is not + defined, the +CVE PATCHED+ field will appear empty in the manifest + file for this package. + The recommended way to define these variables is to use the following syntax:
Signature · 1 line
1.9.1
PATCH v5 5/7 support/scripts/cpedb.py: new CPE XML helper
In reply to Matt WeberPython class which consumes a NIST CPE XML and provides helper functions to access and search the db's data.
---
v5
[Ricardo
- Fixed typo in join/split of cpe str without version
- Removed extra prints as they aren't needed when we have the output reports/stdout
- Updated v4 comments about general flake formatting cleanup
- Incorporated parts of patch 1/2 suggestions for optimizations
[Arnout
- added pre-processing of cpe values into two sets, one with and one without version
- Collectly with Ricardo, decided to move cpe class to this seperate script
v1 -> v4
- No version
--- support/scripts/cpedb.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 support/scripts/cpedb.py
support/scripts/cpedb.py new file +52 -0
@@ -0,0 +1,52 @@ +import sys +import urllib2 +import xmltodict +import gzip +from StringIO import StringIO + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +class CPEDB: + all_cpedb = dict() + all_cpes = set() + all_cpes_no_version = set() + + def get_xml_dict(self): + print("CPE: Fetching xml manifest...") + try: + compressed_cpe_file = urllib2.urlopen(CPE_XML_URL) + print("CPE: Unzipping xml manifest...") + cpe_file = gzip.GzipFile(fileobj=StringIO(compressed_cpe_file.read())).read() + print("CPE: Converting xml manifest to dict...") + self.all_cpedb = xmltodict.parse(cpe_file) + + for cpe in self.all_cpedb['cpe-list']['cpe-item']: + cpe_str = cpe['cpe-23:cpe23-item']['@name'] + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + self.all_cpes.add(cpe_str) + self.all_cpes_no_version.add(cpe_str_no_version) + + except urllib2.HTTPError: + print("CPE: HTTP Error: %s" % CPE_XML_URL) + sys.exit(1) + except urllib2.URLError: + print("CPE: URL Error: %s" % CPE_XML_URL) + sys.exit(1) + + def find_partial(self, cpe_str): + cpe_str_no_version = self.get_cpe_no_version(cpe_str) + if cpe_str_no_version in self.all_cpes_no_version: + return cpe_str_no_version + + def find(self, cpe_str): + if cpe_str in self.all_cpes: + return cpe_str + + def get_cpe_no_version(self, cpe): + return ":".join(cpe.split(":")[:5]) + + def get_nvd_url(self, cpe_str): + return "https://nvd.nist.gov/products/cpe/search/results?keyword=" + \ + urllib2.quote(cpe_str) + \ + "&status=FINAL&orderBy=CPEURI&namingFormat=2.3"
Signature · 1 line
1.9.1
PATCH v5 6/7 support/scripts/pkg-stats: add CPE reporting
In reply to Matt WeberPkg status now includes CPE as an item reported in the html output (stat summary and for each pkg)
---
Changes
v4 -> v5
[Ricardo
- Renamed patch to correctly match file name
- Removed extra prints as they aren't needed when we have the output reports/stdout
- Updated v4 comments about general flake formatting cleanup
[Arnout
- Collectly with Ricardo, decided to move cpe report analysis to a seperate script and breakout a module that's imported for the cpedb class
- Rename cpe_dict to instead be cpedb
v3 -> v4
- Collapsed patch 5 and 6 together into this single patch
[Eric
- added except handling around file io
- fixed condition where buildroot isn't generating a CPE string as part of the infra and output that is the case. (eventually these probably could be fixed but there aren't many at this point)
[Ricardo
- fixed patch naming and resolved flake8 issues
- took the opportunity to also fix other flake8 syntax update suggestions
- added except handling to have proper exits
- cleaned up csv file header skippin
- condensed partial cve string split
- updated help txt as suggested
- reworked output file requirement. Removed -o as required but added check if provided when -c isn't used
v3
- New patch
--- support/scripts/pkg-stats | 80 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 74 insertions(+), 6 deletions(-)
support/scripts/pkg-stats +74 -6
@@ -24,6 +24,7 @@ from collections import defaultdict import re import subprocess import sys +from cpedb import CPEDB INFRA_RE = re.compile("\$\(eval \$\(([a-z-]*)-package\)\)")
@@ -32,6 +33,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_cpe_id = dict() def __init__(self, name, path): self.name = name
@@ -43,6 +45,8 @@ class Package: self.patch_count = 0 self.warnings = 0 self.current_version = None + self.cpe_id = None + self.has_cpe = False def pkgvar(self): return self.name.upper().replace("-", "_")
@@ -116,6 +120,26 @@ class Package: self.warnings = int(m.group(1)) return + def set_cpe_info(self, cpe_dict): + """ + Fills in the .has_cpe field + """ + var = self.pkgvar() + if var in self.all_cpe_id: + self.cpe_id = self.all_cpe_id[var] + if self.cpe_id is None: + # BR infra did not build a CPE ID for this pkg + # as it's most likely a host pkg + return + result = cpe_dict.find(self.cpe_id) + if not result: + result = cpe_dict.find_partial(cpe_dict.get_cpe_no_version(self.cpe_id)) + if result: + self.has_cpe = "Update" + # Unset case for has_cpe is assumed missing/does not exist + else: + self.has_cpe = cpe_dict.get_nvd_url(self.cpe_id) + def __eq__(self, other): return self.path == other.path
@@ -254,6 +278,20 @@ def package_init_make_info(): Package.all_versions[pkgvar] = value + # CPE ID + o = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", + "-s", "printvars", "VARS=%_CPE_ID"]) + for l in o.splitlines(): + # Get variable name and value + pkgvar, value = l.split("=") + + # Strip _CPE_ID + pkgvar = pkgvar[:-7] + if pkgvar in ("LINUX", "LINUX_HEADERS"): + Package.all_cpe_id[pkgvar] = "cpe:2.3:o:" + value + ":*:*:*:*:*:*:*" + else: + Package.all_cpe_id[pkgvar] = "cpe:2.3:a:" + value + ":*:*:*:*:*:*:*" + def calculate_stats(packages): stats = defaultdict(int)
@@ -279,6 +317,12 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 + if pkg.has_cpe == "Update": + stats["update-cpe"] += 1 + elif pkg.has_cpe: + stats["cpe"] += 1 + else: + stats["no-cpe"] += 1 stats["patches"] += pkg.patch_count return stats
@@ -422,6 +466,20 @@ def dump_html_pkg(f, pkg): f.write(" <td class=\"%s\">%d</td>\n" % (" ".join(td_class), pkg.warnings)) + # CPE Valid + td_class = ["centered"] + if not pkg.has_cpe: + td_class.append("wrong") + f.write(" <td class=\"%s\">%s</td>\n" % + (" ".join(td_class), boolean_str(pkg.has_cpe))) + elif pkg.has_cpe == "Update": + td_class.append("wrong") + f.write(" <td class=\"%s\">Update</td>\n" % + (" ".join(td_class))) + else: + td_class.append("correct") + f.write(" <td class=\"%s\"><a href=\"%s\">%s</a></td>\n" % + (" ".join(td_class), pkg.has_cpe, boolean_str(pkg.has_cpe))) f.write(" </tr>\n")
@@ -437,6 +495,7 @@ def dump_html_all_pkgs(f, packages): <td class=\"centered\">Hash file</td> <td class=\"centered\">Current version</td> <td class=\"centered\">Warnings</td> +<td class=\"centered\">CPE Valid</td> </tr> """) for pkg in sorted(packages):
@@ -463,6 +522,12 @@ def dump_html_stats(f, stats): stats["hash"]) f.write(" <tr><td>Packages not having a hash file</td><td>%s</td></tr>\n" % stats["no-hash"]) + f.write(" <tr><td>Packages having a registered CPE</td><td>%s</td></tr>\n" % + stats["cpe"]) + f.write(" <tr><td>Packages needing CPE update</td><td>%s</td></tr>\n" % + stats["update-cpe"]) + f.write(" <tr><td>Packages missing a registered CPE</td><td>%s</td></tr>\n" % + stats["no-cpe"]) f.write(" <tr><td>Total number of patches</td><td>%s</td></tr>\n" % stats["patches"]) f.write("</table>\n")
@@ -499,17 +564,19 @@ def parse_args(): def __main__(): args = parse_args() if args.npackages and args.packages: - print "ERROR: -n and -p are mutually exclusive" + print("ERROR: -n and -p are mutually exclusive") sys.exit(1) if args.packages: package_list = args.packages.split(",") else: package_list = None - print "Build package list ..." + cpedb = CPEDB() + cpedb.get_xml_dict() + print("Build package list ...") packages = get_pkglist(args.npackages, package_list) - print "Getting package make info ..." + print("Getting package make info ...") package_init_make_info() - print "Getting package details ..." + print("Getting package details ...") for pkg in packages: pkg.set_infra() pkg.set_license()
@@ -517,9 +584,10 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() - print "Calculate stats" + pkg.set_cpe_info(cpedb) + print("Calculate stats") stats = calculate_stats(packages) - print "Write HTML" + print("Write HTML") dump_html(packages, stats, args.output)
Signature · 1 line
1.9.1
PATCH v5 6/7 support/scripts/pkg-stats: add CPE reporting
Hello,
Quoted 10 lines · Matt Weber
packages =
get_pkglist(args.npackages, package_list)support/scripts/pkg-stats +2
-2- print "Getting package make info ..." + print("Getting package make info ...") package_init_make_info() - print "Getting package details ..." + print("Getting package details ...") for pkg in packages: pkg.set_infra() pkg.set_license()
support/scripts/pkg-stats def __main__(): · line 590
Quoted 9 lines
support/scripts/pkg-stats +3
-2@@ -517,9 +584,10 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() - print "Calculate stats" + pkg.set_cpe_info(cpedb) + print("Calculate stats") stats = calculate_stats(packages) - print "Write HTML" + print("Write HTML")
I hate silly reviews, and this is a silly review, but this change to use parenthesis in print calls is not related to introducing CPE support, so it should be a separate patch (which I could merge sooner and separately from the CPE discussion).
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
PATCH v5 6/7 support/scripts/pkg-stats: add CPE reporting
Thomas,
Quoted 20 lines · Thomas Petazzoni
Hello,
Quoted 16 lines · Matt Weber
packages =
get_pkglist(args.npackages, package_list)support/scripts/pkg-stats +4
-3- print "Getting package make info ..." + print("Getting package make info ...") package_init_make_info() - print "Getting package details ..." + print("Getting package details ...") for pkg in packages: pkg.set_infra() pkg.set_license()@@ -517,9 +584,10 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() - print "Calculate stats" + pkg.set_cpe_info(cpedb) + print("Calculate stats")
Quoted 8 lines
support/scripts/pkg-stats def __main__(): · line 590
Quoted 3 lines
support/scripts/pkg-stats +1
-1stats = calculate_stats(packages) - print "Write HTML" + print("Write HTML")I hate silly reviews, and this is a silly review, but this change to use parenthesis in print calls is not related to introducing CPE support, so it should be a separate patch (which I could merge sooner and separately from the CPE discussion).
Noted, Ricardo had mentioned similar. See below.
http://patchwork.ozlabs.org/patch/916300/
Matt
PATCH v5 7/7 support/scripts/cpe-report: new script
In reply to Matt WeberThe script supports looking up all the CPEs provided in a make cpe-info csv file export from a target Buildroot build. It checks the current version and suggests a CPE needs update or possibly initial submission to NIST.
---
Changes
v5
[Ricardo
- Updated v4 comments about general flake formatting cleanup
- Incorporated parts of patch 1/2 suggestions for optimizations
[Ricardo/Arnout
- Collectly, decided to move cpe report analysis to this script and use a seperate module cpedb class
[Arnout
- Rename cpe_dict to instead be cpedb
v1 -> v4
- Patch did not exist and was part of pkg-stats file
--- support/scripts/cpe-report | 53 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 support/scripts/cpe-report
support/scripts/cpe-report new file +53 -0
@@ -0,0 +1,53 @@ +#!/usr/bin/env python + +import argparse +import sys +import csv +from cpedb import CPEDB + + +def get_target_cpe_report(cpe_report_file, cpedb): + report_cpe_exact_match = "" + report_cpe_needing_update = "" + report_cpe_missing = "" + + print("CPE: Checking for matches...") + try: + with open(cpe_report_file) as cpe_file: + cpe_list = csv.reader(cpe_file) + next(cpe_list) # make cpe-info has a one line header + for cpe in cpe_list: + result = cpedb.find(cpe[0]) + if not result: + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) + if not result: + report_cpe_missing += cpe[0] + "\n" + else: + report_cpe_needing_update += cpe[0] + "\n" + else: + report_cpe_exact_match += cpe[0] + "\n" + except (OSError, IOError) as e: + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) + sys.exit(1) + + print("CPE: Found EXACT match:\n" + report_cpe_exact_match) + print("CPE: Found but REQUIRES UPDATE:\n" + report_cpe_needing_update) + print("CPE: Not found (proposing the following to be added):\n" + report_cpe_missing) + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-c', dest='cpe_report', action='store', required=True, + help='CPE Report generated by make cpe-info (csv format)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + cpedb = CPEDB() + cpedb.get_xml_dict() + print("Performing Target CPE Report Analysis...") + get_target_cpe_report(args.cpe_report, cpedb) + + +__main__()
Signature · 1 line
1.9.1