Add initial capability to check CVEs for recipes
cover Add initial capability to check CVEs for recipes
From: Mariano Lopez <[email protected]>
This series add the cve-check-tool recipe, a tool used to identify potentially vulnerable software through version matching. It will check if a vulnerability has been addressed by a patch.
Also add the new cve-check class that will add a task for all recipes to check for CVEs using cve-check-tool. This tool can be used by recipe, image (will generate an image report in deploy dir), and with "world" and "universe"
To run it just inherit the class and enter:
bitbake -c cve_check <recipe>
The following changes since commit 23056103c949b498c23b47579e8dd57ce78e6ed9:
uclibc: Do not use immediate expansion operator (2016-02-22 20:42:48 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib mariano/bug7515
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug7515
Mariano Lopez (3):
cve-check-tool: Add recipe
cve-check-tool patch to allow select dir for the db cve-check.bbclass: Add class
meta/classes/cve-check.bbclass | 229 +++++++++++++++++++++ .../change_logic_cve_get_file_parent.patch | 45 ++++ .../cve-check-tool/cve-check-tool_5.6.bb | 61 ++++++ 3 files changed, 335 insertions(+) create mode 100644 meta/classes/cve-check.bbclass create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
Signature · 1 line
2.6.2
PATCH 1/3 cve-check-tool: Add recipe
From: Mariano Lopez <[email protected]>
cve-check-tool is a program to for checking public CVEs. This tool also seek to determine if a vulnerability has been addressed by a patch.
The recipe also includes the do_populate_cve_db task that will populate the database used by the tool. This task is added when the cve-check class has been inherited.
[YOCTO #7515]
Co-authored by Elena Reshetova & Mariano Lopez
--- .../cve-check-tool/cve-check-tool_5.6.bb | 61 ++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb new file +61 -0
@@ -0,0 +1,61 @@ +SUMMARY = "cve-check-tool" +DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\ +The tool will identify potentially vunlnerable software packages within Linux distributions through version matching." +HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool" +SECTION = "Development/Tools" +LICENSE = "GPL-2.0" +LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6" + +SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \ + file://change_logic_cve_get_file_parent.patch" + +SRC_URI[md5sum] = "30f32e6254580162eacfcc437a144463" +SRC_URI[sha256sum] = "d35af2bfa014b9d7cdc9c59ec0bd7df40c22dfcd57244c9099c0aa9bdc9c0cb4" + +DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl" + +inherit pkgconfig autotools + +EXTRA_OECONF = "--disable-static" + +python () { + # If the cve-check class is inherited it is needed to populate the + # CVE database before checking the CVEs for all recipes. + pn = d.getVar("PN", True) + if pn.endswith("-native") and bb.data.inherits_class('cve-check', d): + bb.build.addtask("do_populate_cve_db", "do_build", "do_populate_sysroot", d) +} + +python do_populate_cve_db () { + import subprocess + import time + from bb.utils import export_proxies + + export_proxies(d) + fail_text = "Failed to update database" + error_str = fail_text + cve_dir = d.getVar("CVE_CHECK_DB_DIR", True) + cmd = "cve-check-update -d %s" % cve_dir + bb.debug(1, "Updating cve-check-tool database located in %s" % cve_dir) + try: + popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = popen.communicate() + bb.debug(2, "Command %s returned:\n%s" % (cmd, output.decode())) + error_str = error.decode() + bb.debug(2, "Command %s errors:\n%s" % (cmd, error_str)) + except: + bb.warn("Error in executing cve-check-update: %s" % str(sys.exc_info())) + + if fail_text in error_str: + bb.warn("Failed to update cve-check-tool database, CVEs won't be checked") + else: + utc_time = time.gmtime(time.time()) + format_time = "%Y-%m-%d %H:%M:%S" + with open(d.getVar("CVE_CHECK_TMP_FILE", True), "w") as f: + f.write("CVE database was updated on %s UTC\n\n" + % time.strftime(format_time, utc_time)) +} + +do_populate_cve_db[nostamp] = "1" + +BBCLASSEXTEND = "native"
Signature · 1 line
2.6.2
Re: [PATCH 1/3] cve-check-tool: Add recipe
Quoted 9 lines · On 24 February 2016 at 15:27, <[email protected]> wrote:
meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb new file +6
-0+python () { + # If the cve-check class is inherited it is needed to populate the + # CVE database before checking the CVEs for all recipes. + pn = d.getVar("PN", True) + if pn.endswith("-native") and bb.data.inherits_class('cve-check', d): + bb.build.addtask("do_populate_cve_db", "do_build","
do_populate_sysroot", d)
+}
Wouldn't this be better done as an explicit dependency of the class task?
Ross
PATCH 2/3 cve-check-tool patch to allow select dir for the db
In reply to [email protected]From: Mariano Lopez <[email protected]>
This patch allows to select the directory for the
database used by cve-check-tool.
[YOCTO #7515]
--- .../change_logic_cve_get_file_parent.patch | 45 ++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch
meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patch new file +45 -0
@@ -0,0 +1,45 @@ +From 22cc9186909f98f024d78a08504d0bf532806de0 Mon Sep 17 00:00:00 2001 +From: Mariano Lopez <[email protected]> +Date: Thu, 18 Feb 2016 14:26:02 +0000 +Subject: [PATCH] util.c: Change logic in cve_get_file_parent() + +Function cve_get_file_parent() will try to get the +realpath and the get the dirname. If the file used +to get parent doesn't exist the call will fail. + +This problem is present when using another directory +for the database, realpath() won't find the nvd.db +file and the program will exit quitely. + +This patch will first get the dirname and the get +the realpath to avoid failing when the doesn't exist. + +Upstream-Status: Accepted [Release v5.6.3] + +Signed-off-by: Mariano Lopez <[email protected]> +--- + src/library/util.c | 8 +++----- + 1 file changed, 3 insertions(+), 5 deletions(-) + +diff --git a/src/library/util.c b/src/library/util.c +index 8a20728..4d4a576 100644 +--- a/src/library/util.c ++++ b/src/library/util.c +@@ -184,11 +184,9 @@ bool cve_is_dir(const char *p) + + char *cve_get_file_parent(const char *p) + { +- char *r = realpath(p, NULL); +- if (!r) { +- return NULL; +- } +- return dirname(r); ++ autofree(char) *d = strdup(p); ++ char *r = realpath(dirname(d), NULL); ++ return r; + } + + bool cve_file_set_text(const char *path, char *text) +-- +2.6.2 +
Signature · 1 line
2.6.2
Re: [PATCH 2/3] cve-check-tool patch to allow select dir for the db
Quoted 3 lines · On 24 February 2016 at 15:27, <[email protected]> wrote:
This patch allows to select the directory for the
database used by cve-check-tool.
Squash this into 1/3.
Ross
Re: [PATCH 2/3] cve-check-tool patch to allow select dir for the db
Quoted 10 lines · Ross
On 24 February 2016 at 15:27, <[email protected] <mailto:[email protected]>> wrote:
This patch allows to select the directory for the database used by cve-check-tool.
Squash this into 1/3.
Ross
This was the only patch that I made my own, that is why I sent it as separated patch. If you check the other patches there are co-authors in there. I don't want someone else take the blame if my patch bust something :)
Mariano
PATCH 3/3 cve-check.bbclass: Add class
In reply to [email protected]From: Mariano Lopez <[email protected]>
This class adds a new task for all the recipes to use cve-check-tool in order to look for public CVEs affecting the packages generated.
It is possible to use this class when building an image, building a recipe, or using the "world" or "universe" cases.
In order to use this class it must be inherited at some point and it will add the task automatically to every recipe.
[YOCTO #7515]
Co-authored by Ross Burton & Mariano Lopez
--- meta/classes/cve-check.bbclass | 229 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 meta/classes/cve-check.bbclass
meta/classes/cve-check.bbclass new file +229 -0
@@ -0,0 +1,229 @@ +# +CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db" + +CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve" +CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.log" +CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check" + +CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve" +CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cve" +CVE_CHECK_COPY_FILES ??= "1" +CVE_CHECK_CREATE_MANIFEST ??= "1" + +# Whitelist for packages (PN) +cve_check_pn_whitelist () { + glibc-locale +} + +# Whitelist for CVE and version of package +python cve_check_cve_whitelist () { + {"CVE-2014-2524": ("6.3",), \ + } +} + +python do_cve_check () { + """ + Check recipe for patched and unpatched CVEs + """ + + if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)): + patched_cves = get_patches_cves(d) + patched, unpatched = check_cves(d, patched_cves) + if patched or unpatched: + cve_data = get_cve_info(d, patched + unpatched) + cve_write_data(d, patched, unpatched, cve_data) + else: + bb.note("Failed to update CVE database, skipping CVE check") +} + +addtask cve_check before do_build +do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db" +do_cve_check[nostamp] = "1" + +python cve_check_cleanup () { + """ + Delete the file used to gather all the CVE information. + """ + import bb.utils + + tmp_file = e.data.getVar("CVE_CHECK_TMP_FILE", True) + bb.utils.remove(tmp_file) +} + +addhandler cve_check_cleanup +cve_check_cleanup[eventmask] = "bb.cooker.CookerExit" + +python cve_check_write_rootfs_manifest () { + """ + Create CVE manifest when building an image + """ + + import shutil + from bb.utils import mkdirhier + + if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)) and \ + d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1": + bb.note("Writing rootfs CVE manifest") + deploy_dir = d.getVar("DEPLOY_DIR_IMAGE", True) + link_name = d.getVar("IMAGE_LINK_NAME", True) + manifest_name = d.getVar("CVE_CHECK_MANIFEST", True) + cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE", True) + + shutil.copyfile(cve_tmp_file, manifest_name) + + if manifest_name is not None and os.path.exists(manifest_name): + manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name) + if os.path.exists(manifest_link): + if d.getVar('RM_OLD_IMAGE', True) == "1" and \ + os.path.exists(os.path.realpath(manifest_link)): + os.remove(os.path.realpath(manifest_link)) + os.remove(manifest_link) + os.symlink(os.path.basename(manifest_name), manifest_link) + bb.plain("Image CVE report stored in: %s" % manifest_name) +} + +ROOTFS_POSTPROCESS_COMMAND_prepend = "cve_check_write_rootfs_manifest; " + + +def get_patches_cves(d): + """ + Get patches that solve CVEs using the "CVE: " tag. + """ + + import re + + pn = d.getVar("PN", True) + cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+") + patched_cves = set() + for url in src_patches(d): + patch_file = bb.fetch.decodeurl(url)[2] + with open(patch_file, "r") as f: + patch_text = f.read() + + # Search for the "CVE: " line + match = cve_match.search(patch_text) + if match: + # Get only the CVEs without the "CVE: " tag + cves = patch_text[match.start()+5:match.end()] + for cve in cves.split(): + patched_cves.add(cve) + + return patched_cves + +def check_cves(d, patched_cves): + """ + Run cve-check-tool looking for patched and unpatched CVEs. + """ + + from bb.utils import export_proxies + import ast, csv, tempfile, subprocess, StringIO + + cves_patched = [] + cves_unpatched = [] + bpn = d.getVar("BPN", True) + pv = d.getVar("PV", True) + cves = " ".join(patched_cves) + cve_dir = d.getVar("CVE_CHECK_DB_DIR", True) + cve_whitelist = ast.literal_eval(d.getVar("cve_check_cve_whitelist", True).strip()) + cmd = "cve-check-tool --no-html --csv --not-affected -t faux -d %s" % cve_dir + + # If the recipe has been whitlisted we return empty lists + if d.getVar("PN", True) in d.getVar("cve_check_pn_whitelist", True).split(): + return ([], []) + + # It is needed to export the proxies to download the database using HTTP + export_proxies(d) + # Write the faux CSV file to be used with cve-check-tool + fd, faux = tempfile.mkstemp(prefix="cve-faux-") + with os.fdopen(fd, "w") as f: + f.write("%s,%s,%s," % (bpn, pv, cves)) + + cmd += " %s" % faux + try: + popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + output, error = popen.communicate() + except: + bb.warn("Couldn't check CVEs %s" % str(sys.exc_info())) + finally: + os.remove(faux) + + for row in csv.reader(StringIO.StringIO(output)): + if row[2]: + for cve in row[2].split(): + # Skip if the CVE has been whitlisted for the current version + if pv not in cve_whitelist.get(cve,[]): + cves_unpatched.append(cve) + if row[3]: + for cve in row[3].split(): + cves_patched.append(cve) + + return (cves_patched, cves_unpatched) + +def get_cve_info(d, cves): + """ + Get CVE information from the database used by cve-check-tool. + """ + + try: + import sqlite3 + except ImportError: + from pysqlite2 import dbapi2 as sqlite3 + + cve_data = {} + db_file = d.getVar("CVE_CHECK_DB_FILE", True) + placeholder = ",".join("?" * len(cves)) + query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholder + conn = sqlite3.connect(db_file) + cur = conn.cursor() + for row in cur.execute(query, tuple(cves)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["score"] = row[2] + cve_data[row[0]]["modified"] = row[3] + cve_data[row[0]]["vector"] = row[4] + conn.close() + + return cve_data + +def cve_write_data(d, patched, unpatched, cve_data): + """ + Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and + CVE manifest if enabled. + """ + + from bb.utils import mkdirhier + + cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True) + nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId=" + write_string = "" + mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True)) + + for cve in sorted(cve_data): + write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True) + write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True) + write_string += "CVE: %s\n" % cve + if cve in patched: + write_string += "CVE STATUS: Patched\n" + else: + write_string += "CVE STATUS: Unpatched\n" + bb.warn("Found unpatched CVE, for more information check %s" % cve_file) + write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] + write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"] + write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] + write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) + + with open(cve_file, "w") as f: + f.write(write_string) + + if d.getVar("CVE_CHECK_COPY_FILES", True) == "1": + cve_dir = d.getVar("CVE_CHECK_DIR", True) + mkdirhier(cve_dir) + deploy_file = os.path.join(cve_dir, d.getVar("PN", True)) + with open(deploy_file, "w") as f: + f.write(write_string) + + if d.getVar("CVE_CHECK_CREATE_MANIFEST", True) == "1": + with open(d.getVar("CVE_CHECK_TMP_FILE", True), "a") as f: + f.write("%s" % write_string) +
Signature · 1 line
2.6.2
Re: [PATCH 3/3] cve-check.bbclass: Add class
meta/classes/cve-check.bbclass line 17
Quoted 4 lines · On 24 February 2016 at 15:27, <[email protected]> wrote:
meta/classes/cve-check.bbclass new file +4
-0+# Whitelist for packages (PN) +cve_check_pn_whitelist () { + glibc-locale +}
Why is this a shell function? CVE_CHECK_PN_WHITELIST = "glibc-locale" please.
+# Whitelist for CVE and version of package
meta/classes/cve-check.bbclass line 23
Quoted 4 lines
meta/classes/cve-check.bbclass new file +4
-0+python cve_check_cve_whitelist () { + {"CVE-2014-2524": ("6.3",), \ + } +}
Why is this a Python function? Make it a bare string with implied formatting and it can be extended from outside this class, where as this can't.
Quoted 2 lines
+addtask
cve_checkbeforedo_build
If you're expecting to look at the sources, you'll want to be after do_fetch too.
Quoted 2 lines
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
And cve-check-tool-native:do_populate_sysroot.
meta/classes/cve-check.bbclass line 97
Quoted 10 lines
meta/classes/cve-check.bbclass new file +9
-0+def get_patches_cves(d): + """ + Get patches that solve CVEs using the "CVE: " tag. + """ + + import re + + pn = d.getVar("PN", True) + cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
How does this work as the backslashes are escaping the - and d and d? Use r"" strings.
+ patched_cves = set()
Quoted 4 lines
meta/classes/cve-check.bbclass new file +2
-0+ for url in src_patches(d): + patch_file = bb.fetch.decodeurl(url)[2]
meta/classes/cve-check.bbclass line 110
Quoted 10 lines
meta/classes/cve-check.bbclass new file +10
-0+ with open(patch_file, "r") as f: + patch_text = f.read() + + # Search for the "CVE: " line + match = cve_match.search(patch_text) + if match: + # Get only the CVEs without the "CVE: " tag + cves = patch_text[match.start()+5:match.end()] + for cve in cves.split(): + patched_cves.add(cve)
Breaks for patches such as this in glibc:
meta/recipes-core/glibc/glibc/CVE-2015-9761_1.patch:CVE: CVE-2015-9761 patch #1
I'd probably look for a line that starts with "CVE:" and the use re.findall to find all strings matching r"CVE-\d{4}-\d+"
Quoted 4 lines
+ # It is needed to export the proxies to download the database using
HTTP
+export_proxies(d)
The database has already been downloaded hasn't it?
meta/classes/cve-check.bbclass line 140
Quoted 6 lines
meta/classes/cve-check.bbclass new file +4
-0+ # Write the faux CSV file to be used with cve-check-tool + fd, faux = tempfile.mkstemp(prefix="cve-faux-") + with os.fdopen(fd, "w") as f: + f.write("%s,%s,%s," % (bpn, pv, cves))
Put this inside the try incase the write fails so the file will still be deleted.
Quoted 6 lines
meta/classes/cve-check.bbclass new file +3
-0+ cmd += " %s" % faux + try: + popen = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
+ output, error =popen.communicate()
Don't use the low-level function, the higher level helpers are clearer.
Always write cmd as a list unless you *need* the shell.
try:
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
except CommandCalledException as e:
bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
+def get_cve_info(d, cves):
meta/classes/cve-check.bbclass line 171
Quoted 9 lines
meta/classes/cve-check.bbclass new file +8
-0+ """ + Get CVE information from the database used by cve-check-tool. + """ + + try: + import sqlite3 + except ImportError: + from pysqlite2 import dbapi2 as sqlite3
Isn't the output from cve-check-tool good enough? Would it be nicer to extend the log instead of assuming that the database format won't ever change?
+def cve_write_data(d, patched, unpatched, cve_data):
Quoted 26 lines
meta/classes/cve-check.bbclass new file +21
-0+ """ + Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and + CVE manifest if enabled. + """ + + from bb.utils import mkdirhier + + cve_file = d.getVar("CVE_CHECK_LOCAL_FILE", True) + nvd_link = "https://web.nvd.nist.gov/view/vuln/detail?vulnId=" + write_string = "" + mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True)) + + for cve in sorted(cve_data): + write_string += "PACKAGE NAME: %s\n" % d.getVar("PN", True) + write_string += "PACKAGE VERSION: %s\n" % d.getVar("PV", True) + write_string += "CVE: %s\n" % cve + if cve in patched: + write_string += "CVE STATUS: Patched\n" + else: + write_string += "CVE STATUS: Unpatched\n" + bb.warn("Found unpatched CVE, for more information check %s"%
cve_file)(quoted diff) +2
-0+ write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] + write_string += "CVSS v2 BASE SCORE: %s\n" %cve_data[cve]["score"]
meta/classes/cve-check.bbclass line 217
Quoted 5 lines
meta/classes/cve-check.bbclass new file +5
-0+ write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] + write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve) + + with open(cve_file, "w") as f: + f.write(write_string)
Just write to the file instead of to a temporary string.
Ross
Re: [PATCH 3/3] cve-check.bbclass: Add class
Quoted 7 lines · Ross
On 24 February 2016 at 15:27, <[email protected] <mailto:[email protected]>> wrote:
+do_cve_check[depends] = "cve-check-tool-native:do_populate_cve_db"
And cve-check-tool-native:do_populate_sysroot.
cve-check-tool-native:do_populate_cve_db depends on cve-check-tool-native:do_populate_sysroot, so adding it there would be redundant.
Quoted 12 lines
+def
get_patches_cves(d):
+ """
+ Get patches that solve CVEs using the "CVE: " tag.
+ """
+
+ import re
+
+ pn =d.getVar("PN", True)
+cve_match=re.compile("CVE:( CVE\-\d+\-\d+)+")
Quoted 1 line
How does this work as the backslashes are escaping the - and d and d? Use r"" strings.
The backslashes just escape the "-", the "d" is the same as with the raw string. I don't really see the need to use r"" here.
Quoted 16 lines
+
patched_cves=set()
+ for url insrc_patches(d):
+patch_file= bb.fetch.decodeurl(url)[2]
+ withopen(patch_file, "r")as f:
+patch_text=f.read()
+
+ # Search for the "CVE: " line
+ match =cve_match.search(patch_text)
+ if match:
+ # Get only the CVEs without the "CVE: " tag
+ cves = patch_text[match.start()+5:match.end()]
+ for cve incves.split():
+patched_cves.add(cve)
Quoted 5 lines
Breaks for patches such as this in glibc:
meta/recipes-core/glibc/glibc/CVE-2015-9761_1.patch:CVE: CVE-2015-9761 patch #1
I'd probably look for a line that starts with "CVE:" and the use re.findall to find all strings matching r"CVE-\d{4}-\d+"
What do you mean by break? It does catch the CVE just fine, to test it just revert the glibc 2.23 update. I find cleaner to match the string in a single operation instead of searching for the tag line by line and then match the CVEs.
Quoted 11 lines
+def
get_cve_info(d, cves):
+ """
+ Get CVE information from the database used by cve-check-tool.
+ """
+
+ try:
+ import sqlite3
+ except ImportError:
+ from pysqlite2 import dbapi2 as sqlite3
Quoted 1 line
Isn't the output from cve-check-tool good enough? Would it be nicer to extend the log instead of assuming that the database format won't ever change?
The output from cve-check-tool is only the CVE number, if that is good enough, the query to the database can be removed.
Quoted 37 lines
+def
cve_write_data(d, patched, unpatched, cve_data):
+ """
+ Write CVE information in WORKDIR; and toCVE_CHECK_DIR, and
+ CVE manifest if enabled.
+ """
+
+ from bb.utils import mkdirhier
+
+cve_file=d.getVar("CVE_CHECK_LOCAL_FILE", True)
+nvd_link= "https://web.nvd.nist.gov/view/vuln/detail?vulnId="
+write_string= ""
+mkdirhier(d.getVar("CVE_CHECK_LOCAL_DIR", True))
+
+ for cve insorted(cve_data):
+write_string+= "PACKAGE NAME: %s\n" %d.getVar("PN", True)
+write_string+= "PACKAGE VERSION: %s\n" % d.getVar("PV",
True)
+write_string+= "CVE: %s\n" % cve
+ if cve in patched:
+write_string+= "CVE STATUS: Patched\n"
+ else:
+write_string+= "CVE STATUS: Unpatched\n"
+ bb.warn("Found unpatched CVE, for more information
check %s" %cve_file)
+write_string+= "CVE SUMMARY: %s\n" %
cve_data[cve]["summary"]
+write_string+= "CVSS v2 BASE SCORE: %s\n" %
cve_data[cve]["score"]
+write_string+= "VECTOR: %s\n" % cve_data[cve]["vector"]
+write_string+= "MORE INFORMATION: %s%s\n\n" % (nvd_link,
cve)
+
+ withopen(cve_file, "w")as f:
+f.write(write_string)
Quoted 1 line
Just write to the file instead of to a temporary string.
The temporary string is used for other two files, one could be copied, but the other appends the string content.
Quoted 2 lines
Ross
I have implemented the rest of the comments, just need your input before sending a new version.
Mariano
Quoted 6 lines · On Wed, Feb 24, 2016 at 03:27:05PM +0000, [email protected] wrote:
From: Mariano Lopez <[email protected]>
This series add the cve-check-tool recipe, a tool used to identify potentially vulnerable software through version matching. It will check if a vulnerability has been addressed by a patch.
Quoted 8 lines
Also add the new cve-check class that will add a task for all recipes to check for CVEs using cve-check-tool. This tool can be used by recipe, image (will generate an image report in deploy dir), and with "world" and "universe"
To run it just inherit the class and enter:
bitbake -c
cve_check<recipe>
I tried these on yocto/dizzy but:
ERROR: Task do_cve_check in /home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bb depends upon non-existent task do_populate_cve_db in virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
Is there some simple way to make this work there too?
For testing purposes I tried this only with busybox:
$ cat busybox_%.bbappend inherit cve-check
The cve-check-tool itself needed a few native backports/fixes:
meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb +2 -0
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons" do_install_append () { install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor } + +BBCLASSEXTEND = "native"
meta/recipes-gnome/json-glib/json-glib_1.0.0.bb +2 -0
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097 inherit gnome gettext lib_package EXTRA_OECONF = "--disable-introspection" + +BBCLASSEXTEND = "native"
-Mikko
Quoted 44 lines · Mikko Rapeli
Quoted 6 lines · On Wed, Feb 24, 2016 at 03:27:05PM +0000, [email protected] wrote:
From: Mariano Lopez <[email protected]>
This series add the cve-check-tool recipe, a tool used to identify potentially vulnerable software through version matching. It will check if a vulnerability has been addressed by a patch.
Quoted 8 lines
Also add the new cve-check class that will add a task for all recipes to check for CVEs using cve-check-tool. This tool can be used by recipe, image (will generate an image report in deploy dir), and with "world" and "universe"
To run it just inherit the class and enter:
bitbake -c
cve_check<recipe>I tried these on yocto/dizzy but:
ERROR: Task
do_cve_checkin/home/builder/src/base/poky/meta/recipes-core/busybox/busybox_1.22.1.bbdepends upon non-existent taskdo_populate_cve_dbin virtual:native:/home/builder/src/base/poky/meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bbIs there some simple way to make this work there too?
For testing purposes I tried this only with busybox:
$ cat busybox_%.bbappend inherit cve-checkThe cve-check-tool itself needed a few native backports/fixes:
meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb +2
-0@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons" do_install_append () { install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor } + +BBCLASSEXTEND = "native"meta/recipes-gnome/json-glib/json-glib_1.0.0.bb +0
-0
meta/recipes-gnome/json-glib/json-glib_1.0.0.bb SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097 · line 22
Quoted 5 lines
meta/recipes-gnome/json-glib/json-glib_1.0.0.bb +2
-0@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097 inherit gnome gettext lib_package EXTRA_OECONF = "--disable-introspection" + +BBCLASSEXTEND = "native"
Sorry, I guess this is needed to enable the class properly:
$ grep cve-check conf/local.conf INHERIT += "cve-check"
but there are some other backports needed in python modules...
-Mikko
Quoted 13 lines · Mikko Rapeli
Quoted 12 lines · Mikko Rapeli
Quoted 11 lines · On Wed, Feb 24, 2016 at 03:27:05PM +0000, [email protected] wrote:
From: Mariano Lopez <[email protected]>
This series add the cve-check-tool recipe, a tool used to identify potentially vulnerable software through version matching. It will check if a vulnerability has been addressed by a patch.
Also add the new cve-check class that will add a task for all recipes to check for CVEs using cve-check-tool. This tool can be used by recipe, image (will generate an image report in deploy dir), and with "world" and "universe"
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
To run it just inherit the class and enter:
bitbake -c
cve_check<recipe>I tried these on yocto/dizzy but:
Full changes needed in dizzy are:
bitbake/lib/bb/utils.py +18 -0
@@ -893,3 +893,21 @@ def multiprocessingpool(*args, **kwargs): return multiprocessing.Pool(*args, **kwargs) +# export common proxies variables from datastore to environment +def export_proxies(d): + import os + + variables = ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY', + 'ftp_proxy', 'FTP_PROXY', 'no_proxy', 'NO_PROXY'] + exported = False + + for v in variables: + if v in os.environ.keys(): + exported = True + else: + v_proxy = d.getVar(v, True) + if v_proxy is not None: + os.environ[v] = v_proxy + exported = True + + return exported
meta/recipes-gnome/hicolor-icon-theme/hicolor-icon-theme_0.13.bb +2 -0
@@ -21,3 +21,5 @@ FILES_${PN} += "${datadir}/icons" do_install_append () { install -m 0644 ${WORKDIR}/index.theme ${D}/${datadir}/icons/hicolor } + +BBCLASSEXTEND = "native"
meta/recipes-gnome/json-glib/json-glib_1.0.0.bb +2 -0
@@ -18,3 +18,5 @@ SRC_URI[archive.sha256sum] = "dbf558d2da989ab84a27e4e13daa51ceaa97eb959c2c2f8097 inherit gnome gettext lib_package EXTRA_OECONF = "--disable-introspection" + +BBCLASSEXTEND = "native"
And with this I get nice reports with "bitbake -c cve_check openssl" to tmp/deploy/cve/openssl.
I'll try with a full image build next, but I really, really like this stuff.
Thanks!
-Mikko
For openssh there must be some bugs or tunings needed to match the version numbers used in CVE to ones in yocto. openssh-6.6p1 has zero matches with the check but I think there are several:
downloads/CVE_CHECK$ grep openssh *xml| grep 6\.6\:p1
nvdcve-2.0-2016.xml: <cpe-lang:fact-ref name="cpe:/a:openbsd:openssh:6.6:p1"/> nvdcve-2.0-2016.xml: <vuln:product>cpe:/a:openbsd:openssh:6.6:p1</vuln:product> nvdcve-2.0-2016.xml: <cpe-lang:fact-ref name="cpe:/a:openbsd:openssh:6.6:p1"/> nvdcve-2.0-2016.xml: <vuln:product>cpe:/a:openbsd:openssh:6.6:p1</vuln:product>
How should these tunings be made?
-Mikko
Hi,
On my developer machine the cve-check ran ok for dizzy but on build server with sstate-cache and rmwork enabled it failed with what looks like a race condition when scanning the patch files:
17:45:36 ERROR: Error executing a python function in /home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>
17:45:36 0013: else:
17:45:36 0014: bb.note("Failed to update CVE database, skipping CVE check")
17:45:36 0015:
17:45:36 0016:
17:45:36 *** 0017:do_cve_check(d)
17:45:36 0018:
17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check
17:45:37 0004: Check recipe for patched and unpatched CVEs
17:45:37 0005: """
17:45:37 0006:
17:45:37 0007: if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)):
17:45:37 *** 0008: patched_cves = get_patches_cves(d)
17:45:37 0009: patched, unpatched = check_cves(d, patched_cves)
17:45:37 0010: if patched or unpatched:
17:45:37 0011: cve_data = get_cve_info(d, patched + unpatched)
17:45:37 0012: cve_write_data(d, patched, unpatched, cve_data)
17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves
17:45:37 0009: cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+")
17:45:37 0010: patched_cves = set()
17:45:37 0011: for url in src_patches(d):
17:45:37 0012: patch_file = bb.fetch.decodeurl(url)[2]
17:45:37 *** 0013: with open(patch_file, "r") as f:
17:45:37 0014: patch_text = f.read()
17:45:37 0015:
17:45:37 0016: # Search for the "CVE: " line
17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory: '/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed: do_cve_check
So could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
I could not see that kind of fixes in master.
-Mikko
Quoted 40 lines · On 02/26/2016 02:14 AM, [email protected] wrote:
Hi,
On my developer machine the cve-check ran ok for dizzy but on build server with sstate-cache and rmwork enabled it failed with what looks like a race condition when scanning the patch files:
17:45:36 ERROR: Error executing a python function in
/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>17:45:36 0013: else: 17:45:36 0014: bb.note("Failed to update CVE database, skipping CVE check") 17:45:36 0015: 17:45:36 0016: 17:45:36 *** 0017:do_cve_check(d) 17:45:36 0018: 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check 17:45:37 0004: Check recipe for patched and unpatched CVEs 17:45:37 0005: """ 17:45:37 0006: 17:45:37 0007: if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)): 17:45:37 *** 0008: patched_cves = get_patches_cves(d) 17:45:37 0009: patched, unpatched = check_cves(d, patched_cves) 17:45:37 0010: if patched or unpatched: 17:45:37 0011: cve_data = get_cve_info(d, patched + unpatched) 17:45:37 0012: cve_write_data(d, patched, unpatched, cve_data) 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves 17:45:37 0009: cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+") 17:45:37 0010: patched_cves = set() 17:45:37 0011: for url in src_patches(d): 17:45:37 0012: patch_file = bb.fetch.decodeurl(url)[2] 17:45:37 *** 0013: with open(patch_file, "r") as f: 17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory: '
/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_check
Quoted 6 lines
So could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
I could not see that kind of fixes in master.
-Mikko
The changes in patch series were minimal and actually this part of the code wasn't touched at all. That part of the code will look for all the files in the SRC_URI variable and will look for the "CVE:" tag in order to find patches that solve CVEs.
It seems the problem is with the bitbake fetcher, or the recipe; unfortunately the fetcher is one of the components that most change between releases. Another thing to check is that if actually there is a heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You can check this in the cve_check or patch log in the work directory of the recipe.
Mariano
Quoted 44 lines · Mariano Lopez
Quoted 43 lines · On 02/26/2016 02:14 AM, [email protected] wrote:
Hi,
On my developer machine the cve-check ran ok for dizzy but on build server with sstate-cache and rmwork enabled it failed with what looks like a race condition when scanning the patch files:
17:45:36 ERROR: Error executing a python function in
/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>17:45:36 0013: else: 17:45:36 0014: bb.note("Failed to update CVE database, skipping CVE check") 17:45:36 0015: 17:45:36 0016: 17:45:36 *** 0017:do_cve_check(d) 17:45:36 0018: 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check 17:45:37 0004: Check recipe for patched and unpatched CVEs 17:45:37 0005: """ 17:45:37 0006: 17:45:37 0007: if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)): 17:45:37 *** 0008: patched_cves = get_patches_cves(d) 17:45:37 0009: patched, unpatched = check_cves(d, patched_cves) 17:45:37 0010: if patched or unpatched: 17:45:37 0011: cve_data = get_cve_info(d, patched + unpatched) 17:45:37 0012: cve_write_data(d, patched, unpatched, cve_data) 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves 17:45:37 0009: cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+") 17:45:37 0010: patched_cves = set() 17:45:37 0011: for url in src_patches(d): 17:45:37 0012: patch_file = bb.fetch.decodeurl(url)[2] 17:45:37 *** 0013: with open(patch_file, "r") as f: 17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory: '
/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_checkSo could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Quoted 8 lines
Quoted 3 lines
I could not see that kind of fixes in master.
-Mikko
The changes in patch series were minimal and actually this part of the code wasn't touched at all. That part of the code will look for all the files in the
SRC_URIvariable and will look for the "CVE:" tag in order to find patches that solve CVEs.
Yep, the code seems straight forward.
Quoted 6 lines
It seems the problem is with the bitbake fetcher, or the recipe; unfortunately the fetcher is one of the components that most change between releases. Another thing to check is that if actually there is a heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You can check this in the
cve_checkor patch log in the work directory of the recipe.
Unfortunately the file is there if I check with devshell but I have now four different CI runs with this failure. Only difference to my developer machine is sstate cache. Build machines maintain their own sstate cache.
-Mikko
Quoted 63 lines · Mikko Rapeli
Quoted 44 lines · Mariano Lopez
Quoted 43 lines · On 02/26/2016 02:14 AM, [email protected] wrote:
Hi,
On my developer machine the cve-check ran ok for dizzy but on build server with sstate-cache and rmwork enabled it failed with what looks like a race condition when scanning the patch files:
17:45:36 ERROR: Error executing a python function in
/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>17:45:36 0013: else: 17:45:36 0014: bb.note("Failed to update CVE database, skipping CVE check") 17:45:36 0015: 17:45:36 0016: 17:45:36 *** 0017:do_cve_check(d) 17:45:36 0018: 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check 17:45:37 0004: Check recipe for patched and unpatched CVEs 17:45:37 0005: """ 17:45:37 0006: 17:45:37 0007: if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)): 17:45:37 *** 0008: patched_cves = get_patches_cves(d) 17:45:37 0009: patched, unpatched = check_cves(d, patched_cves) 17:45:37 0010: if patched or unpatched: 17:45:37 0011: cve_data = get_cve_info(d, patched + unpatched) 17:45:37 0012: cve_write_data(d, patched, unpatched, cve_data) 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves 17:45:37 0009: cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+") 17:45:37 0010: patched_cves = set() 17:45:37 0011: for url in src_patches(d): 17:45:37 0012: patch_file = bb.fetch.decodeurl(url)[2] 17:45:37 *** 0013: with open(patch_file, "r") as f: 17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory: '
/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_checkSo could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Quoted 8 lines
Quoted 3 lines
I could not see that kind of fixes in master.
-Mikko
The changes in patch series were minimal and actually this part of the code wasn't touched at all. That part of the code will look for all the files in the
SRC_URIvariable and will look for the "CVE:" tag in order to find patches that solve CVEs.Yep, the code seems straight forward.
Quoted 6 lines
It seems the problem is with the bitbake fetcher, or the recipe; unfortunately the fetcher is one of the components that most change between releases. Another thing to check is that if actually there is a heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You can check this in the
cve_checkor patch log in the work directory of the recipe.
Quoted 3 lines
Unfortunately the file is there if I check with devshell but I have now four different CI runs with this failure. Only difference to my developer machine is sstate cache. Build machines maintain their own sstate cache.
Last two runs were with v2 patches.
-Mikko
Quoted 61 lines · On 02/26/2016 08:57 AM, [email protected] wrote:
Quoted 60 lines · Mikko Rapeli
Quoted 44 lines · Mariano Lopez
Quoted 43 lines · On 02/26/2016 02:14 AM, [email protected] wrote:
Hi,
On my developer machine the cve-check ran ok for dizzy but on build server with sstate-cache and rmwork enabled it failed with what looks like a race condition when scanning the patch files:
17:45:36 ERROR: Error executing a python function in
/home/builder/src/base/poky/meta/recipes-extended/mailx/mailx_12.5.bb:
17:45:36
17:45:36 The stack trace of python calls that resulted in this exception/failure was:
17:45:36 File: 'do_cve_check', lineno: 17, function: <module>17:45:36 0013: else: 17:45:36 0014: bb.note("Failed to update CVE database, skipping CVE check") 17:45:36 0015: 17:45:36 0016: 17:45:36 *** 0017:do_cve_check(d) 17:45:36 0018: 17:45:37 File: 'do_cve_check', lineno: 8, function: do_cve_check 17:45:37 0004: Check recipe for patched and unpatched CVEs 17:45:37 0005: """ 17:45:37 0006: 17:45:37 0007: if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE", True)): 17:45:37 *** 0008: patched_cves = get_patches_cves(d) 17:45:37 0009: patched, unpatched = check_cves(d, patched_cves) 17:45:37 0010: if patched or unpatched: 17:45:37 0011: cve_data = get_cve_info(d, patched + unpatched) 17:45:37 0012: cve_write_data(d, patched, unpatched, cve_data) 17:45:37 File: 'cve-check.bbclass', lineno: 13, function: get_patches_cves 17:45:37 0009: cve_match = re.compile("CVE:( CVE\-\d+\-\d+)+") 17:45:37 0010: patched_cves = set() 17:45:37 0011: for url in src_patches(d): 17:45:37 0012: patch_file = bb.fetch.decodeurl(url)[2] 17:45:37 *** 0013: with open(patch_file, "r") as f: 17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory: '
/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_checkSo could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Quoted 7 lines
Quoted 3 lines
I could not see that kind of fixes in master.
-Mikko
The changes in patch series were minimal and actually this part of the code wasn't touched at all. That part of the code will look for all the files in the
SRC_URIvariable and will look for the "CVE:" tag in order to find patches that solve CVEs.Yep, the code seems straight forward.
Quoted 6 lines
It seems the problem is with the bitbake fetcher, or the recipe; unfortunately the fetcher is one of the components that most change between releases. Another thing to check is that if actually there is a heirloom-mailx_12.5-1.diff file in the paths that the fetcher look for. You can check this in the
cve_checkor patch log in the work directory of the recipe.
Quoted 4 lines
Quoted 3 lines
Unfortunately the file is there if I check with devshell but I have now four different CI runs with this failure. Only difference to my developer machine is sstate cache. Build machines maintain their own sstate cache.
Last two runs were with v2 patches.
Would be possible to run these CI with master to check if you see the error too?
Also, what you can do is to put try: except:, but this won't solve the problem, just will hide it so the build can finish.
Quoted 2 lines
-Mikko
Mariano Lopez
Quoted 5 lines · On 26 February 2016 at 08:14, <[email protected]> wrote:
17:45:37 *** 0013: with
open(patch_file, "r")as f:17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)
Quoted 8 lines
17:45:37 Exception: IOError: [Errno 2] No such file or directory:
'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_checkSo could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Do you have rm_work enabled?
Ross
Quoted 12 lines · Ross
Quoted 10 lines · On 26 February 2016 at 08:14, <[email protected]> wrote:
17:45:37 *** 0013: with
open(patch_file, "r")as f:17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory:
'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_check
Quoted 5 lines
Quoted 3 lines
So could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Do you have
rm_workenabled?
Yes.
-Mikko
Quoted 13 lines · On 02/29/2016 08:19 AM, [email protected] wrote:
Quoted 12 lines · Ross
Quoted 10 lines · On 26 February 2016 at 08:14, <[email protected]> wrote:
17:45:37 *** 0013: with
open(patch_file, "r")as f:17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory:
'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_check
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
So could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Do you have
rm_workenabled?Yes.
-Mikko
I think I have found the problem, when you do devshell it will execute do_unpack and the cve_check task must run after that for some recipes. Try this:
addtask cve_check after do_unpack before do_build
Sorry, to no include a diff, the diff is way bigger than just this line at the moment.
Mariano
Quoted 22 lines · Mariano Lopez
Quoted 19 lines · On 02/29/2016 08:19 AM, [email protected] wrote:
Quoted 12 lines · Ross
Quoted 10 lines · On 26 February 2016 at 08:14, <[email protected]> wrote:
17:45:37 *** 0013: with
open(patch_file, "r")as f:17:45:37 0014: patch_text = f.read() 17:45:37 0015: 17:45:37 0016: # Search for the "CVE: " line 17:45:37 0017: match = cve_match.search(patch_text)17:45:37 Exception: IOError: [Errno 2] No such file or directory:
'/home/builder/src/base/build/tmp/work/corei7-64-linux/mailx/12.5-r2/heirloom-mailx_12.5-1.diff' 17:45:37
17:45:37 ERROR: Function failed:do_cve_checkQuoted 4 lines
Quoted 3 lines
So could this be caused by cve-check changes or is this just a side effect of some other recipe problems?
Do you have
rm_workenabled?Yes.
Quoted 7 lines
Quoted 1 line
-Mikko
I think I have found the problem, when you do devshell it will execute
do_unpackand thecve_checktask must run after that for some recipes. Try this:addtask
cve_checkafterdo_unpackbeforedo_build
Thanks, with this change the scan builds pass on dizzy.
-Mikko
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · akuster
So my questions in the bugs go unanswered. If you don't ever intend on
responding just say so, I do appreciate honestly. I then know where I stand.- armin
Sorry, for not answered the bug earlier, to be honest I hadn't seen your comment earlier.
I just replied to the bug.
Mariano
Quoted 39 lines · On 02/24/2016 07:27 AM, [email protected] wrote:
Quoted 37 lines
From: Mariano Lopez <[email protected]>
This series add the cve-check-tool recipe, a tool used to identify potentially vulnerable software through version matching. It will check if a vulnerability has been addressed by a patch.
Also add the new cve-check class that will add a task for all recipes to check for CVEs using cve-check-tool. This tool can be used by recipe, image (will generate an image report in deploy dir), and with "world" and "universe"
To run it just inherit the class and enter:
bitbake -c
cve_check<recipe>The following changes since commit
23056103c949b498c23b47579e8dd57ce78e6ed9:uclibc: Do not use immediate expansion operator (2016-02-22 20:42:48 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contribmariano/bug7515
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug7515Mariano Lopez (3):
cve-check-tool: Add recipe
cve-check-tool patch to allow select dir for the db cve-check.bbclass: Add classmeta/classes/cve-check.bbclass | 229 +++++++++++++++++++++ .../change_logic_cve_get_file_parent.patch | 45 ++++ .../cve-check-tool/cve-check-tool_5.6.bb | 61 ++++++ 3 files changed, 335 insertions(+)create mode 100644 meta/classes/cve-check.bbclass
create mode 100644meta/recipes-devtools/cve-check-tool/cve-check-tool/change_logic_cve_get_file_parent.patchcreate mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.bb
Signature · 1 line
Mariano Lopez