cve-check: Depends on cve-update-db-native
PATCH 1/3 cve-check: Depends on cve-update-db-native
From: Pierre Le Magourou <[email protected]>
do_populate_cve_db is a native task.
--- meta/classes/cve-check.bbclass | 2 +- meta/recipes-core/meta/cve-update-db.bb | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
meta/classes/cve-check.bbclass +1 -1
@@ -63,7 +63,7 @@ python do_cve_check () { } addtask cve_check after do_unpack before do_build -do_cve_check[depends] = "cve-update-db:do_populate_cve_db" +do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" python cve_check_cleanup () {
meta/recipes-core/meta/cve-update-db.bb +1 -0
@@ -141,3 +141,4 @@ addtask do_populate_cve_db before do_fetch do_populate_cve_db[nostamp] = "1" EXCLUDE_FROM_WORLD = "1" +BBCLASSEXTEND =+ "native"
Signature · 1 line
2.11.0
PATCH 2/3 cve-update-db: Use NVD CPE data to populate PRODUCTS table
From: Pierre Le Magourou <[email protected]>
Instead of using expanded list of affected versions that is not reliable, use the 'cpe_match' node in the 'configurations' json node.
For cve-check to correctly match affected CVE, the sqlite database need to contain operator_start, operator_end and the corresponding versions fields.
--- meta/recipes-core/meta/cve-update-db.bb | 88 +++++++++++++++++++++++++++------ 1 file changed, 74 insertions(+), 14 deletions(-)
meta/recipes-core/meta/cve-update-db.bb +74 -14
@@ -25,7 +25,7 @@ python do_populate_cve_db() { YEAR_START = 2002 db_dir = d.getVar("DL_DIR") + '/CVE_CHECK' - db_file = db_dir + '/nvd-json.db' + db_file = db_dir + '/nvdcve.db' json_tmpfile = db_dir + '/nvd.json.gz' proxy = d.getVar("https_proxy") cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a')
@@ -99,9 +99,76 @@ def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \ - VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)") - c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \ - (PRODUCT, VERSION)") + VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ + VERSION_END TEXT, OPERATOR_END TEXT)") + +def insert_elt(c, db_values): + product_str = db_values[0] + db_values[1] + db_values[2] + db_values[3] + hashstr = hash_djb2(product_str) + db_values.insert(0, hashstr) + query = "insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?, ?, ?)" + c.execute(query, db_values) + +def parse_node_and_insert(c, node, cveId): + # Parse children node if needed + try: + for child in node['children']: + parse_node_and_insert(c, child, cveId) + except: + pass + + # Exit if the cpe_match node does not exists + try: + cpe_match = node['cpe_match'] + except: + return + + for cpe in cpe_match: + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + db_values = [cveId, vendor, product, version, '=', '', ''] + insert_elt(c, db_values) + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + try: + if cpe['versionStartIncluding']: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + except: + pass + try: + if cpe['versionStartExcluding']: + op_start = '>' + v_start = cpe['versionStartExcluding'] + except: + pass + try: + if cpe['versionEndIncluding']: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + except: + pass + try: + if cpe['versionEndExcluding']: + op_end = '<' + v_end = cpe['versionEndExcluding'] + except: + pass + + db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] + insert_elt(c, db_values) def update_db(c, json_filename): import json
@@ -125,16 +192,9 @@ def update_db(c, json_filename): c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]) - for vendor in elt['cve']['affects']['vendor']['vendor_data']: - for product in vendor['product']['product_data']: - for version in product['version']['version_data']: - product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value'] - hashstr = hash_djb2(product_str) - c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)", - [ hashstr, cveId, vendor['vendor_name'], - product['product_name'], version['version_value'], - version['version_affected']]) - + configurations = elt['configurations']['nodes'] + for config in configurations: + parse_node_and_insert(c, config, cveId) addtask do_populate_cve_db before do_fetch
Signature · 1 line
2.11.0
PATCH 3/3 cve-check: Update unpatched CVE matching
In reply to Pierre Le MagourouFrom: Pierre Le Magourou <[email protected]>
Now that cve-update-db added CPE information to NVD database. We can check for unpatched versions with operators '<', '<=', '>', and '>='.
--- meta/classes/cve-check.bbclass | 54 +++++++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 14 deletions(-)
meta/classes/cve-check.bbclass +40 -14
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -189,27 +189,53 @@ def check_cves(d, patched_cves): conn = sqlite3.connect(db_file) c = conn.cursor() - query = """SELECT * FROM PRODUCTS WHERE - (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR - (PRODUCT IS '{0}' AND OPERATOR IS '<=');""" + query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '{0}';" + for product in products: for row in c.execute(query.format(product, pv)): cve = row[1] - version = row[4] - - try: - discardVersion = LooseVersion(version) < LooseVersion(pv) - except: - discardVersion = True + version_start = row[4] + operator_start = row[5] + version_end = row[6] + operator_end = row[7] if pv in cve_whitelist.get(cve, []): bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - elif discardVersion: - bb.debug(2, "Do not consider version %s " % (version)) else: - cves_unpatched.append(cve) + if (operator_start == '=' and pv == version_start): + cves_unpatched.append(cve) + else: + if operator_start: + try: + to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_start, version_start, cve)) + to_append_start = False + else: + to_append_start = False + + if operator_end: + try: + to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_end, version_end, cve)) + to_append_end = False + else: + to_append_end = False + + if operator_start and operator_end: + to_append = to_append_start and to_append_end + else: + to_append = to_append_start or to_append_end + + if to_append: + cves_unpatched.append(cve) bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) conn.close()
@@ -217,7 +243,7 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ - Get CVE information from the database used by cve-check-tool. + Get CVE information from the database. Unfortunately the only way to get CVE info is set the output to html (hard to parse) or query directly the database.
Signature · 1 line
2.11.0
✗ patchtest: failure for "[meta-oe] cve-check: Depends o..." and 2 more
In reply to Pierre Le Magourou== Series Details ==
Series: "[meta-oe] cve-check: Depends o..." and 2 more
Revision: 1
URL : https://patchwork.openembedded.org/series/18529/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Patch [meta-oe,1/3] cve-check: Depends on cve-update-db-native Issue Series sent to the wrong mailing list [test_target_mailing_list] Suggested fix Check the project's README (meta-oe,1/3) and send the patch to the indicated list
* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at a0a96db236)
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
Quoted 10 lines · Pierre Le Magourou
From: Pierre Le Magourou <[email protected]>
do_populate_cve_dbis a native task.Signed-off-by: Pierre Le Magourou <
[email protected]>--- meta/classes/cve-check.bbclass | 2 +- meta/recipes-core/meta/cve-update-db.bb | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/1015
:(
Cheers,
Richard
Hello,
Quoted 12 lines
Quoted 10 lines
From: Pierre Le Magourou <[email protected]>
do_populate_cve_dbis a native task.Signed-off-by: Pierre Le Magourou <
[email protected]>--- meta/classes/cve-check.bbclass | 2 +- meta/recipes-core/meta/cve-update-db.bb | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
The autobuilder job launches cve-update-db and cve-update-db-native in parallel, this cannot work because they both try to update the same sqlite db.
As cve-update-db is only used for the build machine (native only), we only need a cve-update-db-native.bb recipe. I'll send a v2 patch to fix that.
ps: Sorry I just realized I was sending the patches to the wrong destination (meta-oe instead of oe-core)
Pierre