Skip to contentSkip to conversation

cve-update-db: New recipe to update CVE database

PATCH In review Pierre Le Magourou 6 participants 21 messages Jun 19, 2019 – Jun 27, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/4 cve-update-db: New recipe to update CVE database

From: Pierre Le Magourou <[email protected]>

cve-check-tool-native do_populate_cve_db task was using deprecated NVD xml data feeds, cve-update-db uses NVD json data feeds.

Sqlite database schema was updated to take into account CVSSv3 CVE scores and operator in affected product versions.
A new META table was added to store the last modification date of the NVD json data feeds.

---
 meta/recipes-core/meta/cve-update-db.bb | 121 ++++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 meta/recipes-core/meta/cve-update-db.bb
meta/recipes-core/meta/cve-update-db.bb new file +121 -0
new file mode 100644
index 0000000000..522fd23807
@@ -0,0 +1,121 @@
+SUMMARY = "Updates the NVD CVE database"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+PACKAGES = ""
+
+inherit nopackages
+
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+deltask do_populate_sysroot
+
+python do_populate_cve_db() {
+    """
+    Update NVD database with json data feed
+    """
+
+    import sqlite3, urllib3, shutil, gzip, re
+    from datetime import date
+
+    BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
+    YEAR_START = 2002
+    JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz'
+
+    # Connect to database
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    conn = sqlite3.connect(db_file)
+    c = conn.cursor()
+
+    initialize_db(c)
+
+    http = urllib3.PoolManager()
+
+    for year in range(YEAR_START, date.today().year + 1):
+        year_url = BASE_URL + str(year)
+        meta_url = year_url + ".meta"
+        json_url = year_url + ".json.gz"
+
+        # Retrieve meta last modified date
+        with http.request('GET', meta_url, preload_content=False) as r:
+            date_line = str(r.data.splitlines()[0])
+            last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1)
+
+        # Compare with current db last modified date
+        c.execute("select DATE from META where YEAR = '%d'" % year)
+        meta = c.fetchone()
+        if not meta or meta[0] != last_modified:
+            # Update db with current year json file
+            with http.request('GET', json_url, preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile:
+                shutil.copyfileobj(r, tmpfile)
+            with gzip.open(JSON_TMPFILE, 'rt') as jsonfile:
+                update_db(c, jsonfile)
+            c.execute("insert or replace into META values (?, ?)",
+                    [year, last_modified])
+
+    conn.commit()
+    conn.close()
+
+    with open(d.getVar("CVE_CHECK_TMP_FILE"), 'a'):
+        os.utime(d.getVar("CVE_CHECK_TMP_FILE"), None)
+}
+
+# DJB2 hash algorithm
+def hash_djb2(s):
+    hash = 5381
+    for x in s:
+        hash = (( hash << 5) + hash) + ord(x)
+
+    return hash & 0xFFFFFFFF
+
+def initialize_db(c):
+    c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
+        SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
+    c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \
+        VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)")
+    c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \
+        (PRODUCT, VERSION)")
+
+def update_db(c, json_filename):
+    import json
+    root = json.load(json_filename)
+
+    for elt in root['CVE_Items']:
+        if not elt['impact']:
+            continue
+
+        cveId = elt['cve']['CVE_data_meta']['ID']
+        cveDesc = elt['cve']['description']['description_data'][0]['value']
+        date = elt['lastModifiedDate']
+        accessVector = elt['impact']['baseMetricV2']['cvssV2']['accessVector']
+        cvssv2 = elt['impact']['baseMetricV2']['cvssV2']['baseScore']
+
+        try:
+            cvssv3 = elt['impact']['baseMetricV3']['cvssV3']['baseScore']
+        except:
+            cvssv3 = 0.0
+
+        c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)",
+                [cveId, cveDesc, cvssv2, cvssv3, date, accessVector])
+
+        for vendor in elt['cve']['affects']['vendor']['vendor_data']:
+            for product in vendor['product']['product_data']:
+                for version in product['version']['version_data']:
+                    product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value']
+                    hashstr = hash_djb2(product_str)
+                    c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)",
+                            [ hashstr, cveId, vendor['vendor_name'],
+                                product['product_name'], version['version_value'],
+                                version['version_affected']])
+
+
+
+addtask do_populate_cve_db before do_cve_check
+do_populate_cve_db[nostamp] = "1"
+
+EXCLUDE_FROM_WORLD = "1"
Signature · 1 line
2.11.0
#1 lore raw

PATCH 2/4 cve-check: Remove dependency to cve-check-tool-native

From: Pierre Le Magourou <[email protected]>

Use the new update-cve-db recipe to update database.

---
 meta/classes/cve-check.bbclass | 71 ++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 45 deletions(-)
meta/classes/cve-check.bbclass +26 -45
index 743bc08a4f..28619c7bd4 100644
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
 CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
-CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db"
 
 CVE_CHECK_LOG ?= "${T}/cve.log"
 CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -62,7 +62,7 @@ python do_cve_check () {
 }
 
 addtask cve_check after do_unpack before do_build
-do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db"
+do_cve_check[depends] = "cve-update-db:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
 python cve_check_cleanup () {
@@ -163,61 +163,40 @@ def get_patches_cves(d):
 
 def check_cves(d, patched_cves):
     """
-    Run cve-check-tool looking for patched and unpatched CVEs.
+    Connect to the NVD database and find unpatched cves.
     """
-
     import ast, csv, tempfile, subprocess, io
 
-    cves_patched = []
     cves_unpatched = []
     bpn = d.getVar("CVE_PRODUCT")
     # If this has been unset then we're not scanning for CVEs here (for example, image recipes)
     if not bpn:
         return ([], [])
     pv = d.getVar("CVE_VERSION").split("+git")[0]
-    cves = " ".join(patched_cves)
-    cve_db_dir = d.getVar("CVE_CHECK_DB_DIR")
     cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
-    cve_cmd = "cve-check-tool"
-    cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir]
 
     # If the recipe has been whitlisted we return empty lists
     if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split():
         bb.note("Recipe has been whitelisted, skipping check")
         return ([], [])
 
-    try:
-        # Write the faux CSV file to be used with cve-check-tool
-        fd, faux = tempfile.mkstemp(prefix="cve-faux-")
-        with os.fdopen(fd, "w") as f:
-            for pn in bpn.split():
-                f.write("%s,%s,%s,\n" % (pn, pv, cves))
-        cmd.append(faux)
-
-        output = subprocess.check_output(cmd).decode("utf-8")
-        bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output))
-    except subprocess.CalledProcessError as e:
-        bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
-    finally:
-        os.remove(faux)
-
-    for row in csv.reader(io.StringIO(output)):
-        # Third row has the unpatched CVEs
-        if row[2]:
-            for cve in row[2].split():
-                # Skip if the CVE has been whitlisted for the current version
-                if pv in cve_whitelist.get(cve,[]):
-                    bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
-                else:
-                    cves_unpatched.append(cve)
-                    bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
-        # Fourth row has patched CVEs
-        if row[3]:
-            for cve in row[3].split():
-                cves_patched.append(cve)
-                bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve))
-
-    return (cves_patched, cves_unpatched)
+    import sqlite3
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    conn = sqlite3.connect(db_file)
+    c = conn.cursor()
+    query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';"
+    for row in c.execute(query % (bpn,pv)):
+        cve = row[1]
+        if pv in cve_whitelist.get(cve,[]):
+            bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
+        elif cve in patched_cves:
+            bb.note("%s has been patched" % (cve))
+        else:
+            cves_unpatched.append(cve)
+            bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
+    conn.close()
+
+    return (list(patched_cves), cves_unpatched)
 
 def get_cve_info(d, cves):
     """
@@ -241,9 +220,10 @@ def get_cve_info(d, cves):
     for row in cur.execute(query, tuple(cves)):
         cve_data[row[0]] = {}
         cve_data[row[0]]["summary"] = row[1]
-        cve_data[row[0]]["score"] = row[2]
-        cve_data[row[0]]["modified"] = row[3]
-        cve_data[row[0]]["vector"] = row[4]
+        cve_data[row[0]]["scorev2"] = row[2]
+        cve_data[row[0]]["scorev3"] = row[3]
+        cve_data[row[0]]["modified"] = row[4]
+        cve_data[row[0]]["vector"] = row[5]
     conn.close()
 
     return cve_data
@@ -270,7 +250,8 @@ def cve_write_data(d, patched, unpatched, cve_data):
             unpatched_cves.append(cve)
             write_string += "CVE STATUS: Unpatched\n"
         write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
-        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"]
+        write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"]
         write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
         write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
 
Signature · 1 line
2.11.0
#2 lore raw

Re: [PATCH 2/4] cve-check: Remove dependency to cve-check-tool-native

Does this mean we can delete the cve-check-tool recipe itself too?

Ross

Quoted 144 lines · Pierre Le Magourou

From: Pierre Le Magourou <[email protected]>

Use the new update-cve-db recipe to update database.

---
 meta/classes/cve-check.bbclass | 71 ++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 45 deletions(-)
meta/classes/cve-check.bbclass +26 -46
index 743bc08a4f..28619c7bd4 100644
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}"
 CVE_VERSION ??= "${PV}"
 
 CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
-CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"
+CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db"
 
 CVE_CHECK_LOG ?= "${T}/cve.log"
 CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -62,7 +62,7 @@ python do_cve_check () {
 }
 
 addtask cve_check after do_unpack before do_build
-do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db"
+do_cve_check[depends] = "cve-update-db:do_populate_cve_db"
 do_cve_check[nostamp] = "1"
 
 python cve_check_cleanup () {
@@ -163,61 +163,40 @@ def get_patches_cves(d):
 
 def check_cves(d, patched_cves):
     """
-    Run cve-check-tool looking for patched and unpatched CVEs.
+    Connect to the NVD database and find unpatched cves.
     """
-
     import ast, csv, tempfile, subprocess, io
 
-    cves_patched = []
     cves_unpatched = []
     bpn = d.getVar("CVE_PRODUCT")
     # If this has been unset then we're not scanning for CVEs here (for example, image recipes)
     if not bpn:
         return ([], [])
     pv = d.getVar("CVE_VERSION").split("+git")[0]
-    cves = " ".join(patched_cves)
-    cve_db_dir = d.getVar("CVE_CHECK_DB_DIR")
     cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
-    cve_cmd = "cve-check-tool"
-    cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir]
 
     # If the recipe has been whitlisted we return empty lists
     if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split():
         bb.note("Recipe has been whitelisted, skipping check")
         return ([], [])
 
-    try:
-        # Write the faux CSV file to be used with cve-check-tool
-        fd, faux = tempfile.mkstemp(prefix="cve-faux-")
-        with os.fdopen(fd, "w") as f:
-            for pn in bpn.split():
-                f.write("%s,%s,%s,\n" % (pn, pv, cves))
-        cmd.append(faux)
-
-        output = subprocess.check_output(cmd).decode("utf-8")
-        bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output))
-    except subprocess.CalledProcessError as e:
-        bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output))
-    finally:
-        os.remove(faux)
-
-    for row in csv.reader(io.StringIO(output)):
-        # Third row has the unpatched CVEs
-        if row[2]:
-            for cve in row[2].split():
-                # Skip if the CVE has been whitlisted for the current version
-                if pv in cve_whitelist.get(cve,[]):
-                    bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
-                else:
-                    cves_unpatched.append(cve)
-                    bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
-        # Fourth row has patched CVEs
-        if row[3]:
-            for cve in row[3].split():
-                cves_patched.append(cve)
-                bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve))
-
-    return (cves_patched, cves_unpatched)
+    import sqlite3
+    db_file = d.getVar("CVE_CHECK_DB_FILE")
+    conn = sqlite3.connect(db_file)
+    c = conn.cursor()
+    query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';"
+    for row in c.execute(query % (bpn,pv)):
+        cve = row[1]
+        if pv in cve_whitelist.get(cve,[]):
+            bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
+        elif cve in patched_cves:
+            bb.note("%s has been patched" % (cve))
+        else:
+            cves_unpatched.append(cve)
+            bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
+    conn.close()
+
+    return (list(patched_cves), cves_unpatched)
 
 def get_cve_info(d, cves):
     """
@@ -241,9 +220,10 @@ def get_cve_info(d, cves):
     for row in cur.execute(query, tuple(cves)):
         cve_data[row[0]] = {}
         cve_data[row[0]]["summary"] = row[1]
-        cve_data[row[0]]["score"] = row[2]
-        cve_data[row[0]]["modified"] = row[3]
-        cve_data[row[0]]["vector"] = row[4]
+        cve_data[row[0]]["scorev2"] = row[2]
+        cve_data[row[0]]["scorev3"] = row[3]
+        cve_data[row[0]]["modified"] = row[4]
+        cve_data[row[0]]["vector"] = row[5]
     conn.close()
 
     return cve_data
@@ -270,7 +250,8 @@ def cve_write_data(d, patched, unpatched, cve_data):
             unpatched_cves.append(cve)
             write_string += "CVE STATUS: Unpatched\n"
         write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"]
-        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"]
+        write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"]
+        write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"]
         write_string += "VECTOR: %s\n" % cve_data[cve]["vector"]
         write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
 
--

2.11.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw

Re: [PATCH 2/4] cve-check: Remove dependency to cve-check-tool-native

Quoted 2 lines

Does this mean we can delete the cve-check-tool recipe itself too?

Yes, with this patch cve-check class does not need cve-check-tool anymore.

Pierre

#4 lore raw

PATCH 3/4 cve-check: Manage CVE_PRODUCT with more than one name

In reply to Pierre Le Magourou

From: Pierre Le Magourou <[email protected]>

In some rare cases (eg. curl recipe) the CVE_PRODUCT contains more than one name.

---
 meta/classes/cve-check.bbclass | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)
meta/classes/cve-check.bbclass +14 -11
index 28619c7bd4..e7540b8c1f 100644
@@ -168,9 +168,10 @@ def check_cves(d, patched_cves):
     import ast, csv, tempfile, subprocess, io
 
     cves_unpatched = []
-    bpn = d.getVar("CVE_PRODUCT")
+    # CVE_PRODUCT can contain more than one product (eg. curl/libcurl)
+    bpn = d.getVar("CVE_PRODUCT").split()
     # If this has been unset then we're not scanning for CVEs here (for example, image recipes)
-    if not bpn:
+    if len(bpn) == 0:
         return ([], [])
     pv = d.getVar("CVE_VERSION").split("+git")[0]
     cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
@@ -184,16 +185,18 @@ def check_cves(d, patched_cves):
     db_file = d.getVar("CVE_CHECK_DB_FILE")
     conn = sqlite3.connect(db_file)
     c = conn.cursor()
+
     query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';"
-    for row in c.execute(query % (bpn,pv)):
-        cve = row[1]
-        if pv in cve_whitelist.get(cve,[]):
-            bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve))
-        elif cve in patched_cves:
-            bb.note("%s has been patched" % (cve))
-        else:
-            cves_unpatched.append(cve)
-            bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve))
+    for idx in range(len(bpn)):
+        for row in c.execute(query % (bpn[idx],pv)):
+            cve = row[1]
+            if pv in cve_whitelist.get(cve,[]):
+                bb.note("%s-%s has been whitelisted for %s" % (bpn[idx], pv, cve))
+            elif cve in patched_cves:
+                bb.note("%s has been patched" % (cve))
+            else:
+                cves_unpatched.append(cve)
+                bb.debug(2, "%s-%s is not patched for %s" % (bpn[idx], pv, cve))
     conn.close()
 
     return (list(patched_cves), cves_unpatched)
Signature · 1 line
2.11.0
#5 lore raw

PATCH 4/4 cve-check: Consider CVE that affects versions with less than operator

In reply to Pierre Le Magourou

From: Pierre Le Magourou <[email protected]>

In the NVD json CVE feed, affected versions can be strictly matched to a version, but they can also be matched with the operator '<='.

Add a new condition in the sqlite query to match affected versions that are defined with the operator '<='. Then use LooseVersion to discard all versions that are not relevant.

---
 meta/classes/cve-check.bbclass | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
meta/classes/cve-check.bbclass +14 -2
index e7540b8c1f..379f7121cc 100644
@@ -166,6 +166,7 @@ def check_cves(d, patched_cves):
     Connect to the NVD database and find unpatched cves.
     """
     import ast, csv, tempfile, subprocess, io
+    from distutils.version import LooseVersion
 
     cves_unpatched = []
     # CVE_PRODUCT can contain more than one product (eg. curl/libcurl)
@@ -186,14 +187,25 @@ def check_cves(d, patched_cves):
     conn = sqlite3.connect(db_file)
     c = conn.cursor()
 
-    query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';"
+    query = """SELECT * FROM PRODUCTS WHERE
+               (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR
+               (PRODUCT IS '{0}' AND OPERATOR IS '<=');"""
     for idx in range(len(bpn)):
-        for row in c.execute(query % (bpn[idx],pv)):
+        for row in c.execute(query.format(bpn[idx],pv)):
             cve = row[1]
+            version = row[4]
+
+            try:
+                discardVersion = LooseVersion(version) < LooseVersion(pv)
+            except:
+                discardVersion = True
+
             if pv in cve_whitelist.get(cve,[]):
                 bb.note("%s-%s has been whitelisted for %s" % (bpn[idx], pv, cve))
             elif cve in patched_cves:
                 bb.note("%s has been patched" % (cve))
+            elif discardVersion:
+                bb.debug(2, "Do not consider version %s " % (version))
             else:
                 cves_unpatched.append(cve)
                 bb.debug(2, "%s-%s is not patched for %s" % (bpn[idx], pv, cve))
Signature · 1 line
2.11.0
#6 lore raw
In reply to Pierre Le Magourou

Not sure which of the changes is responsible, but this is new:
WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE (CVE-2015-1773)

https://nvd.nist.gov/vuln/detail/CVE-2015-1773

Note that the flex tool is completely unrelated to Apache Flex.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#7 lore raw
Quoted 9 lines

Not sure which of the changes is responsible, but this is new:
WARNING: flex-native-2.6.0-r0 do_cve_check: Found unpatched CVE (CVE-2015-1773)

https://nvd.nist.gov/vuln/detail/CVE-2015-1773

Note that the flex tool is completely unrelated to Apache Flex.

I see, the 4/4 patch is responsible for that (Consider CVE that affects versions with less than operator). It takes into account the comparison operator in the json NVD file (new 'version_affected' field that was not in the XML data feed). So this CVE matches because 2.6.0 <= 4.14.0. But it should not match because it concerns another product (flex_project/flex vs Apache/flex).

There is indeed a problem I didn't manage. The CVE_PRODUCT variable we use in cve-check only takes the product name (here 'flex') into account, we should also consider the vendor name (here 'flex_project').

Without this patch (4/4), the behaviour should be the same as before.

Pierre

#9 lore raw
Quoted 9 lines · On Fri, 21 Jun 2019 at 13:11, <[email protected]> wrote:

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

On my Debian 9 build container python3-urllib3 wasn't installed by default or via other dependencies.

Should this rather be provided via python3-native?

Alex

#10 lore raw
Quoted 6 lines · Alexander Kanavin
Quoted 4 lines · On Fri, 21 Jun 2019 at 13:11, <[email protected]> wrote:

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.

Quoted 7 lines
Quoted 5 lines

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

On my Debian 9 build container python3-urllib3 wasn't installed by default or via other dependencies.

Should this rather be provided via python3-native?

Hmm, possibly? I cherry-picked the patches to sumo and saw this missing dependency in my container.

Did poky master switch from using host python to native after sumo?

-Mikko

#11 lore raw
Quoted 6 lines · On Fri, 21 Jun 2019 at 13:48, <[email protected]> wrote:

Hmm, possibly? I cherry-picked the patches to sumo and saw this missing dependency in my container.

Did poky master switch from using host python to native after sumo?

poky uses host python for some things and native python for other things. Generally where 'core python' is enough then we use the host python, but when additional libraries are needed, then native python plus those libraries is a better choice, as this doesn't require adding to the required host packages list, and allows better control over how they're built.

Alex

#12 lore raw
Quoted 9 lines · Alexander Kanavin
Quoted 6 lines · On Fri, 21 Jun 2019 at 13:48, <[email protected]> wrote:

Hmm, possibly? I cherry-picked the patches to sumo and saw this missing dependency in my container.

Did poky master switch from using host python to native after sumo?

Quoted 6 lines

poky uses host python for some things and native python for other things. Generally where 'core python' is enough then we use the host python, but when additional libraries are needed, then native python plus those libraries is a better choice, as this doesn't require adding to the required host packages list, and allows better control over how they're built.

Currently do_populate_cve_db is executed using host python, and the DEPENDS field is empty, and INHIBIT_DEFAULT_DEPS = "1". python-urllib3 is also not available in poky, but only in meta-openembedded.

I'm fine with additional python3-urllib3 dependency to bitbake build host tools when running CVE check builds. Just wanted to document it.

-Mikko

#13 lore raw
In reply to [email protected]
Quoted 4 lines · On Fri, 21 Jun 2019 at 12:11, <[email protected]> wrote:

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

Somehow I didn't notice it, thanks.

Pierre, can you rewrite this to use standard library instead of urllib3?

Ross

#14 lore raw
Quoted 9 lines · Ross
Quoted 4 lines · On Fri, 21 Jun 2019 at 12:11, <[email protected]> wrote:

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

Somehow I didn't notice it, thanks.

Pierre, can you rewrite this to use standard library instead of urllib3?

Also, the CVE db is updated using this custom task without link to do_fetch, which means a fetchall task would not update the database for off line NO_NETWORK builds.

Could the task be added as dependency to do_fetch() or are there some other side effects?

-Mikko

#15 lore raw

Hi,

Quoted 7 lines

Also, the CVE db is updated using this custom task without link to do_fetch, which means a fetchall task would not update the database for off line NO_NETWORK builds.

Could the task be added as dependency to do_fetch() or are there some other side effects?

Yes I can do that, I don't think this will cause side effects.

Pierre

#16 lore raw
Quoted 9 lines · Pierre Le Magourou
Quoted 7 lines

Also, the CVE db is updated using this custom task without link to do_fetch, which means a fetchall task would not update the database for off line NO_NETWORK builds.

Could the task be added as dependency to do_fetch() or are there some other side effects?

Yes I can do that, I don't think this will cause side effects.

Oh, you'll also want to set the proxies appropriately.

Ross

#17 lore raw
In reply to Burton, Ross

Hi,

Quoted 8 lines
Quoted 4 lines

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

Somehow I didn't notice it, thanks.

Pierre, can you rewrite this to use standard library instead of urllib3?

Yes of course,
I only need urrlib to fetch the NVD json feeds, so I don't need urllib3, I can use standard library.

Pierre

#18 lore raw

Thanks! :)

Ross

Quoted 17 lines · Pierre Le Magourou

Hi,

Quoted 8 lines
Quoted 4 lines

This adds python3 urllib3 (python3-urllib3 in Debian) to build environment dependencies. It's the first user of urllib3 in poky, AFAIK. Maybe documentation could be updated too, e.g.
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#brief-build-system-packages

Somehow I didn't notice it, thanks.

Pierre, can you rewrite this to use standard library instead of urllib3?

Yes of course,
I only need urrlib to fetch the NVD json feeds, so I don't need urllib3, I can use standard library.

Pierre

#19 lore raw
In reply to Pierre Le Magourou
Quoted 43 lines · Pierre Le Magourou

From: Pierre Le Magourou <[email protected]>

cve-check-tool-native do_populate_cve_db task was using deprecated NVD
xml data feeds, cve-update-db uses NVD json data feeds.

Sqlite database schema was updated to take into account CVSSv3 CVE scores and operator in affected product versions.
A new META table was added to store the last modification date of the NVD json data feeds.

Signed-off-by: Pierre Le Magourou <
[email protected]>

---
 meta/recipes-core/meta/cve-update-db.bb | 121
++++++++++++++++++++++++++++++++
 1 file changed, 121 insertions(+)
 create mode 100644 meta/recipes-core/meta/cve-update-db.bb
meta/recipes- renamed from meta/recipes-core/meta/cve-update-db.bb +0 -0

core/meta/cve-update-db.bb
new file mode 100644
index 0000000000..522fd23807

meta/recipes-core/meta/cve-update-db.bb new file +17 -0
@@ -0,0 +1,121 @@
+SUMMARY = "Updates the NVD CVE database"
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+PACKAGES = ""
+
+inherit nopackages
+
+deltask do_fetch
+deltask do_unpack
+deltask do_patch
+deltask do_configure
+deltask do_compile
+deltask do_install
+deltask do_populate_sysroot
+
+python do_populate_cve_db() {

meta/recipes-core/meta/cve-update-db.bb line 27

Quoted 10 lines
meta/recipes-core/meta/cve-update-db.bb new file +10 -0
+    """
+    Update NVD database with json data feed
+    """
+
+    import sqlite3, urllib3, shutil, gzip, re
+    from datetime import date
+
+    BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
+    YEAR_START = 2002
+    JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz'

It looks like CVE_CHECK_DB_DIR has no default value which resulted in:

https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b

We only started seeing that error after your later patch to add back the do_fetch task. build-appliance is trying to collect up all the sources it may need.

Cheers,

Richard

#20 lore raw

Hi,

Quoted 8 lines

It looks like CVE_CHECK_DB_DIR has no default value which resulted in:

https://autobuilder.yoctoproject.org/typhoon/#/builders/23/builds/988/steps/7/logs/step1b

We only started seeing that error after your later patch to add back the do_fetch task. build-appliance is trying to collect up all the sources it may need.

I see the problem, it happens when cve-update-db do_fetch task is executed and cve-check class is not inherited. I sent a v2 patch that sets default value to CVE_CHECK_DB_DIR.

Pierre