CVE check backport
RFC PATCH 9/47 cve-update-db: do_populate_cve_db depends on do_fetch
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
To be able to populate NVD database on a fetchall
(bitbake <image> --run-all=fetch), set the do_populate_cve_db task to be executed before do_fetch.
Do not get CVE_CHECK_DB_DIR, CVE_CHECK_DB_FILE and CVE_CHECK_TMP_FILE variable because do_populate_cve_db can be called in a context where cve-check class is not loaded.
(From OE-Core rev: 975793e3825a2a9ca6dc0e43577f680214cb7993)
--- meta/recipes-core/meta/cve-update-db.bb | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
meta/recipes-core/meta/cve-update-db.bb +13 -8
@@ -6,7 +6,6 @@ PACKAGES = "" inherit nopackages -deltask do_fetch deltask do_unpack deltask do_patch deltask do_configure
@@ -24,11 +23,16 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz' + + db_dir = d.getVar("DL_DIR") + '/CVE_CHECK' + db_file = db_dir + '/nvd-json.db' + json_tmpfile = db_dir + '/nvd.json.gz' proxy = d.getVar("https_proxy") + if not os.path.isdir(db_dir): + os.mkdir(db_dir) + # Connect to database - db_file = d.getVar("CVE_CHECK_DB_FILE") conn = sqlite3.connect(db_file) c = conn.cursor()
@@ -55,9 +59,9 @@ python do_populate_cve_db() { req = urllib.request.Request(json_url) if proxy: req.set_proxy(proxy, 'https') - with urllib.request.urlopen(req) as r, open(JSON_TMPFILE, 'wb') as tmpfile: + with urllib.request.urlopen(req) as r, open(json_tmpfile, 'wb') as tmpfile: shutil.copyfileobj(r, tmpfile) - with gzip.open(JSON_TMPFILE, 'rt') as jsonfile: + with gzip.open(json_tmpfile, 'rt') as jsonfile: update_db(c, jsonfile) c.execute("insert or replace into META values (?, ?)", [year, last_modified])
@@ -65,8 +69,9 @@ python do_populate_cve_db() { conn.commit() conn.close() - with open(d.getVar("CVE_CHECK_TMP_FILE"), 'a'): - os.utime(d.getVar("CVE_CHECK_TMP_FILE"), None) + cve_check_tmp_file = d.getVar("TMPDIR") + '/cve_check' + with open(cve_check_tmp_file, 'a'): + os.utime(cve_check_tmp_file, None) } # DJB2 hash algorithm
@@ -120,7 +125,7 @@ def update_db(c, json_filename): -addtask do_populate_cve_db before do_cve_check +addtask do_populate_cve_db before do_fetch do_populate_cve_db[nostamp] = "1" EXCLUDE_FROM_WORLD = "1"
Signature · 1 line
1.9.1
RFC PATCH 10/47 cve-update-db: Catch request.urlopen errors.
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
If the NVD url is not accessible, print a warning on top of the CVE report, and continue. The database will not be fully updated, but cve_check can still run on the previous database.
(From OE-Core rev: 0325dd72714f0b447558084f481b77f0ec850eed)
--- meta/classes/cve-check.bbclass | 5 +++-- meta/recipes-core/meta/cve-update-db.bb | 30 +++++++++++++++++++++--------- 2 files changed, 24 insertions(+), 11 deletions(-)
meta/classes/cve-check.bbclass +3 -2
@@ -51,14 +51,15 @@ python do_cve_check () { Check recipe for patched and unpatched CVEs """ - if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")): + if os.path.exists(d.getVar("CVE_CHECK_DB_FILE")): patched_cves = get_patches_cves(d) patched, unpatched = check_cves(d, patched_cves) if patched or unpatched: cve_data = get_cve_info(d, patched + unpatched) cve_write_data(d, patched, unpatched, cve_data) else: - bb.note("Failed to update CVE database, skipping CVE check") + bb.note("No CVE database found, skipping CVE check") + } addtask cve_check after do_unpack before do_build
meta/recipes-core/meta/cve-update-db.bb +21 -9
@@ -28,6 +28,7 @@ python do_populate_cve_db() { db_file = db_dir + '/nvd-json.db' json_tmpfile = db_dir + '/nvd.json.gz' proxy = d.getVar("https_proxy") + cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a') if not os.path.isdir(db_dir): os.mkdir(db_dir)
@@ -47,9 +48,13 @@ python do_populate_cve_db() { req = urllib.request.Request(meta_url) if proxy: req.set_proxy(proxy, 'https') - with urllib.request.urlopen(req) as r: - date_line = str(r.read().splitlines()[0]) - last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) + try: + with urllib.request.urlopen(req, timeout=1) as r: + date_line = str(r.read().splitlines()[0]) + last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) + except: + cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') + break # Compare with current db last modified date c.execute("select DATE from META where YEAR = '%d'" % year)
@@ -59,19 +64,26 @@ python do_populate_cve_db() { req = urllib.request.Request(json_url) if proxy: req.set_proxy(proxy, 'https') - with urllib.request.urlopen(req) as r, open(json_tmpfile, 'wb') as tmpfile: - shutil.copyfileobj(r, tmpfile) + try: + with urllib.request.urlopen(req, timeout=1) as r, \ + open(json_tmpfile, 'wb') as tmpfile: + shutil.copyfileobj(r, tmpfile) + except: + cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') + break + with gzip.open(json_tmpfile, 'rt') as jsonfile: update_db(c, jsonfile) c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + # Update success, set the date to cve_check file. + if year == date.today().year: + cve_f.write('CVE database update : %s\n\n' % date.today()) + + cve_f.close() conn.commit() conn.close() - - cve_check_tmp_file = d.getVar("TMPDIR") + '/cve_check' - with open(cve_check_tmp_file, 'a'): - os.utime(cve_check_tmp_file, None) } # DJB2 hash algorithm
Signature · 1 line
1.9.1
RFC PATCH 11/47 cve-check: Depends on cve-update-db-native
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
do_populate_cve_db is a native task.
(From OE-Core rev: 4078da92b49946848cddebe1735f301af161e162)
Conflicts:
meta/conf/distro/include/maintainers.inc
--- meta/classes/cve-check.bbclass | 2 +- meta/conf/distro/include/maintainers.inc | 2 + meta/recipes-core/meta/cve-update-db-native.bb | 143 +++++++++++++++++++++++++ meta/recipes-core/meta/cve-update-db.bb | 143 ------------------------- 4 files changed, 146 insertions(+), 144 deletions(-) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb delete mode 100644 meta/recipes-core/meta/cve-update-db.bb
meta/classes/cve-check.bbclass +1 -1
@@ -63,7 +63,7 @@ python do_cve_check () { } addtask cve_check after do_unpack before do_build -do_cve_check[depends] = "cve-update-db:do_populate_cve_db" +do_cve_check[depends] = "cve-update-db-native:do_populate_cve_db" do_cve_check[nostamp] = "1" python cve_check_cleanup () {
meta/conf/distro/include/maintainers.inc +2 -0
@@ -122,7 +122,9 @@ RECIPE_MAINTAINER_pn-cryptodev-module = "Robert Yang <[email protected]> RECIPE_MAINTAINER_pn-cryptodev-tests = "Robert Yang <[email protected]>" RECIPE_MAINTAINER_pn-cups = "Chen Qi <[email protected]>" RECIPE_MAINTAINER_pn-curl = "Armin Kuster <[email protected]>" +RECIPE_MAINTAINER_pn-cve-check-tool = "Ross Burton <[email protected]>" RECIPE_MAINTAINER_pn-cve-update-db = "Ross Burton <[email protected]>" +RECIPE_MAINTAINER_pn-cve-update-db-native = "Ross Burton <[email protected]>" RECIPE_MAINTAINER_pn-cwautomacros = "Maxin B. John <[email protected]>" RECIPE_MAINTAINER_pn-damageproto = "Armin Kuster <[email protected]>" RECIPE_MAINTAINER_pn-db = "Mark Hatle <[email protected]>"
meta/recipes-core/meta/cve-update-db-native.bb new file +143 -0
143 lines not on this page · raw message
meta/recipes-core/meta/cve-update-db.bb deleted +0 -143
143 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 12/47 cve-check: Update unpatched CVE matching
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
Now that cve-update-db added CPE information to NVD database. We can check for unpatched versions with operators '<', '<=', '>', and '>='.
(From OE-Core rev: bc0195be1b15bcffe60127bc5e8b7011a853c2ed)
--- meta/classes/cve-check.bbclass | 54 +++++++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 14 deletions(-)
meta/classes/cve-check.bbclass +40 -14
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -189,27 +189,53 @@ def check_cves(d, patched_cves): conn = sqlite3.connect(db_file) c = conn.cursor() - query = """SELECT * FROM PRODUCTS WHERE - (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR - (PRODUCT IS '{0}' AND OPERATOR IS '<=');""" + query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '{0}';" + for product in products: for row in c.execute(query.format(product, pv)): cve = row[1] - version = row[4] - - try: - discardVersion = LooseVersion(version) < LooseVersion(pv) - except: - discardVersion = True + version_start = row[4] + operator_start = row[5] + version_end = row[6] + operator_end = row[7] if pv in cve_whitelist.get(cve, []): bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) - elif discardVersion: - bb.debug(2, "Do not consider version %s " % (version)) else: - cves_unpatched.append(cve) + if (operator_start == '=' and pv == version_start): + cves_unpatched.append(cve) + else: + if operator_start: + try: + to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) + to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_start, version_start, cve)) + to_append_start = False + else: + to_append_start = False + + if operator_end: + try: + to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) + to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) + except: + bb.note("%s: Failed to compare %s %s %s for %s" % + (product, pv, operator_end, version_end, cve)) + to_append_end = False + else: + to_append_end = False + + if operator_start and operator_end: + to_append = to_append_start and to_append_end + else: + to_append = to_append_start or to_append_end + + if to_append: + cves_unpatched.append(cve) bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) conn.close()
@@ -217,7 +243,7 @@ def check_cves(d, patched_cves): def get_cve_info(d, cves): """ - Get CVE information from the database used by cve-check-tool. + Get CVE information from the database. Unfortunately the only way to get CVE info is set the output to html (hard to parse) or query directly the database.
Signature · 1 line
1.9.1
RFC PATCH 18/47 cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
CVE_CHECK_WHITELIST does not contain version anymore, as it was not used. This variable should be set per recipe.
(From OE-Core rev: 7069302a4ccbb5b72e1902f284cf078516fd7294)
--- meta/classes/cve-check.bbclass | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
meta/classes/cve-check.bbclass +11 -11
@@ -39,15 +39,12 @@ CVE_CHECK_CREATE_MANIFEST ??= "1" # Whitelist for packages (PN) CVE_CHECK_PN_WHITELIST ?= "" -# Whitelist for CVE and version of package. If a CVE is found then the PV is -# compared with the version list, and if found the CVE is considered -# patched. -# -# The value should be valid Python in this format: -# { -# 'CVE-2014-2524': ('6.3','5.2') -# } -CVE_CHECK_CVE_WHITELIST ?= "{}" +# Whitelist for CVE. If a CVE is found, then it is considered patched. +# The value is a string containing space separated CVE values: +# +# CVE_CHECK_WHITELIST = 'CVE-2014-2524 CVE-2018-1234' +# +CVE_CHECK_WHITELIST ?= "" python do_cve_check () { """
@@ -185,7 +182,10 @@ def check_cves(d, patched_cves): bb.note("Recipe has been whitelisted, skipping check") return ([], []) - cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) + old_cve_whitelist = d.getVar("CVE_CHECK_CVE_WHITELIST") + if old_cve_whitelist: + bb.warn("CVE_CHECK_CVE_WHITELIST is deprecated, please use CVE_CHECK_WHITELIST.") + cve_whitelist = d.getVar("CVE_CHECK_WHITELIST").split() import sqlite3 db_file = d.getVar("CVE_CHECK_DB_FILE")
@@ -206,7 +206,7 @@ def check_cves(d, patched_cves): version_end = row[6] operator_end = row[7] - if pv in cve_whitelist.get(cve, []): + if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) elif cve in patched_cves: bb.note("%s has been patched" % (cve))
Signature · 1 line
1.9.1
RFC PATCH 13/47 cve-check: remove redundant readline CVE whitelisting
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
CVE-2014-2524 is a readline CVE that was fixed in 6.3patch3 onwards, but the tooling wasn't able to detect this version. As we now ship readline 8 we don't need to manually whitelist it, and if we did then the whitelisting should be in the readline recipe.
(From OE-Core rev: 07bb8b25e172aa5c8ae96b6e8eb4ac901b835219)
--- meta/classes/cve-check.bbclass | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
meta/classes/cve-check.bbclass +9 -4
@@ -41,10 +41,15 @@ CVE_CHECK_PN_WHITELIST = "\ glibc-locale \ " -# Whitelist for CVE and version of package -CVE_CHECK_CVE_WHITELIST = "{\ - 'CVE-2014-2524': ('6.3','5.2',), \ -}" +# Whitelist for CVE and version of package. If a CVE is found then the PV is +# compared with the version list, and if found the CVE is considered +# patched. +# +# The value should be valid Python in this format: +# { +# 'CVE-2014-2524': ('6.3','5.2') +# } +CVE_CHECK_CVE_WHITELIST ?= "{}" python do_cve_check () { """
Signature · 1 line
1.9.1
RFC PATCH 15/47 glibc: exclude child recipes from CVE scanning
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
As glibc will be scanned for CVEs, we don't need to scan glibc-locale, glibc-mtrace, and glibc-scripts which are all separate recipes for technical reasons.
Exclude the recipes by setting CVE_PRODUCT in the recipe, instead of using the global whitelist.
(From OE-Core rev: 1f9a963b9ff7ebe052ba54b9fcbdf7d09478dd17)
--- meta/classes/cve-check.bbclass | 4 +--- meta/recipes-core/glibc/glibc-locale.inc | 3 +++ meta/recipes-core/glibc/glibc-mtrace.inc | 3 +++ meta/recipes-core/glibc/glibc-scripts.inc | 3 +++ 4 files changed, 10 insertions(+), 3 deletions(-)
meta/classes/cve-check.bbclass +1 -3
@@ -37,9 +37,7 @@ CVE_CHECK_COPY_FILES ??= "1" CVE_CHECK_CREATE_MANIFEST ??= "1" # Whitelist for packages (PN) -CVE_CHECK_PN_WHITELIST = "\ - glibc-locale \ -" +CVE_CHECK_PN_WHITELIST ?= "" # Whitelist for CVE and version of package. If a CVE is found then the PV is # compared with the version list, and if found the CVE is considered
meta/recipes-core/glibc/glibc-locale.inc +3 -0
@@ -95,3 +95,6 @@ do_install () { inherit libc-package BBCLASSEXTEND = "nativesdk" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = ""
meta/recipes-core/glibc/glibc-mtrace.inc +3 -0
@@ -11,3 +11,6 @@ do_install() { install -d -m 0755 ${D}${bindir} install -m 0755 ${SRC}/mtrace ${D}${bindir}/ } + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = ""
meta/recipes-core/glibc/glibc-scripts.inc +3 -0
@@ -18,3 +18,6 @@ do_install() { # sotruss script requires sotruss-lib.so (given by libsotruss package), # to produce trace of the library calls. RDEPENDS_${PN} += "libsotruss" + +# Don't scan for CVEs as glibc will be scanned +CVE_PRODUCT = ""
Signature · 1 line
1.9.1
RFC PATCH 16/47 cve-check.bbclass: initialize to_append
In reply to Mikko RapeliFixes build failure with core-image-minimal:
Exception: UnboundLocalError: local variable 'to_append' referenced before assignment
(From OE-Core rev: 270ac00cb43d0614dfe1c95f960c76e9e5fa20d4)
--- meta/classes/cve-check.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes/cve-check.bbclass +1 -0
@@ -207,6 +207,7 @@ def check_cves(d, patched_cves): elif cve in patched_cves: bb.note("%s has been patched" % (cve)) else: + to_append = False if (operator_start == '=' and pv == version_start): cves_unpatched.append(cve) else:
Signature · 1 line
1.9.1
RFC PATCH 17/47 cve-check: allow comparison of Vendor as well as Product
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
Some product names are too vague to be searched without also matching the vendor, for example Flex could be the parser compiler we ship, or Adobe Flex, or Apache Flex, or IBM Flex.
If entries in CVE_PRODUCT contain a colon then split it as vendor:product to improve the search.
Also don't use .format() to construct SQL as that can lead to security issues. Instead, use ? placeholders and lets sqlite3 handle the escaping.
(From OE-Core rev: e6bf90009877d00243417898700d2320fd87b39c)
--- meta/classes/cve-check.bbclass | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
meta/classes/cve-check.bbclass +8 -4
@@ -190,12 +190,16 @@ def check_cves(d, patched_cves): import sqlite3 db_file = d.getVar("CVE_CHECK_DB_FILE") conn = sqlite3.connect(db_file) - c = conn.cursor() - - query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '{0}';" for product in products: - for row in c.execute(query.format(product, pv)): + c = conn.cursor() + if ":" in product: + vendor, product = product.split(":", 1) + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ? AND VENDOR IS ?", (product, vendor)) + else: + c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) + + for row in c: cve = row[1] version_start = row[4] operator_start = row[5]
Signature · 1 line
1.9.1
cover CVE check backport
Hi,
Request for comments, call for help, LTS too?
Yocto 2.5 sumo isn't actively maintained by the Yocto Project anymore. But that does not mean that support for it needs to stop.
I use sumo and due to various reasons like BSP layers, binary compatibility, contracts etc can't update to newer release or to master branch. I suspect I'm not alone.
sumo CVE checking machinery is broken due to changes in
NIST and NVD (see
https://nvd.nist.gov/general/news/XML-Vulnerability-Feed-Retirement and
https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release ) so some backports from poky master/zeus are needed to fix the tooling. Thanks to Anuj, Chen, Chin, Pierre, Ross and others who fixed these on master branch!
The tooling will expose that sumo is severely lacking in security patches, but the tooling is a start for anyone interested, like me, to fill the gaps and publish patches for bitbake recipes we care about.
Could sumo be an LTS? Well I hope so. The LTS proposal
http://lists.openembedded.org/pipermail/openembedded-architecture/2019-October/001665.html
https://docs.google.com/document/d/1AwAFDf52f_FoXksbHEVUMlu4hpcI0JMGVG-Kj_sUkyc/edit
from Yocto Project is great. Maybe as part of that work, someone could setup a really minimal set of QA on Yocto Project side to also test patches aiming at yocto 2.5 sumo. If not, would be really nice if someone could collect patches into sumo-next or sumo-contrib branch where us users could be in charge of all Quality Assurance.
So, comments and review are welcome. Patches even more so!
Patches were tested on an x86 product tree where full stack CVE analysis produces good results. Then I ported them to pure poky sumo and ran core-image-minimal build. Tried running "bitbake world" build which also succeeds. The results show following bitbake target recipes from poky with unpatched CVEs (ignored native, SDK and cross tools for now):
build/tmp/deploy/cve$ grep -l "Unpatched" * | egrep -v -- "-native|nativesdk-|-cross" | sort apt
aspell
binutils
bluez5
busybox
bzip2
cairo
cups
curl
db
dropbear
elfutils
epiphany
expat
file
gcc
gcc-runtime
gcc-sanitizers
gcc-source-7.3.0
ghostscript
git
glib-2.0
glibc
gnupg
gnutls
go
gstreamer1.0
libarchive
libcomps
libcroco
libexif
libgcc
libgcrypt
libid3tag
libjpeg-turbo
libpcap
libpcre
libpng
librsvg
libsndfile1
libsolv
libvorbis
libx11
libxkbcommon
libxslt
lighttpd
lz4
nasm
ncurses
openssh
openssl
pango
patch
pcmanfm
perl
python
python3
qemu
shadow
sqlite3
sudo
sysstat
systemd
tar
tiff
unzip
webkitgtk
wget
wpa-supplicant
xdg-utils
xserver-xorg
zip
Sampling on the data shows that
* openssl 1.0.2p is missing patch for CVE-2019-1559
* openssh 7.6p1 is missing a lot more patches
* gcc is missing patches for CVE-2018-12886 on ARM and CVE-2019-15847 on POWER9
* libpng is missing patch for CVE-2018-14048
* libjpeg-turbo is missing patch for CVE-2018-14498
* libgcrypt is missing patch for CVE-2018-6829
etc.
About CVE checking in yocto:
* enable with 'INHERIT += "cve-check"' in conf/local.conf
* see the resulting reports in tmp/deploy/cve/ directory for all compiled recipes
* there is also an image specific summary but I saw it included native and nativesdk recipe data too
* for applying CVE patches, white listing, setting product names etc see the meta/classes/cve-check.bbclass and examples in this patchset and in master branch
* note that only recompiled recipes will be analyzed for CVEs so things from sstate cache will be ignored, a clean build without cache may be needed when enabling the check
ps. sumo still comes with gcc 7.3 and my patch to update to 7.4 with lots of bug fixes has not been applied from
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278049.html
I've been using gcc 7.4 in several x86 and arm64 projects so I would also apply this update to any sumo tree out there.
Cheers,
-Mikko
Anuj Mittal (2):
openssl: set CVE vendor to openssl
rsync: fix CVEs for included zlib
Chen Qi (9):
flac: also add flac to CVE_PRODUCT
xserver-xorg: set CVE_PRODUCT
nasm: add CVE_PRODUCT
dropbear: set CVE_PRODUCT
libsdl: set CVE_PRODUCT
ghostscript: set CVE_PRODUCT
squashfs-tools: set CVE_PRODUCT
libxfont2: set CVE_PRODUCT
webkitgtk: set CVE_PRODUCT
Chin Huat Ang (1):
cve-update-db-native: fix https proxy issues
Mikko Rapeli (1):
cve-check.bbclass: initialize to_append
Pierre Le Magourou (13):
cve-update-db: New recipe to update CVE database
cve-check: Remove dependency to cve-check-tool-native
cve-check: Manage CVE_PRODUCT with more than one name
cve-check: Consider CVE that affects versions with less than operator
cve-update-db: Use std library instead of urllib3
cve-update-db: Manage proxy if needed.
cve-update-db: do_populate_cve_db depends on do_fetch
cve-update-db: Catch request.urlopen errors.
cve-check: Depends on cve-update-db-native
cve-check: Update unpatched CVE matching
cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST
cve-update-db: Use NVD CPE data to populate PRODUCTS table
cve-update-db-native: Remove hash column from database.
Ross Burton (21):
cve-check: be idiomatic
cve-check: remove redundant readline CVE whitelisting
cve-check-tool: remove
glibc: exclude child recipes from CVE scanning
cve-check: allow comparison of Vendor as well as Product
cve-update-db-native: use SQL placeholders instead of format strings
cve-update-db-native: use os.path.join instead of +
cve-update-db: actually inherit native
cve-update-db-native: use executemany() to optimise CPE insertion
cve-update-db-native: improve metadata parsing
cve-update-db-native: clean up JSON fetching
cve-check: ensure all known CVEs are in the report
cve-check: failure to parse versions should be more visible
flex: set CVE_PRODUCT to include vendor
libpam: set CVE_PRODUCT
procps: whitelist CVE-2018-1121
libpng: whitelist CVE-2019-17371
ed: set CVE vendor to avoid false positives
boost: set CVE vendor to Boost
subversion: set CVE vendor to Apache
git: set CVE vendor to git-scm
meta/classes/cve-check.bbclass | 147 ++++++++----- meta/conf/distro/include/maintainers.inc | 2 + .../recipes-connectivity/openssl/openssl_1.0.2p.bb | 2 + .../recipes-connectivity/openssl/openssl_1.1.0i.bb | 2 + meta/recipes-core/dropbear/dropbear.inc | 2 + meta/recipes-core/glibc/glibc-locale.inc | 3 + meta/recipes-core/glibc/glibc-mtrace.inc | 3 + meta/recipes-core/glibc/glibc-scripts.inc | 3 + meta/recipes-core/meta/cve-update-db-native.bb | 190 +++++++++++++++++ .../cve-check-tool/cve-check-tool_5.6.4.bb | 62 ------ ...01-Fix-freeing-memory-allocated-by-sqlite.patch | 50 ----- ...ow-overriding-default-CA-certificate-file.patch | 215 ------------------- ...ogress-in-percent-when-downloading-CVE-db.patch | 135 ------------ ...are-computed-vs-expected-sha256-digit-str.patch | 52 ----- .../check-for-malloc_trim-before-using-it.patch | 51 ----- meta/recipes-devtools/flex/flex_2.6.0.bb | 3 + meta/recipes-devtools/git/git.inc | 2 + meta/recipes-devtools/nasm/nasm_2.13.03.bb | 2 + .../rsync/files/CVE-2016-9840.patch | 75 +++++++ .../rsync/files/CVE-2016-9841.patch | 228 +++++++++++++++++++++ .../rsync/files/CVE-2016-9842.patch | 33 +++ .../rsync/files/CVE-2016-9843.patch | 53 +++++ meta/recipes-devtools/rsync/rsync_3.1.3.bb | 7 +- .../squashfs-tools/squashfs-tools_git.bb | 2 + .../subversion/subversion_1.9.7.bb | 2 + meta/recipes-extended/ed/ed_1.14.2.bb | 2 + .../ghostscript/ghostscript_9.21.bb | 3 + meta/recipes-extended/pam/libpam_1.3.0.bb | 2 + meta/recipes-extended/procps/procps_3.3.12.bb | 3 + meta/recipes-graphics/libsdl/libsdl_1.2.15.bb | 2 + meta/recipes-graphics/libsdl2/libsdl2_2.0.8.bb | 2 + meta/recipes-graphics/xorg-lib/libxfont2_2.0.3.bb | 2 + .../recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 + meta/recipes-multimedia/flac/flac_1.3.2.bb | 2 +- meta/recipes-multimedia/libpng/libpng_1.6.34.bb | 3 + meta/recipes-sato/webkit/webkitgtk_2.18.6.bb | 2 + meta/recipes-support/boost/boost.inc | 2 + 37 files changed, 731 insertions(+), 622 deletions(-) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9840.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9841.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9842.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9843.patch
Signature · 1 line
1.9.1
RFC PATCH 14/47 cve-check-tool: remove
From: Ross Burton <[email protected]>
(From OE-Core rev: 5388ed6d1378d647a65912dbd537f9ef3cb5760a)
--- .../cve-check-tool/cve-check-tool_5.6.4.bb | 62 ------ ...01-Fix-freeing-memory-allocated-by-sqlite.patch | 50 ----- ...ow-overriding-default-CA-certificate-file.patch | 215 --------------------- ...ogress-in-percent-when-downloading-CVE-db.patch | 135 ------------- ...are-computed-vs-expected-sha256-digit-str.patch | 52 ----- .../check-for-malloc_trim-before-using-it.patch | 51 ----- 6 files changed, 565 deletions(-) delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch
meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb deleted +0 -62
@@ -1,62 +0,0 @@ -SUMMARY = "cve-check-tool" -DESCRIPTION = "cve-check-tool is a tool for checking known (public) CVEs.\ -The tool will identify potentially vunlnerable software packages within Linux distributions through version matching." -HOMEPAGE = "https://github.com/ikeydoherty/cve-check-tool" -SECTION = "Development/Tools" -LICENSE = "GPL-2.0+" -LIC_FILES_CHKSUM = "file://LICENSE;md5=e8c1458438ead3c34974bc0be3a03ed6" - -SRC_URI = "https://github.com/ikeydoherty/${BPN}/releases/download/v${PV}/${BP}.tar.xz \ - file://check-for-malloc_trim-before-using-it.patch \ - file://0001-print-progress-in-percent-when-downloading-CVE-db.patch \ - file://0001-curl-allow-overriding-default-CA-certificate-file.patch \ - file://0001-update-Compare-computed-vs-expected-sha256-digit-str.patch \ - file://0001-Fix-freeing-memory-allocated-by-sqlite.patch \ - " - -SRC_URI[md5sum] = "c5f4247140fc9be3bf41491d31a34155" -SRC_URI[sha256sum] = "b8f283be718af8d31232ac1bfc10a0378fb958aaaa49af39168f8acf501e6a5b" - -UPSTREAM_CHECK_URI = "https://github.com/ikeydoherty/cve-check-tool/releases" - -DEPENDS = "libcheck glib-2.0 json-glib curl libxml2 sqlite3 openssl ca-certificates" - -RDEPENDS_${PN} = "ca-certificates" - -inherit pkgconfig autotools - -EXTRA_OECONF = "--disable-coverage --enable-relative-plugins" -CFLAGS_append = " -Wno-error=pedantic" - -do_populate_cve_db() { - if [ "${BB_NO_NETWORK}" = "1" ] ; then - bbwarn "BB_NO_NETWORK is set; Can't update cve-check-tool database, new CVEs won't be detected" - return - fi - - # In case we don't inherit cve-check class, use default values defined in the class. - cve_dir="${CVE_CHECK_DB_DIR}" - cve_file="${CVE_CHECK_TMP_FILE}" - - [ -z "${cve_dir}" ] && cve_dir="${DL_DIR}/CVE_CHECK" - [ -z "${cve_file}" ] && cve_file="${TMPDIR}/cve_check" - - unused="${@bb.utils.export_proxies(d)}" - bbdebug 2 "Updating cve-check-tool database located in $cve_dir" - # --cacert works around curl-native not finding the CA bundle - if cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-certificates.crt -d "$cve_dir" ; then - printf "CVE database was updated on %s UTC\n\n" "$(LANG=C date --utc +'%F %T')" > "$cve_file" - else - bbwarn "Error in executing cve-check-update" - if [ "${@'1' if bb.data.inherits_class('cve-check', d) else '0'}" -ne 0 ] ; then - bbwarn "Failed to update cve-check-tool database, CVEs won't be checked" - fi - fi -} - -addtask populate_cve_db after do_populate_sysroot -do_populate_cve_db[depends] = "cve-check-tool-native:do_populate_sysroot" -do_populate_cve_db[nostamp] = "1" -do_populate_cve_db[progress] = "percent" - -BBCLASSEXTEND = "native nativesdk"
meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch deleted +0 -50
50 lines not on this page · raw message
meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch deleted +0 -215
215 lines not on this page · raw message
meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch deleted +0 -135
135 lines not on this page · raw message
meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch deleted +0 -52
52 lines not on this page · raw message
meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch deleted +0 -51
51 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 2/47 cve-check: Remove dependency to cve-check-tool-native
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
Use the new update-cve-db recipe to update database.
(From OE-Core rev: bc144b028f6f51252f4359248f6921028bcb6780)
--- meta/classes/cve-check.bbclass | 71 ++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 45 deletions(-)
meta/classes/cve-check.bbclass +26 -45
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd-json.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -62,7 +62,7 @@ python do_cve_check () { } addtask cve_check after do_unpack before do_build -do_cve_check[depends] = "cve-check-tool-native:do_populate_sysroot cve-check-tool-native:do_populate_cve_db" +do_cve_check[depends] = "cve-update-db:do_populate_cve_db" do_cve_check[nostamp] = "1" python cve_check_cleanup () {
@@ -163,61 +163,40 @@ def get_patches_cves(d): def check_cves(d, patched_cves): """ - Run cve-check-tool looking for patched and unpatched CVEs. + Connect to the NVD database and find unpatched cves. """ - import ast, csv, tempfile, subprocess, io - cves_patched = [] cves_unpatched = [] bpn = d.getVar("CVE_PRODUCT") # If this has been unset then we're not scanning for CVEs here (for example, image recipes) if not bpn: return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] - cves = " ".join(patched_cves) - cve_db_dir = d.getVar("CVE_CHECK_DB_DIR") cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) - cve_cmd = "cve-check-tool" - cmd = [cve_cmd, "--no-html", "--skip-update", "--csv", "--not-affected", "-t", "faux", "-d", cve_db_dir] # If the recipe has been whitlisted we return empty lists if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split(): bb.note("Recipe has been whitelisted, skipping check") return ([], []) - try: - # Write the faux CSV file to be used with cve-check-tool - fd, faux = tempfile.mkstemp(prefix="cve-faux-") - with os.fdopen(fd, "w") as f: - for pn in bpn.split(): - f.write("%s,%s,%s,\n" % (pn, pv, cves)) - cmd.append(faux) - - output = subprocess.check_output(cmd).decode("utf-8") - bb.debug(2, "Output of command %s:\n%s" % ("\n".join(cmd), output)) - except subprocess.CalledProcessError as e: - bb.warn("Couldn't check for CVEs: %s (output %s)" % (e, e.output)) - finally: - os.remove(faux) - - for row in csv.reader(io.StringIO(output)): - # Third row has the unpatched CVEs - if row[2]: - for cve in row[2].split(): - # Skip if the CVE has been whitlisted for the current version - if pv in cve_whitelist.get(cve,[]): - bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve)) - else: - cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve)) - # Fourth row has patched CVEs - if row[3]: - for cve in row[3].split(): - cves_patched.append(cve) - bb.debug(2, "%s-%s is patched for %s" % (bpn, pv, cve)) - - return (cves_patched, cves_unpatched) + import sqlite3 + db_file = d.getVar("CVE_CHECK_DB_FILE") + conn = sqlite3.connect(db_file) + c = conn.cursor() + query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';" + for row in c.execute(query % (bpn,pv)): + cve = row[1] + if pv in cve_whitelist.get(cve,[]): + bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve)) + elif cve in patched_cves: + bb.note("%s has been patched" % (cve)) + else: + cves_unpatched.append(cve) + bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve)) + conn.close() + + return (list(patched_cves), cves_unpatched) def get_cve_info(d, cves): """
@@ -241,9 +220,10 @@ def get_cve_info(d, cves): for row in cur.execute(query, tuple(cves)): cve_data[row[0]] = {} cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["score"] = row[2] - cve_data[row[0]]["modified"] = row[3] - cve_data[row[0]]["vector"] = row[4] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] conn.close() return cve_data
@@ -270,7 +250,8 @@ def cve_write_data(d, patched, unpatched, cve_data): unpatched_cves.append(cve) write_string += "CVE STATUS: Unpatched\n" write_string += "CVE SUMMARY: %s\n" % cve_data[cve]["summary"] - write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["score"] + write_string += "CVSS v2 BASE SCORE: %s\n" % cve_data[cve]["scorev2"] + write_string += "CVSS v3 BASE SCORE: %s\n" % cve_data[cve]["scorev3"] write_string += "VECTOR: %s\n" % cve_data[cve]["vector"] write_string += "MORE INFORMATION: %s%s\n\n" % (nvd_link, cve)
Signature · 1 line
1.9.1
RFC PATCH 3/47 cve-check: Manage CVE_PRODUCT with more than one name
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
In some rare cases (eg. curl recipe) the CVE_PRODUCT contains more than one name.
(From OE-Core rev: 7f62a20b32a3d42f04ec58786a7d0db68ef1bb05)
--- meta/classes/cve-check.bbclass | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-)
meta/classes/cve-check.bbclass +14 -11
@@ -168,9 +168,10 @@ def check_cves(d, patched_cves): import ast, csv, tempfile, subprocess, io cves_unpatched = [] - bpn = d.getVar("CVE_PRODUCT") + # CVE_PRODUCT can contain more than one product (eg. curl/libcurl) + bpn = d.getVar("CVE_PRODUCT").split() # If this has been unset then we're not scanning for CVEs here (for example, image recipes) - if not bpn: + if len(bpn) == 0: return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST"))
@@ -184,16 +185,18 @@ def check_cves(d, patched_cves): db_file = d.getVar("CVE_CHECK_DB_FILE") conn = sqlite3.connect(db_file) c = conn.cursor() + query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';" - for row in c.execute(query % (bpn,pv)): - cve = row[1] - if pv in cve_whitelist.get(cve,[]): - bb.note("%s-%s has been whitelisted for %s" % (bpn, pv, cve)) - elif cve in patched_cves: - bb.note("%s has been patched" % (cve)) - else: - cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (bpn, pv, cve)) + for idx in range(len(bpn)): + for row in c.execute(query % (bpn[idx],pv)): + cve = row[1] + if pv in cve_whitelist.get(cve,[]): + bb.note("%s-%s has been whitelisted for %s" % (bpn[idx], pv, cve)) + elif cve in patched_cves: + bb.note("%s has been patched" % (cve)) + else: + cves_unpatched.append(cve) + bb.debug(2, "%s-%s is not patched for %s" % (bpn[idx], pv, cve)) conn.close() return (list(patched_cves), cves_unpatched)
Signature · 1 line
1.9.1
RFC PATCH 4/47 cve-check: Consider CVE that affects versions with less than operator
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
In the NVD json CVE feed, affected versions can be strictly matched to a version, but they can also be matched with the operator '<='.
Add a new condition in the sqlite query to match affected versions that are defined with the operator '<='. Then use LooseVersion to discard all versions that are not relevant.
(From OE-Core rev: 3bf63bc60848d91e90c23f6d854d22b78832aa2d)
--- meta/classes/cve-check.bbclass | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
meta/classes/cve-check.bbclass +14 -2
@@ -166,6 +166,7 @@ def check_cves(d, patched_cves): Connect to the NVD database and find unpatched cves. """ import ast, csv, tempfile, subprocess, io + from distutils.version import LooseVersion cves_unpatched = [] # CVE_PRODUCT can contain more than one product (eg. curl/libcurl)
@@ -186,14 +187,25 @@ def check_cves(d, patched_cves): conn = sqlite3.connect(db_file) c = conn.cursor() - query = "SELECT * FROM PRODUCTS WHERE PRODUCT IS '%s' AND VERSION IS '%s';" + query = """SELECT * FROM PRODUCTS WHERE + (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR + (PRODUCT IS '{0}' AND OPERATOR IS '<=');""" for idx in range(len(bpn)): - for row in c.execute(query % (bpn[idx],pv)): + for row in c.execute(query.format(bpn[idx],pv)): cve = row[1] + version = row[4] + + try: + discardVersion = LooseVersion(version) < LooseVersion(pv) + except: + discardVersion = True + if pv in cve_whitelist.get(cve,[]): bb.note("%s-%s has been whitelisted for %s" % (bpn[idx], pv, cve)) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) + elif discardVersion: + bb.debug(2, "Do not consider version %s " % (version)) else: cves_unpatched.append(cve) bb.debug(2, "%s-%s is not patched for %s" % (bpn[idx], pv, cve))
Signature · 1 line
1.9.1
RFC PATCH 6/47 cve-update-db: Use std library instead of urllib3
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
urllib3 was used in this recipe but it was not set as a dependency. As it is not specifically needed, rewrite the recipe with urllib from the standard library.
(From OE-Core rev: c0eabd30d7b9c2517f4ec9229640be421ecc8a5e)
--- meta/recipes-core/meta/cve-update-db.bb | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
meta/recipes-core/meta/cve-update-db.bb +4 -6
@@ -19,7 +19,7 @@ python do_populate_cve_db() { Update NVD database with json data feed """ - import sqlite3, urllib3, shutil, gzip, re + import sqlite3, urllib, shutil, gzip, re from datetime import date BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
@@ -33,16 +33,14 @@ python do_populate_cve_db() { initialize_db(c) - http = urllib3.PoolManager() - for year in range(YEAR_START, date.today().year + 1): year_url = BASE_URL + str(year) meta_url = year_url + ".meta" json_url = year_url + ".json.gz" # Retrieve meta last modified date - with http.request('GET', meta_url, preload_content=False) as r: - date_line = str(r.data.splitlines()[0]) + with urllib.request.urlopen(meta_url) as r: + date_line = str(r.read().splitlines()[0]) last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) # Compare with current db last modified date
@@ -50,7 +48,7 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Update db with current year json file - with http.request('GET', json_url, preload_content=False) as r, open(JSON_TMPFILE, 'wb') as tmpfile: + with urllib.request.urlopen(json_url) as r, open(JSON_TMPFILE, 'wb') as tmpfile: shutil.copyfileobj(r, tmpfile) with gzip.open(JSON_TMPFILE, 'rt') as jsonfile: update_db(c, jsonfile)
Signature · 1 line
1.9.1
RFC PATCH 7/47 cve-check: be idiomatic
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
Instead of generating a series of indexes via range(len(list)), just iterate the list.
(From OE-Core rev: 27eb839ee651c2d584db42d23bcf5dd764eb33f1)
--- meta/classes/cve-check.bbclass | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
meta/classes/cve-check.bbclass +9 -8
@@ -170,18 +170,19 @@ def check_cves(d, patched_cves): cves_unpatched = [] # CVE_PRODUCT can contain more than one product (eg. curl/libcurl) - bpn = d.getVar("CVE_PRODUCT").split() + products = d.getVar("CVE_PRODUCT").split() # If this has been unset then we're not scanning for CVEs here (for example, image recipes) - if len(bpn) == 0: + if not products: return ([], []) pv = d.getVar("CVE_VERSION").split("+git")[0] - cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) # If the recipe has been whitlisted we return empty lists if d.getVar("PN") in d.getVar("CVE_CHECK_PN_WHITELIST").split(): bb.note("Recipe has been whitelisted, skipping check") return ([], []) + cve_whitelist = ast.literal_eval(d.getVar("CVE_CHECK_CVE_WHITELIST")) + import sqlite3 db_file = d.getVar("CVE_CHECK_DB_FILE") conn = sqlite3.connect(db_file)
@@ -190,8 +191,8 @@ def check_cves(d, patched_cves): query = """SELECT * FROM PRODUCTS WHERE (PRODUCT IS '{0}' AND VERSION = '{1}' AND OPERATOR IS '=') OR (PRODUCT IS '{0}' AND OPERATOR IS '<=');""" - for idx in range(len(bpn)): - for row in c.execute(query.format(bpn[idx],pv)): + for product in products: + for row in c.execute(query.format(product, pv)): cve = row[1] version = row[4]
@@ -200,15 +201,15 @@ def check_cves(d, patched_cves): except: discardVersion = True - if pv in cve_whitelist.get(cve,[]): - bb.note("%s-%s has been whitelisted for %s" % (bpn[idx], pv, cve)) + if pv in cve_whitelist.get(cve, []): + bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) elif discardVersion: bb.debug(2, "Do not consider version %s " % (version)) else: cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (bpn[idx], pv, cve)) + bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) conn.close() return (list(patched_cves), cves_unpatched)
Signature · 1 line
1.9.1
RFC PATCH 8/47 cve-update-db: Manage proxy if needed.
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
If https_proxy environment variable is defined, manage proxy to be able to download meta and json data feeds from https://nvd.nist.gov
(From OE-Core rev: 09be21f4d1793b1e26e78391f51bfc0a27b76deb)
--- meta/recipes-core/meta/cve-update-db.bb | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
meta/recipes-core/meta/cve-update-db.bb +9 -2
@@ -25,6 +25,7 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 JSON_TMPFILE = d.getVar("CVE_CHECK_DB_DIR") + '/nvd.json.gz' + proxy = d.getVar("https_proxy") # Connect to database db_file = d.getVar("CVE_CHECK_DB_FILE")
@@ -39,7 +40,10 @@ python do_populate_cve_db() { json_url = year_url + ".json.gz" # Retrieve meta last modified date - with urllib.request.urlopen(meta_url) as r: + req = urllib.request.Request(meta_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r: date_line = str(r.read().splitlines()[0]) last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1)
@@ -48,7 +52,10 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Update db with current year json file - with urllib.request.urlopen(json_url) as r, open(JSON_TMPFILE, 'wb') as tmpfile: + req = urllib.request.Request(json_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r, open(JSON_TMPFILE, 'wb') as tmpfile: shutil.copyfileobj(r, tmpfile) with gzip.open(JSON_TMPFILE, 'rt') as jsonfile: update_db(c, jsonfile)
Signature · 1 line
1.9.1
RFC PATCH 40/47 procps: whitelist CVE-2018-1121
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
This CVE is about race conditions in 'ps' which make it unsuitable for security audits. As these race conditions are unavoidable ps shouldn't be used for security auditing, so this isn't a valid CVE.
(From OE-Core rev: b3fa0654abf9ac32f683ac174e453ea5e64b6cb8)
Conflicts:
meta/recipes-extended/procps/procps_3.3.15.bb --- meta/recipes-extended/procps/procps_3.3.12.bb | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-extended/procps/procps_3.3.12.bb +3 -0
@@ -64,3 +64,6 @@ python __anonymous() { d.setVarFlag('ALTERNATIVE_LINK_NAME', prog, '%s/%s' % (d.getVar('base_sbindir'), prog)) } +# 'ps' isn't suitable for use as a security tool so whitelist this CVE. +# https://bugzilla.redhat.com/show_bug.cgi?id=1575473#c3 +CVE_CHECK_WHITELIST += "CVE-2018-1121"
Signature · 1 line
1.9.1
RFC PATCH 41/47 libpng: whitelist CVE-2019-17371
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
This is actually a memory leak in gif2png 2.x, so whitelist it in the libpng recipe.
(From OE-Core rev: 341e43ebd935daeb592cb073bf00f80c49a8ec2d)
Conflicts:
meta/recipes-multimedia/libpng/libpng_1.6.37.bb --- meta/recipes-multimedia/libpng/libpng_1.6.34.bb | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-multimedia/libpng/libpng_1.6.34.bb +3 -0
@@ -30,3 +30,6 @@ PACKAGES =+ "${PN}-tools" FILES_${PN}-tools = "${bindir}/png-fix-itxt ${bindir}/pngfix ${bindir}/pngcp" BBCLASSEXTEND = "native nativesdk" + +# CVE-2019-17371 is actually a memory leak in gif2png 2.x +CVE_CHECK_WHITELIST += "CVE-2019-17371"
Signature · 1 line
1.9.1
RFC PATCH 42/47 openssl: set CVE vendor to openssl
In reply to Mikko RapeliFrom: Anuj Mittal <[email protected]>
Differentiate it from openssl gem for Ruby.
(From OE-Core rev: 2ec481b19d6c9c20ce6573de77ae89e576d6b8cb)
Conflicts:
meta/recipes-connectivity/openssl/openssl_1.1.1c.bb
--- meta/recipes-connectivity/openssl/openssl_1.0.2p.bb | 2 ++ meta/recipes-connectivity/openssl/openssl_1.1.0i.bb | 2 ++ 2 files changed, 4 insertions(+)
meta/recipes-connectivity/openssl/openssl_1.0.2p.bb +2 -0
@@ -342,3 +342,5 @@ RDEPENDS_${PN}-misc = "${@bb.utils.filter('PACKAGECONFIG', 'perl', d)}" RDEPENDS_${PN}-ptest += "${PN}-misc make perl perl-module-filehandle bc" BBCLASSEXTEND = "native nativesdk" + +CVE_PRODUCT = "openssl:openssl"
meta/recipes-connectivity/openssl/openssl_1.1.0i.bb +2 -0
@@ -168,3 +168,5 @@ FILES_${PN}-engines = "${libdir}/engines-1.1" RDEPENDS_${PN}-ptest += "perl-module-file-spec-functions bash python" BBCLASSEXTEND = "native nativesdk" + +CVE_PRODUCT = "openssl:openssl"
Signature · 1 line
1.9.1
RFC PATCH 43/47 rsync: fix CVEs for included zlib
In reply to Mikko RapeliFrom: Anuj Mittal <[email protected]>
rsync includes its own copy of zlib and doesn't recommend linking with the system version [1].
Import CVE fixes that impact zlib version 1.2.8 [2] that is currently used by rsync.
[1] https://git.samba.org/rsync.git/?p=rsync.git;a=blob;f=zlib/README.rsync
[2] https://nvd.nist.gov/vuln/search/results?form_type=Advanced&cves=on&cpe_version=cpe%3a%2fa%3agnu%3azlib%3a1.2.8
(From OE-Core rev: a55fbb4cb489853dfb0b4553f6e187c3f3633f48)
Conflicts:
meta/recipes-devtools/rsync/rsync_3.1.3.bb --- .../rsync/files/CVE-2016-9840.patch | 75 +++++++ .../rsync/files/CVE-2016-9841.patch | 228 +++++++++++++++++++++ .../rsync/files/CVE-2016-9842.patch | 33 +++ .../rsync/files/CVE-2016-9843.patch | 53 +++++ meta/recipes-devtools/rsync/rsync_3.1.3.bb | 7 +- 5 files changed, 395 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9840.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9841.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9842.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9843.patch
meta/recipes-devtools/rsync/files/CVE-2016-9840.patch new file +75 -0
@@ -0,0 +1,75 @@ +From 6a043145ca6e9c55184013841a67b2fef87e44c0 Mon Sep 17 00:00:00 2001 +From: Mark Adler <[email protected]> +Date: Wed, 21 Sep 2016 23:35:50 -0700 +Subject: [PATCH] Remove offset pointer optimization in inftrees.c. + +inftrees.c was subtracting an offset from a pointer to an array, +in order to provide a pointer that allowed indexing starting at +the offset. This is not compliant with the C standard, for which +the behavior of a pointer decremented before its allocated memory +is undefined. Per the recommendation of a security audit of the +zlib code by Trail of Bits and TrustInSoft, in support of the +Mozilla Foundation, this tiny optimization was removed, in order +to avoid the possibility of undefined behavior. + +CVE: CVE-2016-9840 +Upstream-Status: Backport +Signed-off-by: Anuj Mittal <[email protected]> +--- + inftrees.c | 18 ++++++++---------- + 1 file changed, 8 insertions(+), 10 deletions(-) + +diff --git a/zlib/inftrees.c b/zlib/inftrees.c +index 22fcd666..0d2670d5 100644 +--- a/zlib/inftrees.c ++++ b/zlib/inftrees.c +@@ -54,7 +54,7 @@ unsigned short FAR *work; + code FAR *next; /* next available space in table */ + const unsigned short FAR *base; /* base value table to use */ + const unsigned short FAR *extra; /* extra bits table to use */ +- int end; /* use base and extra for symbol > end */ ++ unsigned match; /* use base and extra for symbol >= match */ + unsigned short count[MAXBITS+1]; /* number of codes of each length */ + unsigned short offs[MAXBITS+1]; /* offsets in table for each length */ + static const unsigned short lbase[31] = { /* Length codes 257..285 base */ +@@ -181,19 +181,17 @@ unsigned short FAR *work; + switch (type) { + case CODES: + base = extra = work; /* dummy value--not used */ +- end = 19; ++ match = 20; + break; + case LENS: + base = lbase; +- base -= 257; + extra = lext; +- extra -= 257; +- end = 256; ++ match = 257; + break; + default: /* DISTS */ + base = dbase; + extra = dext; +- end = -1; ++ match = 0; + } + + /* initialize state for loop */ +@@ -216,13 +214,13 @@ unsigned short FAR *work; + for (;;) { + /* create table entry */ + here.bits = (unsigned char)(len - drop); +- if ((int)(work[sym]) < end) { ++ if (work[sym] + 1 < match) { + here.op = (unsigned char)0; + here.val = work[sym]; + } +- else if ((int)(work[sym]) > end) { +- here.op = (unsigned char)(extra[work[sym]]); +- here.val = base[work[sym]]; ++ else if (work[sym] >= match) { ++ here.op = (unsigned char)(extra[work[sym] - match]); ++ here.val = base[work[sym] - match]; + } + else { + here.op = (unsigned char)(32 + 64); /* end of block */
meta/recipes-devtools/rsync/files/CVE-2016-9841.patch new file +228 -0
228 lines not on this page · raw message
meta/recipes-devtools/rsync/files/CVE-2016-9842.patch new file +33 -0
@@ -0,0 +1,33 @@ +From e54e1299404101a5a9d0cf5e45512b543967f958 Mon Sep 17 00:00:00 2001 +From: Mark Adler <[email protected]> +Date: Sat, 5 Sep 2015 17:45:55 -0700 +Subject: [PATCH] Avoid shifts of negative values inflateMark(). + +The C standard says that bit shifts of negative integers is +undefined. This casts to unsigned values to assure a known +result. + +CVE: CVE-2016-9842 +Upstream-Status: Backport +Signed-off-by: Anuj Mittal <[email protected]> +--- + inflate.c | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +diff --git a/zlib/inflate.c b/zlib/inflate.c +index 2889e3a0..a7184167 100644 +--- a/zlib/inflate.c ++++ b/zlib/inflate.c +@@ -1506,9 +1506,10 @@ z_streamp strm; + { + struct inflate_state FAR *state; + +- if (strm == Z_NULL || strm->state == Z_NULL) return -1L << 16; ++ if (strm == Z_NULL || strm->state == Z_NULL) ++ return (long)(((unsigned long)0 - 1) << 16); + state = (struct inflate_state FAR *)strm->state; +- return ((long)(state->back) << 16) + ++ return (long)(((unsigned long)((long)state->back)) << 16) + + (state->mode == COPY ? state->length : + (state->mode == MATCH ? state->was - state->length : 0)); + }
meta/recipes-devtools/rsync/files/CVE-2016-9843.patch new file +53 -0
53 lines not on this page · raw message
meta/recipes-devtools/rsync/rsync_3.1.3.bb +6 -1
@@ -1,6 +1,11 @@ require rsync.inc -SRC_URI += "file://makefile-no-rebuild.patch" +SRC_URI += "file://makefile-no-rebuild.patch \ + file://CVE-2016-9840.patch \ + file://CVE-2016-9841.patch \ + file://CVE-2016-9842.patch \ + file://CVE-2016-9843.patch \ +" SRC_URI[md5sum] = "1581a588fde9d89f6bc6201e8129afaf" SRC_URI[sha256sum] = "55cc554efec5fdaad70de921cd5a5eeb6c29a95524c715f3bbf849235b0800c0"
Signature · 1 line
1.9.1
RFC PATCH 44/47 ed: set CVE vendor to avoid false positives
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
(From OE-Core rev: 2c3d689e4f78d8ea00b1bd2239af80c8fe038074)
Conflicts:
meta/recipes-extended/ed/ed_1.15.bb --- meta/recipes-extended/ed/ed_1.14.2.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-extended/ed/ed_1.14.2.bb +2 -0
@@ -8,6 +8,8 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=0c7051aef9219dc7237f206c5c4179a7 \ SECTION = "base" +CVE_PRODUCT = "gnu:ed" + # LSB states that ed should be in /bin/ bindir = "${base_bindir}"
Signature · 1 line
1.9.1
RFC PATCH 45/47 boost: set CVE vendor to Boost
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
There's a Boost module for Drupal.
(From OE-Core rev: 30ff8bb6502d45549c698be052a1caf4cb5c611f)
--- meta/recipes-support/boost/boost.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-support/boost/boost.inc +2 -0
@@ -2,6 +2,8 @@ SUMMARY = "Free peer-reviewed portable C++ source libraries" SECTION = "libs" DEPENDS = "bjam-native zlib bzip2" +CVE_PRODUCT = "boost:boost" + ARM_INSTRUCTION_SET_armv4 = "arm" ARM_INSTRUCTION_SET_armv5 = "arm"
Signature · 1 line
1.9.1
RFC PATCH 46/47 subversion: set CVE vendor to Apache
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
There's a Jenkins plugin for Subversion.
(From OE-Core rev: ac115c3b5f1dcb95fb7d39537693fe0dcd330451)
Conflicts:
meta/recipes-devtools/subversion/subversion_1.12.0.bb
--- meta/recipes-devtools/subversion/subversion_1.9.7.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-devtools/subversion/subversion_1.9.7.bb +2 -0
@@ -10,6 +10,8 @@ BBCLASSEXTEND = "native" inherit gettext pkgconfig +CVE_PRODUCT = "apache:subversion" + SRC_URI = "${APACHE_MIRROR}/${BPN}/${BPN}-${PV}.tar.bz2 \ file://disable_macos.patch \ file://serf.m4-Regex-modified-to-allow-D-in-paths.patch \
Signature · 1 line
1.9.1
RFC PATCH 47/47 git: set CVE vendor to git-scm
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
There's a Jenkins plugin for Git.
(From OE-Core rev: f2adf5e4d3e9afc6d45665bbe728c69d195a46ef)
--- meta/recipes-devtools/git/git.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-devtools/git/git.inc +2 -0
@@ -15,6 +15,8 @@ S = "${WORKDIR}/git-${PV}" LIC_FILES_CHKSUM = "file://COPYING;md5=7c0d7ef03a7eb04ce795b0f60e68e7e1" +CVE_PRODUCT = "git-scm:git" + PACKAGECONFIG ??= "" PACKAGECONFIG[cvsserver] = "" PACKAGECONFIG[svn] = ""
Signature · 1 line
1.9.1
RFC PATCH 1/47 cve-update-db: New recipe to update CVE database
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
cve-check-tool-native do_populate_cve_db task was using deprecated NVD xml data feeds, cve-update-db uses NVD json data feeds.
Sqlite database schema was updated to take into account CVSSv3 CVE scores and operator in affected product versions.
A new META table was added to store the last modification date of the NVD json data feeds.
(From OE-Core rev: 546d14135c50c6a571dfbf3baf6e9b22ce3d58e0)
Conflicts:
meta/conf/distro/include/maintainers.inc
--- meta/conf/distro/include/maintainers.inc | 2 +- meta/recipes-core/meta/cve-update-db.bb | 121 +++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-core/meta/cve-update-db.bb
meta/conf/distro/include/maintainers.inc +1 -1
@@ -122,7 +122,7 @@ RECIPE_MAINTAINER_pn-cryptodev-module = "Robert Yang <[email protected]> RECIPE_MAINTAINER_pn-cryptodev-tests = "Robert Yang <[email protected]>" RECIPE_MAINTAINER_pn-cups = "Chen Qi <[email protected]>" RECIPE_MAINTAINER_pn-curl = "Armin Kuster <[email protected]>" -RECIPE_MAINTAINER_pn-cve-check-tool = "Ross Burton <[email protected]>" +RECIPE_MAINTAINER_pn-cve-update-db = "Ross Burton <[email protected]>" RECIPE_MAINTAINER_pn-cwautomacros = "Maxin B. John <[email protected]>" RECIPE_MAINTAINER_pn-damageproto = "Armin Kuster <[email protected]>" RECIPE_MAINTAINER_pn-db = "Mark Hatle <[email protected]>"
meta/recipes-core/meta/cve-update-db.bb new file +121 -0
121 lines not on this page · raw message
Signature · 1 line
1.9.1
RFC PATCH 39/47 libpam: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
(From OE-Core rev: f1d5273d53d66b217f3d4975f5cb5eb367b1aab1)
Conflicts:
meta/recipes-extended/pam/libpam_1.3.1.bb --- meta/recipes-extended/pam/libpam_1.3.0.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-extended/pam/libpam_1.3.0.bb +2 -0
@@ -49,6 +49,8 @@ S = "${WORKDIR}/Linux-PAM-${PV}" inherit autotools gettext pkgconfig +CVE_PRODUCT = "linux-pam" + PACKAGECONFIG[audit] = "--enable-audit,--disable-audit,audit," PACKAGES += "${PN}-runtime ${PN}-xtests"
Signature · 1 line
1.9.1
RFC PATCH 37/47 flex: set CVE_PRODUCT to include vendor
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
There are many projects called Flex and they have CVEs, so also set the vendor to remove these false positives.
(From OE-Core rev: 0598ccdcb31e16f1d1227197591b10ba441fcfe2)
--- meta/recipes-devtools/flex/flex_2.6.0.bb | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-devtools/flex/flex_2.6.0.bb +3 -0
@@ -68,3 +68,6 @@ do_install_ptest() { -e 's/^builddir = \(.*\)/builddir = ./' -e 's/^top_builddir = \(.*\)/top_builddir = ./' \ -i ${D}${PTEST_PATH}/Makefile } + +# Not Apache Flex, or Adobe Flex, or IBM Flex. +CVE_PRODUCT = "flex_project:flex"
Signature · 1 line
1.9.1
RFC PATCH 5/47 flac: also add flac to CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
flac uses both 'flac' and 'libflac' as cve product.
(From OE-Core rev: 3a043a078f6cc89bcc097823fa37cd1311805ae7)
--- meta/recipes-multimedia/flac/flac_1.3.2.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-multimedia/flac/flac_1.3.2.bb +1 -1
@@ -20,7 +20,7 @@ SRC_URI = "http://downloads.xiph.org/releases/flac/${BP}.tar.xz \ SRC_URI[md5sum] = "454f1bfa3f93cc708098d7890d0499bd" SRC_URI[sha256sum] = "91cfc3ed61dc40f47f050a109b08610667d73477af6ef36dcad31c31a4a8d53f" -CVE_PRODUCT = "libflac" +CVE_PRODUCT = "libflac flac" inherit autotools gettext
Signature · 1 line
1.9.1
RFC PATCH 38/47 webkitgtk: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 43aaa117386490c822b824974fb095bd0d3ce1a3)
Conflicts:
meta/recipes-sato/webkit/webkitgtk_2.24.0.bb --- meta/recipes-sato/webkit/webkitgtk_2.18.6.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-sato/webkit/webkitgtk_2.18.6.bb +2 -0
@@ -32,6 +32,8 @@ inherit cmake pkgconfig gobject-introspection perlnative distro_features_check u # depends on libxt REQUIRED_DISTRO_FEATURES = "x11" +CVE_PRODUCT = "webkitgtk webkitgtk\+" + DEPENDS = "zlib libsoup-2.4 curl libxml2 cairo libxslt libxt libidn libgcrypt \ gtk+3 gstreamer1.0 gstreamer1.0-plugins-base flex-native gperf-native sqlite3 \ pango icu bison-native gawk intltool-native libwebp \
Signature · 1 line
1.9.1
RFC PATCH 22/47 cve-update-db-native: use os.path.join instead of +
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
(From OE-Core rev: 4b301030cf9cf7a981dcff85a50e915c045e3130)
--- meta/recipes-core/meta/cve-update-db-native.bb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +4 -4
@@ -24,11 +24,11 @@ python do_populate_cve_db() { BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-" YEAR_START = 2002 - db_dir = d.getVar("DL_DIR") + '/CVE_CHECK' - db_file = db_dir + '/nvdcve_1.0.db' - json_tmpfile = db_dir + '/nvd.json.gz' + db_dir = os.path.join(d.getVar("DL_DIR"), 'CVE_CHECK') + db_file = os.path.join(db_dir, 'nvdcve_1.0.db') + json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') proxy = d.getVar("https_proxy") - cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a') + cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') if not os.path.isdir(db_dir): os.mkdir(db_dir)
Signature · 1 line
1.9.1
RFC PATCH 23/47 cve-update-db: actually inherit native
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
The recipe was called -native but didn't inherit native.
(From OE-Core rev: f0d822fad2a163d1ee32ed3b4c0359245140e19b)
--- meta/recipes-core/meta/cve-update-db-native.bb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +1 -2
@@ -2,9 +2,8 @@ SUMMARY = "Updates the NVD CVE database" LICENSE = "MIT" INHIBIT_DEFAULT_DEPS = "1" -PACKAGES = "" -inherit nopackages +inherit native deltask do_unpack deltask do_patch
Signature · 1 line
1.9.1
RFC PATCH 25/47 cve-update-db-native: improve metadata parsing
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
The metadata parser is fragile: first it coerces a bytes() to a str() (so the string is b'LastModifiedDate:2019...'), assumes the first line is the date, and then uses a regex to parse (which then includes the trailing quote as part of the date).
Clean this up by parsing the bytes as UTF-8 (ASCII is probably fine, but this is safer), iterate through the lines and split on colons to find the right key/value pair.
(From OE-Core rev: bb4e53af33d6ca1e9346464adbdc1b39c47530f3)
--- meta/recipes-core/meta/cve-update-db-native.bb | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +10 -8
@@ -17,7 +17,7 @@ python do_populate_cve_db() { Update NVD database with json data feed """ - import sqlite3, urllib, shutil, gzip, re + import sqlite3, urllib, shutil, gzip from datetime import date BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
@@ -47,13 +47,15 @@ python do_populate_cve_db() { req = urllib.request.Request(meta_url) if proxy: req.set_proxy(proxy, 'https') - try: - with urllib.request.urlopen(req, timeout=1) as r: - date_line = str(r.read().splitlines()[0]) - last_modified = re.search('lastModifiedDate:(.*)', date_line).group(1) - except: - cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break + with urllib.request.urlopen(req) as r: + for l in r.read().decode("utf-8").splitlines(): + key, value = l.split(":", 1) + if key == "lastModifiedDate": + last_modified = value + break + else: + bb.warn("Cannot parse CVE metadata, update failed") + return # Compare with current db last modified date c.execute("select DATE from META where YEAR = ?", (year,))
Signature · 1 line
1.9.1
RFC PATCH 26/47 cve-update-db-native: clean up JSON fetching
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
Currently the code fetches the compressed JSON, writes it to a temporary file, uncompresses that with gzip and passes the fake file object to update_db().
Instead, uncompress the gzip'd data in memory and pass the JSON directly to update_db().
(From OE-Core rev: 9422745979256c442f533770203f62ec071c18fb)
--- meta/recipes-core/meta/cve-update-db-native.bb | 29 +++++++++++--------------- 1 file changed, 12 insertions(+), 17 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +12 -17
@@ -62,25 +62,20 @@ python do_populate_cve_db() { meta = c.fetchone() if not meta or meta[0] != last_modified: # Clear products table entries corresponding to current year - cve_year = 'CVE-' + str(year) + '%' - c.execute("delete from PRODUCTS where ID like ?", (cve_year,)) + c.execute("delete from PRODUCTS where ID like ?", ('CVE-%d%%' % year,)) # Update db with current year json file - req = urllib.request.Request(json_url) - if proxy: - req.set_proxy(proxy, 'https') try: - with urllib.request.urlopen(req, timeout=1) as r, \ - open(json_tmpfile, 'wb') as tmpfile: - shutil.copyfileobj(r, tmpfile) - except: + req = urllib.request.Request(json_url) + if proxy: + req.set_proxy(proxy, 'https') + with urllib.request.urlopen(req) as r: + update_db(c, gzip.decompress(r.read())) + c.execute("insert or replace into META values (?, ?)", [year, last_modified]) + except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n') - break - - with gzip.open(json_tmpfile, 'rt') as jsonfile: - update_db(c, jsonfile) - c.execute("insert or replace into META values (?, ?)", - [year, last_modified]) + bb.warn("Cannot parse CVE data (%s), update failed" % e.reason) + return # Update success, set the date to cve_check file. if year == date.today().year:
@@ -143,9 +138,9 @@ def parse_node_and_insert(c, node, cveId): c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) -def update_db(c, json_filename): +def update_db(c, jsondata): import json - root = json.load(json_filename) + root = json.loads(jsondata) for elt in root['CVE_Items']: if not elt['impact']:
Signature · 1 line
1.9.1
RFC PATCH 27/47 cve-update-db-native: fix https proxy issues
In reply to Mikko RapeliFrom: Chin Huat Ang <[email protected]>
When https_proxy is set, use proxy opener to open CVE metadata and database URLs, otherwise fallback to the urllib.request.urlopen.
Also fix a minor issue where the json database which has been gzip decompressed as byte object should be decoded as utf-8 string as expected by update_db.
(From OE-Core rev: 95438d52b732bec217301fbfc2fb019bbc3707c8)
--- meta/recipes-core/meta/cve-update-db-native.bb | 41 +++++++++++++++++++------- 1 file changed, 30 insertions(+), 11 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +30 -11
@@ -17,7 +17,7 @@ python do_populate_cve_db() { Update NVD database with json data feed """ - import sqlite3, urllib, shutil, gzip + import sqlite3, urllib, urllib.parse, shutil, gzip from datetime import date BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-"
@@ -27,6 +27,16 @@ python do_populate_cve_db() { db_file = os.path.join(db_dir, 'nvdcve_1.0.db') json_tmpfile = os.path.join(db_dir, 'nvd.json.gz') proxy = d.getVar("https_proxy") + + if proxy: + # instantiate an opener but do not install it as the global + # opener unless if we're really sure it's applicable for all + # urllib requests + proxy_handler = urllib.request.ProxyHandler({'https': proxy}) + proxy_opener = urllib.request.build_opener(proxy_handler) + else: + proxy_opener = None + cve_f = open(os.path.join(d.getVar("TMPDIR"), 'cve_check'), 'a') if not os.path.isdir(db_dir):
@@ -44,11 +54,17 @@ python do_populate_cve_db() { json_url = year_url + ".json.gz" # Retrieve meta last modified date - req = urllib.request.Request(meta_url) - if proxy: - req.set_proxy(proxy, 'https') - with urllib.request.urlopen(req) as r: - for l in r.read().decode("utf-8").splitlines(): + + response = None + + if proxy_opener: + response = proxy_opener.open(meta_url) + else: + req = urllib.request.Request(meta_url) + response = urllib.request.urlopen(req) + + if response: + for l in response.read().decode("utf-8").splitlines(): key, value = l.split(":", 1) if key == "lastModifiedDate": last_modified = value
@@ -66,11 +82,14 @@ python do_populate_cve_db() { # Update db with current year json file try: - req = urllib.request.Request(json_url) - if proxy: - req.set_proxy(proxy, 'https') - with urllib.request.urlopen(req) as r: - update_db(c, gzip.decompress(r.read())) + if proxy_opener: + response = proxy_opener.open(json_url) + else: + req = urllib.request.Request(json_url) + response = urllib.request.urlopen(req) + + if response: + update_db(c, gzip.decompress(response.read()).decode('utf-8')) c.execute("insert or replace into META values (?, ?)", [year, last_modified]) except urllib.error.URLError as e: cve_f.write('Warning: CVE db update error, CVE data is outdated.\n\n')
Signature · 1 line
1.9.1
RFC PATCH 28/47 cve-check: ensure all known CVEs are in the report
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
CVEs that are whitelisted or were not vulnerable when there are version comparisons were not included in the report, so alter the logic to ensure that all relevant CVEs are in the report for completeness.
(From OE-Core rev: 98256ff05fcfe9d5ccad360582c36eafb577c264)
--- meta/classes/cve-check.bbclass | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
meta/classes/cve-check.bbclass +7 -2
@@ -208,12 +208,14 @@ def check_cves(d, patched_cves): if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve)) + # TODO: this should be in the report as 'whitelisted' + patched_cves.add(cve) elif cve in patched_cves: bb.note("%s has been patched" % (cve)) else: to_append = False if (operator_start == '=' and pv == version_start): - cves_unpatched.append(cve) + to_append = True else: if operator_start: try:
@@ -243,8 +245,11 @@ def check_cves(d, patched_cves): to_append = to_append_start or to_append_end if to_append: + bb.note("%s-%s is vulnerable to %s" % (product, pv, cve)) cves_unpatched.append(cve) - bb.debug(2, "%s-%s is not patched for %s" % (product, pv, cve)) + else: + bb.note("%s-%s is not vulnerable to %s" % (product, pv, cve)) + patched_cves.add(cve) conn.close() return (list(patched_cves), cves_unpatched)
Signature · 1 line
1.9.1
RFC PATCH 29/47 cve-check: failure to parse versions should be more visible
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
--- meta/classes/cve-check.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/cve-check.bbclass +2 -2
@@ -222,7 +222,7 @@ def check_cves(d, patched_cves): to_append_start = (operator_start == '>=' and LooseVersion(pv) >= LooseVersion(version_start)) to_append_start |= (operator_start == '>' and LooseVersion(pv) > LooseVersion(version_start)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_start, version_start, cve)) to_append_start = False else:
@@ -233,7 +233,7 @@ def check_cves(d, patched_cves): to_append_end = (operator_end == '<=' and LooseVersion(pv) <= LooseVersion(version_end)) to_append_end |= (operator_end == '<' and LooseVersion(pv) < LooseVersion(version_end)) except: - bb.note("%s: Failed to compare %s %s %s for %s" % + bb.warn("%s: Failed to compare %s %s %s for %s" % (product, pv, operator_end, version_end, cve)) to_append_end = False else:
Signature · 1 line
1.9.1
RFC PATCH 30/47 xserver-xorg: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 8995f2c7d6f2f6f760811976af77e949d505a5d8)
--- meta/recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-graphics/xorg-xserver/xserver-xorg.inc +2 -0
@@ -19,6 +19,8 @@ XORG_PN = "xorg-server" SRC_URI = "${XORG_MIRROR}/individual/xserver/${XORG_PN}-${PV}.tar.bz2" SRC_URI += "file://macro_tweak.patch" +CVE_PRODUCT = "xorg-server" + S = "${WORKDIR}/${XORG_PN}-${PV}" inherit autotools pkgconfig
Signature · 1 line
1.9.1
RFC PATCH 31/47 nasm: add CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: e61c42ee49029ae8ffec58128dd083031305d9e5)
Conflicts:
meta/recipes-devtools/nasm/nasm_2.14.02.bb --- meta/recipes-devtools/nasm/nasm_2.13.03.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-devtools/nasm/nasm_2.13.03.bb +2 -0
@@ -33,3 +33,5 @@ do_install() { BBCLASSEXTEND = "native" DEPENDS = "groff-native" + +CVE_PRODUCT = "netwide_assembler"
Signature · 1 line
1.9.1
RFC PATCH 32/47 dropbear: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 3c247a4a166cabf7ddfea403cf272b3fb4e00872)
--- meta/recipes-core/dropbear/dropbear.inc | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-core/dropbear/dropbear.inc +2 -0
@@ -37,6 +37,8 @@ RDEPENDS_${PN} += "${@bb.utils.contains('DISTRO_FEATURES', 'pam', '${PAM_PLUGINS inherit autotools update-rc.d systemd +CVE_PRODUCT = "dropbear_ssh" + INITSCRIPT_NAME = "dropbear" INITSCRIPT_PARAMS = "defaults 10"
Signature · 1 line
1.9.1
RFC PATCH 33/47 libsdl: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 1f0cca19014fef24a359d400c96d178463b2760f)
Conflicts:
meta/recipes-graphics/libsdl2/libsdl2_2.0.9.bb --- meta/recipes-graphics/libsdl/libsdl_1.2.15.bb | 2 ++ meta/recipes-graphics/libsdl2/libsdl2_2.0.8.bb | 2 ++ 2 files changed, 4 insertions(+)
meta/recipes-graphics/libsdl/libsdl_1.2.15.bb +2 -0
@@ -31,6 +31,8 @@ BINCONFIG = "${bindir}/sdl-config" inherit autotools lib_package binconfig-disabled pkgconfig +CVE_PRODUCT = "simple_directmedia_layer sdl" + EXTRA_OECONF = "--disable-static --enable-cdrom --enable-threads --enable-timers \ --enable-file --disable-oss --disable-esd --disable-arts \ --disable-diskaudio --disable-nas \
meta/recipes-graphics/libsdl2/libsdl2_2.0.8.bb +2 -0
@@ -26,6 +26,8 @@ SRC_URI[sha256sum] = "edc77c57308661d576e843344d8638e025a7818bff73f8fbfab09c3c5f inherit autotools lib_package binconfig pkgconfig +CVE_PRODUCT = "simple_directmedia_layer sdl" + EXTRA_OECONF = "--disable-oss --disable-esd --disable-arts \ --disable-diskaudio --disable-nas --disable-esd-shared --disable-esdtest \ --disable-video-dummy \
Signature · 1 line
1.9.1
RFC PATCH 34/47 ghostscript: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 721e69aa12dd9ee22618ef13f29fb6d28eeab9af)
Conflicts:
meta/recipes-extended/ghostscript/ghostscript_9.26.bb
--- meta/recipes-extended/ghostscript/ghostscript_9.21.bb | 3 +++ 1 file changed, 3 insertions(+)
meta/recipes-extended/ghostscript/ghostscript_9.21.bb +3 -0
@@ -134,3 +134,6 @@ BBCLASSEXTEND = "native" # ghostscript does not supports "arc" COMPATIBLE_HOST = "^(?!arc).*" + +# some entries in NVD uses gpl_ghostscript +CVE_PRODUCT = "ghostscript gpl_ghostscript"
Signature · 1 line
1.9.1
RFC PATCH 35/47 squashfs-tools: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 8f03a33f61a94e9b8d8232283204588ce18b45a0)
--- meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb +2 -0
@@ -40,3 +40,5 @@ do_install () { ARM_INSTRUCTION_SET = "arm" BBCLASSEXTEND = "native nativesdk" + +CVE_PRODUCT = "squashfs"
Signature · 1 line
1.9.1
RFC PATCH 36/47 libxfont2: set CVE_PRODUCT
In reply to Mikko RapeliFrom: Chen Qi <[email protected]>
(From OE-Core rev: 066fa83eeaaa34e5b901dc4b82ad607d0fa78f0b)
--- meta/recipes-graphics/xorg-lib/libxfont2_2.0.3.bb | 2 ++ 1 file changed, 2 insertions(+)
meta/recipes-graphics/xorg-lib/libxfont2_2.0.3.bb +2 -0
@@ -20,3 +20,5 @@ SRC_URI[sha256sum] = "0e8ab7fd737ccdfe87e1f02b55f221f0bd4503a1c5f28be4ed6a54586b PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}" PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6," + +CVE_PRODUCT = "libxfont libxfont2"
Signature · 1 line
1.9.1
RFC PATCH 24/47 cve-update-db-native: use executemany() to optimise CPE insertion
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
Instead of calling execute() repeatedly, rewrite the function to be a generator and use executemany() for performance.
(From OE-Core rev: b309840b6aa3423b909a43499356e929c8761318)
--- meta/recipes-core/meta/cve-update-db-native.bb | 85 ++++++++++---------------- 1 file changed, 32 insertions(+), 53 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +32 -53
@@ -97,70 +97,49 @@ def initialize_db(c): VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") -def insert_elt(c, db_values): - query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)" - c.execute(query, db_values) - def parse_node_and_insert(c, node, cveId): # Parse children node if needed - try: - for child in node['children']: - parse_node_and_insert(c, child, cveId) - except: - pass - - # Exit if the cpe_match node does not exists - try: - cpe_match = node['cpe_match'] - except: - return - - for cpe in cpe_match: - if not cpe['vulnerable']: - return - cpe23 = cpe['cpe23Uri'].split(':') - vendor = cpe23[3] - product = cpe23[4] - version = cpe23[5] - - if version != '*': - # Version is defined, this is a '=' match - db_values = [cveId, vendor, product, version, '=', '', ''] - insert_elt(c, db_values) - else: - # Parse start version, end version and operators - op_start = '' - op_end = '' - v_start = '' - v_end = '' - - try: - if cpe['versionStartIncluding']: + for child in node.get('children', ()): + parse_node_and_insert(c, child, cveId) + + def cpe_generator(): + for cpe in node.get('cpe_match', ()): + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + yield [cveId, vendor, product, version, '=', '', ''] + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + if 'versionStartIncluding' in cpe: op_start = '>=' v_start = cpe['versionStartIncluding'] - except: - pass - try: - if cpe['versionStartExcluding']: + + if 'versionStartExcluding' in cpe: op_start = '>' v_start = cpe['versionStartExcluding'] - except: - pass - try: - if cpe['versionEndIncluding']: + + if 'versionEndIncluding' in cpe: op_end = '<=' v_end = cpe['versionEndIncluding'] - except: - pass - try: - if cpe['versionEndExcluding']: + + if 'versionEndExcluding' in cpe: op_end = '<' v_end = cpe['versionEndExcluding'] - except: - pass - db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] - insert_elt(c, db_values) + yield [cveId, vendor, product, v_start, op_start, v_end, op_end] + + c.executemany("insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)", cpe_generator()) def update_db(c, json_filename): import json
Signature · 1 line
1.9.1
RFC PATCH 20/47 cve-update-db: Use NVD CPE data to populate PRODUCTS table
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
Instead of using expanded list of affected versions that is not reliable, use the 'cpe_match' node in the 'configurations' json node.
For cve-check to correctly match affected CVE, the sqlite database need to contain operator_start, operator_end and the corresponding versions fields.
(From OE-Core rev: f7676e9a38d595564922e5f59acbc69c2109a78f)
--- meta/recipes-core/meta/cve-update-db-native.bb | 88 ++++++++++++++++++++++---- 1 file changed, 74 insertions(+), 14 deletions(-)
meta/recipes-core/meta/cve-update-db-native.bb +74 -14
@@ -25,7 +25,7 @@ python do_populate_cve_db() { YEAR_START = 2002 db_dir = d.getVar("DL_DIR") + '/CVE_CHECK' - db_file = db_dir + '/nvd-json.db' + db_file = db_dir + '/nvdcve.db' json_tmpfile = db_dir + '/nvd.json.gz' proxy = d.getVar("https_proxy") cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a')
@@ -99,9 +99,76 @@ def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \ - VENDOR TEXT, PRODUCT TEXT, VERSION TEXT, OPERATOR TEXT)") - c.execute("CREATE INDEX IF NOT EXISTS PRODUCT_IDX ON PRODUCTS \ - (PRODUCT, VERSION)") + VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ + VERSION_END TEXT, OPERATOR_END TEXT)") + +def insert_elt(c, db_values): + product_str = db_values[0] + db_values[1] + db_values[2] + db_values[3] + hashstr = hash_djb2(product_str) + db_values.insert(0, hashstr) + query = "insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?, ?, ?)" + c.execute(query, db_values) + +def parse_node_and_insert(c, node, cveId): + # Parse children node if needed + try: + for child in node['children']: + parse_node_and_insert(c, child, cveId) + except: + pass + + # Exit if the cpe_match node does not exists + try: + cpe_match = node['cpe_match'] + except: + return + + for cpe in cpe_match: + if not cpe['vulnerable']: + return + cpe23 = cpe['cpe23Uri'].split(':') + vendor = cpe23[3] + product = cpe23[4] + version = cpe23[5] + + if version != '*': + # Version is defined, this is a '=' match + db_values = [cveId, vendor, product, version, '=', '', ''] + insert_elt(c, db_values) + else: + # Parse start version, end version and operators + op_start = '' + op_end = '' + v_start = '' + v_end = '' + + try: + if cpe['versionStartIncluding']: + op_start = '>=' + v_start = cpe['versionStartIncluding'] + except: + pass + try: + if cpe['versionStartExcluding']: + op_start = '>' + v_start = cpe['versionStartExcluding'] + except: + pass + try: + if cpe['versionEndIncluding']: + op_end = '<=' + v_end = cpe['versionEndIncluding'] + except: + pass + try: + if cpe['versionEndExcluding']: + op_end = '<' + v_end = cpe['versionEndExcluding'] + except: + pass + + db_values = [cveId, vendor, product, v_start, op_start, v_end, op_end] + insert_elt(c, db_values) def update_db(c, json_filename): import json
@@ -125,16 +192,9 @@ def update_db(c, json_filename): c.execute("insert or replace into NVD values (?, ?, ?, ?, ?, ?)", [cveId, cveDesc, cvssv2, cvssv3, date, accessVector]) - for vendor in elt['cve']['affects']['vendor']['vendor_data']: - for product in vendor['product']['product_data']: - for version in product['version']['version_data']: - product_str = cveId+vendor['vendor_name']+product['product_name']+version['version_value'] - hashstr = hash_djb2(product_str) - c.execute("insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?)", - [ hashstr, cveId, vendor['vendor_name'], - product['product_name'], version['version_value'], - version['version_affected']]) - + configurations = elt['configurations']['nodes'] + for config in configurations: + parse_node_and_insert(c, config, cveId) addtask do_populate_cve_db before do_fetch
Signature · 1 line
1.9.1
RFC PATCH 21/47 cve-update-db-native: Remove hash column from database.
In reply to Mikko RapeliFrom: Pierre Le Magourou <[email protected]>
djb2 hash algorithm was found to do collisions, so the database was sometime missing data. Remove this hash mechanism, clear and populate elements from scratch in PRODUCTS table if the current year needs an update.
(From OE-Core rev: 78de2cb39d74b030cd4ec811bf6f9a6daa003d19)
--- meta/classes/cve-check.bbclass | 12 ++++++------ meta/recipes-core/meta/cve-update-db-native.bb | 21 +++++++-------------- 2 files changed, 13 insertions(+), 20 deletions(-)
meta/classes/cve-check.bbclass +6 -6
@@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}" CVE_VERSION ??= "${PV}" CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK" -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve.db" +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.0.db" CVE_CHECK_LOG ?= "${T}/cve.log" CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
@@ -200,11 +200,11 @@ def check_cves(d, patched_cves): c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,)) for row in c: - cve = row[1] - version_start = row[4] - operator_start = row[5] - version_end = row[6] - operator_end = row[7] + cve = row[0] + version_start = row[3] + operator_start = row[4] + version_end = row[5] + operator_end = row[6] if cve in cve_whitelist: bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
meta/recipes-core/meta/cve-update-db-native.bb +7 -14
@@ -25,7 +25,7 @@ python do_populate_cve_db() { YEAR_START = 2002 db_dir = d.getVar("DL_DIR") + '/CVE_CHECK' - db_file = db_dir + '/nvdcve.db' + db_file = db_dir + '/nvdcve_1.0.db' json_tmpfile = db_dir + '/nvd.json.gz' proxy = d.getVar("https_proxy") cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a')
@@ -60,6 +60,10 @@ python do_populate_cve_db() { c.execute("select DATE from META where YEAR = ?", (year,)) meta = c.fetchone() if not meta or meta[0] != last_modified: + # Clear products table entries corresponding to current year + cve_year = 'CVE-' + str(year) + '%' + c.execute("delete from PRODUCTS where ID like ?", (cve_year,)) + # Update db with current year json file req = urllib.request.Request(json_url) if proxy:
@@ -86,27 +90,16 @@ python do_populate_cve_db() { conn.close() } -# DJB2 hash algorithm -def hash_djb2(s): - hash = 5381 - for x in s: - hash = (( hash << 5) + hash) + ord(x) - - return hash & 0xFFFFFFFF - def initialize_db(c): c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)") c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \ SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)") - c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \ + c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \ VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \ VERSION_END TEXT, OPERATOR_END TEXT)") def insert_elt(c, db_values): - product_str = db_values[0] + db_values[1] + db_values[2] + db_values[3] - hashstr = hash_djb2(product_str) - db_values.insert(0, hashstr) - query = "insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?, ?, ?)" + query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)" c.execute(query, db_values) def parse_node_and_insert(c, node, cveId):
Signature · 1 line
1.9.1
RFC PATCH 19/47 cve-update-db-native: use SQL placeholders instead of format strings
In reply to Mikko RapeliFrom: Ross Burton <[email protected]>
(From OE-Core rev: 91770338f76ef35f3c4eeac216eb9d2b3188e575)
--- meta/recipes-core/meta/cve-update-db-native.bb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/meta/cve-update-db-native.bb +1 -1
@@ -57,7 +57,7 @@ python do_populate_cve_db() { break # Compare with current db last modified date - c.execute("select DATE from META where YEAR = '%d'" % year) + c.execute("select DATE from META where YEAR = ?", (year,)) meta = c.fetchone() if not meta or meta[0] != last_modified: # Update db with current year json file
Signature · 1 line
1.9.1
✗ patchtest: failure for CVE check backport
In reply to Mikko Rapeli== Series Details ==
Series: CVE check backport
Revision: 1
URL : https://patchwork.openembedded.org/series/20979/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Patch [RFC, CFH, sumo, 13/47] cve-check: remove redundant readline CVE whitelisting Issue Missing or incorrectly formatted CVE tag in included patch file [test_cve_tag_format] Suggested fix Correct or include the CVE tag on cve patch with format: "CVE: CVE-YYYY-XXXX"
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
Hello Mikko;
Quoted 20 lines · Mikko Rapeli
Hi,
Request for comments, call for help, LTS too?
Yocto 2.5 sumo isn't actively maintained by the Yocto Project anymore. But that does not mean that support for it needs to stop.
I use sumo and due to various reasons like BSP layers, binary compatibility, contracts etc can't update to newer release or to master branch. I suspect I'm not alone.
sumo CVE checking machinery is broken due to changes in
NIST and NVD (see
https://nvd.nist.gov/general/news/XML-Vulnerability-Feed-Retirement and
https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release ) so some backports from poky master/zeus are needed to fix the tooling. Thanks to Anuj, Chen, Chin, Pierre, Ross and others who fixed these on master branch!
Quoted 6 lines
The tooling will expose that sumo is severely lacking in security patches, but the tooling is a start for anyone interested, like me, to fill the gaps and publish patches for bitbake recipes we care about.
Could sumo be an LTS?
No, IMHO. We don't have the appropriate system in place to do so.
Quoted 5 lines
Well I hope so. The LTS proposal
http://lists.openembedded.org/pipermail/openembedded-architecture/2019-October/001665.html
https://docs.google.com/document/d/1AwAFDf52f_FoXksbHEVUMlu4hpcI0JMGVG-Kj_sUkyc/edit
from Yocto Project is great. Maybe as part of that work, someone could setup a really minimal set of QA on Yocto Project side to also test
QA resources have been a donation from Intel and Windriver above their membership fees. I don't fee right asking them to run QA.
Quoted 3 lines
patches aiming at yocto 2.5 sumo. If not, would be really nice if someone could collect patches into sumo-next or sumo-contrib branch where us users could be in charge of all Quality Assurance.
I have collected other patches for sumo and built them locally but I have no way to inform Richard they pass an AB builds or automated testing for them to get into mainline sumo.
I am placing them into
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/sumo-community
Quoted 2 lines
So, comments and review are welcome. Patches even more so!
i will look at them when i get back from vacation.
Thanks for sending them,
regards,
Armin
Quoted 226 lines
Patches were tested on an x86 product tree where full stack CVE analysis produces good results. Then I ported them to pure poky sumo and ran core-image-minimal build. Tried running "bitbake world" build which also succeeds. The results show following bitbake target recipes from poky with unpatched CVEs (ignored native, SDK and cross tools for now):
build/tmp/deploy/cve$ grep -l "Unpatched" * | egrep -v -- "-native|nativesdk-|-cross" | sort apt
aspell
binutils
bluez5
busybox
bzip2
cairo
cups
curl
db
dropbear
elfutils
epiphany
expat
file
gcc
gcc-runtime
gcc-sanitizers
gcc-source-7.3.0
ghostscript
git
glib-2.0
glibc
gnupg
gnutls
go
gstreamer1.0
libarchive
libcomps
libcroco
libexif
libgcc
libgcrypt
libid3tag
libjpeg-turbo
libpcap
libpcre
libpng
librsvg
libsndfile1
libsolv
libvorbis
libx11
libxkbcommon
libxslt
lighttpd
lz4
nasm
ncurses
openssh
openssl
pango
patch
pcmanfm
perl
python
python3
qemu
shadow
sqlite3
sudo
sysstat
systemd
tar
tiff
unzip
webkitgtk
wget
wpa-supplicant
xdg-utils
xserver-xorg
zipSampling on the data shows that
* openssl 1.0.2p is missing patch for CVE-2019-1559
* openssh 7.6p1 is missing a lot more patches
* gcc is missing patches for CVE-2018-12886 on ARM and CVE-2019-15847 on POWER9
* libpng is missing patch for CVE-2018-14048
* libjpeg-turbo is missing patch for CVE-2018-14498
* libgcrypt is missing patch for CVE-2018-6829
etc.About CVE checking in yocto:
* enable with 'INHERIT += "cve-check"' in
conf/local.conf
* see the resulting reports in tmp/deploy/cve/ directory for all compiled recipes
* there is also an image specific summary but I saw it included native and nativesdk recipe data too
* for applying CVE patches, white listing, setting product names etc see the meta/classes/cve-check.bbclass and examples in this patchset and in master branch
* note that only recompiled recipes will be analyzed for CVEs so things from sstate cache will be ignored, a clean build without cache may be needed when enabling the checkps. sumo still comes with gcc 7.3 and my patch to update to 7.4 with lots of bug fixes has not been applied from
http://lists.openembedded.org/pipermail/openembedded-core/2019-January/278049.html
I've been using gcc 7.4 in several x86 and arm64 projects so I would also apply this update to any sumo tree out there.Cheers,
-Mikko
Anuj Mittal (2):
openssl: set CVE vendor to openssl
rsync: fix CVEs for included zlibChen Qi (9):
flac: also add flac toCVE_PRODUCT
xserver-xorg: setCVE_PRODUCT
nasm: addCVE_PRODUCT
dropbear: setCVE_PRODUCT
libsdl: setCVE_PRODUCT
ghostscript: setCVE_PRODUCT
squashfs-tools: setCVE_PRODUCT
libxfont2: setCVE_PRODUCT
webkitgtk: setCVE_PRODUCTChin Huat Ang (1):
cve-update-db-native: fix https proxy issuesMikko Rapeli (1):
cve-check.bbclass: initializeto_appendPierre Le Magourou (13):
cve-update-db: New recipe to update CVE database
cve-check: Remove dependency to cve-check-tool-native
cve-check: ManageCVE_PRODUCTwith more than one name
cve-check: Consider CVE that affects versions with less than operator
cve-update-db: Use std library instead of urllib3
cve-update-db: Manage proxy if needed.
cve-update-db:do_populate_cve_dbdepends ondo_fetch
cve-update-db: Catch request.urlopen errors.
cve-check: Depends on cve-update-db-native
cve-check: Update unpatched CVE matching
cve-check: ReplaceCVE_CHECK_CVE_WHITELISTbyCVE_CHECK_WHITELIST
cve-update-db: Use NVD CPE data to populate PRODUCTS table
cve-update-db-native: Remove hash column from database.Ross Burton (21):
cve-check: be idiomatic
cve-check: remove redundant readline CVE whitelisting
cve-check-tool: remove
glibc: exclude child recipes from CVE scanning
cve-check: allow comparison of Vendor as well as Product
cve-update-db-native: use SQL placeholders instead of format strings
cve-update-db-native: use os.path.join instead of +
cve-update-db: actually inherit native
cve-update-db-native: useexecutemany()to optimise CPE insertion
cve-update-db-native: improve metadata parsing
cve-update-db-native: clean up JSON fetching
cve-check: ensure all known CVEs are in the report
cve-check: failure to parse versions should be more visible
flex: setCVE_PRODUCTto include vendor
libpam: setCVE_PRODUCT
procps: whitelist CVE-2018-1121
libpng: whitelist CVE-2019-17371
ed: set CVE vendor to avoid false positives
boost: set CVE vendor to Boost
subversion: set CVE vendor to Apache
git: set CVE vendor to git-scmmeta/classes/cve-check.bbclass | 147 ++++++++----- meta/conf/distro/include/maintainers.inc | 2 + .../recipes-connectivity/openssl/openssl_1.0.2p.bb | 2 + .../recipes-connectivity/openssl/openssl_1.1.0i.bb | 2 + meta/recipes-core/dropbear/dropbear.inc | 2 + meta/recipes-core/glibc/glibc-locale.inc | 3 + meta/recipes-core/glibc/glibc-mtrace.inc | 3 + meta/recipes-core/glibc/glibc-scripts.inc | 3 + meta/recipes-core/meta/cve-update-db-native.bb | 190 +++++++++++++++++ .../cve-check-tool/cve-check-tool_5.6.4.bb | 62 ------ ...01-Fix-freeing-memory-allocated-by-sqlite.patch | 50 ----- ...ow-overriding-default-CA-certificate-file.patch | 215 ------------------- ...ogress-in-percent-when-downloading-CVE-db.patch | 135 ------------ ...are-computed-vs-expected-sha256-digit-str.patch | 52 ----- .../check-for-malloc_trim-before-using-it.patch | 51 ----- meta/recipes-devtools/flex/flex_2.6.0.bb | 3 + meta/recipes-devtools/git/git.inc | 2 + meta/recipes-devtools/nasm/nasm_2.13.03.bb | 2 + .../rsync/files/CVE-2016-9840.patch | 75 +++++++ .../rsync/files/CVE-2016-9841.patch | 228 +++++++++++++++++++++ .../rsync/files/CVE-2016-9842.patch | 33 +++ .../rsync/files/CVE-2016-9843.patch | 53 +++++ meta/recipes-devtools/rsync/rsync_3.1.3.bb | 7 +- .../squashfs-tools/squashfs-tools_git.bb | 2 + .../subversion/subversion_1.9.7.bb | 2 + meta/recipes-extended/ed/ed_1.14.2.bb | 2 + .../ghostscript/ghostscript_9.21.bb | 3 + meta/recipes-extended/pam/libpam_1.3.0.bb | 2 + meta/recipes-extended/procps/procps_3.3.12.bb | 3 + meta/recipes-graphics/libsdl/libsdl_1.2.15.bb | 2 + meta/recipes-graphics/libsdl2/libsdl2_2.0.8.bb | 2 + meta/recipes-graphics/xorg-lib/libxfont2_2.0.3.bb | 2 + .../recipes-graphics/xorg-xserver/xserver-xorg.inc | 2 + meta/recipes-multimedia/flac/flac_1.3.2.bb | 2 +- meta/recipes-multimedia/libpng/libpng_1.6.34.bb | 3 + meta/recipes-sato/webkit/webkitgtk_2.18.6.bb | 2 + meta/recipes-support/boost/boost.inc | 2 + 37 files changed, 731 insertions(+), 622 deletions(-) create mode 100644 meta/recipes-core/meta/cve-update-db-native.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/cve-check-tool_5.6.4.bb delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-Fix-freeing-memory-allocated-by-sqlite.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-curl-allow-overriding-default-CA-certificate-file.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-print-progress-in-percent-when-downloading-CVE-db.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/0001-update-Compare-computed-vs-expected-sha256-digit-str.patch delete mode 100644 meta/recipes-devtools/cve-check-tool/files/check-for-malloc_trim-before-using-it.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9840.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9841.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9842.patch create mode 100644 meta/recipes-devtools/rsync/files/CVE-2016-9843.patch
Hi,
Quoted 1 line · On Wed, Nov 06, 2019 at 01:46:09PM -0800, akuster808 wrote:
Hello Mikko;
<snip>
Quoted 6 lines
I have collected other patches for sumo and built them locally but I have no way to inform Richard they pass an AB builds or automated testing for them to get into mainline sumo.
I am placing them into
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/sumo-community
Wow, I had no idea of this tree. Looks like the content is quite liberal with version updates which may be necessary to maintain decent CVE fix status with the resources at hand.
What is the relationship to thud?
Comparing oe-core sumo branch to contrib/stable/sumo-community shows for example:
meta-selftest/conf/layer.conf +1 -1
@@ -9,4 +9,4 @@ BBFILE_COLLECTIONS += "selftest" BBFILE_PATTERN_selftest = "^${LAYERDIR}/" BBFILE_PRIORITY_selftest = "5" -LAYERSERIES_COMPAT_selftest = "sumo" +LAYERSERIES_COMPAT_selftest = "thud"
meta-skeleton/conf/layer.conf +1 -1
@@ -14,4 +14,4 @@ LAYERVERSION_skeleton = "1" LAYERDEPENDS_skeleton = "core" -LAYERSERIES_COMPAT_skeleton = "sumo" +LAYERSERIES_COMPAT_skeleton = "thud"
meta/conf/layer.conf +2 -2
@@ -7,12 +7,12 @@ BBFILE_COLLECTIONS += "core" BBFILE_PATTERN_core = "^${LAYERDIR}/" BBFILE_PRIORITY_core = "5" -LAYERSERIES_CORENAMES = "sumo" +LAYERSERIES_CORENAMES = "thud" # This should only be incremented on significant changes that will # cause compatibility issues with other layers LAYERVERSION_core = "11" -LAYERSERIES_COMPAT_core = "sumo" +LAYERSERIES_COMPAT_core = "thud"
...
Cheers,
-Mikko
Quoted 1 line · Mikko Rapeli
Hi,
Hi Mikko,
Quoted 4 lines
...
I use sumo and due to various reasons like BSP layers, binary compatibility, contracts etc can't update to newer release or to master branch. I suspect I'm not alone.
I might end up with similar reasons, but for warrior. And might end up doing similar longer term updates for warrior. (not yet 100% certain)
Quoted 6 lines
...
The tooling will expose that sumo is severely lacking in security patches, but the tooling is a start for anyone interested, like me, to fill the gaps and publish patches for bitbake recipes we care about.
...
Thud is officially still community maintained, as long as this is true the point could be made that everything that gets fixed in sumo should also get fixed in thud.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedHi,
Quoted 5 lines · Adrian Bunk
Quoted 1 line · Mikko Rapeli
Hi,
Hi Mikko,
Quoted 8 lines
Quoted 4 lines
...
I use sumo and due to various reasons like BSP layers, binary compatibility, contracts etc can't update to newer release or to master branch. I suspect I'm not alone.I might end up with similar reasons, but for warrior. And might end up doing similar longer term updates for warrior. (not yet 100% certain)
I'm skipping warrior but going to zeus in addition to sumo. After insipiration from Yocto Project Summit I hope to run master branch in some projects with regular updates, and eventually aligning to some stable release again. Hopefully an LTS one :)
Quoted 10 lines
Quoted 6 lines
...
The tooling will expose that sumo is severely lacking in security patches, but the tooling is a start for anyone interested, like me, to fill the gaps and publish patches for bitbake recipes we care about.
...Thud is officially still community maintained, as long as this is true the point could be made that everything that gets fixed in sumo should also get fixed in thud.
So to keep sumo alive, we should the also keep zeus, warrior and thud, and of course master branch first. For some issues this actually works when the exact same CVE patch applies, but the open question then is testing.
How should a developer test a patch before submitting it, or multiple versions of it?
I'm testing in project tree with CI and target tests, then compile testing on master. qemu ptest runs would be nice but not sure how to get a stable or useful test set for various branches.
To make things more complicated, the project trees sadly contain more backports, fixes and workarounds which are not suitable for upstreaming into stable or even master branches.
-Mikko
Quoted 1 line · On Thu, Nov 07, 2019 at 12:13:51PM +0000, [email protected] wrote:
Hi,
Hi Mikko,
Quoted 11 lines · Adrian Bunk
Quoted 10 lines · Mikko Rapeli
Quoted 1 line
Hi,
Hi Mikko,
Quoted 4 lines
...
I use sumo and due to various reasons like BSP layers, binary compatibility, contracts etc can't update to newer release or to master branch. I suspect I'm not alone.
Quoted 8 lines
Quoted 3 lines
I might end up with similar reasons, but for warrior. And might end up doing similar longer term updates for warrior. (not yet 100% certain)
I'm skipping warrior but going to zeus in addition to sumo. After insipiration from Yocto Project Summit I hope to run master branch in some projects with regular updates, and eventually aligning to some stable release again. Hopefully an LTS one :)
everyone is currently running projects on different releases.
Let's hope LTS will happen, and that with a properly communicated LTS schedule most distributions and users will switch to the LTS releases just like what happened with Ubuntu.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
...
The tooling will expose that sumo is severely lacking in security patches, but the tooling is a start for anyone interested, like me, to fill the gaps and publish patches for bitbake recipes we care about.
...
Quoted 8 lines
Quoted 3 lines
Thud is officially still community maintained, as long as this is true the point could be made that everything that gets fixed in sumo should also get fixed in thud.
So to keep sumo alive, we should the also keep zeus, warrior and thud, and of course master branch first. For some issues this actually works when the exact same CVE patch applies, but the open question then is testing.
...
When a branch is EOL it is documented to be dead.
But upgrading to a more recent non-EOL branch, e.g. sumo to thud, should not result in losing (security) fixes.
The root problem is that "community support" for a stable branch in practice often means "no support".
If sumo is supported but thud is not, this should at least be made visible to users.
Quoted 1 line
-Mikko
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 4 lines · On Wed, 2019-11-06 at 13:46 -0800, akuster808 wrote:
On 11/6/19 7:37 AM, Mikko Rapeli wrote:
QA resources have been a donation from Intel and Windriver above their membership fees. I don't fee right asking them to run QA.
Quoted 8 lines
Quoted 4 lines
patches aiming at yocto 2.5 sumo. If not, would be really nice if someone could collect patches into sumo-next or sumo-contrib branch where us
users could be in charge of all Quality Assurance.I have collected other patches for sumo and built them locally but I have no way to inform Richard they pass an AB builds or automated testing for them to get into mainline sumo.
Given the discussions around LTS and the fact we'd need to do *something* with the autobuilder to help support it, I've been experimenting.
I created a sumo-next branch with a uninative update and Mikko's patches and then managed to get it to build (and pass) on the autobuilder.
I did directly merge a bitbake fix to the 1.38 branch to avoid test failures too.
The autobuilder successfully builds a-full for sumo-next. Compared to a normal build the differences are:
* unsupported workers are removed from the pool for sumo so it only builds on a subset of the infrastructure
* no buildperf tests
* there are no supported fedora workers left so no oe-selftest-fedora
* no ptest image execution (we never did this with sumo)
* no test result output (wasn't present in sumo and never backported)
I'm prepared to merge sumo-next on the basis of these results and use it as a model for how we could continue to get critical patches into older releases. More work would be needed to see if any older releases could work.
At this point the TSC needs to discuss LTS and what our branch policy may be going forward but this does add useful data to that discussion and may help influence those discussions.
[For reference I did have to seek advice from upstream buildbot on how to avoid a bug and its taken quite some experimenting to find the magic to make this work. I did have to update the autobuilder-helper sumo branch to be in sync too].
Cheers,
Richard
Quoted 32 lines · Richard Purdie
Quoted 10 lines · On Wed, 2019-11-06 at 13:46 -0800, akuster808 wrote:
On 11/6/19 7:37 AM, Mikko Rapeli wrote:
QA resources have been a donation from Intel and Windriver above their membership fees. I don't fee right asking them to run QA.Quoted 4 lines
patches aiming at yocto 2.5 sumo. If not, would be really nice if someone could collect patches into sumo-next or sumo-contrib branch where us
users could be in charge of all Quality Assurance.I have collected other patches for sumo and built them locally but I have no way to inform Richard they pass an AB builds or automated testing for them to get into mainline sumo.
Given the discussions around LTS and the fact we'd need to do *something* with the autobuilder to help support it, I've been experimenting.
I created a sumo-next branch with a uninative update and Mikko's patches and then managed to get it to build (and pass) on the autobuilder.
I did directly merge a bitbake fix to the 1.38 branch to avoid test failures too.
The autobuilder successfully builds a-full for sumo-next. Compared to a normal build the differences are:
* unsupported workers are removed from the pool for sumo so it only builds on a subset of the infrastructure
* no buildperf tests
* there are no supported fedora workers left so no oe-selftest-fedora
* no ptest image execution (we never did this with sumo)
* no test result output (wasn't present in sumo and never backported)
Quoted 3 lines
I'm prepared to merge sumo-next on the basis of these results and use it as a model for how we could continue to get critical patches into older releases.
Are you taking the other patches also submitted for sumo ?
- armin
Quoted 16 lines
More work would be needed to see if any older releases could work.
At this point the TSC needs to discuss LTS and what our branch policy may be going forward but this does add useful data to that discussion and may help influence those discussions.
[For reference I did have to seek advice from upstream buildbot on how to avoid a bug and its taken quite some experimenting to find the magic to make this work. I did have to update the autobuilder-helper sumo branch to be in sync too].
Cheers,
Richard
Quoted 42 lines · On Thu, 2019-11-07 at 07:55 -0800, akuster808 wrote:
Quoted 40 lines · Richard Purdie
Quoted 9 lines · On Wed, 2019-11-06 at 13:46 -0800, akuster808 wrote:
On 11/6/19 7:37 AM, Mikko Rapeli wrote:
QA resources have been a donation from Intel and Windriver above their membership fees. I don't fee right asking them to run QA.Quoted 6 lines
patches aiming at yocto 2.5 sumo. If not, would be really nice if
someone could collect patches into sumo-next or sumo-contrib branch
where us
users could be in charge of all Quality Assurance.Quoted 5 lines
I have collected other patches for sumo and built them locally but I
have no way to inform Richard they pass an AB builds or automated
testing for them to get into mainline sumo.Given the discussions around LTS and the fact we'd need to do *something* with the autobuilder to help support it, I've been experimenting.
I created a sumo-next branch with a uninative update and Mikko's patches and then managed to get it to build (and pass) on the autobuilder.
I did directly merge a bitbake fix to the 1.38 branch to avoid test failures too.
The autobuilder successfully builds a-full for sumo-next. Compared to a
normal build the differences are:* unsupported workers are removed from the pool for sumo so it only
builds on a subset of the infrastructure
* no buildperf tests
* there are no supported fedora workers left so no oe-selftest- fedora
* no ptest image execution (we never did this with sumo)
* no test result output (wasn't present in sumo and never backported)
Quoted 6 lines
Quoted 5 lines
I'm prepared to merge sumo-next on the basis of these results and use
it as a model for how we could continue to get critical patches into
older releases.Are you taking the other patches also submitted for sumo ?
I am worried about what the bigger picture for this looks like but we could try testing them. I think the TSC needs to discuss this.
Cheers,
Richard
Quoted 6 lines · Richard Purdie
On Thu, 2019-11-07 at 07:55 -0800, akuster808 wrote:
...Quoted 1 line
Are you taking the other patches also submitted for sumo ?
I am worried about what the bigger picture for this looks like but we could try testing them. I think the TSC needs to discuss this.
How were community supported branches supposed to work?
All branches from 2.1 (sic) are documented as supported in the wiki.
Quoted 3 lines
Cheers,
Richard
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 5 lines · Adrian Bunk
Quoted 4 lines · Richard Purdie
On Thu, 2019-11-07 at 07:55 -0800, akuster808 wrote:
...Quoted 1 line
Are you taking the other patches also submitted for sumo ?
Quoted 7 lines
Quoted 3 lines
I am worried about what the bigger picture for this looks like but we
could try testing them. I think the TSC needs to discuss this.How were community supported branches supposed to work?
All branches from 2.1 (sic) are documented as supported in the wiki.
That was changed recently and I kind of wish we'd waited until we'd followed through with a consistent plan which covers the spectrum of EOL/community/stable/LTS.
As I've said in a few places, the TSC really needs to figure this out and its complicated by the LTS discussions. Those discussions are happening but aren't simple.
Cheers,
Richard
Quoted 10 lines · Richard Purdie
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Richard Purdie
On Thu, 2019-11-07 at 07:55 -0800, akuster808 wrote:
...Quoted 1 line
Are you taking the other patches also submitted for sumo ?
I am worried about what the bigger picture for this looks like but we
could try testing them. I think the TSC needs to discuss this.
Quoted 5 lines
Quoted 3 lines
How were community supported branches supposed to work?
All branches from 2.1 (sic) are documented as supported in the wiki.
That was changed recently
What was changed recently was that the pre-2.1 branches were marked EOL:
https://wiki.yoctoproject.org/wiki/index.php?title=Releases&diff=62334&oldid=62324
Non-stable branches are documented as community-supported since 2014:
https://wiki.yoctoproject.org/wiki/index.php?title=Releases&diff=12747&oldid=12743
Quoted 7 lines
and I kind of wish we'd waited until we'd
followed through with a consistent plan which covers the spectrum of EOL/community/stable/LTS.As I've said in a few places, the TSC really needs to figure this out and its complicated by the LTS discussions. Those discussions are happening but aren't simple.
Past releases and future releases might be separate topics.
Please keep in mind that many people already have to support products on existing stable branches, working under the assumption that patches submitted by the community will be accepted.
Closing future non-LTS branches early might be OK if this is part of a clearly communicated EOL schedule for future LTS and non-LTS releases, like it is clear when Ubuntu releases will be released and for how long they are supported.
But this visibility on upstream support is needed before deciding on a Yocto release for a product. Future LTS releases are irrelevant for existing products that cannot move to a different stable branch.
Quoted 3 lines
Cheers,
Richard
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 51 lines · Adrian Bunk
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Adrian Bunk
Quoted 8 lines · Richard Purdie
On Thu, 2019-11-07 at 07:55 -0800, akuster808 wrote:
...Quoted 1 line
Are you taking the other patches also submitted for sumo ?
I am worried about what the bigger picture for this looks like but
we
could try testing them. I think the TSC needs to discuss this.Quoted 6 lines
Quoted 4 lines
How were community supported branches supposed to work?
All branches from 2.1 (sic) are documented as supported in the wiki.
That was changed recently
What was changed recently was that the pre-2.1 branches were marked EOL:
https://wiki.yoctoproject.org/wiki/index.php?title=Releases&diff=62334&oldid=62324Non-stable branches are documented as community-supported since 2014:
https://wiki.yoctoproject.org/wiki/index.php?title=Releases&diff=12747&oldid=12743Quoted 9 lines
and I kind of wish we'd waited until we'd
followed through with a consistent plan which covers the spectrum of
EOL/community/stable/LTS.As I've said in a few places, the TSC really needs to figure this out
and its complicated by the LTS discussions. Those discussions are happening but aren't simple.Past releases and future releases might be separate topics.
Please keep in mind that many people already have to support products on existing stable branches, working under the assumption that patches
submitted by the community will be accepted.Closing future non-LTS branches early might be OK if this is part of a
clearly communicated EOL schedule for future LTS and non-LTS releases,
like it is clear when Ubuntu releases will be released and for how long
they are supported.
Quoted 5 lines
But this visibility on upstream support is needed before deciding on a Yocto release for a product. Future LTS releases are irrelevant for
existing products that cannot move to a different stable branch.
current and prior two releases are actively maintained. So that should have been the consideration when selecting a release for production in past. see
https://wiki.yoctoproject.org/wiki/Stable_branch_maintenance
Quoted 14 lines
Quoted 3 lines
Cheers,
Richard
cu
AdrianSignature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Quoted 23 lines · Khem Raj
Quoted 1 line · Adrian Bunk
On Mon, Nov 11, 2019 at 01:12:47PM +0000, Richard Purdie wrote:
...
Quoted 20 lines
Quoted 4 lines
As I've said in a few places, the TSC really needs to figure this out
and its complicated by the LTS discussions. Those discussions are happening but aren't simple.Past releases and future releases might be separate topics.
Please keep in mind that many people already have to support products on existing stable branches, working under the assumption that patches
submitted by the community will be accepted.Closing future non-LTS branches early might be OK if this is part of a
clearly communicated EOL schedule for future LTS and non-LTS releases,
like it is clear when Ubuntu releases will be released and for how long
they are supported.
Quoted 6 lines
Quoted 4 lines
But this visibility on upstream support is needed before deciding on a Yocto release for a product. Future LTS releases are irrelevant for
existing products that cannot move to a different stable branch.current and prior two releases are actively maintained.
"master and the latest 2 stable branches"
(AFAIK thud is no longer actively maintained)
Quoted 5 lines
So that should
have been the consideration when selecting a release for production in past. seehttps://wiki.yoctoproject.org/wiki/Stable_branch_maintenance
Which also says "but well-tested patches may still be accepted for them".
The question is exactly how the patch acceptance for the community maintained stable branches works for people who are maintaining products on these older branches.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed