Skip to contentSkip to conversation

Patch review

PATCH In review Steve Sakoman 1 participant 8 messages Apr 28, 2022 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this set of patches for dunfell and have comments back by end of day Monday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3587

The following changes since commit 8e81d38048c953d0823abf04d5b2506cd988f0bb:

build-appliance-image: Update to dunfell head revision (2022-04-25 15:58:54 +0100)

are available in the Git repository at:

git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Dmitry Baryshkov (1):
linux-firmware: correct license for ar3k firmware

Marta Rybczynska (1):
cve-check: add json format

Richard Purdie (1):
perf-build-test/report: Drop phantomjs and html email reports support

Ross Burton (1):
boost: don't specify gcc version

Steve Sakoman (1):
scripts/contrib/oe-build-perf-report-email.py: remove obsolete check
for phantomjs and optipng

sana kazi (1):
tiff: Fix CVE-2022-0891

 meta/classes/cve-check.bbclass                | 144 +++++++++++-
 meta/lib/oe/cve_check.py                      |  16 ++
 .../linux-firmware/linux-firmware_20220411.bb |   4 +-
 .../libtiff/files/CVE-2022-0891.patch         | 217 ++++++++++++++++++
 meta/recipes-multimedia/libtiff/tiff_4.1.0.bb |   1 +
 meta/recipes-support/boost/boost.inc          |   2 +-
 scripts/contrib/build-perf-test-wrapper.sh    |  15 +-
 scripts/contrib/oe-build-perf-report-email.py | 167 +-------------
 8 files changed, 388 insertions(+), 178 deletions(-)
 create mode 100644 meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/6 tiff: Fix CVE-2022-0891

From: sana kazi <[email protected]>

Fix CVE-2022-0891 for tiff

---
 .../libtiff/files/CVE-2022-0891.patch         | 217 ++++++++++++++++++
 meta/recipes-multimedia/libtiff/tiff_4.1.0.bb |   1 +
 2 files changed, 218 insertions(+)
 create mode 100644 meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch
meta/recipes-multimedia/libtiff/files/CVE-2022-0891.patch new file +217 -0
new file mode 100644
index 0000000000..e2f1bd3056
@@ -0,0 +1,217 @@
+From 232282fd8f9c21eefe8d2d2b96cdbbb172fe7b7c Mon Sep 17 00:00:00 2001
+From: Su Laus <[email protected]>
+Date: Tue, 8 Mar 2022 17:02:44 +0000
+Subject: [PATCH] tiffcrop: fix issue #380 and #382 heap buffer overflow in
+ extractImageSection
+
+CVE: CVE-2022-0891
+Upstream-Status: Backport [https://sources.debian.org/src/tiff/4.1.0+git191117-2%7Edeb10u4/debian/patches/CVE-2022-0891.patch/]
+Comment: No change in any hunk
+Signed-off-by: Sana Kazi <[email protected]>
+---
+ tools/tiffcrop.c | 92 +++++++++++++++++++-----------------------------
+ 1 file changed, 36 insertions(+), 56 deletions(-)
+
+diff --git a/tools/tiffcrop.c b/tools/tiffcrop.c
+index f2e5474a..e62bcc71 100644
+--- a/tools/tiffcrop.c
++++ b/tools/tiffcrop.c
+@@ -105,8 +105,8 @@
+  *                of messages to monitor progess without enabling dump logs.
+  */
+ 
+-static   char tiffcrop_version_id[] = "2.4";
+-static   char tiffcrop_rev_date[] = "12-13-2010";
++static   char tiffcrop_version_id[] = "2.4.1";
++static   char tiffcrop_rev_date[] = "03-03-2010";
+ 
+ #include "tif_config.h"
+ #include "tiffiop.h"
+@@ -6670,10 +6670,10 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+   uint32    img_length;
+ #endif
+-  uint32    j, shift1, shift2, trailing_bits;
++  uint32    j, shift1, trailing_bits;
+   uint32    row, first_row, last_row, first_col, last_col;
+   uint32    src_offset, dst_offset, row_offset, col_offset;
+-  uint32    offset1, offset2, full_bytes;
++  uint32    offset1, full_bytes;
+   uint32    sect_width;
+ #ifdef DEVELMODE
+   uint32    sect_length;
+@@ -6683,7 +6683,6 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+   int      k;
+   unsigned char bitset;
+-  static char *bitarray = NULL;
+ #endif
+ 
+   img_width = image->width;
+@@ -6701,17 +6700,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+   dst_offset = 0;
+ 
+ #ifdef DEVELMODE
+-  if (bitarray == NULL)
+-    {
+-    if ((bitarray = (char *)malloc(img_width)) == NULL)
+-      {
+-      TIFFError ("", "DEBUG: Unable to allocate debugging bitarray");
+-      return (-1);
+-      }
+-    }
++  char bitarray[39];
+ #endif
+ 
+-  /* rows, columns, width, length are expressed in pixels */
++  /* rows, columns, width, length are expressed in pixels
++   * first_row, last_row, .. are index into image array starting at 0 to width-1,
++   * last_col shall be also extracted.  */
+   first_row = section->y1;
+   last_row  = section->y2;
+   first_col = section->x1;
+@@ -6721,9 +6715,14 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #ifdef DEVELMODE
+   sect_length = last_row - first_row + 1;
+ #endif
+-  img_rowsize = ((img_width * bps + 7) / 8) * spp;
+-  full_bytes = (sect_width * spp * bps) / 8;   /* number of COMPLETE bytes per row in section */
+-  trailing_bits = (sect_width * bps) % 8;
++    /* The read function loadImage() used copy separate plane data into a buffer as interleaved
++     * samples rather than separate planes so the same logic works to extract regions
++     * regardless of the way the data are organized in the input file.
++     * Furthermore, bytes and bits are arranged in buffer according to COMPRESSION=1 and FILLORDER=1 
++     */
++    img_rowsize = (((img_width * spp * bps) + 7) / 8);    /* row size in full bytes of source image */
++    full_bytes = (sect_width * spp * bps) / 8;            /* number of COMPLETE bytes per row in section */
++    trailing_bits = (sect_width * spp * bps) % 8;         /* trailing bits within the last byte of destination buffer */
+ 
+ #ifdef DEVELMODE
+     TIFFError ("", "First row: %d, last row: %d, First col: %d, last col: %d\n",
+@@ -6736,10 +6735,9 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ 
+   if ((bps % 8) == 0)
+     {
+-    col_offset = first_col * spp * bps / 8;
++    col_offset = (first_col * spp * bps) / 8;
+     for (row = first_row; row <= last_row; row++)
+       {
+-      /* row_offset = row * img_width * spp * bps / 8; */
+       row_offset = row * img_rowsize;
+       src_offset = row_offset + col_offset;
+ 
+@@ -6752,14 +6750,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+     }
+   else
+     { /* bps != 8 */
+-    shift1  = spp * ((first_col * bps) % 8);
+-    shift2  = spp * ((last_col * bps) % 8);
++    shift1 = ((first_col * spp * bps) % 8);           /* shift1 = bits to skip in the first byte of source buffer*/
+     for (row = first_row; row <= last_row; row++)
+       {
+       /* pull out the first byte */
+       row_offset = row * img_rowsize;
+-      offset1 = row_offset + (first_col * bps / 8);
+-      offset2 = row_offset + (last_col * bps / 8);
++      offset1 = row_offset + ((first_col * spp * bps) / 8);   /* offset1 = offset into source of byte with first bits to be extracted */
+ 
+ #ifdef DEVELMODE
+       for (j = 0, k = 7; j < 8; j++, k--)
+@@ -6771,12 +6767,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+       sprintf(&bitarray[9], " ");
+       for (j = 10, k = 7; j < 18; j++, k--)
+         {
+-        bitset = *(src_buff + offset2) & (((unsigned char)1 << k)) ? 1 : 0;
++        bitset = *(src_buff + offset1 + full_bytes) & (((unsigned char)1 << k)) ? 1 : 0;
+         sprintf(&bitarray[j], (bitset) ? "1" : "0");
+         }
+       bitarray[18] = '\0';
+-      TIFFError ("", "Row: %3d Offset1: %d,  Shift1: %d,    Offset2: %d,  Shift2:  %d\n", 
+-                 row, offset1, shift1, offset2, shift2); 
++      TIFFError ("", "Row: %3d Offset1: %"PRIu32",  Shift1: %"PRIu32",    Offset2: %"PRIu32",  Trailing_bits:  %"PRIu32"\n", 
++                 row, offset1, shift1, offset1+full_bytes, trailing_bits); 
+ #endif
+ 
+       bytebuff1 = bytebuff2 = 0;
+@@ -6800,11 +6796,12 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ 
+         if (trailing_bits != 0)
+           {
+-	  bytebuff2 = src_buff[offset2] & ((unsigned char)255 << (7 - shift2));
++      /* Only copy higher bits of samples and mask lower bits of not wanted column samples to zero */
++	  bytebuff2 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (8 - trailing_bits));
+           sect_buff[dst_offset] = bytebuff2;
+ #ifdef DEVELMODE
+ 	  TIFFError ("", "        Trailing bits src offset:  %8d, Dst offset: %8d\n", 
+-                              offset2, dst_offset); 
++          offset1 + full_bytes, dst_offset);
+           for (j = 30, k = 7; j < 38; j++, k--)
+             {
+             bitset = *(sect_buff + dst_offset) & (((unsigned char)1 << k)) ? 1 : 0;
+@@ -6823,8 +6820,10 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #endif
+         for (j = 0; j <= full_bytes; j++) 
+           {
+-	  bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1);
+-	  bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (7 - shift1));
++          /* Skip the first shift1 bits and shift the source up by shift1 bits before save to destination.*/
++          /* Attention: src_buff size needs to be some bytes larger than image size, because could read behind image here. */
++          bytebuff1 = src_buff[offset1 + j] & ((unsigned char)255 >> shift1);
++          bytebuff2 = src_buff[offset1 + j + 1] & ((unsigned char)255 << (8 - shift1));
+           sect_buff[dst_offset + j] = (bytebuff1 << shift1) | (bytebuff2 >> (8 - shift1));
+           }
+ #ifdef DEVELMODE
+@@ -6840,36 +6839,17 @@ extractImageSection(struct image_data *image, struct pageseg *section,
+ #endif
+         dst_offset += full_bytes;
+ 
++        /* Copy the trailing_bits for the last byte in the destination buffer. 
++           Could come from one ore two bytes of the source buffer. */
+         if (trailing_bits != 0)
+           {
+ #ifdef DEVELMODE
+-	    TIFFError ("", "        Trailing bits   src offset: %8d, Dst offset: %8d\n", offset1 + full_bytes, dst_offset); 
+-#endif
+-	  if (shift2 > shift1)
+-            {
+-	    bytebuff1 = src_buff[offset1 + full_bytes] & ((unsigned char)255 << (7 - shift2));
+-            bytebuff2 = bytebuff1 & ((unsigned char)255 << shift1);
+-            sect_buff[dst_offset] = bytebuff2;
+-#ifdef DEVELMODE
+-	    TIFFError ("", "        Shift2 > Shift1\n"); 
++          TIFFError("", "        Trailing bits %4"PRIu32"   src offset: %8"PRIu32", Dst offset: %8"PRIu32"\n", trailing_bits, offset1 + full_bytes, dst_offset);
+ #endif
++          /* More than necessary bits are already copied into last destination buffer, 
++           * only masking of last byte in destination buffer is necessary.*/ 
++          sect_buff[dst_offset] &= ((uint8_t)0xFF << (8 - trailing_bits));
+             }
+-          else
+-            {
+-	    if (shift2 < shift1)
+-              {
+-              bytebuff2 = ((unsigned char)255 << (shift1 - shift2 - 1));
+-	      sect_buff[dst_offset] &= bytebuff2;
+-#ifdef DEVELMODE
+-	      TIFFError ("", "        Shift2 < Shift1\n"); 
+-#endif
+-              }
+-#ifdef DEVELMODE
+-            else
+-	      TIFFError ("", "        Shift2 == Shift1\n"); 
+-#endif
+-            }
+-	  }
+ #ifdef DEVELMODE
+ 	  sprintf(&bitarray[28], " ");
+ 	  sprintf(&bitarray[29], " ");
+@@ -7022,7 +7002,7 @@ writeImageSections(TIFF *in, TIFF *out, struct image_data *image,
+     width  = sections[i].x2 - sections[i].x1 + 1;
+     length = sections[i].y2 - sections[i].y1 + 1;
+     sectsize = (uint32)
+-	    ceil((width * image->bps + 7) / (double)8) * image->spp * length;
++	    ceil((width * image->bps * image->spp + 7) / (double)8) * length;
+     /* allocate a buffer if we don't have one already */
+     if (createImageSection(sectsize, sect_buff_ptr))
+       {
+-- 
+GitLab
meta/recipes-multimedia/libtiff/tiff_4.1.0.bb +1 -0
index 9db247ecc7..e3ffb12f9e 100644
@@ -18,6 +18,7 @@ SRC_URI = "http://download.osgeo.org/libtiff/tiff-${PV}.tar.gz \
            file://0001-tiffset-fix-global-buffer-overflow-for-ASCII-tags-wh.patch \
            file://561599c99f987dc32ae110370cfdd7df7975586b.patch \
            file://eecb0712f4c3a5b449f70c57988260a667ddbdef.patch \
+           file://CVE-2022-0891.patch \
           "
 SRC_URI[md5sum] = "2165e7aba557463acc0664e71a3ed424"
 SRC_URI[sha256sum] = "5d29f32517dadb6dbcd1255ea5bbc93a2b54b94fbf83653b4d65c7d6775b8634"
Signature · 1 line
2.25.1
#2 lore raw

PATCH 2/6 boost: don't specify gcc version

In reply to Steve Sakoman

From: Ross Burton <[email protected]>

There's no need to specify an ancient GCC version here as Boost will probe it.

---
 meta/recipes-support/boost/boost.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/boost/boost.inc +1 -1
index 829e728b6d..1c13fb3599 100644
@@ -165,7 +165,7 @@ do_configure() {
 
 	# D2194:Fixing the failure of "error: duplicate initialization of gcc with the following parameters" during compilation.
 	rm -f ${WORKDIR}/user-config.jam
-	echo 'using gcc : 4.3.1 : ${CXX} : <cflags>"${CFLAGS}" <cxxflags>"${CXXFLAGS}" <linkflags>"${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam
+	echo 'using gcc : : ${CXX} : <cflags>"${CFLAGS}" <cxxflags>"${CXXFLAGS}" <linkflags>"${LDFLAGS}" ;' >> ${WORKDIR}/user-config.jam
 
 	# If we want Python then we need to tell Boost *exactly* where to find it
 	if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then
Signature · 1 line
2.25.1
#3 lore raw

PATCH 3/6 linux-firmware: correct license for ar3k firmware

In reply to Steve Sakoman

From: Dmitry Baryshkov <[email protected]>

According to the WHENCE file, some a3k firmware files are licensed under the special ar3k license, while others are licensed under the more generic Atheros license. Document this by adding extending the LICENSE_${PN}-ar3k and depending on both of them.

(cherry picked from commit 8e651814af706285d64b532095fcd6f5f02629ba)
---
 meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/recipes-kernel/linux-firmware/linux-firmware_20220411.bb +2 -2
index 04a6cefaa1..ad8dbac17f 100644
@@ -352,7 +352,7 @@ FILES_${PN}-carl9170 = " \
 RDEPENDS_${PN}-carl9170 += "${PN}-gplv2-license"
 
 # For QualCommAthos
-LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k"
+LICENSE_${PN}-ar3k = "Firmware-qualcommAthos_ar3k & Firmware-atheros_firmware"
 LICENSE_${PN}-ar3k-license = "Firmware-qualcommAthos_ar3k"
 LICENSE_${PN}-ath10k = "Firmware-qualcommAthos_ath10k"
 LICENSE_${PN}-ath10k-license = "Firmware-qualcommAthos_ath10k"
@@ -376,7 +376,7 @@ FILES_${PN}-qca = " \
   ${nonarch_base_libdir}/firmware/qca \
 "
 
-RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license"
+RDEPENDS_${PN}-ar3k += "${PN}-ar3k-license ${PN}-atheros-license"
 RDEPENDS_${PN}-ath10k += "${PN}-ath10k-license"
 RDEPENDS_${PN}-ath11k += "${PN}-ath10k-license"
 RDEPENDS_${PN}-qca += "${PN}-ath10k-license"
Signature · 1 line
2.25.1
#4 lore raw

PATCH 4/6 cve-check: add json format

In reply to Steve Sakoman

From: Marta Rybczynska <[email protected]>

Backport to dunfell from master df567de36ae5964bee433ebb97e8bf702034994a

Add an option to output the CVE check in a JSON-based format. This format is easier to parse in software than the original text-based one and allows post-processing by other tools.

Output formats are now handed by CVE_CHECK_FORMAT_TEXT and CVE_CHECK_FORMAT_JSON. The text format is enabled by default to maintain compatibility, while the JSON format is disabled by default.

The JSON output format gets generated in a similar way to the text format with the exception of the manifest: appending to JSON arrays requires parsing the file. Because of that we first write JSON fragments and then assemble them in one pass at the end.

---
 meta/classes/cve-check.bbclass | 144 ++++++++++++++++++++++++++++++++-
 meta/lib/oe/cve_check.py       |  16 ++++
 2 files changed, 159 insertions(+), 1 deletion(-)
meta/classes/cve-check.bbclass +143 -1
index 75c5b92b96..a7156cbdfb 100644
@@ -34,15 +34,27 @@ CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
 CVE_CHECK_SUMMARY_DIR ?= "${LOG_DIR}/cve"
 CVE_CHECK_SUMMARY_FILE_NAME ?= "cve-summary"
 CVE_CHECK_SUMMARY_FILE ?= "${CVE_CHECK_SUMMARY_DIR}/${CVE_CHECK_SUMMARY_FILE_NAME}"
+CVE_CHECK_SUMMARY_FILE_NAME_JSON = "cve-summary.json"
+CVE_CHECK_SUMMARY_INDEX_PATH = "${CVE_CHECK_SUMMARY_DIR}/cve-summary-index.txt"
+
+CVE_CHECK_LOG_JSON ?= "${T}/cve.json"
 
 CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"
 CVE_CHECK_RECIPE_FILE ?= "${CVE_CHECK_DIR}/${PN}"
+CVE_CHECK_RECIPE_FILE_JSON ?= "${CVE_CHECK_DIR}/${PN}_cve.json"
 CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cve"
+CVE_CHECK_MANIFEST_JSON ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.json"
 CVE_CHECK_COPY_FILES ??= "1"
 CVE_CHECK_CREATE_MANIFEST ??= "1"
 
 CVE_CHECK_REPORT_PATCHED ??= "1"
 
+# Provide text output
+CVE_CHECK_FORMAT_TEXT ??= "1"
+
+# Provide JSON output - disabled by default for backward compatibility
+CVE_CHECK_FORMAT_JSON ??= "0"
+
 # Whitelist for packages (PN)
 CVE_CHECK_PN_WHITELIST ?= ""
 
@@ -118,6 +130,7 @@ python cve_check_cleanup () {
     Delete the file used to gather all the CVE information.
     """
     bb.utils.remove(e.data.getVar("CVE_CHECK_TMP_FILE"))
+    bb.utils.remove(e.data.getVar("CVE_CHECK_SUMMARY_INDEX_PATH"))
 }
 
 addhandler cve_check_cleanup
@@ -129,11 +142,15 @@ python cve_check_write_rootfs_manifest () {
     """
 
     import shutil
+    from oe.cve_check import cve_check_merge_jsons
 
     if d.getVar("CVE_CHECK_COPY_FILES") == "1":
         deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE")
         if os.path.exists(deploy_file):
             bb.utils.remove(deploy_file)
+        deploy_file_json = d.getVar("CVE_CHECK_RECIPE_FILE_JSON")
+        if os.path.exists(deploy_file_json):
+            bb.utils.remove(deploy_file_json)
 
     if os.path.exists(d.getVar("CVE_CHECK_TMP_FILE")):
         bb.note("Writing rootfs CVE manifest")
@@ -152,6 +169,26 @@ python cve_check_write_rootfs_manifest () {
                 os.remove(manifest_link)
             os.symlink(os.path.basename(manifest_name), manifest_link)
             bb.plain("Image CVE report stored in: %s" % manifest_name)
+
+    if os.path.exists(d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")):
+        import json
+        bb.note("Generating JSON CVE manifest")
+        deploy_dir = d.getVar("DEPLOY_DIR_IMAGE")
+        link_name = d.getVar("IMAGE_LINK_NAME")
+        manifest_name = d.getVar("CVE_CHECK_MANIFEST_JSON")
+        index_file = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")
+        manifest = {"version":"1", "package": []}
+        with open(index_file) as f:
+            filename = f.readline()
+            while filename:
+                with open(filename.rstrip()) as j:
+                    data = json.load(j)
+                    cve_check_merge_jsons(manifest, data)
+                filename = f.readline()
+
+        with open(manifest_name, "w") as f:
+            json.dump(manifest, f, indent=2)
+        bb.plain("Image CVE report stored in: %s" % manifest_name)
 }
 
 ROOTFS_POSTPROCESS_COMMAND_prepend = "${@'cve_check_write_rootfs_manifest; ' if d.getVar('CVE_CHECK_CREATE_MANIFEST') == '1' else ''}"
@@ -337,7 +374,7 @@ def get_cve_info(d, cves):
     conn.close()
     return cve_data
 
-def cve_write_data(d, patched, unpatched, whitelisted, cve_data):
+def cve_write_data_text(d, patched, unpatched, whitelisted, cve_data):
     """
     Write CVE information in WORKDIR; and to CVE_CHECK_DIR, and
     CVE manifest if enabled.
@@ -403,3 +440,108 @@ def cve_write_data(d, patched, unpatched, whitelisted, cve_data):
 
             with open(d.getVar("CVE_CHECK_TMP_FILE"), "a") as f:
                 f.write("%s" % write_string)
+
+def cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file):
+    """
+    Write CVE information in the JSON format: to WORKDIR; and to
+    CVE_CHECK_DIR, if CVE manifest if enabled, write fragment
+    files that will be assembled at the end in cve_check_write_rootfs_manifest.
+    """
+
+    import json
+
+    write_string = json.dumps(output, indent=2)
+    with open(direct_file, "w") as f:
+        bb.note("Writing file %s with CVE information" % direct_file)
+        f.write(write_string)
+
+    if d.getVar("CVE_CHECK_COPY_FILES") == "1":
+        bb.utils.mkdirhier(os.path.dirname(deploy_file))
+        with open(deploy_file, "w") as f:
+            f.write(write_string)
+
+    if d.getVar("CVE_CHECK_CREATE_MANIFEST") == "1":
+        cvelogpath = d.getVar("CVE_CHECK_SUMMARY_DIR")
+        index_path = d.getVar("CVE_CHECK_SUMMARY_INDEX_PATH")
+        bb.utils.mkdirhier(cvelogpath)
+        fragment_file = os.path.basename(deploy_file)
+        fragment_path = os.path.join(cvelogpath, fragment_file)
+        with open(fragment_path, "w") as f:
+            f.write(write_string)
+        with open(index_path, "a+") as f:
+            f.write("%s\n" % fragment_path)
+
+def cve_write_data_json(d, patched, unpatched, ignored, cve_data):
+    """
+    Prepare CVE data for the JSON format, then write it.
+    """
+
+    output = {"version":"1", "package": []}
+    nvd_link = "https://nvd.nist.gov/vuln/detail/"
+
+    fdir_name  = d.getVar("FILE_DIRNAME")
+    layer = fdir_name.split("/")[-3]
+
+    include_layers = d.getVar("CVE_CHECK_LAYER_INCLUDELIST").split()
+    exclude_layers = d.getVar("CVE_CHECK_LAYER_EXCLUDELIST").split()
+
+    if exclude_layers and layer in exclude_layers:
+        return
+
+    if include_layers and layer not in include_layers:
+        return
+
+    unpatched_cves = []
+
+    package_version = "%s%s" % (d.getVar("EXTENDPE"), d.getVar("PV"))
+    package_data = {
+        "name" : d.getVar("PN"),
+        "layer" : layer,
+        "version" : package_version
+    }
+    cve_list = []
+
+    for cve in sorted(cve_data):
+        is_patched = cve in patched
+        status = "Unpatched"
+        if is_patched and (d.getVar("CVE_CHECK_REPORT_PATCHED") != "1"):
+            continue
+        if cve in ignored:
+            status = "Ignored"
+        elif is_patched:
+            status = "Patched"
+        else:
+            # default value of status is Unpatched
+            unpatched_cves.append(cve)
+
+        issue_link = "%s%s" % (nvd_link, cve)
+
+        cve_item = {
+            "id" : cve,
+            "summary" : cve_data[cve]["summary"],
+            "scorev2" : cve_data[cve]["scorev2"],
+            "scorev3" : cve_data[cve]["scorev3"],
+            "vector" : cve_data[cve]["vector"],
+            "status" : status,
+            "link": issue_link
+        }
+        cve_list.append(cve_item)
+
+    package_data["issue"] = cve_list
+    output["package"].append(package_data)
+
+    direct_file = d.getVar("CVE_CHECK_LOG_JSON")
+    deploy_file = d.getVar("CVE_CHECK_RECIPE_FILE_JSON")
+    manifest_file = d.getVar("CVE_CHECK_SUMMARY_FILE_NAME_JSON")
+
+    cve_check_write_json_output(d, output, direct_file, deploy_file, manifest_file)
+
+def cve_write_data(d, patched, unpatched, ignored, cve_data):
+    """
+    Write CVE data in each enabled format.
+    """
+
+    if d.getVar("CVE_CHECK_FORMAT_TEXT") == "1":
+        cve_write_data_text(d, patched, unpatched, ignored, cve_data)
+    if d.getVar("CVE_CHECK_FORMAT_JSON") == "1":
+        cve_write_data_json(d, patched, unpatched, ignored, cve_data)
meta/lib/oe/cve_check.py +16 -0
index a1d7c292af..1d3c775bbe 100644
@@ -63,3 +63,19 @@ def _cmpkey(release, patch_l, pre_l, pre_v):
     else:
         _pre = float(pre_v) if pre_v else float('-inf')
     return _release, _patch, _pre
+
+def cve_check_merge_jsons(output, data):
+    """
+    Merge the data in the "package" property to the main data file
+    output
+    """
+    if output["version"] != data["version"]:
+        bb.error("Version mismatch when merging JSON outputs")
+        return
+
+    for product in output["package"]:
+        if product["name"] == data["package"][0]["name"]:
+            bb.error("Error adding the same package twice")
+            return
+
+    output["package"].append(data["package"][0])
Signature · 1 line
2.25.1
#5 lore raw

PATCH 5/6 perf-build-test/report: Drop phantomjs and html email reports support

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

phantomjs isn't reliable and we've moved to sharing the reports via a webserver. Update the scripts to more match those being used in the autobuilder helper where the html email support was removed.

(cherry picked from commit b5c131006e3fad0a15e6cdf81f71dc1e96647028)
---
 scripts/contrib/build-perf-test-wrapper.sh    |  15 +-
 scripts/contrib/oe-build-perf-report-email.py | 154 +-----------------
 2 files changed, 8 insertions(+), 161 deletions(-)
scripts/contrib/build-perf-test-wrapper.sh +2 -13
index fa71d4a2e9..0a85e6e708 100755
@@ -87,21 +87,10 @@ if [ $# -ne 0 ]; then
     exit 1
 fi
 
-if [ -n "$email_to" ]; then
-    if ! [ -x "$(command -v phantomjs)" ]; then
-        echo "ERROR: Sending email needs phantomjs."
-        exit 1
-    fi
-    if ! [ -x "$(command -v optipng)" ]; then
-        echo "ERROR: Sending email needs optipng."
-        exit 1
-    fi
-fi
-
 # Open a file descriptor for flock and acquire lock
 LOCK_FILE="/tmp/oe-build-perf-test-wrapper.lock"
 if ! exec 3> "$LOCK_FILE"; then
-    echo "ERROR: Unable to open lock file"
+    echo "ERROR: Unable to open loemack file"
     exit 1
 fi
 if ! flock -n 3; then
@@ -226,7 +215,7 @@ if [ -n "$results_repo" ]; then
     if [ -n "$email_to" ]; then
         echo "Emailing test report"
         os_name=`get_os_release_var PRETTY_NAME`
-        "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt --html $report_html "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}"
+        "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text $report_txt "${OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS[@]}"
     fi
 
     # Upload report files, unless we're on detached head
scripts/contrib/oe-build-perf-report-email.py +6 -148

195 lines not on this page · raw message

Signature · 1 line
2.25.1
#6 lore raw

PATCH 6/6 scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng

In reply to Steve Sakoman

Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028 perf-build-test/report: Drop phantomjs and html email reports support

---
 scripts/contrib/oe-build-perf-report-email.py | 13 -------------
 1 file changed, 13 deletions(-)
scripts/contrib/oe-build-perf-report-email.py +0 -13
index c900720f6e..7192113c28 100755
@@ -27,17 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 log = logging.getLogger('oe-build-perf-report')
 
 
-def check_utils():
-    """Check that all needed utils are installed in the system"""
-    missing = []
-    for cmd in ('phantomjs', 'optipng'):
-        if not shutil.which(cmd):
-            missing.append(cmd)
-    if missing:
-        log.error("The following tools are missing: %s", ' '.join(missing))
-        sys.exit(1)
-
-
 def parse_args(argv):
     """Parse command line arguments"""
     description = """Email build perf test report"""
@@ -101,8 +90,6 @@ def main(argv=None):
     if args.debug:
         log.setLevel(logging.DEBUG)
 
-    check_utils()
-
     if args.outdir:
         outdir = args.outdir
         if not os.path.exists(outdir):
Signature · 1 line
2.25.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Re: [OE-core][dunfell 6/6] scripts/contrib/oe-build-perf-report-email.py: remove obsolete check for phantomjs and optipng

This patch is in master-next, but not yet master. I decided to run the dunfell review process in parallel with master. I will wait till the patch hits master before sending a pull request (and will add the cherry-pick notation)

Steve

Quoted 52 lines · Steve Sakoman via lists.openembedded.org

Use of those tools was removed in b5c131006e3fad0a15e6cdf81f71dc1e96647028 perf-build-test/report: Drop phantomjs and html email reports support

---
 scripts/contrib/oe-build-perf-report-email.py | 13 -------------
 1 file changed, 13 deletions(-)
scripts/contrib/oe-build-perf-report-email.py +0 -14
index c900720f6e..7192113c28 100755
@@ -27,17 +27,6 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 log = logging.getLogger('oe-build-perf-report')
 
 
-def check_utils():
-    """Check that all needed utils are installed in the system"""
-    missing = []
-    for cmd in ('phantomjs', 'optipng'):
-        if not shutil.which(cmd):
-            missing.append(cmd)
-    if missing:
-        log.error("The following tools are missing: %s", ' '.join(missing))
-        sys.exit(1)
-
-
 def parse_args(argv):
     """Parse command line arguments"""
     description = """Email build perf test report"""
@@ -101,8 +90,6 @@ def main(argv=None):
     if args.debug:
         log.setLevel(logging.DEBUG)
 
-    check_utils()
-
     if args.outdir:
         outdir = args.outdir
         if not os.path.exists(outdir):
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#164984): https://lists.openembedded.org/g/openembedded-core/message/164984
Mute This Topic: https://lists.openembedded.org/mt/90764072/3620601
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-