Skip to contentSkip to conversation

Patch review

PATCH New Steve Sakoman 1 participant 16 messages Dec 4, 2024

lore
Share & tools
lore
#0 lore raw

cover Patch review

Please review this set of changes for scarthgap and have comments back by end of day Friday, December 6

Passed a-full on autobuilder:

https://valkyrie.yoctoproject.org/#/builders/29/builds/582

The following changes since commit e0c4f2f69723f64a29ba35c4de5fb1a7c79ef718:

rootfs: Ensure run-postinsts is not uninstalled for read-only-rootfs-delayed-postinsts (2024-12-02 10:18:14 -0800)

are available in the Git repository at:

https://git.openembedded.org/openembedded-core-contrib stable/scarthgap-nut
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/scarthgap-nut

Changqing Li (1):
rxvt-unicode.inc: disable the terminfo installation by setting TIC to

    :

Hongxu Jia (2):
ovmf: fix CVE-2024-38796
ovmf: fix CVE-2024-1298

Richard Purdie (12):
do_package/sstate/sstatesig: Change timestamp clamping to hash output
only
selftest/reproducible: Drop rawlogs
selftest/reproducible: Clean up pathnames
resulttool: Allow store to filter to specific revisions
resulttool: Use single space indentation in json output oeqa/utils/gitarchive: Return tag name and improve exclude handling
resulttool: Fix passthrough of --all files in store mode
resulttool: Add --logfile-archive option to store mode
resulttool: Handle ltp rawlogs as well as ptest
resulttool: Clean up repoducible build logs
resulttool: Trim the precision of duration information
resulttool: Improve repo layout for oeselftest results

 meta/classes-global/sstate.bbclass            | 16 ----
 meta/lib/oe/sstatesig.py                      |  7 +-
 meta/lib/oeqa/core/runner.py                  |  2 +-
 meta/lib/oeqa/selftest/cases/reproducible.py  |  8 +-
 meta/lib/oeqa/utils/gitarchive.py             |  4 +-
 ...ential-UINT32-overflow-in-S3-ResumeC.patch | 51 +++++++++++++
 ...-Fix-overflow-issue-in-BasePeCoffLib.patch | 36 +++++++++
 meta/recipes-core/ovmf/ovmf_git.bb            |  2 +
 .../rxvt-unicode/rxvt-unicode.inc             |  3 +-
 scripts/lib/resulttool/manualexecution.py     |  2 +-
 scripts/lib/resulttool/report.py              |  2 +-
 scripts/lib/resulttool/resultutils.py         | 76 +++++++++++++++----
 scripts/lib/resulttool/store.py               | 26 ++++++-
 13 files changed, 188 insertions(+), 47 deletions(-)
 create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch
 create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/15 ovmf: fix CVE-2024-38796

From: Hongxu Jia <[email protected]>

Backport fix from upstream to resolve CVE-2024-38796

https://github.com/tianocore/edk2/commit/c95233b8525ca6828921affd1496146cff262e65

---
 ...-Fix-overflow-issue-in-BasePeCoffLib.patch | 36 +++++++++++++++++++
 meta/recipes-core/ovmf/ovmf_git.bb            |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch
meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch new file +36 -0
new file mode 100644
index 0000000000..c6e15c5069
@@ -0,0 +1,36 @@
+From 5f7bd3f3c4747d5bb2733f017f8c5b93b63a74e3 Mon Sep 17 00:00:00 2001
+From: Doug Flick <[email protected]>
+Date: Fri, 22 Nov 2024 13:03:33 +0800
+Subject: [PATCH] MdePkg: Fix overflow issue in BasePeCoffLib
+
+The RelocDir->Size is a UINT32 value, and RelocDir->VirtualAddress is
+also a UINT32 value. The current code does not check for overflow when
+adding RelocDir->Size to RelocDir->VirtualAddress. This patch adds a
+check to ensure that the addition does not overflow.
+
+Signed-off-by: Doug Flick <[email protected]>
+Authored-by: sriraamx gobichettipalayam <[email protected]>
+
+CVE: CVE-2024-38796
+Upstream-Status: Backport [https://github.com/tianocore/edk2/commit/c95233b8525ca6828921affd1496146cff262e65]
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ MdePkg/Library/BasePeCoffLib/BasePeCoff.c | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/MdePkg/Library/BasePeCoffLib/BasePeCoff.c b/MdePkg/Library/BasePeCoffLib/BasePeCoff.c
+index 86ff2e7..128090d 100644
+--- a/MdePkg/Library/BasePeCoffLib/BasePeCoff.c
++++ b/MdePkg/Library/BasePeCoffLib/BasePeCoff.c
+@@ -1054,7 +1054,7 @@ PeCoffLoaderRelocateImage (
+     RelocDir = &Hdr.Te->DataDirectory[0];
+   }
+ 
+-  if ((RelocDir != NULL) && (RelocDir->Size > 0)) {
++  if ((RelocDir != NULL) && (RelocDir->Size > 0) && (RelocDir->Size - 1 < MAX_UINT32 - RelocDir->VirtualAddress)) {
+     RelocBase    = (EFI_IMAGE_BASE_RELOCATION *)PeCoffLoaderImageAddress (ImageContext, RelocDir->VirtualAddress, TeStrippedOffset);
+     RelocBaseEnd = (EFI_IMAGE_BASE_RELOCATION *)PeCoffLoaderImageAddress (
+                                                   ImageContext,
+-- 
+2.34.1
+
meta/recipes-core/ovmf/ovmf_git.bb +1 -0
index 35ca8d1834..cc2ac4268c 100644
@@ -24,6 +24,7 @@ SRC_URI = "gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https \
            file://0002-BaseTools-makefile-adjust-to-build-in-under-bitbake.patch \
            file://0003-debug-prefix-map.patch \
            file://0004-reproducible.patch \
+           file://0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch \
            "
 
 PV = "edk2-stable202402"
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/15 ovmf: fix CVE-2024-1298

In reply to Steve Sakoman

From: Hongxu Jia <[email protected]>

Backport fix from upstream to resolve CVE-2024-1298

https://github.com/tianocore/edk2/commit/284dbac43da752ee34825c8b3f6f9e8281cb5a19

---
 ...ential-UINT32-overflow-in-S3-ResumeC.patch | 51 +++++++++++++++++++
 meta/recipes-core/ovmf/ovmf_git.bb            |  1 +
 2 files changed, 52 insertions(+)
 create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch
meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch new file +51 -0
new file mode 100644
index 0000000000..264820138f
@@ -0,0 +1,51 @@
+From 150ea3ea4c821b133a782eeb33ef2a9c8fd8d7c3 Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <[email protected]>
+Date: Fri, 22 Nov 2024 13:05:57 +0800
+Subject: [PATCH] MdeModulePkg: Potential UINT32 overflow in S3 ResumeCount
+
+REF: https://bugzilla.tianocore.org/show_bug.cgi?id=4677
+
+Attacker able to modify physical memory and ResumeCount.
+System will crash/DoS when ResumeCount reaches its MAX_UINT32.
+
+Cc: Zhiguang Liu <[email protected]>
+Cc: Dandan Bi <[email protected]>
+Cc: Liming Gao <[email protected]>
+
+Signed-off-by: Pakkirisamy ShanmugavelX <[email protected]>
+Reviewed-by: Liming Gao <[email protected]>
+
+CVE: CVE-2024-1298
+Upstream-Status: Backport [https://github.com/tianocore/edk2/commit/284dbac43da752ee34825c8b3f6f9e8281cb5a19]
+Signed-off-by: Hongxu Jia <[email protected]>
+---
+ .../FirmwarePerformancePei.c                         | 12 ++++++++----
+ 1 file changed, 8 insertions(+), 4 deletions(-)
+
+diff --git a/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c b/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c
+index 2f2b2a8..2ba9215 100644
+--- a/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c
++++ b/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c
+@@ -112,11 +112,15 @@ FpdtStatusCodeListenerPei (
+   //
+   S3ResumeTotal = MultU64x32 (AcpiS3ResumeRecord->AverageResume, AcpiS3ResumeRecord->ResumeCount);
+   AcpiS3ResumeRecord->ResumeCount++;
+-  AcpiS3ResumeRecord->AverageResume = DivU64x32 (S3ResumeTotal + AcpiS3ResumeRecord->FullResume, AcpiS3ResumeRecord->ResumeCount);
++  if (AcpiS3ResumeRecord->ResumeCount > 0) {
++    AcpiS3ResumeRecord->AverageResume = DivU64x32 (S3ResumeTotal + AcpiS3ResumeRecord->FullResume, AcpiS3ResumeRecord->ResumeCount);
++    DEBUG ((DEBUG_INFO, "\nFPDT: S3 Resume Performance - AverageResume = 0x%x\n", AcpiS3ResumeRecord->AverageResume));
++  } else {
++    DEBUG ((DEBUG_ERROR, "\nFPDT: S3 ResumeCount reaches the MAX_UINT32 value. S3 ResumeCount record reset to Zero."));
++  }
+ 
+-  DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - ResumeCount   = %d\n", AcpiS3ResumeRecord->ResumeCount));
+-  DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - FullResume    = %ld\n", AcpiS3ResumeRecord->FullResume));
+-  DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - AverageResume = %ld\n", AcpiS3ResumeRecord->AverageResume));
++  DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - ResumeCount   = 0x%x\n", AcpiS3ResumeRecord->ResumeCount));
++  DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - FullResume    = 0x%x\n", AcpiS3ResumeRecord->FullResume));
+ 
+   //
+   // Update S3 Suspend Performance Record.
+-- 
+2.34.1
+
meta/recipes-core/ovmf/ovmf_git.bb +1 -0
index cc2ac4268c..ada6ee72db 100644
@@ -25,6 +25,7 @@ SRC_URI = "gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https \
            file://0003-debug-prefix-map.patch \
            file://0004-reproducible.patch \
            file://0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch \
+           file://0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch \
            "
 
 PV = "edk2-stable202402"
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/15 do_package/sstate/sstatesig: Change timestamp clamping to hash output only

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

The code was changing the timestamps of the files in the do_package output, particularly the files added for debug sources. This was to do two things:

a) make do_package sstate more reproducible
b) ensure better hash equivalence matching

Unfortuately the debug source files are hardlinks into the source tree for efficiency so touching these, touches a lot of files in ${B} and ${S}. This causes unpredictable effects if compile is run again for example, or could cause compiling in the install task.

The hash equivalence matching is of key importance but we can mimic that using clamping of the file timestamps in the depsig output used to generate the hashes.

This patch drops the global timestamp clamping, instead allowing the files to retain their creation timestamps into sstate. This makes do_package sstate slightly less reproducibile. We could clamp the sstate timestamps but that would lead to two different sets of timestamps depending on whether the data came from sstate or not. I'd prefer to have consistent code behaviour, rather than differing behavhour depending on whether data came from sstate or not.

If we wanted to have reproducibiliy and fix the "corruption" of S/B and have consistent codepaths, the only other option would be two copies of the sources, which could end up huge and seems the least desireable option.

This patch therefore drops the timestamp clamping in the sstate files and tweaks the depsig data generation to clamp the timestamps for do_package instead since this seems the best compromise.

I validated that rpm/deb/ipk files still generate correctly as before.

(cherry picked from commit 475759fdab7200488b2a568b2ba1aa31a456d113)
---
 meta/classes-global/sstate.bbclass | 16 ----------------
 meta/lib/oe/sstatesig.py           |  7 ++++++-
 2 files changed, 6 insertions(+), 17 deletions(-)
meta/classes-global/sstate.bbclass +0 -16
index 93df5fa9e6..567797305f 100644
@@ -648,15 +648,6 @@ def sstate_package(ss, d):
 
     tmpdir = d.getVar('TMPDIR')
 
-    fixtime = False
-    if ss['task'] == "package":
-        fixtime = True
-
-    def fixtimestamp(root, path):
-        f = os.path.join(root, path)
-        if os.lstat(f).st_mtime > sde:
-            os.utime(f, (sde, sde), follow_symlinks=False)
-
     sstatebuild = d.expand("${WORKDIR}/sstate-build-%s/" % ss['task'])
     sde = int(d.getVar("SOURCE_DATE_EPOCH") or time.time())
     d.setVar("SSTATE_CURRTASK", ss['task'])
@@ -671,8 +662,6 @@ def sstate_package(ss, d):
         # to sstate tasks but there aren't many of these so better just avoid them entirely.
         for walkroot, dirs, files in os.walk(state[1]):
             for file in files + dirs:
-                if fixtime:
-                    fixtimestamp(walkroot, file)
                 srcpath = os.path.join(walkroot, file)
                 if not os.path.islink(srcpath):
                     continue
@@ -694,11 +683,6 @@ def sstate_package(ss, d):
         bb.utils.mkdirhier(plain)
         bb.utils.mkdirhier(pdir)
         bb.utils.rename(plain, pdir)
-        if fixtime:
-            fixtimestamp(pdir, "")
-            for walkroot, dirs, files in os.walk(pdir):
-                for file in files + dirs:
-                    fixtimestamp(walkroot, file)
 
     d.setVar('SSTATE_BUILDDIR', sstatebuild)
     d.setVar('SSTATE_INSTDIR', sstatebuild)
meta/lib/oe/sstatesig.py +6 -1
index a46e5502ab..d818fce8f1 100644
@@ -524,6 +524,7 @@ def OEOuthashBasic(path, sigfile, task, d):
     if task == "package":
         include_timestamps = True
         include_root = False
+        source_date_epoch = float(d.getVar("SOURCE_DATE_EPOCH"))
     hash_version = d.getVar('HASHEQUIV_HASH_VERSION')
     extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA")
 
@@ -615,7 +616,11 @@ def OEOuthashBasic(path, sigfile, task, d):
                         raise Exception(msg).with_traceback(e.__traceback__)
 
                 if include_timestamps:
-                    update_hash(" %10d" % s.st_mtime)
+                    # Need to clamp to SOURCE_DATE_EPOCH
+                    if s.st_mtime > source_date_epoch:
+                        update_hash(" %10d" % source_date_epoch)
+                    else:
+                        update_hash(" %10d" % s.st_mtime)
 
                 update_hash(" ")
                 if stat.S_ISBLK(s.st_mode) or stat.S_ISCHR(s.st_mode):
Signature · 1 line
2.34.1
#4 lore raw

PATCH 4/15 rxvt-unicode.inc: disable the terminfo installation by setting TIC to :

In reply to Steve Sakoman

From: Changqing Li <[email protected]>

Without this change, TIC is the native tic in recipe-sysroot-native. By default, native tic has set its default terminfo path to native path:
${datadir}/terminfo; $HOME/.terminfo

When sstate cache is used, the cached native tic's terminfo path could be a path not exist on current host, then native tic will try to install terminfo to HOME dir, cause host contamination.

Disable the terminfo installation by setting TIC to :

(cherry picked from commit fe35ead2c3135a18c346e7baa31d34b15c3e2d95)
---
 meta/recipes-sato/rxvt-unicode/rxvt-unicode.inc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-sato/rxvt-unicode/rxvt-unicode.inc +2 -1
index 016614b19c..aa6d908538 100644
@@ -28,7 +28,8 @@ EXTRA_OECONF = "--enable-xim \
 		--with-codesets=eu --enable-pointer-blank \
 		--enable-text-blink --enable-rxvt-scroll \
 		--enable-combining --disable-perl \
-		--with-x=${STAGING_DIR_HOST}${prefix}"
+		--with-x=${STAGING_DIR_HOST}${prefix} \
+		ac_cv_path_TIC=:"
 
 PACKAGECONFIG ??= ""
 PACKAGECONFIG[startup] = "--enable-startup-notification,--disable-startup-notification,startup-notification,"
Signature · 1 line
2.34.1
#5 lore raw

PATCH 5/15 selftest/reproducible: Drop rawlogs

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

The "rawlogs" data consists of a long string of results data which is already in a structured data format. I can't see this is adding much value in duplciating the data but it does create a huge string with a lot of long problematic pathnames and inflates the results data size.

I suggest we drop this data as obsolete and not necessary.

(cherry picked from commit 5b2c70fab2ffa409b861d83f048b65d458d03a90)
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 6 ------
 1 file changed, 6 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +0 -6
index 021e894012..139c43c7e0 100644
@@ -177,12 +177,8 @@ class ReproducibleTests(OESelftestTestCase):
             self.sstate_targets = bb_vars['OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS'].split()
 
         self.extraresults = {}
-        self.extraresults.setdefault('reproducible.rawlogs', {})['log'] = ''
         self.extraresults.setdefault('reproducible', {}).setdefault('files', {})
 
-    def append_to_log(self, msg):
-        self.extraresults['reproducible.rawlogs']['log'] += msg
-
     def compare_packages(self, reference_dir, test_dir, diffutils_sysroot):
         result = PackageCompareResults(self.oeqa_reproducible_excluded_packages)
 
@@ -295,8 +291,6 @@ class ReproducibleTests(OESelftestTestCase):
 
                 self.logger.info('Reproducibility summary for %s: %s' % (c, result))
 
-                self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total))
-
                 self.write_package_list(package_class, 'missing', result.missing)
                 self.write_package_list(package_class, 'different', result.different)
                 self.write_package_list(package_class, 'different_excluded', result.different_excluded)
Signature · 1 line
2.34.1
#6 lore raw

PATCH 6/15 selftest/reproducible: Clean up pathnames

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

There are several problems with these paths. Firstly they contain full system paths which depend upon where the test was run. These are pretty pointless and just take up a lot of space making the results files large.

Secondly, they contain the same path twice. The reference and target path will always be the same thing in two different locations.

Strip off the prefix and remove the duplication. This does change the output data but that can't really be avoided. It does shrink the results data and makes it more readable.

(cherry picked from commit 81a44de36e864b08687451fd85aeba7c529fd7f7)
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +1 -1
index 139c43c7e0..7c5c1a996c 100644
@@ -205,7 +205,7 @@ class ReproducibleTests(OESelftestTestCase):
 
     def write_package_list(self, package_class, name, packages):
         self.extraresults['reproducible']['files'].setdefault(package_class, {})[name] = [
-                {'reference': p.reference, 'test': p.test} for p in packages]
+                p.reference.split("/./")[1] for p in packages]
 
     def copy_file(self, source, dest):
         bb.utils.mkdirhier(os.path.dirname(dest))
Signature · 1 line
2.34.1
#7 lore raw

PATCH 7/15 resulttool: Allow store to filter to specific revisions

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

We have a challenge on the autobuilder where test results from both OE-Core and poky are being mixed together during result storage which is confusing the data. Add a way to filter to specific revisions as the least worst way to fix the various issues this is causing.

(cherry picked from commit 3f276a0dc65341668788853be2cf27ab6aa12b13)
---
 scripts/lib/resulttool/store.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/resulttool/store.py +5 -0
index e0951f0a8f..430213bbfe 100644
@@ -65,6 +65,9 @@ def store(args, logger):
 
         for r in revisions:
             results = revisions[r]
+            if args.revision and r[0] != args.revision:
+                logger.info('skipping %s as non-matching' % r[0])
+                continue
             keywords = {'commit': r[0], 'branch': r[1], "commit_count": r[2]}
             subprocess.check_call(["find", tempdir, "!", "-path", "./.git/*", "-delete"])
             resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
@@ -102,3 +105,5 @@ def register_commands(subparsers):
                               help='add executed-by configuration to each result file')
     parser_build.add_argument('-t', '--extra-test-env', default='',
                               help='add extra test environment data to each result file configuration')
+    parser_build.add_argument('-r', '--revision', default='',
+                              help='only store data for the specified revision')
Signature · 1 line
2.34.1
#8 lore raw

PATCH 8/15 resulttool: Use single space indentation in json output

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Using 4 space indentation in resulted in hundreds of megabytes of extra file size in general use. Reduce this to make filesizes more managable and reduce the processing cost. Some level of indentation and spacing does make the files more readable and allows use of git diff so we need to retain some of it.

(cherry picked from commit a274cdcaf852cca9497f0358f44dda99c06aacbe)
---
 meta/lib/oeqa/core/runner.py              | 2 +-
 scripts/lib/resulttool/manualexecution.py | 2 +-
 scripts/lib/resulttool/report.py          | 2 +-
 scripts/lib/resulttool/resultutils.py     | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
meta/lib/oeqa/core/runner.py +1 -1
index a86a706bd9..b683d9b80a 100644
@@ -357,7 +357,7 @@ class OETestResultJSONHelper(object):
             os.makedirs(write_dir, exist_ok=True)
         test_results = self._get_existing_testresults_if_available(write_dir)
         test_results[result_id] = {'configuration': configuration, 'result': test_result}
-        json_testresults = json.dumps(test_results, sort_keys=True, indent=4)
+        json_testresults = json.dumps(test_results, sort_keys=True, indent=1)
         self._write_file(write_dir, self.testresult_filename, json_testresults)
         if has_bb:
             bb.utils.unlockfile(lf)
scripts/lib/resulttool/manualexecution.py +1 -1
index ecb27c5933..ae0861ac6b 100755
@@ -22,7 +22,7 @@ def load_json_file(f):
 def write_json_file(f, json_data):
     os.makedirs(os.path.dirname(f), exist_ok=True)
     with open(f, 'w') as filedata:
-        filedata.write(json.dumps(json_data, sort_keys=True, indent=4))
+        filedata.write(json.dumps(json_data, sort_keys=True, indent=1))
 
 class ManualTestRunner(object):
 
scripts/lib/resulttool/report.py +1 -1
index a349510ab8..1c100b00ab 100644
@@ -256,7 +256,7 @@ class ResultsTextReport(object):
                 if selected_test_case_only:
                     print_selected_testcase_result(raw_results, selected_test_case_only)
                 else:
-                    print(json.dumps(raw_results, sort_keys=True, indent=4))
+                    print(json.dumps(raw_results, sort_keys=True, indent=1))
             else:
                 print('Could not find raw test result for %s' % raw_test)
             return 0
scripts/lib/resulttool/resultutils.py +1 -1
index c5521d81bd..8fd4e0a9cf 100644
@@ -169,7 +169,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p
         if not ptestjson:
             resultsout = strip_ptestresults(results[res])
         with open(dst, 'w') as f:
-            f.write(json.dumps(resultsout, sort_keys=True, indent=4))
+            f.write(json.dumps(resultsout, sort_keys=True, indent=1))
         for res2 in results[res]:
             if ptestlogs and 'result' in results[res][res2]:
                 seriesresults = results[res][res2]['result']
Signature · 1 line
2.34.1
#9 lore raw

PATCH 9/15 oeqa/utils/gitarchive: Return tag name and improve exclude handling

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Tweak the gitarchive exclude handling not to error if excluded files don't match.

Also return the tagname created so that other code can then use it.

(cherry picked from commit 1adba3430faffdf6217b6a00533a3b48a9388abc)
---
 meta/lib/oeqa/utils/gitarchive.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/utils/gitarchive.py +3 -1
index 10cb267dfa..a826646059 100644
@@ -67,7 +67,7 @@ def git_commit_data(repo, data_dir, branch, message, exclude, notes, log):
 
         # Remove files that are excluded
         if exclude:
-            repo.run_cmd(['rm', '--cached'] + [f for f in exclude], env_update)
+            repo.run_cmd(['rm', '--cached', '--ignore-unmatch'] + [f for f in exclude], env_update)
 
         tree = repo.run_cmd('write-tree', env_update)
 
@@ -202,6 +202,8 @@ def gitarchive(data_dir, git_dir, no_create, bare, commit_msg_subject, commit_ms
         log.info("Pushing data to remote")
         data_repo.run_cmd(cmd)
 
+    return tag_name
+
 # Container class for tester revisions
 TestedRev = namedtuple('TestedRev', 'commit commit_number tags')
 
Signature · 1 line
2.34.1
#10 lore raw

PATCH 10/15 resulttool: Fix passthrough of --all files in store mode

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

When using store mode, --all was broken as not all files were being preserved. Fix this by limiting the scope of the git rm command.

(cherry picked from commit 9604561d2022b6c76b1cb4186d40800d1affdd2b)
---
 scripts/lib/resulttool/store.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/resulttool/store.py +1 -1
index 430213bbfe..903e29627a 100644
@@ -69,7 +69,7 @@ def store(args, logger):
                 logger.info('skipping %s as non-matching' % r[0])
                 continue
             keywords = {'commit': r[0], 'branch': r[1], "commit_count": r[2]}
-            subprocess.check_call(["find", tempdir, "!", "-path", "./.git/*", "-delete"])
+            subprocess.check_call(["find", tempdir, "-name", "testresults.json", "!", "-path", "./.git/*", "-delete"])
             resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
 
             logger.info('Storing test result into git repository %s' % args.git_dir)
Signature · 1 line
2.34.1
#11 lore raw

PATCH 11/15 resulttool: Add --logfile-archive option to store mode

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Storing the log files inside the testresults git repo isn't scaling and isn't really appropriate use of a git repository. Allow these to be optionally stored in a separate filesystem location so the git repo can remain managable.

(cherry picked from commit 1afc0f3d7e93fa8496be241e9622d3b9a6904bd5)
---
 scripts/lib/resulttool/store.py | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)
scripts/lib/resulttool/store.py +17 -2
index 903e29627a..578910d234 100644
@@ -74,12 +74,25 @@ def store(args, logger):
 
             logger.info('Storing test result into git repository %s' % args.git_dir)
 
-            gitarchive.gitarchive(tempdir, args.git_dir, False, False,
+            excludes = []
+            if args.logfile_archive:
+                excludes = ['*.log', "*.log.zst"]
+
+            tagname = gitarchive.gitarchive(tempdir, args.git_dir, False, False,
                                   "Results of {branch}:{commit}", "branch: {branch}\ncommit: {commit}", "{branch}",
                                   False, "{branch}/{commit_count}-g{commit}/{tag_number}",
                                   'Test run #{tag_number} of {branch}:{commit}', '',
-                                  [], [], False, keywords, logger)
+                                  excludes, [], False, keywords, logger)
 
+            if args.logfile_archive:
+                logdir = args.logfile_archive + "/" + tagname
+                shutil.copytree(tempdir, logdir)
+                for root, dirs,  files in os.walk(logdir):
+                    for name in files:
+                        if not name.endswith(".log"):
+                            continue
+                        f = os.path.join(root, name)
+                        subprocess.run(["zstd", f, "--rm"], check=True, capture_output=True)
     finally:
         subprocess.check_call(["rm", "-rf",  tempdir])
 
@@ -107,3 +120,5 @@ def register_commands(subparsers):
                               help='add extra test environment data to each result file configuration')
     parser_build.add_argument('-r', '--revision', default='',
                               help='only store data for the specified revision')
+    parser_build.add_argument('-l', '--logfile-archive', default='',
+                              help='directory to separately archive log files along with a copy of the results')
Signature · 1 line
2.34.1
#12 lore raw

PATCH 12/15 resulttool: Handle ltp rawlogs as well as ptest

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Improve the rawlogs handling to include ltp logs as well as the ptest ones to reduce the size of the results git repos.

(cherry picked from commit a0a1954d559609c2c1ca16936d0d68eb3c4c6b45)
---
 scripts/lib/resulttool/resultutils.py | 33 ++++++++++++++++-----------
 1 file changed, 20 insertions(+), 13 deletions(-)
scripts/lib/resulttool/resultutils.py +20 -13
index 8fd4e0a9cf..b9b93afaa6 100644
@@ -14,8 +14,11 @@ import scriptpath
 import copy
 import urllib.request
 import posixpath
+import logging
 scriptpath.add_oe_lib_path()
 
+logger = logging.getLogger('resulttool')
+
 flatten_map = {
     "oeselftest": [],
     "runtime": [],
@@ -38,6 +41,12 @@ store_map = {
     "manual": ['TEST_TYPE', 'TEST_MODULE', 'MACHINE', 'IMAGE_BASENAME']
 }
 
+rawlog_sections = {
+    "ptestresult.rawlogs": "ptest",
+    "ltpresult.rawlogs": "ltp",
+    "ltpposixresult.rawlogs": "ltpposix"
+}
+
 def is_url(p):
     """
     Helper for determining if the given path is a URL
@@ -108,15 +117,14 @@ def filter_resultsdata(results, resultid):
                  newresults[r][i] = results[r][i]
     return newresults
 
-def strip_ptestresults(results):
+def strip_logs(results):
     newresults = copy.deepcopy(results)
-    #for a in newresults2:
-    #  newresults = newresults2[a]
     for res in newresults:
         if 'result' not in newresults[res]:
             continue
-        if 'ptestresult.rawlogs' in newresults[res]['result']:
-            del newresults[res]['result']['ptestresult.rawlogs']
+        for logtype in rawlog_sections:
+            if logtype in newresults[res]['result']:
+                del newresults[res]['result'][logtype]
         if 'ptestresult.sections' in newresults[res]['result']:
             for i in newresults[res]['result']['ptestresult.sections']:
                 if 'log' in newresults[res]['result']['ptestresult.sections'][i]:
@@ -155,9 +163,6 @@ def generic_get_rawlogs(sectname, results):
         return None
     return decode_log(results[sectname]['log'])
 
-def ptestresult_get_rawlogs(results):
-    return generic_get_rawlogs('ptestresult.rawlogs', results)
-
 def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, ptestlogs=False):
     for res in results:
         if res:
@@ -167,16 +172,18 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p
         os.makedirs(os.path.dirname(dst), exist_ok=True)
         resultsout = results[res]
         if not ptestjson:
-            resultsout = strip_ptestresults(results[res])
+            resultsout = strip_logs(results[res])
         with open(dst, 'w') as f:
             f.write(json.dumps(resultsout, sort_keys=True, indent=1))
         for res2 in results[res]:
             if ptestlogs and 'result' in results[res][res2]:
                 seriesresults = results[res][res2]['result']
-                rawlogs = ptestresult_get_rawlogs(seriesresults)
-                if rawlogs is not None:
-                    with open(dst.replace(fn, "ptest-raw.log"), "w+") as f:
-                        f.write(rawlogs)
+                for logtype in rawlog_sections:
+                    logdata = generic_get_rawlogs(logtype, seriesresults)
+                    if logdata is not None:
+                        logger.info("Extracting " + rawlog_sections[logtype] + "-raw.log")
+                        with open(dst.replace(fn, rawlog_sections[logtype] + "-raw.log"), "w+") as f:
+                            f.write(logdata)
                 if 'ptestresult.sections' in seriesresults:
                     for i in seriesresults['ptestresult.sections']:
                         sectionlog = ptestresult_get_log(seriesresults, i)
Signature · 1 line
2.34.1
#13 lore raw

PATCH 13/15 resulttool: Clean up repoducible build logs

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

We've improved the data stored for reproduicible builds. Teach resulttool how to apply those cleanups when reprocessing data so we can reduce results file sizes and make the data easier to process.

(cherry picked from commit b799c57ae6d61c1b1c7035c8a2c4ba6ee08d1a81)
---
 scripts/lib/resulttool/resultutils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
scripts/lib/resulttool/resultutils.py +22 -0
index b9b93afaa6..9cba8639a3 100644
@@ -131,6 +131,27 @@ def strip_logs(results):
                     del newresults[res]['result']['ptestresult.sections'][i]['log']
     return newresults
 
+def handle_cleanups(results):
+    # Remove pointless path duplication from old format reproducibility results
+    for res2 in results:
+        try:
+            section = results[res2]['result']['reproducible']['files']
+            for pkgtype in section:
+                for filelist in section[pkgtype].copy():
+                    if section[pkgtype][filelist] and type(section[pkgtype][filelist][0]) == dict:
+                        newlist = []
+                        for entry in section[pkgtype][filelist]:
+                            newlist.append(entry["reference"].split("/./")[1])
+                        section[pkgtype][filelist] = newlist
+
+        except KeyError:
+            pass
+    # Remove pointless duplicate rawlogs data
+    try:
+        del results[res2]['result']['reproducible.rawlogs']
+    except KeyError:
+        pass
+
 def decode_log(logdata):
     if isinstance(logdata, str):
         return logdata
@@ -173,6 +194,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p
         resultsout = results[res]
         if not ptestjson:
             resultsout = strip_logs(results[res])
+        handle_cleanups(resultsout)
         with open(dst, 'w') as f:
             f.write(json.dumps(resultsout, sort_keys=True, indent=1))
         for res2 in results[res]:
Signature · 1 line
2.34.1
#14 lore raw

PATCH 14/15 resulttool: Trim the precision of duration information

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

The duration values have pointless amounts of precision. Removing some of the least significant digits reduces result size and makes the results easier to read.

(cherry picked from commit a789a2e6d97bb8efd663226a17db8d1ca6c1e40f)
---
 scripts/lib/resulttool/resultutils.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
scripts/lib/resulttool/resultutils.py +17 -0
index 9cba8639a3..760e426de0 100644
@@ -131,6 +131,22 @@ def strip_logs(results):
                     del newresults[res]['result']['ptestresult.sections'][i]['log']
     return newresults
 
+# For timing numbers, crazy amounts of precision don't make sense and just confuse
+# the logs. For numbers over 1, trim to 3 decimal places, for numbers less than 1,
+# trim to 4 significant digits
+def trim_durations(results):
+    for res in results:
+        if 'result' not in results[res]:
+            continue
+        for entry in results[res]['result']:
+            if 'duration' in results[res]['result'][entry]:
+                duration = results[res]['result'][entry]['duration']
+                if duration > 1:
+                    results[res]['result'][entry]['duration'] = float("%.3f" % duration)
+                elif duration < 1:
+                    results[res]['result'][entry]['duration'] = float("%.4g" % duration)
+    return results
+
 def handle_cleanups(results):
     # Remove pointless path duplication from old format reproducibility results
     for res2 in results:
@@ -194,6 +210,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p
         resultsout = results[res]
         if not ptestjson:
             resultsout = strip_logs(results[res])
+        trim_durations(resultsout)
         handle_cleanups(resultsout)
         with open(dst, 'w') as f:
             f.write(json.dumps(resultsout, sort_keys=True, indent=1))
Signature · 1 line
2.34.1
#15 lore raw

PATCH 15/15 resulttool: Improve repo layout for oeselftest results

In reply to Steve Sakoman

From: Richard Purdie <[email protected]>

Having all oe-selftest results on top of each other results in a large 640MB json file which is hard to use. Split the results out per machine and test type.

This also stops the toolchain raw logs from overwriting each other meaning more than one MACHINE is preserved.

(cherry picked from commit 4b890f04bc7d147b4a11b824a84f3d2abd75ac54)
---
 scripts/lib/resulttool/resultutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/resulttool/resultutils.py +1 -1
index 760e426de0..b8fc79a6ac 100644
@@ -34,7 +34,7 @@ regression_map = {
     "manual": ['TEST_TYPE', 'TEST_MODULE', 'IMAGE_BASENAME', 'MACHINE']
 }
 store_map = {
-    "oeselftest": ['TEST_TYPE'],
+    "oeselftest": ['TEST_TYPE', 'TESTSERIES', 'MACHINE'],
     "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'],
     "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
     "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
Signature · 1 line
2.34.1