Patch review
cover Patch review
Please review this set of changes for scarthgap and have comments back by end of day Friday, December 6
Passed a-full on autobuilder:
https://valkyrie.yoctoproject.org/#/builders/29/builds/582
The following changes since commit e0c4f2f69723f64a29ba35c4de5fb1a7c79ef718:
rootfs: Ensure run-postinsts is not uninstalled for read-only-rootfs-delayed-postinsts (2024-12-02 10:18:14 -0800)
are available in the Git repository at:
https://git.openembedded.org/openembedded-core-contrib stable/scarthgap-nut
https://git.openembedded.org/openembedded-core-contrib/log/?h=stable/scarthgap-nut
Changqing Li (1):
rxvt-unicode.inc: disable the terminfo installation by setting TIC to
:
Hongxu Jia (2):
ovmf: fix CVE-2024-38796
ovmf: fix CVE-2024-1298
Richard Purdie (12):
do_package/sstate/sstatesig: Change timestamp clamping to hash output
only
selftest/reproducible: Drop rawlogs
selftest/reproducible: Clean up pathnames
resulttool: Allow store to filter to specific revisions
resulttool: Use single space indentation in json output oeqa/utils/gitarchive: Return tag name and improve exclude handling
resulttool: Fix passthrough of --all files in store mode
resulttool: Add --logfile-archive option to store mode
resulttool: Handle ltp rawlogs as well as ptest
resulttool: Clean up repoducible build logs
resulttool: Trim the precision of duration information
resulttool: Improve repo layout for oeselftest results
meta/classes-global/sstate.bbclass | 16 ---- meta/lib/oe/sstatesig.py | 7 +- meta/lib/oeqa/core/runner.py | 2 +- meta/lib/oeqa/selftest/cases/reproducible.py | 8 +- meta/lib/oeqa/utils/gitarchive.py | 4 +- ...ential-UINT32-overflow-in-S3-ResumeC.patch | 51 +++++++++++++ ...-Fix-overflow-issue-in-BasePeCoffLib.patch | 36 +++++++++ meta/recipes-core/ovmf/ovmf_git.bb | 2 + .../rxvt-unicode/rxvt-unicode.inc | 3 +- scripts/lib/resulttool/manualexecution.py | 2 +- scripts/lib/resulttool/report.py | 2 +- scripts/lib/resulttool/resultutils.py | 76 +++++++++++++++---- scripts/lib/resulttool/store.py | 26 ++++++- 13 files changed, 188 insertions(+), 47 deletions(-) create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch
Signature · 1 line
2.34.1
PATCH 1/15 ovmf: fix CVE-2024-38796
From: Hongxu Jia <[email protected]>
Backport fix from upstream to resolve CVE-2024-38796
https://github.com/tianocore/edk2/commit/c95233b8525ca6828921affd1496146cff262e65
--- ...-Fix-overflow-issue-in-BasePeCoffLib.patch | 36 +++++++++++++++++++ meta/recipes-core/ovmf/ovmf_git.bb | 1 + 2 files changed, 37 insertions(+) create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch
meta/recipes-core/ovmf/ovmf/0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch new file +36 -0
@@ -0,0 +1,36 @@ +From 5f7bd3f3c4747d5bb2733f017f8c5b93b63a74e3 Mon Sep 17 00:00:00 2001 +From: Doug Flick <[email protected]> +Date: Fri, 22 Nov 2024 13:03:33 +0800 +Subject: [PATCH] MdePkg: Fix overflow issue in BasePeCoffLib + +The RelocDir->Size is a UINT32 value, and RelocDir->VirtualAddress is +also a UINT32 value. The current code does not check for overflow when +adding RelocDir->Size to RelocDir->VirtualAddress. This patch adds a +check to ensure that the addition does not overflow. + +Signed-off-by: Doug Flick <[email protected]> +Authored-by: sriraamx gobichettipalayam <[email protected]> + +CVE: CVE-2024-38796 +Upstream-Status: Backport [https://github.com/tianocore/edk2/commit/c95233b8525ca6828921affd1496146cff262e65] +Signed-off-by: Hongxu Jia <[email protected]> +--- + MdePkg/Library/BasePeCoffLib/BasePeCoff.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/MdePkg/Library/BasePeCoffLib/BasePeCoff.c b/MdePkg/Library/BasePeCoffLib/BasePeCoff.c +index 86ff2e7..128090d 100644 +--- a/MdePkg/Library/BasePeCoffLib/BasePeCoff.c ++++ b/MdePkg/Library/BasePeCoffLib/BasePeCoff.c +@@ -1054,7 +1054,7 @@ PeCoffLoaderRelocateImage ( + RelocDir = &Hdr.Te->DataDirectory[0]; + } + +- if ((RelocDir != NULL) && (RelocDir->Size > 0)) { ++ if ((RelocDir != NULL) && (RelocDir->Size > 0) && (RelocDir->Size - 1 < MAX_UINT32 - RelocDir->VirtualAddress)) { + RelocBase = (EFI_IMAGE_BASE_RELOCATION *)PeCoffLoaderImageAddress (ImageContext, RelocDir->VirtualAddress, TeStrippedOffset); + RelocBaseEnd = (EFI_IMAGE_BASE_RELOCATION *)PeCoffLoaderImageAddress ( + ImageContext, +-- +2.34.1 +
meta/recipes-core/ovmf/ovmf_git.bb +1 -0
@@ -24,6 +24,7 @@ SRC_URI = "gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https \ file://0002-BaseTools-makefile-adjust-to-build-in-under-bitbake.patch \ file://0003-debug-prefix-map.patch \ file://0004-reproducible.patch \ + file://0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch \ " PV = "edk2-stable202402"
Signature · 1 line
2.34.1
PATCH 2/15 ovmf: fix CVE-2024-1298
In reply to Steve SakomanFrom: Hongxu Jia <[email protected]>
Backport fix from upstream to resolve CVE-2024-1298
https://github.com/tianocore/edk2/commit/284dbac43da752ee34825c8b3f6f9e8281cb5a19
--- ...ential-UINT32-overflow-in-S3-ResumeC.patch | 51 +++++++++++++++++++ meta/recipes-core/ovmf/ovmf_git.bb | 1 + 2 files changed, 52 insertions(+) create mode 100644 meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch
meta/recipes-core/ovmf/ovmf/0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch new file +51 -0
@@ -0,0 +1,51 @@ +From 150ea3ea4c821b133a782eeb33ef2a9c8fd8d7c3 Mon Sep 17 00:00:00 2001 +From: Hongxu Jia <[email protected]> +Date: Fri, 22 Nov 2024 13:05:57 +0800 +Subject: [PATCH] MdeModulePkg: Potential UINT32 overflow in S3 ResumeCount + +REF: https://bugzilla.tianocore.org/show_bug.cgi?id=4677 + +Attacker able to modify physical memory and ResumeCount. +System will crash/DoS when ResumeCount reaches its MAX_UINT32. + +Cc: Zhiguang Liu <[email protected]> +Cc: Dandan Bi <[email protected]> +Cc: Liming Gao <[email protected]> + +Signed-off-by: Pakkirisamy ShanmugavelX <[email protected]> +Reviewed-by: Liming Gao <[email protected]> + +CVE: CVE-2024-1298 +Upstream-Status: Backport [https://github.com/tianocore/edk2/commit/284dbac43da752ee34825c8b3f6f9e8281cb5a19] +Signed-off-by: Hongxu Jia <[email protected]> +--- + .../FirmwarePerformancePei.c | 12 ++++++++---- + 1 file changed, 8 insertions(+), 4 deletions(-) + +diff --git a/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c b/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c +index 2f2b2a8..2ba9215 100644 +--- a/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c ++++ b/MdeModulePkg/Universal/Acpi/FirmwarePerformanceDataTablePei/FirmwarePerformancePei.c +@@ -112,11 +112,15 @@ FpdtStatusCodeListenerPei ( + // + S3ResumeTotal = MultU64x32 (AcpiS3ResumeRecord->AverageResume, AcpiS3ResumeRecord->ResumeCount); + AcpiS3ResumeRecord->ResumeCount++; +- AcpiS3ResumeRecord->AverageResume = DivU64x32 (S3ResumeTotal + AcpiS3ResumeRecord->FullResume, AcpiS3ResumeRecord->ResumeCount); ++ if (AcpiS3ResumeRecord->ResumeCount > 0) { ++ AcpiS3ResumeRecord->AverageResume = DivU64x32 (S3ResumeTotal + AcpiS3ResumeRecord->FullResume, AcpiS3ResumeRecord->ResumeCount); ++ DEBUG ((DEBUG_INFO, "\nFPDT: S3 Resume Performance - AverageResume = 0x%x\n", AcpiS3ResumeRecord->AverageResume)); ++ } else { ++ DEBUG ((DEBUG_ERROR, "\nFPDT: S3 ResumeCount reaches the MAX_UINT32 value. S3 ResumeCount record reset to Zero.")); ++ } + +- DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - ResumeCount = %d\n", AcpiS3ResumeRecord->ResumeCount)); +- DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - FullResume = %ld\n", AcpiS3ResumeRecord->FullResume)); +- DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - AverageResume = %ld\n", AcpiS3ResumeRecord->AverageResume)); ++ DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - ResumeCount = 0x%x\n", AcpiS3ResumeRecord->ResumeCount)); ++ DEBUG ((DEBUG_INFO, "FPDT: S3 Resume Performance - FullResume = 0x%x\n", AcpiS3ResumeRecord->FullResume)); + + // + // Update S3 Suspend Performance Record. +-- +2.34.1 +
meta/recipes-core/ovmf/ovmf_git.bb +1 -0
@@ -25,6 +25,7 @@ SRC_URI = "gitsm://github.com/tianocore/edk2.git;branch=master;protocol=https \ file://0003-debug-prefix-map.patch \ file://0004-reproducible.patch \ file://0001-MdePkg-Fix-overflow-issue-in-BasePeCoffLib.patch \ + file://0001-MdeModulePkg-Potential-UINT32-overflow-in-S3-ResumeC.patch \ " PV = "edk2-stable202402"
Signature · 1 line
2.34.1
PATCH 3/15 do_package/sstate/sstatesig: Change timestamp clamping to hash output only
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
The code was changing the timestamps of the files in the do_package output, particularly the files added for debug sources. This was to do two things:
a) make do_package sstate more reproducible
b) ensure better hash equivalence matching
Unfortuately the debug source files are hardlinks into the source tree for efficiency so touching these, touches a lot of files in ${B} and ${S}. This causes unpredictable effects if compile is run again for example, or could cause compiling in the install task.
The hash equivalence matching is of key importance but we can mimic that using clamping of the file timestamps in the depsig output used to generate the hashes.
This patch drops the global timestamp clamping, instead allowing the files to retain their creation timestamps into sstate. This makes do_package sstate slightly less reproducibile. We could clamp the sstate timestamps but that would lead to two different sets of timestamps depending on whether the data came from sstate or not. I'd prefer to have consistent code behaviour, rather than differing behavhour depending on whether data came from sstate or not.
If we wanted to have reproducibiliy and fix the "corruption" of S/B and have consistent codepaths, the only other option would be two copies of the sources, which could end up huge and seems the least desireable option.
This patch therefore drops the timestamp clamping in the sstate files and tweaks the depsig data generation to clamp the timestamps for do_package instead since this seems the best compromise.
I validated that rpm/deb/ipk files still generate correctly as before.
--- meta/classes-global/sstate.bbclass | 16 ---------------- meta/lib/oe/sstatesig.py | 7 ++++++- 2 files changed, 6 insertions(+), 17 deletions(-)
meta/classes-global/sstate.bbclass +0 -16
@@ -648,15 +648,6 @@ def sstate_package(ss, d): tmpdir = d.getVar('TMPDIR') - fixtime = False - if ss['task'] == "package": - fixtime = True - - def fixtimestamp(root, path): - f = os.path.join(root, path) - if os.lstat(f).st_mtime > sde: - os.utime(f, (sde, sde), follow_symlinks=False) - sstatebuild = d.expand("${WORKDIR}/sstate-build-%s/" % ss['task']) sde = int(d.getVar("SOURCE_DATE_EPOCH") or time.time()) d.setVar("SSTATE_CURRTASK", ss['task'])
@@ -671,8 +662,6 @@ def sstate_package(ss, d): # to sstate tasks but there aren't many of these so better just avoid them entirely. for walkroot, dirs, files in os.walk(state[1]): for file in files + dirs: - if fixtime: - fixtimestamp(walkroot, file) srcpath = os.path.join(walkroot, file) if not os.path.islink(srcpath): continue
@@ -694,11 +683,6 @@ def sstate_package(ss, d): bb.utils.mkdirhier(plain) bb.utils.mkdirhier(pdir) bb.utils.rename(plain, pdir) - if fixtime: - fixtimestamp(pdir, "") - for walkroot, dirs, files in os.walk(pdir): - for file in files + dirs: - fixtimestamp(walkroot, file) d.setVar('SSTATE_BUILDDIR', sstatebuild) d.setVar('SSTATE_INSTDIR', sstatebuild)
meta/lib/oe/sstatesig.py +6 -1
@@ -524,6 +524,7 @@ def OEOuthashBasic(path, sigfile, task, d): if task == "package": include_timestamps = True include_root = False + source_date_epoch = float(d.getVar("SOURCE_DATE_EPOCH")) hash_version = d.getVar('HASHEQUIV_HASH_VERSION') extra_sigdata = d.getVar("HASHEQUIV_EXTRA_SIGDATA")
@@ -615,7 +616,11 @@ def OEOuthashBasic(path, sigfile, task, d): raise Exception(msg).with_traceback(e.__traceback__) if include_timestamps: - update_hash(" %10d" % s.st_mtime) + # Need to clamp to SOURCE_DATE_EPOCH + if s.st_mtime > source_date_epoch: + update_hash(" %10d" % source_date_epoch) + else: + update_hash(" %10d" % s.st_mtime) update_hash(" ") if stat.S_ISBLK(s.st_mode) or stat.S_ISCHR(s.st_mode):
Signature · 1 line
2.34.1
PATCH 4/15 rxvt-unicode.inc: disable the terminfo installation by setting TIC to :
In reply to Steve SakomanFrom: Changqing Li <[email protected]>
Without this change, TIC is the native tic in recipe-sysroot-native. By default, native tic has set its default terminfo path to native path:
${datadir}/terminfo; $HOME/.terminfo
When sstate cache is used, the cached native tic's terminfo path could be a path not exist on current host, then native tic will try to install terminfo to HOME dir, cause host contamination.
Disable the terminfo installation by setting TIC to :
--- meta/recipes-sato/rxvt-unicode/rxvt-unicode.inc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-sato/rxvt-unicode/rxvt-unicode.inc +2 -1
@@ -28,7 +28,8 @@ EXTRA_OECONF = "--enable-xim \ --with-codesets=eu --enable-pointer-blank \ --enable-text-blink --enable-rxvt-scroll \ --enable-combining --disable-perl \ - --with-x=${STAGING_DIR_HOST}${prefix}" + --with-x=${STAGING_DIR_HOST}${prefix} \ + ac_cv_path_TIC=:" PACKAGECONFIG ??= "" PACKAGECONFIG[startup] = "--enable-startup-notification,--disable-startup-notification,startup-notification,"
Signature · 1 line
2.34.1
PATCH 5/15 selftest/reproducible: Drop rawlogs
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
The "rawlogs" data consists of a long string of results data which is already in a structured data format. I can't see this is adding much value in duplciating the data but it does create a huge string with a lot of long problematic pathnames and inflates the results data size.
I suggest we drop this data as obsolete and not necessary.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 6 ------ 1 file changed, 6 deletions(-)
meta/lib/oeqa/selftest/cases/reproducible.py +0 -6
@@ -177,12 +177,8 @@ class ReproducibleTests(OESelftestTestCase): self.sstate_targets = bb_vars['OEQA_REPRODUCIBLE_TEST_SSTATE_TARGETS'].split() self.extraresults = {} - self.extraresults.setdefault('reproducible.rawlogs', {})['log'] = '' self.extraresults.setdefault('reproducible', {}).setdefault('files', {}) - def append_to_log(self, msg): - self.extraresults['reproducible.rawlogs']['log'] += msg - def compare_packages(self, reference_dir, test_dir, diffutils_sysroot): result = PackageCompareResults(self.oeqa_reproducible_excluded_packages)
@@ -295,8 +291,6 @@ class ReproducibleTests(OESelftestTestCase): self.logger.info('Reproducibility summary for %s: %s' % (c, result)) - self.append_to_log('\n'.join("%s: %s" % (r.status, r.test) for r in result.total)) - self.write_package_list(package_class, 'missing', result.missing) self.write_package_list(package_class, 'different', result.different) self.write_package_list(package_class, 'different_excluded', result.different_excluded)
Signature · 1 line
2.34.1
PATCH 6/15 selftest/reproducible: Clean up pathnames
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
There are several problems with these paths. Firstly they contain full system paths which depend upon where the test was run. These are pretty pointless and just take up a lot of space making the results files large.
Secondly, they contain the same path twice. The reference and target path will always be the same thing in two different locations.
Strip off the prefix and remove the duplication. This does change the output data but that can't really be avoided. It does shrink the results data and makes it more readable.
--- meta/lib/oeqa/selftest/cases/reproducible.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/reproducible.py +1 -1
@@ -205,7 +205,7 @@ class ReproducibleTests(OESelftestTestCase): def write_package_list(self, package_class, name, packages): self.extraresults['reproducible']['files'].setdefault(package_class, {})[name] = [ - {'reference': p.reference, 'test': p.test} for p in packages] + p.reference.split("/./")[1] for p in packages] def copy_file(self, source, dest): bb.utils.mkdirhier(os.path.dirname(dest))
Signature · 1 line
2.34.1
PATCH 7/15 resulttool: Allow store to filter to specific revisions
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
We have a challenge on the autobuilder where test results from both OE-Core and poky are being mixed together during result storage which is confusing the data. Add a way to filter to specific revisions as the least worst way to fix the various issues this is causing.
--- scripts/lib/resulttool/store.py | 5 +++++ 1 file changed, 5 insertions(+)
scripts/lib/resulttool/store.py +5 -0
@@ -65,6 +65,9 @@ def store(args, logger): for r in revisions: results = revisions[r] + if args.revision and r[0] != args.revision: + logger.info('skipping %s as non-matching' % r[0]) + continue keywords = {'commit': r[0], 'branch': r[1], "commit_count": r[2]} subprocess.check_call(["find", tempdir, "!", "-path", "./.git/*", "-delete"]) resultutils.save_resultsdata(results, tempdir, ptestlogs=True)
@@ -102,3 +105,5 @@ def register_commands(subparsers): help='add executed-by configuration to each result file') parser_build.add_argument('-t', '--extra-test-env', default='', help='add extra test environment data to each result file configuration') + parser_build.add_argument('-r', '--revision', default='', + help='only store data for the specified revision')
Signature · 1 line
2.34.1
PATCH 8/15 resulttool: Use single space indentation in json output
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Using 4 space indentation in resulted in hundreds of megabytes of extra file size in general use. Reduce this to make filesizes more managable and reduce the processing cost. Some level of indentation and spacing does make the files more readable and allows use of git diff so we need to retain some of it.
--- meta/lib/oeqa/core/runner.py | 2 +- scripts/lib/resulttool/manualexecution.py | 2 +- scripts/lib/resulttool/report.py | 2 +- scripts/lib/resulttool/resultutils.py | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
meta/lib/oeqa/core/runner.py +1 -1
@@ -357,7 +357,7 @@ class OETestResultJSONHelper(object): os.makedirs(write_dir, exist_ok=True) test_results = self._get_existing_testresults_if_available(write_dir) test_results[result_id] = {'configuration': configuration, 'result': test_result} - json_testresults = json.dumps(test_results, sort_keys=True, indent=4) + json_testresults = json.dumps(test_results, sort_keys=True, indent=1) self._write_file(write_dir, self.testresult_filename, json_testresults) if has_bb: bb.utils.unlockfile(lf)
scripts/lib/resulttool/manualexecution.py +1 -1
@@ -22,7 +22,7 @@ def load_json_file(f): def write_json_file(f, json_data): os.makedirs(os.path.dirname(f), exist_ok=True) with open(f, 'w') as filedata: - filedata.write(json.dumps(json_data, sort_keys=True, indent=4)) + filedata.write(json.dumps(json_data, sort_keys=True, indent=1)) class ManualTestRunner(object):
scripts/lib/resulttool/report.py +1 -1
@@ -256,7 +256,7 @@ class ResultsTextReport(object): if selected_test_case_only: print_selected_testcase_result(raw_results, selected_test_case_only) else: - print(json.dumps(raw_results, sort_keys=True, indent=4)) + print(json.dumps(raw_results, sort_keys=True, indent=1)) else: print('Could not find raw test result for %s' % raw_test) return 0
scripts/lib/resulttool/resultutils.py +1 -1
@@ -169,7 +169,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p if not ptestjson: resultsout = strip_ptestresults(results[res]) with open(dst, 'w') as f: - f.write(json.dumps(resultsout, sort_keys=True, indent=4)) + f.write(json.dumps(resultsout, sort_keys=True, indent=1)) for res2 in results[res]: if ptestlogs and 'result' in results[res][res2]: seriesresults = results[res][res2]['result']
Signature · 1 line
2.34.1
PATCH 9/15 oeqa/utils/gitarchive: Return tag name and improve exclude handling
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Tweak the gitarchive exclude handling not to error if excluded files don't match.
Also return the tagname created so that other code can then use it.
--- meta/lib/oeqa/utils/gitarchive.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oeqa/utils/gitarchive.py +3 -1
@@ -67,7 +67,7 @@ def git_commit_data(repo, data_dir, branch, message, exclude, notes, log): # Remove files that are excluded if exclude: - repo.run_cmd(['rm', '--cached'] + [f for f in exclude], env_update) + repo.run_cmd(['rm', '--cached', '--ignore-unmatch'] + [f for f in exclude], env_update) tree = repo.run_cmd('write-tree', env_update)
@@ -202,6 +202,8 @@ def gitarchive(data_dir, git_dir, no_create, bare, commit_msg_subject, commit_ms log.info("Pushing data to remote") data_repo.run_cmd(cmd) + return tag_name + # Container class for tester revisions TestedRev = namedtuple('TestedRev', 'commit commit_number tags')
Signature · 1 line
2.34.1
PATCH 10/15 resulttool: Fix passthrough of --all files in store mode
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
When using store mode, --all was broken as not all files were being preserved. Fix this by limiting the scope of the git rm command.
--- scripts/lib/resulttool/store.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/resulttool/store.py +1 -1
@@ -69,7 +69,7 @@ def store(args, logger): logger.info('skipping %s as non-matching' % r[0]) continue keywords = {'commit': r[0], 'branch': r[1], "commit_count": r[2]} - subprocess.check_call(["find", tempdir, "!", "-path", "./.git/*", "-delete"]) + subprocess.check_call(["find", tempdir, "-name", "testresults.json", "!", "-path", "./.git/*", "-delete"]) resultutils.save_resultsdata(results, tempdir, ptestlogs=True) logger.info('Storing test result into git repository %s' % args.git_dir)
Signature · 1 line
2.34.1
PATCH 11/15 resulttool: Add --logfile-archive option to store mode
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Storing the log files inside the testresults git repo isn't scaling and isn't really appropriate use of a git repository. Allow these to be optionally stored in a separate filesystem location so the git repo can remain managable.
--- scripts/lib/resulttool/store.py | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
scripts/lib/resulttool/store.py +17 -2
@@ -74,12 +74,25 @@ def store(args, logger): logger.info('Storing test result into git repository %s' % args.git_dir) - gitarchive.gitarchive(tempdir, args.git_dir, False, False, + excludes = [] + if args.logfile_archive: + excludes = ['*.log', "*.log.zst"] + + tagname = gitarchive.gitarchive(tempdir, args.git_dir, False, False, "Results of {branch}:{commit}", "branch: {branch}\ncommit: {commit}", "{branch}", False, "{branch}/{commit_count}-g{commit}/{tag_number}", 'Test run #{tag_number} of {branch}:{commit}', '', - [], [], False, keywords, logger) + excludes, [], False, keywords, logger) + if args.logfile_archive: + logdir = args.logfile_archive + "/" + tagname + shutil.copytree(tempdir, logdir) + for root, dirs, files in os.walk(logdir): + for name in files: + if not name.endswith(".log"): + continue + f = os.path.join(root, name) + subprocess.run(["zstd", f, "--rm"], check=True, capture_output=True) finally: subprocess.check_call(["rm", "-rf", tempdir])
@@ -107,3 +120,5 @@ def register_commands(subparsers): help='add extra test environment data to each result file configuration') parser_build.add_argument('-r', '--revision', default='', help='only store data for the specified revision') + parser_build.add_argument('-l', '--logfile-archive', default='', + help='directory to separately archive log files along with a copy of the results')
Signature · 1 line
2.34.1
PATCH 12/15 resulttool: Handle ltp rawlogs as well as ptest
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Improve the rawlogs handling to include ltp logs as well as the ptest ones to reduce the size of the results git repos.
--- scripts/lib/resulttool/resultutils.py | 33 ++++++++++++++++----------- 1 file changed, 20 insertions(+), 13 deletions(-)
scripts/lib/resulttool/resultutils.py +20 -13
@@ -14,8 +14,11 @@ import scriptpath import copy import urllib.request import posixpath +import logging scriptpath.add_oe_lib_path() +logger = logging.getLogger('resulttool') + flatten_map = { "oeselftest": [], "runtime": [],
@@ -38,6 +41,12 @@ store_map = { "manual": ['TEST_TYPE', 'TEST_MODULE', 'MACHINE', 'IMAGE_BASENAME'] } +rawlog_sections = { + "ptestresult.rawlogs": "ptest", + "ltpresult.rawlogs": "ltp", + "ltpposixresult.rawlogs": "ltpposix" +} + def is_url(p): """ Helper for determining if the given path is a URL
@@ -108,15 +117,14 @@ def filter_resultsdata(results, resultid): newresults[r][i] = results[r][i] return newresults -def strip_ptestresults(results): +def strip_logs(results): newresults = copy.deepcopy(results) - #for a in newresults2: - # newresults = newresults2[a] for res in newresults: if 'result' not in newresults[res]: continue - if 'ptestresult.rawlogs' in newresults[res]['result']: - del newresults[res]['result']['ptestresult.rawlogs'] + for logtype in rawlog_sections: + if logtype in newresults[res]['result']: + del newresults[res]['result'][logtype] if 'ptestresult.sections' in newresults[res]['result']: for i in newresults[res]['result']['ptestresult.sections']: if 'log' in newresults[res]['result']['ptestresult.sections'][i]:
@@ -155,9 +163,6 @@ def generic_get_rawlogs(sectname, results): return None return decode_log(results[sectname]['log']) -def ptestresult_get_rawlogs(results): - return generic_get_rawlogs('ptestresult.rawlogs', results) - def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, ptestlogs=False): for res in results: if res:
@@ -167,16 +172,18 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p os.makedirs(os.path.dirname(dst), exist_ok=True) resultsout = results[res] if not ptestjson: - resultsout = strip_ptestresults(results[res]) + resultsout = strip_logs(results[res]) with open(dst, 'w') as f: f.write(json.dumps(resultsout, sort_keys=True, indent=1)) for res2 in results[res]: if ptestlogs and 'result' in results[res][res2]: seriesresults = results[res][res2]['result'] - rawlogs = ptestresult_get_rawlogs(seriesresults) - if rawlogs is not None: - with open(dst.replace(fn, "ptest-raw.log"), "w+") as f: - f.write(rawlogs) + for logtype in rawlog_sections: + logdata = generic_get_rawlogs(logtype, seriesresults) + if logdata is not None: + logger.info("Extracting " + rawlog_sections[logtype] + "-raw.log") + with open(dst.replace(fn, rawlog_sections[logtype] + "-raw.log"), "w+") as f: + f.write(logdata) if 'ptestresult.sections' in seriesresults: for i in seriesresults['ptestresult.sections']: sectionlog = ptestresult_get_log(seriesresults, i)
Signature · 1 line
2.34.1
PATCH 13/15 resulttool: Clean up repoducible build logs
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
We've improved the data stored for reproduicible builds. Teach resulttool how to apply those cleanups when reprocessing data so we can reduce results file sizes and make the data easier to process.
--- scripts/lib/resulttool/resultutils.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
scripts/lib/resulttool/resultutils.py +22 -0
@@ -131,6 +131,27 @@ def strip_logs(results): del newresults[res]['result']['ptestresult.sections'][i]['log'] return newresults +def handle_cleanups(results): + # Remove pointless path duplication from old format reproducibility results + for res2 in results: + try: + section = results[res2]['result']['reproducible']['files'] + for pkgtype in section: + for filelist in section[pkgtype].copy(): + if section[pkgtype][filelist] and type(section[pkgtype][filelist][0]) == dict: + newlist = [] + for entry in section[pkgtype][filelist]: + newlist.append(entry["reference"].split("/./")[1]) + section[pkgtype][filelist] = newlist + + except KeyError: + pass + # Remove pointless duplicate rawlogs data + try: + del results[res2]['result']['reproducible.rawlogs'] + except KeyError: + pass + def decode_log(logdata): if isinstance(logdata, str): return logdata
@@ -173,6 +194,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p resultsout = results[res] if not ptestjson: resultsout = strip_logs(results[res]) + handle_cleanups(resultsout) with open(dst, 'w') as f: f.write(json.dumps(resultsout, sort_keys=True, indent=1)) for res2 in results[res]:
Signature · 1 line
2.34.1
PATCH 14/15 resulttool: Trim the precision of duration information
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
The duration values have pointless amounts of precision. Removing some of the least significant digits reduces result size and makes the results easier to read.
--- scripts/lib/resulttool/resultutils.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
scripts/lib/resulttool/resultutils.py +17 -0
@@ -131,6 +131,22 @@ def strip_logs(results): del newresults[res]['result']['ptestresult.sections'][i]['log'] return newresults +# For timing numbers, crazy amounts of precision don't make sense and just confuse +# the logs. For numbers over 1, trim to 3 decimal places, for numbers less than 1, +# trim to 4 significant digits +def trim_durations(results): + for res in results: + if 'result' not in results[res]: + continue + for entry in results[res]['result']: + if 'duration' in results[res]['result'][entry]: + duration = results[res]['result'][entry]['duration'] + if duration > 1: + results[res]['result'][entry]['duration'] = float("%.3f" % duration) + elif duration < 1: + results[res]['result'][entry]['duration'] = float("%.4g" % duration) + return results + def handle_cleanups(results): # Remove pointless path duplication from old format reproducibility results for res2 in results:
@@ -194,6 +210,7 @@ def save_resultsdata(results, destdir, fn="testresults.json", ptestjson=False, p resultsout = results[res] if not ptestjson: resultsout = strip_logs(results[res]) + trim_durations(resultsout) handle_cleanups(resultsout) with open(dst, 'w') as f: f.write(json.dumps(resultsout, sort_keys=True, indent=1))
Signature · 1 line
2.34.1
PATCH 15/15 resulttool: Improve repo layout for oeselftest results
In reply to Steve SakomanFrom: Richard Purdie <[email protected]>
Having all oe-selftest results on top of each other results in a large 640MB json file which is hard to use. Split the results out per machine and test type.
This also stops the toolchain raw logs from overwriting each other meaning more than one MACHINE is preserved.
--- scripts/lib/resulttool/resultutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/resulttool/resultutils.py +1 -1
@@ -34,7 +34,7 @@ regression_map = { "manual": ['TEST_TYPE', 'TEST_MODULE', 'IMAGE_BASENAME', 'MACHINE'] } store_map = { - "oeselftest": ['TEST_TYPE'], + "oeselftest": ['TEST_TYPE', 'TESTSERIES', 'MACHINE'], "runtime": ['TEST_TYPE', 'DISTRO', 'MACHINE', 'IMAGE_BASENAME'], "sdk": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'], "sdkext": ['TEST_TYPE', 'MACHINE', 'SDKMACHINE', 'IMAGE_BASENAME'],
Signature · 1 line
2.34.1