classes: add new retain class for retaining build results
cover classes: add new retain class for retaining build results
Add a fairly simple class to tar up the work directory on task failure to a configurable location - useful when the build environment is ephemeral (e.g. an autobuilder/build pipeline). The directories that are retained can be configured, and you can also choose to save directories on every build as well if desired.
[This is a significant reworking of a prototype I sent as an RFC back in 2020 (AKA "workdir_save"). Changes since then include deferring tarball creation until the end, more config flexibility, and the addition of test cases.]
The following changes since commit 662396533177b72cc1d83e95841b27f7e42dcb20:
create-spdx-3.0/populate_sdk_base: Add SDK_CLASSES inherit mechanism to fix tarball SPDX manifests (2024-07-26 12:26:24 +0100)
are available in the git repository at:
https://git.yoctoproject.org/poky-contrib paule/retain
https://git.yoctoproject.org/poky-contrib/log/?h=paule/retain
(should have been openembedded-core-contrib I know, but for some reason I wasn't able to push there.)
Paul Eggleton (1):
classes: add new retain class for retaining build results
meta/classes-global/retain.bbclass | 162 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/retain.py | 219 +++++++++++++++++++++++++++++++++ 2 files changed, 381 insertions(+) create mode 100644 meta/classes-global/retain.bbclass create mode 100644 meta/lib/oeqa/selftest/cases/retain.py
Signature · 1 line
1.8.3.1
PATCH 1/1 classes: add new retain class for retaining build results
From: Paul Eggleton <[email protected]>
If you are running your builds inside an environment where you don't have access to the build tree (e.g. an autobuilder where you can only download final artifacts such as images), then debugging build failures can be difficult - you can't examine log files, the source tree or output files. When enabled, by default this class will retain the work directory for any recipe that has a task failure in the form of a tarball, and can also be configured to save other directories on failure or always.
It puts these tarballs in a configurable location (${TMPDIR}/retained by default), where they can be picked up by a separate process and made available as downloadable artifacts.
--- meta/classes-global/retain.bbclass | 162 ++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/retain.py | 219 +++++++++++++++++++++++++++++++++ 2 files changed, 381 insertions(+) create mode 100644 meta/classes-global/retain.bbclass create mode 100644 meta/lib/oeqa/selftest/cases/retain.py
meta/classes-global/retain.bbclass new file +162 -0
@@ -0,0 +1,162 @@ +# Creates a tarball of the work directory for a recipe when one of its +# tasks fails, or any other nominated directories. +# Useful in cases where the environment in which builds are run is +# ephemeral or otherwise inaccessible for examination during +# debugging. +# +# To enable, simply add the following to your configuration: +# +# INHERIT += "retain" +# +# You can specify the recipe-specific directories to save upon failure +# or always (space-separated) e.g.: +# +# RETAIN_DIRS_FAILURE = "${WORKDIR}" # default +# RETAIN_DIRS_ALWAYS = "${T}" +# +# Naturally you can use overrides to limit it to a specific recipe: +# RETAIN_DIRS_ALWAYS:pn-somerecipe = "${T}" +# +# You can also specify global (non-recipe-specific) directories to save: +# +# RETAIN_DIRS_GLOBAL_FAILURE = "${LOG_DIR}" +# RETAIN_DIRS_GLOBAL_ALWAYS = "${BUILDSTATS_BASE}" +# +# If you wish to use a different tarball name prefix you can do so by +# adding a : followed by the desired prefix (no spaces) in any of the +# RETAIN_DIRS variables. e.g. to use "buildlogs" as the prefix for the +# tarball of ${T} you would do this: +# +# RETAIN_DIRS_FAILURE = "${T}:buildlogs" +# +# Notes: +# * For this to be useful you also need corresponding logic in your build +# orchestration tool to pick up any files written out to RETAIN_OUTDIR +# (with the other assumption being that no files are present there at +# the start of the build, since there is no logic to purge old files). +# * Work directories can be quite large, so saving them can take some time +# and of course space. +# * Tarball creation is deferred to the end of the build, thus you will +# get the state at the end, not immediately upon failure. +# * Extra directories must naturally be populated at the time the retain +# class goes to save them (build completion); to try ensure this for +# things that are also saved on build completion (e.g. buildstats), put +# the INHERIT += "retain" after the INHERIT += lines for the class that +# is writing out the data that you wish to save. +# * The tarballs have the tarball name as a top-level directory so that +# multiple tarballs can be extracted side-by-side easily. +# +# Copyright (c) 2020, 2024 Microsoft Corporation +# +# SPDX-License-Identifier: GPL-2.0-only +# + +RETAIN_OUTDIR ?= "${TMPDIR}/retained" +RETAIN_DIRS_FAILURE ?= "${WORKDIR}" +RETAIN_DIRS_ALWAYS ?= "" +RETAIN_DIRS_GLOBAL_FAILURE ?= "" +RETAIN_DIRS_GLOBAL_ALWAYS ?= "" +RETAIN_TARBALL_SUFFIX ?= "${DATETIME}.tar.gz" +RETAIN_ENABLED ?= "1" + + +def retain_retain_dir(desc, tarprefix, path, tarbasepath, d): + import datetime + + outdir = d.getVar('RETAIN_OUTDIR') + bb.utils.mkdirhier(outdir) + suffix = d.getVar('RETAIN_TARBALL_SUFFIX') + tarname = '%s_%s' % (tarprefix, suffix) + tarfp = os.path.join(outdir, '%s' % tarname) + tardir = os.path.relpath(path, tarbasepath) + cmdargs = ['tar', 'cfa', tarfp] + # Prefix paths within the tarball with the tarball name so that + # multiple tarballs can be extracted side-by-side + cmdargs += ['--transform', 's:^:%s/:' % tarname] + cmdargs += [tardir] + try: + bb.process.run(cmdargs, cwd=tarbasepath) + except bb.process.ExecutionError as e: + # It is possible for other tasks to be writing to the workdir + # while we are tarring it up, in which case tar will return 1, + # but we don't care in this situation (tar returns 2 for other + # errors so we we will see those) + if e.exitcode != 1: + bb.warn('retain: error saving %s: %s' % (desc, str(e))) + + +addhandler retain_task_handler +retain_task_handler[eventmask] = "bb.build.TaskFailed bb.build.TaskSucceeded" + +addhandler retain_build_handler +retain_build_handler[eventmask] = "bb.event.BuildStarted bb.event.BuildCompleted" + +python retain_task_handler() { + if d.getVar('RETAIN_ENABLED') != '1': + return + + dirs = d.getVar('RETAIN_DIRS_ALWAYS') + if isinstance(e, bb.build.TaskFailed): + dirs += ' ' + d.getVar('RETAIN_DIRS_FAILURE') + + dirs = dirs.strip().split() + if dirs: + outdir = d.getVar('RETAIN_OUTDIR') + bb.utils.mkdirhier(outdir) + dirlist_file = os.path.join(outdir, 'retain_dirs.list') + pn = d.getVar('PN') + taskname = d.getVar('BB_CURRENTTASK') + with open(dirlist_file, 'a') as f: + for entry in dirs: + f.write('%s %s %s\n' % (pn, taskname, entry)) +} + +python retain_build_handler() { + outdir = d.getVar('RETAIN_OUTDIR') + dirlist_file = os.path.join(outdir, 'retain_dirs.list') + + if isinstance(e, bb.event.BuildStarted): + if os.path.exists(dirlist_file): + os.remove(dirlist_file) + return + + if d.getVar('RETAIN_ENABLED') != '1': + return + + savedirs = {} + try: + with open(dirlist_file, 'r') as f: + for line in f: + pn, _, path = line.rstrip().split() + if not path in savedirs: + savedirs[path] = pn + os.remove(dirlist_file) + except FileNotFoundError: + pass + + if e.getFailures(): + for path in (d.getVar('RETAIN_DIRS_GLOBAL_FAILURE') or '').strip().split(): + savedirs[path] = '' + + for path in (d.getVar('RETAIN_DIRS_GLOBAL_ALWAYS') or '').strip().split(): + savedirs[path] = '' + + if savedirs: + bb.plain('NOTE: retain: retaining build output...') + count = 0 + for path, pn in savedirs.items(): + if ':' in path: + path, itemname = path.rsplit(':', 1) + else: + itemname = os.path.basename(path) + if pn: + itemname = pn + '_' + itemname + if os.path.exists(path): + retain_retain_dir(itemname, itemname, path, os.path.dirname(path), d) + count += 1 + else: + bb.warn('retain: path %s does not currently exist' % path) + if count: + item = 'archive' if count == 1 else 'archives' + bb.plain('NOTE: retain: saved %d %s to %s' % (count, item, outdir)) +}
meta/lib/oeqa/selftest/cases/retain.py new file +219 -0
219 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Hi Paul,
Quoted 14 lines · Paul Eggleton via lists.openembedded.org
From: Paul Eggleton <[email protected]>
If you are running your builds inside an environment where you don't have access to the build tree (e.g. an autobuilder where you can only download final artifacts such as images), then debugging build failures
can be difficult - you can't examine log files, the source tree or output files. When enabled, by default this class will retain the work
directory for any recipe that has a task failure in the form of a tarball, and can also be configured to save other directories on failure
or always.
Quoted 6 lines
It puts these tarballs in a configurable location (${TMPDIR}/retained by
default), where they can be picked up by a separate process and made available as downloadable artifacts.Signed-off-by Paul Eggleton <[email protected]>
Thanks for the patch, looks like it could be useful for some contexts.
I ran it through the autobuilder and whilst one selftest passed there were failures too:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6961/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3662/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/7016/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/7026/steps/14/logs/stdio
Maybe everything after the first run failed so a build from sstate issue at a total guess?
Cheers,
Richard