classes: add workdir_save class
cover classes: add workdir_save class
This is fairly trivial but I've found it useful in my setup - I'd like to know if others would as well. Improvement suggestions welcome.
Please review the following changes for suitability for inclusion. If you have any objections or suggestions for improvement, please respond to the patches. If you agree with the changes, please provide your Acked-by.
The following changes since commit b3c96103a5063eeefb0c537227eab3f77616b9c0:
libnl: Extend for native/nativesdk (2020-07-08 10:56:11 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/workdir_save
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/workdir_save
Paul Eggleton (1):
classes: add workdir_save class
meta/classes/workdir_save.bbclass | 54 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 meta/classes/workdir_save.bbclass
Signature · 1 line
1.8.3.1
RFC PATCH 1/1 classes: add workdir_save class
If you are running your builds inside an environment where you don't have access to the build tree (e.g. an autobuilder where you can only download final artifacts such as images), then debugging build failures can be difficult - you can't examine log files, the source tree or output files. When enabled, this class triggers on task failure and saves a tarball of the work directory for the task's recipe and puts it in a configurable location, where it can be picked up by a separate process and made available as a downloadable artifact.
--- meta/classes/workdir_save.bbclass | 54 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 meta/classes/workdir_save.bbclass
meta/classes/workdir_save.bbclass new file +54 -0
@@ -0,0 +1,54 @@ +# Save the work directory for a recipe when one of its tasks fails. +# Useful in cases where the environment in which builds are run is +# ephemeral or otherwise inaccessible for examination during +# debugging. +# +# To enable, simply add the following to your configuration: +# +# INHERIT += "workdir_save" +# +# Notes: +# * For this to work you also need corresponding logic in your build +# orchestration tool to pick up any files written out to WORKDIR_SAVE_DIR +# (with the other assumption being that no files are present there at +# the start of the build). +# * Work directories can be quite large, so saving them can take some time +# and of course space. +# +# Copyright (c) 2020 Microsoft Corporation +# +# SPDX-License-Identifier: GPL-2.0-only +# + +WORKDIR_SAVE_DIR ?= "${TMPDIR}/workdir_save" +WORKDIR_SAVE_ENABLED ?= "1" + +addhandler workdir_save_handler +workdir_save_handler[eventmask] = "bb.build.TaskFailed" + +python workdir_save_handler() { + import datetime + if d.getVar('WORKDIR_SAVE_ENABLED') != '1': + return + + base_workdir = d.getVar('BASE_WORKDIR') + workdir = d.getVar('WORKDIR') + outdir = d.getVar('WORKDIR_SAVE_DIR') + bb.utils.mkdirhier(outdir) + pn = d.getVar('PN') + tstamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S") + tarfn = 'workdir_%s_%s.tar.gz' % (pn, tstamp) + tarfp = os.path.join(outdir, tarfn) + taskname = d.getVar('BB_CURRENTTASK') + bb.plain('NOTE: Saving workdir for failed task %s.do_%s to %s' % (pn, taskname, tarfp)) + try: + bb.process.run(['tar', 'czf', tarfp, + os.path.relpath(workdir, base_workdir)], cwd=base_workdir) + except bb.process.ExecutionError as e: + # It is possible for other tasks to be writing to the workdir + # while we are tarring it up, in which case tar will return 1, + # but we don't care in this situation (tar returns 2 for other + # errors so we we will see those) + if e.exitcode != 1: + bb.warn('workdir save error: %s' % str(e)) +}
Signature · 1 line
1.8.3.1
Quoted 12 lines · Paul Eggleton
If you are running your builds inside an environment where you don't have access to the build tree (e.g. an autobuilder where you can only download final artifacts such as images), then debugging build failures
can be difficult - you can't examine log files, the source tree or output
files. When enabled, this class triggers on task failure and saves a tarball of the work directory for the task's recipe and puts it in a configurable location, where it can be picked up by a separate process
and made available as a downloadable artifact.
Quoted 1 line
Signed-off-by Paul Eggleton <[email protected]>
Seems like a useful idea. I think the name may need "failed" in the name to make it clearer what it does. We also need a section for the classes chapter in the manual.
As another data point, OEQA is using OEQA_DEBUGGING_SAVED_OUTPUT to trigger saving information for reproducible builds...
Cheers,
Richard
Quoted 27 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Richard Purdie
Sent: den 8 juli 2020 20:12
To: Paul Eggleton <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [RFC PATCH 1/1] classes: addworkdir_saveclassQuoted 12 lines · Paul Eggleton
If you are running your builds inside an environment where you don't have access to the build tree (e.g. an autobuilder where you can only download final artifacts such as images), then debugging build failures
can be difficult - you can't examine log files, the source tree or output
files. When enabled, this class triggers on task failure and saves a tarball of the work directory for the task's recipe and puts it in a configurable location, where it can be picked up by a separate process
and made available as a downloadable artifact.Quoted 1 line
Signed-off-by Paul Eggleton <[email protected]>Seems like a useful idea. I think the name may need "failed" in the name to make it clearer what it does. We also need a section for the classes chapter in the manual.
Quoted 6 lines
As another data point, OEQA is using
OEQA_DEBUGGING_SAVED_OUTPUTto trigger saving information for reproducible builds...Cheers,
Richard
How about "save_workdir_on_failure.bbclass"?
//Peter
Quoted 9 lines · Peter Kjellerstedt
Quoted 7 lines · Richard Purdie
Seems like a useful idea. I think the name may need "failed" in the name to make it clearer what it does. We also need a section for the classes chapter in the manual.
As another data point, OEQA is using
OEQA_DEBUGGING_SAVED_OUTPUTto trigger saving information for reproducible builds...How about "save_workdir_on_failure.bbclass"?
I'd like to keep the name short if possible, it'll end up in the variable names as well. Thinking ahead we might want to save other information in future beyond the workdir, so we could possibly drop that from the name. What about "failure_save" or "failure_saver"? Also, with underscores or without?
Paul