travis-ci: include the trash directories of failed tests in the trace log
PATCH 1/1 travis-ci: include the trash directories of failed tests in the trace log
The trash directory of a failed test might contain invaluable information about the cause of the failure, but we have no access to the trash directories of Travis CI build jobs. The only feedback we get from there is the build job's trace log, so...
Modify 'ci/print-test-failures.sh' to create a tar.gz archive of the trash directory of each failed test, encode that archive with base64, and print the resulting block of ASCII text, so it gets embedded in the trace log. Furthermore, run tests with '--immediate' to faithfully preserve the failed state.
Extracting the trash directories from the trace log turned out to be a bit of a hassle, partly because of the size of these logs (usually resulting in several hundreds or even thousands of lines of base64-encoded text), and partly because these logs have CRLF, CRCRLF and occasionally even CRCRCRLF line endings, which cause 'base64 -d' from coreutils to complain about "invalid input". For convenience add a small script 'ci/util/extract-trash-dirs.sh', which will extract and unpack all base64-encoded trash directories embedded in the log fed to its standard input, and include an example command to be copy-pasted into a terminal to do it all at the end of the failure report.
A few of our tests create sizeable trash directories, so limit the size of each included base64-encoded block, let's say, to 1MB. And just in case something fundamental gets broken and a lot of tests fail at once, don't include trash directories when the combined size of the included base64-encoded blocks would exceed 1MB.
---
Notes:
This is an improved version of the PoC mentioned some months ago at:
https://public-inbox.org/git/[email protected]/
I'm still not sure whether this is too clever or too ugly, or both, but it did actually prove to be useful since then: it's very likely that we wouldn't have 2f3cbcd8c5 (tests: make forging GPG signed commits and tags more robust, 2018-06-04) without it.
The output looks like this; with this patch on top of yesterday's 'pu' (6f49f36eba), which happens to have multiple test failures:
https://travis-ci.org/szeder/git/jobs/410471312#L3010
(Note: the Travis CI webpage will likely fold up the failure output before you could take a closer look at it; then click on the line containing '$ ci/print-test-failures.sh' and scroll down for a while.)
ci/lib-travisci.sh | 2 +- ci/print-test-failures.sh | 55 +++++++++++++++++++++++++++++++++-- ci/util/extract-trash-dirs.sh | 50 +++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+), 3 deletions(-) create mode 100755 ci/util/extract-trash-dirs.sh
ci/lib-travisci.sh +1 -1
@@ -97,7 +97,7 @@ fi export DEVELOPER=1 export DEFAULT_TEST_TARGET=prove export GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save" -export GIT_TEST_OPTS="--verbose-log -x" +export GIT_TEST_OPTS="--verbose-log -x --immediate" export GIT_TEST_CLONE_2GB=YesPlease if [ "$jobname" = linux-gcc ]; then export CC=gcc-8
ci/print-test-failures.sh +53 -2
71 lines not on this page · raw message
ci/util/extract-trash-dirs.sh new file +50 -0
50 lines not on this page · raw message
Signature · 1 line
2.18.0.408.g42635c01bc
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Lars Schneider
Quoted 7 lines · SZEDER Gábor
The trash directory of a failed test might contain invaluable information about the cause of the failure, but we have no access to the trash directories of Travis CI build jobs. The only feedback we get from there is the build job's trace log, so...
100% agree that keeping the trash directory is valuable.
Quoted 7 lines
Quoted 5 lines
Modify '
ci/print-test-failures.sh' to create a tar.gz archive of the trash directory of each failed test, encode that archive with base64, and print the resulting block of ASCII text, so it gets embedded in the trace log. Furthermore, run tests with '--immediate' to faithfully preserve the failed state.I feel that might be complicated to work with.
It works quite conveniently in practice, that was one of the points of the next paragraph.
Quoted 2 lines
How about this:
We create an publicly available Git repo (e.g. github.com/git/git-test-trash`)
I'm not sure I understand. A publicly _writable_ repository? That sounds like asking for trouble.
Quoted 5 lines
and then we push the trash directory of every failed test run to this repo. Then we link the commit in the logs to help people find them.
Wouldn't that make it easier to access the trash dir?
With this patch, if there are any trash directories embedded in the log, then the log will show a command near the very end that you can copy-paste into a terminal, and bam! in less than 3 seconds you have all those trash directories downloaded and unpacked.
I'm not sure how it could be made any easier than that.
Quoted 19 lines
Quoted 19 lines
Extracting the trash directories from the trace log turned out to be a bit of a hassle, partly because of the size of these logs (usually resulting in several hundreds or even thousands of lines of base64-encoded text), and partly because these logs have CRLF, CRCRLF and occasionally even CRCRCRLF line endings, which cause 'base64 -d' from coreutils to complain about "invalid input". For convenience add a small script '
ci/util/extract-trash-dirs.sh', which will extract and unpack all base64-encoded trash directories embedded in the log fed to its standard input, and include an example command to be copy-pasted into a terminal to do it all at the end of the failure report.A few of our tests create sizeable trash directories, so limit the size of each included base64-encoded block, let's say, to 1MB. And just in case something fundamental gets broken and a lot of tests fail at once, don't include trash directories when the combined size of the included base64-encoded blocks would exceed 1MB.
Signed-off-by SZEDER Gábor <[email protected]>---