Skip to contentSkip to conversation

travis: Improve log browseability

PATCH In review Wainer dos Santos Moschetta 3 participants 7 messages Dec 30, 2019 – Jan 9, 2020

lore
Share & tools
lore
#0 lore raw

cover travis: Improve log browseability

While I was working on [1] I needed to check what dependencies were missing on Travis environment to build a kernel. The test was coded so that a build failure would cancel the test. As a result, and because all tests pass'ed (or cancel'ed) I could not see the Avocado logs (consequentely not able to check the missing packages). That was the main reason for the Patch 01 in this series. Also it seems pointless to print the log of the tests that pass.

The patch 02 is a tentative to improve the browsing of build vs test logs. Honestly I was expecting that Travis was able to put the output of steps each on toggled text boxes (see the 'before_script' section, each command output has a
toggle). But that isn't the behavior of the 'script' section. At least it prints the end of the build command ('The command "BUILD_RC=0 && make -j3 || BUILD_RC=$?" exited with 0.') which can be grep'ed.

Git:
- Tree: https://github.com/wainersm/qemu
- Brach: travis_improved_browse

CI:
- Travis (FAIL): https://travis-ci.org/wainersm/qemu/builds/630948777
QEMU keep failing with --without-default-devices

References:
[1] https://www.mail-archive.com/[email protected]/msg663432.html

Wainer dos Santos Moschetta (2):
travis.yml: avocado: Print logs of non-pass tests only travis.yml: Detach build and test steps

 .travis.yml | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH 1/2 travis.yml: avocado: Print logs of non-pass tests only

The acceptance tests build on Travis is configured to print the entire Avocado's job log in case any test fail. Usually one is interested on failed tests only though. So this change the Travis configuration in order to show the log of tests which status is different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests will have the log printed too because it can help to debug some condition on CI environment which is not being fulfilled.

---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
.travis.yml +2 -2
index 376b7d6dfa..e7fdcb238c 100644
@@ -262,8 +262,8 @@ matrix:
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
         - TEST_CMD="make check-acceptance"
-      after_failure:
-        - cat tests/results/latest/job.log
+      after_script:
+        - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
       addons:
         apt:
           packages:
Signature · 1 line
2.23.0
Reviewed-by Tested-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

Quoted 7 lines · Wainer dos Santos Moschetta

The acceptance tests build on Travis is configured to print the entire Avocado's job log in case any test fail. Usually one is interested on failed tests only though. So this change the Travis configuration in order to show the log of tests which status is different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests will have the log printed too because it can help to debug some condition on CI environment which is not being fulfilled.

Nice!

Tested with a broken test:
https://travis-ci.org/philmd/qemu/jobs/631341382

Quoted 21 lines
---
  .travis.yml | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
.travis.yml +2 -2
index 376b7d6dfa..e7fdcb238c 100644
@@ -262,8 +262,8 @@ matrix:
      - env:
          - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
          - TEST_CMD="make check-acceptance"
-      after_failure:
-        - cat tests/results/latest/job.log
+      after_script:
+        - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
        addons:
          apt:
            packages:
#3 lore raw

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

In reply to Wainer dos Santos Moschetta
Quoted 26 lines · Wainer dos Santos Moschetta

The acceptance tests build on Travis is configured to print the entire Avocado's job log in case any test fail. Usually one is interested on failed tests only though. So this change the Travis configuration in order to show the log of tests which status is different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests will have the log printed too because it can help to debug some condition on CI environment which is not being fulfilled.

---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
.travis.yml +2 -2
index 376b7d6dfa..e7fdcb238c 100644
@@ -262,8 +262,8 @@ matrix:
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
         - TEST_CMD="make check-acceptance"
-      after_failure:
-        - cat tests/results/latest/job.log
+      after_script:
+        - python3 -c 'import json; r =
 json.load(open("tests/results/latest/results.json"));
Quoted 2 lines

[print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

Shame there is no scriptlet for this sort of thing packaged with avocado. Anyway I'll give it a spin and see if any failures come up.

Quoted 3 lines

addons:
apt:
packages:

Signature · 1 line
Alex Bennée
#4 lore raw

Re: [PATCH 1/2] travis.yml: avocado: Print logs of non-pass tests only

Quoted 24 lines · Alex Bennée
Quoted 22 lines · Wainer dos Santos Moschetta

The acceptance tests build on Travis is configured to print the entire Avocado's job log in case any test fail. Usually one is interested on failed tests only though. So this change the Travis configuration in order to show the log of tests which status is different from 'PASS' and 'SKIP' only. Note that 'CANCEL'-ed tests will have the log printed too because it can help to debug some condition on CI environment which is not being fulfilled.

---
  .travis.yml | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
.travis.yml +0 -1
index 376b7d6dfa..e7fdcb238c 100644
@@ -262,8 +262,8 @@ matrix:
      - env:
          - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu,ppc-softmmu,ppc64-softmmu,m68k-softmmu,sparc-softmmu"
          - TEST_CMD="make check-acceptance"
-      after_failure:
Quoted 8 lines
Quoted 6 lines
.travis.yml +2 -1
-        - cat tests/results/latest/job.log
+      after_script:
+        - python3 -c 'import json; r =
 json.load(open("tests/results/latest/results.json"));

[print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat

Shame there is no scriptlet for this sort of thing packaged with avocado. Anyway I'll give it a spin and see if any failures come up.

Good point. I'm working on such as scriptlet ('plugin' on Avocado jargon) but I think we can stick with this change.

Thanks!

- Wainer

Quoted 5 lines
Quoted 4 lines

addons:
apt:
packages:

#5 lore raw

PATCH 2/2 travis.yml: Detach build and test steps

In reply to Wainer dos Santos Moschetta

Currently build and test commands are a single step in a Travis's script block. In order to see the output of the tests one needs to scroll down the log to find where the build messages ended and the limit is not clear. If they were in different steps then Travis would print the result build command, which can be easily grep'ed.

So this change is made to detach those commands
to ease the visualization of the output.

Note that all steps on the script block is executed regardless if one previous has failed. To overcome it, let's save the return code of the build then check whether succeed or failed on the test step.

---
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
.travis.yml +2 -1
index e7fdcb238c..fcc9de368b 100644
@@ -94,7 +94,8 @@ before_script:
   - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
   - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
 script:
-  - make -j3 && travis_retry ${TEST_CMD}
+  - BUILD_RC=0 && make -j3 || BUILD_RC=$?
+  - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
 after_script:
   - if command -v ccache ; then ccache --show-stats ; fi
 
Signature · 1 line
2.23.0
Reviewed-by Tested-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] travis.yml: Detach build and test steps

Quoted 7 lines · Wainer dos Santos Moschetta

Currently build and test commands are a single step in a Travis's script block. In order to see the output of the tests one needs to scroll down the log to find where the build messages ended and the limit is not clear. If they were in different steps then Travis would print the result build command, which can be easily grep'ed.

Quoted 7 lines

So this change is made to detach those commands
to ease the visualization of the output.

Note that all steps on the script block is executed regardless if one previous has failed. To overcome it, let's save the return code of the build then check whether succeed or failed on the test step.

Tested with a broken test:
https://travis-ci.org/philmd/qemu/jobs/631341382

Quoted 21 lines
---
  .travis.yml | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
.travis.yml +2 -1
index e7fdcb238c..fcc9de368b 100644
@@ -94,7 +94,8 @@ before_script:
    - mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR}
    - ${SRC_DIR}/configure ${BASE_CONFIG} ${CONFIG} || { cat config.log && exit 1; }
  script:
-  - make -j3 && travis_retry ${TEST_CMD}
+  - BUILD_RC=0 && make -j3 || BUILD_RC=$?
+  - if [ "$BUILD_RC" -eq 0 ] ; then travis_retry ${TEST_CMD} ; else $(exit $BUILD_RC); fi
  after_script:
    - if command -v ccache ; then ccache --show-stats ; fi