Skip to contentSkip to conversation

Acceptance Tests: improve usage on GitLab CI

PATCH In review Cleber Rosa 5 participants 12 messages Oct 9, 2020 – Oct 19, 2020

lore
Share & tools
lore
#0 lore raw

cover Acceptance Tests: improve usage on GitLab CI

This attempts to improve a bit the execution of acceptance tests in both GitLab CI, and in restricted Python environments.

For the GitLab's rendering of test results, a sample can be seen here:

https://gitlab.com/cleber.gnu/qemu/-/pipelines/200639030/test_report

Cleber Rosa (3):
Acceptance tests: bump pycdlib version for easier installation Acceptance tests: do not show canceled test logs on GitLab CI Acceptance tests: show test report on GitLab CI

 .gitlab-ci.yml         | 7 ++++++-
 tests/requirements.txt | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.4
#1 lore raw

PATCH 1/3 Acceptance tests: bump pycdlib version for easier installation

On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.

pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.

---
 tests/requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
index 036691c922..a1c631fa59 100644
@@ -2,4 +2,4 @@
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 avocado-framework==81.0
-pycdlib==1.9.0
+pycdlib==1.11.0
Signature · 1 line
2.25.4
Reviewed-by Tested-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation

Quoted 11 lines · Cleber Rosa

On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.

pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.

This BZ is different. The correct URL is:
https://bugs.launchpad.net/qemu/+bug/1897783

Quoted 17 lines
---
  tests/requirements.txt | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
index 036691c922..a1c631fa59 100644
@@ -2,4 +2,4 @@
  # in the tests/venv Python virtual environment. For more info,
  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
  avocado-framework==81.0
-pycdlib==1.9.0
+pycdlib==1.11.0
#3 lore raw

Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation

Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Cleber Rosa

On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.

pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.

Quoted 6 lines
Quoted 2 lines

This BZ is different. The correct URL is:
https://bugs.launchpad.net/qemu/+bug/1897783

Good catch, thanks!

- Cleber.

Reviewed-by on v1 1/3
#4 lore raw

Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation

In reply to Cleber Rosa
Quoted 29 lines · Cleber Rosa

On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.

pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.

---
 tests/requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -2
index 036691c922..a1c631fa59 100644
@@ -2,4 +2,4 @@
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 avocado-framework==81.0
-pycdlib==1.9.0
+pycdlib==1.11.0
--
Quoted 2 lines

2.25.4

#5 lore raw

PATCH 2/3 Acceptance tests: do not show canceled test logs on GitLab CI

In reply to Cleber Rosa

Tests resulting in "CANCEL" in Avocado are usually canceled on purpose, and are almost identical to "SKIP". The logs for canceled tests are adding a lot of noise to the logs being shown on GitLab CI, and causing distraction from real failures.

As a side note, this "after script" is scheduled for removal once the feature is implemented within Avocado itself.

Reference: https://github.com/avocado-framework/avocado/issues/4266

---
 .gitlab-ci.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
index a51c89554f..bed5fe6161 100644
@@ -63,7 +63,7 @@ include:
       fi
   after_script:
     - cd build
-    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
     - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
 build-system-ubuntu:
Signature · 1 line
2.25.4
Reviewed-by Tested-by on v1 2/3
#6 lore raw

Re: [PATCH 2/3] Acceptance tests: do not show canceled test logs on GitLab CI

Quoted 4 lines · Cleber Rosa

Tests resulting in "CANCEL" in Avocado are usually canceled on purpose, and are almost identical to "SKIP". The logs for canceled tests are adding a lot of noise to the logs being shown on GitLab CI, and causing distraction from real failures.

Thanks, big improvement!!!

Quoted 24 lines

As a side note, this "after script" is scheduled for removal once the feature is implemented within Avocado itself.

Reference: https://github.com/avocado-framework/avocado/issues/4266

---
  .gitlab-ci.yml | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
index a51c89554f..bed5fe6161 100644
@@ -63,7 +63,7 @@ include:
        fi
    after_script:
      - cd build
-    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
+    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
      - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
  build-system-ubuntu:
#7 lore raw

PATCH 3/3 Acceptance tests: show test report on GitLab CI

In reply to Cleber Rosa

Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.

---
 .gitlab-ci.yml | 5 +++++
 1 file changed, 5 insertions(+)
.gitlab-ci.yml +5 -0
index bed5fe6161..7d4b2ced9e 100644
@@ -53,6 +53,11 @@ include:
     paths:
       - ${CI_PROJECT_DIR}/avocado-cache
     policy: pull-push
+  artifacts:
+    paths:
+      - build/tests/results/latest/results.xml
+    reports:
+      junit: build/tests/results/latest/results.xml
   before_script:
     - mkdir -p ~/.config/avocado
     - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
Signature · 1 line
2.25.4
Reviewed-by Tested-by on v1 3/3
#8 lore raw

Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI

Quoted 2 lines · Cleber Rosa

Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.

https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report

Quoted 23 lines
---
  .gitlab-ci.yml | 5 +++++
  1 file changed, 5 insertions(+)
.gitlab-ci.yml +5 -0
index bed5fe6161..7d4b2ced9e 100644
@@ -53,6 +53,11 @@ include:
      paths:
        - ${CI_PROJECT_DIR}/avocado-cache
      policy: pull-push
+  artifacts:
+    paths:
+      - build/tests/results/latest/results.xml
+    reports:
+      junit: build/tests/results/latest/results.xml
    before_script:
      - mkdir -p ~/.config/avocado
      - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
Reviewed-by
#9 lore raw

Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI

Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 2 lines · Cleber Rosa

Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.

https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report

That's awesome - I wonder if the tap reports can also be read?

Signature · 1 line
Alex Bennée
#10 lore raw

Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI

Quoted 7 lines · Alex Bennée
Quoted 4 lines · Philippe Mathieu-Daudé
Quoted 2 lines · Cleber Rosa

Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.

Quoted 7 lines
Quoted 4 lines

https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report

That's awesome - I wonder if the tap reports can also be read?

Not according to:

https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreports

But, I'm working on further integrating Avocado to other types of tests in QEMU, so users can choose to run test jobs with any combination of test types.

A very rough PoC with some unit tests (that generate TAP) being run by Avocado, and the JUnit report being displayed, can be seen here:

https://gitlab.com/cleber.gnu/qemu/-/pipelines/201609047/test_report

The grand vision is, among other things, to be able run tests related to a given subsystem or feature, say "migration", no matter what type of tests they are.

Cheers,
- Cleber.

Quoted 5 lines
Signature · 1 line
Alex Bennée
#11 lore raw
In reply to Cleber Rosa
Quoted 7 lines · Cleber Rosa

This attempts to improve a bit the execution of acceptance tests in both GitLab CI, and in restricted Python environments.

For the GitLab's rendering of test results, a sample can be seen here:

https://gitlab.com/cleber.gnu/qemu/-/pipelines/200639030/test_report

Quoted 9 lines

Cleber Rosa (3):
Acceptance tests: bump pycdlib version for easier installation Acceptance tests: do not show canceled test logs on GitLab CI Acceptance tests: show test report on GitLab CI

 .gitlab-ci.yml         | 7 ++++++-
 tests/requirements.txt | 2 +-
 2 files changed, 7 insertions(+), 2 deletions(-)

Thanks, queued (with the bug URL fixed).

Thomas