Acceptance Tests: improve usage on GitLab CI
cover Acceptance Tests: improve usage on GitLab CI
This attempts to improve a bit the execution of acceptance tests in both GitLab CI, and in restricted Python environments.
For the GitLab's rendering of test results, a sample can be seen here:
https://gitlab.com/cleber.gnu/qemu/-/pipelines/200639030/test_report
Cleber Rosa (3):
Acceptance tests: bump pycdlib version for easier installation Acceptance tests: do not show canceled test logs on GitLab CI Acceptance tests: show test report on GitLab CI
.gitlab-ci.yml | 7 ++++++- tests/requirements.txt | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.4
PATCH 1/3 Acceptance tests: bump pycdlib version for easier installation
On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.
pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.
--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/requirements.txt +1 -1
@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==81.0 -pycdlib==1.9.0 +pycdlib==1.11.0
Signature · 1 line
2.25.4
Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation
Quoted 11 lines · Cleber Rosa
On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.
pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.
Cc Bug 1897783 <[email protected]>
This BZ is different. The correct URL is:
https://bugs.launchpad.net/qemu/+bug/1897783
Quoted 17 lines
Reported-by Philippe Mathieu-Daudé <[email protected]>Signed-off-by Cleber Rosa <[email protected]>--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/requirements.txt +1
-1@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==81.0 -pycdlib==1.9.0 +pycdlib==1.11.0
Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation
Quoted 10 lines · Philippe Mathieu-Daudé
Quoted 9 lines · Cleber Rosa
On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.
pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.
Quoted 6 lines
Quoted 2 lines
Cc Bug 1897783 <[email protected]>This BZ is different. The correct URL is:
https://bugs.launchpad.net/qemu/+bug/1897783
Good catch, thanks!
- Cleber.
Re: [PATCH 1/3] Acceptance tests: bump pycdlib version for easier installation
In reply to Cleber RosaQuoted 29 lines · Cleber Rosa
On with certain versions of "pip", package installations will attempt to create wheels. And, on environments without a "complete" Python installation (as described in the acceptance tests requirements docs), that will fail.
pycdlib, starting with version 1.11.0, is now being made available as wheels, so its instalation on those constrained environments is now possible.
Cc Bug 1897783 <[email protected]>Reported-by Philippe Mathieu-Daudé <[email protected]>Signed-off-by Cleber Rosa <[email protected]>--- tests/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/requirements.txt +1
-2@@ -2,4 +2,4 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 avocado-framework==81.0 -pycdlib==1.9.0 +pycdlib==1.11.0 --
Quoted 2 lines
2.25.4
PATCH 2/3 Acceptance tests: do not show canceled test logs on GitLab CI
In reply to Cleber RosaTests resulting in "CANCEL" in Avocado are usually canceled on purpose, and are almost identical to "SKIP". The logs for canceled tests are adding a lot of noise to the logs being shown on GitLab CI, and causing distraction from real failures.
As a side note, this "after script" is scheduled for removal once the feature is implemented within Avocado itself.
Reference: https://github.com/avocado-framework/avocado/issues/4266
--- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
.gitlab-ci.yml +1 -1
@@ -63,7 +63,7 @@ include: fi after_script: - cd build - - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat + - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat - du -chs ${CI_PROJECT_DIR}/avocado-cache build-system-ubuntu:
Signature · 1 line
2.25.4
Re: [PATCH 2/3] Acceptance tests: do not show canceled test logs on GitLab CI
Quoted 4 lines · Cleber Rosa
Tests resulting in "CANCEL" in Avocado are usually canceled on purpose, and are almost identical to "SKIP". The logs for canceled tests are adding a lot of noise to the logs being shown on GitLab CI, and causing distraction from real failures.
Thanks, big improvement!!!
Quoted 24 lines
As a side note, this "after script" is scheduled for removal once the feature is implemented within Avocado itself.
Reference: https://github.com/avocado-framework/avocado/issues/4266
Signed-off-by Cleber Rosa <[email protected]>--- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-).gitlab-ci.yml +1
-1@@ -63,7 +63,7 @@ include: fi after_script: - cd build - - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat + - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat - du -chs ${CI_PROJECT_DIR}/avocado-cache build-system-ubuntu:
PATCH 3/3 Acceptance tests: show test report on GitLab CI
In reply to Cleber RosaAvocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.
--- .gitlab-ci.yml | 5 +++++ 1 file changed, 5 insertions(+)
.gitlab-ci.yml +5 -0
@@ -53,6 +53,11 @@ include: paths: - ${CI_PROJECT_DIR}/avocado-cache policy: pull-push + artifacts: + paths: + - build/tests/results/latest/results.xml + reports: + junit: build/tests/results/latest/results.xml before_script: - mkdir -p ~/.config/avocado - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
Signature · 1 line
2.25.4
Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI
Quoted 2 lines · Cleber Rosa
Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.
https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report
Quoted 23 lines
Signed-off-by Cleber Rosa <[email protected]>--- .gitlab-ci.yml | 5 +++++ 1 file changed, 5 insertions(+).gitlab-ci.yml +5
-0@@ -53,6 +53,11 @@ include: paths: - ${CI_PROJECT_DIR}/avocado-cache policy: pull-push + artifacts: + paths: + - build/tests/results/latest/results.xml + reports: + junit: build/tests/results/latest/results.xml before_script: - mkdir -p ~/.config/avocado - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
Alex Bennée
Reviewed-byRe: [PATCH 3/3] Acceptance tests: show test report on GitLab CI
Quoted 8 lines · Philippe Mathieu-Daudé
Quoted 2 lines · Cleber Rosa
Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.
https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report
Reviewed-by Philippe Mathieu-Daudé <[email protected]>
That's awesome - I wonder if the tap reports can also be read?
Signature · 1 line
Alex Bennée
Re: [PATCH 3/3] Acceptance tests: show test report on GitLab CI
Quoted 7 lines · Alex Bennée
Quoted 4 lines · Philippe Mathieu-Daudé
Quoted 2 lines · Cleber Rosa
Avocado will, by default, produce JUnit files. Let's ask GitLab to present those in the web UI.
Quoted 7 lines
Quoted 4 lines
https://gitlab.com/philmd/qemu/-/pipelines/200764992/test_report
Reviewed-by Philippe Mathieu-Daudé <[email protected]>That's awesome - I wonder if the tap reports can also be read?
Not according to:
https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreports
But, I'm working on further integrating Avocado to other types of tests in QEMU, so users can choose to run test jobs with any combination of test types.
A very rough PoC with some unit tests (that generate TAP) being run by Avocado, and the JUnit report being displayed, can be seen here:
https://gitlab.com/cleber.gnu/qemu/-/pipelines/201609047/test_report
The grand vision is, among other things, to be able run tests related to a given subsystem or feature, say "migration", no matter what type of tests they are.
Cheers,
- Cleber.
Quoted 5 lines
Reviewed-by Alex Bennée <[email protected]>Signature · 1 line
Alex Bennée
Quoted 7 lines · Cleber Rosa
This attempts to improve a bit the execution of acceptance tests in both GitLab CI, and in restricted Python environments.
For the GitLab's rendering of test results, a sample can be seen here:
https://gitlab.com/cleber.gnu/qemu/-/pipelines/200639030/test_report
Quoted 9 lines
Cleber Rosa (3):
Acceptance tests: bump pycdlib version for easier installation Acceptance tests: do not show canceled test logs on GitLab CI Acceptance tests: show test report on GitLab CI.gitlab-ci.yml | 7 ++++++- tests/requirements.txt | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-)
Thanks, queued (with the bug URL fixed).
Thomas