Test more in less time in the Gitlab-CI
PATCH v2 2/4 tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab
In reply to Thomas HuthThese tests always time out on Gitlab, not sure what's happening here. Let's disable them until somebody has enough spare time to debug the issues.
--- tests/acceptance/machine_rx_gdbsim.py | 4 ++++ tests/acceptance/replay_kernel.py | 1 + 2 files changed, 5 insertions(+)
tests/acceptance/machine_rx_gdbsim.py +4 -0
@@ -8,6 +8,9 @@ # This work is licensed under the terms of the GNU GPL, version 2 or # later. See the COPYING file in the top-level directory. +import os + +from avocado import skipIf from avocado_qemu import Test from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern
@@ -42,6 +45,7 @@ class RxGdbSimMachine(Test): # FIXME limit baudrate on chardev, else we type too fast #exec_command_and_wait_for_pattern(self, 'version', gcc_version) + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_linux_sash(self): """ Boots a Linux kernel and checks that the console is operational.
tests/acceptance/replay_kernel.py +1 -0
@@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest): self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1) + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_arm_cubieboard_initrd(self): """ :avocado: tags=arch:arm
Signature · 1 line
2.18.1
PATCH v2 3/4 gitlab-ci.yml: Add build-system-debian and build-system-centos jobs
In reply to Thomas HuthWe were missing the two new targets avr-softmmu and rx-softmmu in the gitlab-CI so far, and did not add some of the "other endianess" targets like sh4eb-softmmu yet.
Since the current build-system-* jobs run already for a very long time, let's do not add these missing targets there, but introduce two new additional build jobs, one running with Debian and one running with CentOS, and add the new targets there. Also move some targets from the old build-system-* jobs to these new targets, to distribute the load and reduce the runtime of the CI.
--- .gitlab-ci.yml | 88 +++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 73 insertions(+), 15 deletions(-)
.gitlab-ci.yml +73 -15
@@ -53,68 +53,126 @@ include: - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat - du -chs $HOME/avocado/data/cache -build-system-ubuntu-main: +build-system-ubuntu: <<: *native_build_job_definition variables: IMAGE: ubuntu2004 - TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu - moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu - riscv64-softmmu sparc-softmmu + TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu + moxie-softmmu microblazeel-softmmu mips64el-softmmu MAKE_CHECK_ARGS: check-build artifacts: paths: - build -check-system-ubuntu-main: +check-system-ubuntu: <<: *native_test_job_definition needs: - - job: build-system-ubuntu-main + - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check -acceptance-system-ubuntu-main: +acceptance-system-ubuntu: <<: *native_test_job_definition needs: - - job: build-system-ubuntu-main + - job: build-system-ubuntu artifacts: true variables: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance <<: *post_acceptance -build-system-fedora-alt: +build-system-debian: + <<: *native_build_job_definition + variables: + IMAGE: debian-amd64 + TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu + riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu + MAKE_CHECK_ARGS: check-build + artifacts: + paths: + - build + +check-system-debian: + <<: *native_test_job_definition + needs: + - job: build-system-debian + artifacts: true + variables: + IMAGE: debian-amd64 + MAKE_CHECK_ARGS: check + +acceptance-system-debian: + <<: *native_test_job_definition + needs: + - job: build-system-debian + artifacts: true + variables: + IMAGE: debian-amd64 + MAKE_CHECK_ARGS: check-acceptance + <<: *post_acceptance + +build-system-fedora: <<: *native_build_job_definition variables: IMAGE: fedora TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu - riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu - xtensa-softmmu nios2-softmmu or1k-softmmu + xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu MAKE_CHECK_ARGS: check-build artifacts: paths: - build -check-system-fedora-alt: +check-system-fedora: <<: *native_test_job_definition needs: - - job: build-system-fedora-alt + - job: build-system-fedora artifacts: true variables: IMAGE: fedora MAKE_CHECK_ARGS: check -acceptance-system-fedora-alt: +acceptance-system-fedora: <<: *native_test_job_definition needs: - - job: build-system-fedora-alt + - job: build-system-fedora artifacts: true variables: IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance <<: *post_acceptance +build-system-centos: + <<: *native_build_job_definition + variables: + IMAGE: centos8 + TARGETS: ppc64-softmmu lm32-softmmu or1k-softmmu s390x-softmmu + x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu + MAKE_CHECK_ARGS: check-build + artifacts: + paths: + - build + +check-system-centos: + <<: *native_test_job_definition + needs: + - job: build-system-centos + artifacts: true + variables: + IMAGE: centos8 + MAKE_CHECK_ARGS: check + +acceptance-system-centos: + <<: *native_test_job_definition + needs: + - job: build-system-centos + artifacts: true + variables: + IMAGE: centos8 + MAKE_CHECK_ARGS: check-acceptance + <<: *post_acceptance + build-disabled: <<: *native_build_job_definition variables:
Signature · 1 line
2.18.1
v2 cover Test more in less time in the Gitlab-CI
This patch series adds two new "build-system" pipelines to the Gitlab-CI, one based on Debian and one on CentOS. We then use these build pipelines to test the targets that were missing so far (e.g. the two new targets rx-softmmu and avr-softmmu), and move some of the targets from the other build-system pipelines here, too, so that the total testing time gets shorter (at least 5 minutes from what I've seen so far).
Unfortunately, two of the avocado-based acceptance tests that would get enabled by this are not working on gitlab, so they are disabled in the second patch. It does not seem to be a caching issue (I've checked it with the fourth patch), maybe it's something related to ports not being available in the container, but I did not investigate further. They can be enabled back later again when someone has some more time for debugging.
v2:
- Added Reviewed-bys from v1
- Added fourth patch to fix the caching issue
Thomas Huth (4):
tests/docker: Add python3-venv and netcat to the debian-amd64
container
tests/acceptance: Disable the rx sash and arm cubieboard replay test
on Gitlab
gitlab-ci.yml: Add build-system-debian and build-system-centos jobs
gitlab-ci: Fix Avocado cache usage
.gitlab-ci.yml | 109 +++++++++++++++---- tests/acceptance/machine_rx_gdbsim.py | 4 + tests/acceptance/replay_kernel.py | 1 + tests/docker/dockerfiles/debian-amd64.docker | 4 +- 4 files changed, 98 insertions(+), 20 deletions(-)
Signature · 1 line
2.18.1
PATCH v2 4/4 gitlab-ci: Fix Avocado cache usage
In commit 6957fd98dc ("gitlab: add avocado asset caching") we tried to save the Avocado cache (as in commit c1073e44b4 with Travis-CI) however it doesn't work as expected. For some reason Avocado uses /root/avocado_cache/ which we can not select later.
Manually generate a Avocado config to force the use of the current job's directory.
This patch is based on an earlier version from Philippe Mathieu-Daudé.
--- .gitlab-ci.yml | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-)
.gitlab-ci.yml +19 -6
@@ -47,11 +47,24 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.post_acceptance_template: &post_acceptance +.acceptance_template: &acceptance_definition + cache: + key: "${CI_JOB_NAME}-cache" + paths: + - ${CI_PROJECT_DIR}/avocado-cache + policy: pull-push + before_script: + - mkdir -p ~/.config/avocado + - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf + - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']" + >> ~/.config/avocado/avocado.conf + - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then + du -chs ${CI_PROJECT_DIR}/avocado-cache ; + fi after_script: - cd build - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat - - du -chs $HOME/avocado/data/cache + - du -chs ${CI_PROJECT_DIR}/avocado-cache build-system-ubuntu: <<: *native_build_job_definition
@@ -81,7 +94,7 @@ acceptance-system-ubuntu: variables: IMAGE: ubuntu2004 MAKE_CHECK_ARGS: check-acceptance - <<: *post_acceptance + <<: *acceptance_definition build-system-debian: <<: *native_build_job_definition
@@ -111,7 +124,7 @@ acceptance-system-debian: variables: IMAGE: debian-amd64 MAKE_CHECK_ARGS: check-acceptance - <<: *post_acceptance + <<: *acceptance_definition build-system-fedora: <<: *native_build_job_definition
@@ -141,7 +154,7 @@ acceptance-system-fedora: variables: IMAGE: fedora MAKE_CHECK_ARGS: check-acceptance - <<: *post_acceptance + <<: *acceptance_definition build-system-centos: <<: *native_build_job_definition
@@ -171,7 +184,7 @@ acceptance-system-centos: variables: IMAGE: centos8 MAKE_CHECK_ARGS: check-acceptance - <<: *post_acceptance + <<: *acceptance_definition build-disabled: <<: *native_build_job_definition
Signature · 1 line
2.18.1
PATCH v2 1/4 tests/docker: Add python3-venv and netcat to the debian-amd64 container
In reply to Thomas HuthWithout python3-venv, I get the following message when trying to run the acceptance tests within the debian container:
The virtual environment was not created successfully because ensurepip is not available. On Debian/Ubuntu systems, you need to install the python3-venv package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the python3-venv package, recreate your virtual environment.
Let's do it as the message suggests.
And while we're at it, also add netcat here since it is required for some of the acceptance tests.
--- tests/docker/dockerfiles/debian-amd64.docker | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tests/docker/dockerfiles/debian-amd64.docker +3 -1
@@ -20,7 +20,9 @@ RUN apt update && \ librdmacm-dev \ libsasl2-dev \ libsnappy-dev \ - libvte-dev + libvte-dev \ + netcat-openbsd \ + python3-venv # virgl RUN apt update && \
Signature · 1 line
2.18.1
Re: [PATCH v2 2/4] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab
In reply to Thomas HuthQuoted 7 lines · Thomas Huth
These tests always time out on Gitlab, not sure what's happening here. Let's disable them until somebody has enough spare time to debug the issues.
Reviewed-by Wainer dos Santos Moschetta <[email protected]>Reviewed-by Cleber Rosa <[email protected]>Signed-off-by Thomas Huth <[email protected]>
Quoted 39 lines
--- tests/acceptance/machine_rx_gdbsim.py | 4 ++++ tests/acceptance/replay_kernel.py | 1 + 2 files changed, 5 insertions(+)tests/acceptance/machine_rx_gdbsim.py +4
-0@@ -8,6 +8,9 @@ # This work is licensed under the terms of the GNU GPL, version 2 or # later. See the COPYING file in the top-level directory. +import os + +from avocado import skipIf from avocado_qemu import Test from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern@@ -42,6 +45,7 @@ class RxGdbSimMachine(Test): # FIXME limit baudrate on chardev, else we type too fast #exec_command_and_wait_for_pattern(self, 'version', gcc_version) + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_linux_sash(self): """ Boots a Linux kernel and checks that the console is operational.tests/acceptance/replay_kernel.py +1
-0@@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest): self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1) + @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_arm_cubieboard_initrd(self): """ :avocado: tags=arch:arm
Signature · 1 line
Alex Bennée
Re: [PATCH v2 1/4] tests/docker: Add python3-venv and netcat to the debian-amd64 container
In reply to Thomas HuthQuoted 10 lines · Thomas Huth
Without python3-venv, I get the following message when trying to run the acceptance tests within the debian container:
The virtual environment was not created successfully because ensurepip is not available. On Debian/Ubuntu systems, you need to install the python3-venv package using the following command.
apt-get install python3-venv
You may need to use sudo with that command. After installing the python3-venv package, recreate your virtual environment.
Quoted 6 lines
Let's do it as the message suggests.
And while we're at it, also add netcat here since it is required for some of the acceptance tests.
Signed-off-by Thomas Huth <[email protected]>
Quoted 19 lines
--- tests/docker/dockerfiles/debian-amd64.docker | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)tests/docker/dockerfiles/debian-amd64.docker +3
-1@@ -20,7 +20,9 @@ RUN apt update && \ librdmacm-dev \ libsasl2-dev \ libsnappy-dev \ - libvte-dev + libvte-dev \ + netcat-openbsd \ + python3-venv # virgl RUN apt update && \
Signature · 1 line
Alex Bennée
Re: [PATCH v2 3/4] gitlab-ci.yml: Add build-system-debian and build-system-centos jobs
In reply to Thomas HuthQuoted 11 lines · Thomas Huth
We were missing the two new targets avr-softmmu and rx-softmmu in the gitlab-CI so far, and did not add some of the "other endianess" targets like sh4eb-softmmu yet.
Since the current build-system-* jobs run already for a very long time, let's do not add these missing targets there, but introduce two new additional build jobs, one running with Debian and one running with CentOS, and add the new targets there. Also move some targets from the old build-system-* jobs to these new targets, to distribute the load and reduce the runtime of the CI.Signed-off-by Thomas Huth <[email protected]>
Signature · 1 line
Alex Bennée
Re: [PATCH v2 4/4] gitlab-ci: Fix Avocado cache usage
In reply to Thomas HuthQuoted 9 lines · Thomas Huth
In commit
6957fd98dc("gitlab: add avocado asset caching") we tried to save the Avocado cache (as in commitc1073e44b4with Travis-CI) however it doesn't work as expected. For some reason Avocado uses/root/avocado_cache/which we can not select later.Manually generate a Avocado config to force the use of the current job's directory.
This patch is based on an earlier version from Philippe Mathieu-Daudé.
Maybe add a Based-on: <msgid>?
Quoted 16 lines
Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci.yml | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-).gitlab-ci.yml +1
-1@@ -47,11 +47,24 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS -.post_acceptance_template: &post_acceptance +.acceptance_template: &acceptance_definition
.gitlab-ci.yml include: · line 60
Quoted 10 lines
.gitlab-ci.yml +10
-0+ cache: + key: "${CI_JOB_NAME}-cache" + paths: + - ${CI_PROJECT_DIR}/avocado-cache + policy: pull-push + before_script: + - mkdir -p ~/.config/avocado + - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf + - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']" + >> ~/.config/avocado/avocado.conf
I was hoping there was a neater way to do this with the multiline commands but whatever:
Signature · 1 line
Alex Bennée
Re: [PATCH v2 4/4] gitlab-ci: Fix Avocado cache usage
Quoted 8 lines · Alex Bennée
Quoted 5 lines · Thomas Huth
In commit
6957fd98dc("gitlab: add avocado asset caching") we tried to save the Avocado cache (as in commitc1073e44b4with Travis-CI) however it doesn't work as expected. For some reason Avocado uses/root/avocado_cache/which we can not select later.
Quoted 6 lines
Quoted 4 lines
Manually generate a Avocado config to force the use of the current job's directory.
This patch is based on an earlier version from Philippe Mathieu-Daudé.
Maybe add a Based-on: <msgid>?
Isn't that tag rather used for expressing dependencies between patch series? See
https://wiki.qemu.org/Contribute/SubmitAPatch#Base_patches_against_current_git_master
?
Quoted 14 lines
Quoted 14 lines
Signed-off-by Thomas Huth <[email protected]>--- .gitlab-ci.yml | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-).gitlab-ci.yml +0
-0@@ -47,11 +47,24 @@ include: - find . -type f -exec touch {} + - make $MAKE_CHECK_ARGS
Quoted 15 lines
.gitlab-ci.yml include: · line 60
Quoted 12 lines
.gitlab-ci.yml +11
-1-.post_acceptance_template: &post_acceptance +.acceptance_template: &acceptance_definition + cache: + key: "${CI_JOB_NAME}-cache" + paths: + - ${CI_PROJECT_DIR}/avocado-cache + policy: pull-push + before_script: + - mkdir -p ~/.config/avocado + - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf + - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']" + >> ~/.config/avocado/avocado.confI was hoping there was a neater way to do this with the multiline commands but whatever:
I'm still fighting YAML syntax when it comes to such cases ... so I'm open to suggestions!
Quoted 1 line
Reviewed-by Alex Bennée <[email protected]>
Thanks a lot!
Thomas